Method for monitoring a protected area based on a model object dataset with at least one 3D model object

Abstract

The invention relates to a method for monitoring a protected area based on a model object dataset with at least one 3D model object, comprising the steps of: capturing a 3D dataset of the protected area using a 3D image sensor, detecting at least one detection object in the captured 3D dataset, processing the 3D dataset of the detected detection object using a training-based recognition algorithm, wherein the training-based recognition algorithm determines a geometric transformation for at least one of the 3D model objects to achieve congruence between the 3D model object and the detection object, applying the geometric transformation to the 3D model object, and comparing the transformed 3D model object with the detection object using a comparison algorithm, wherein a muting mode is triggered in the event of a match and a safety mode is triggered in the event of a deviation.

Description

[0001] The invention relates to a method for monitoring a protection area based on a model object data set with at least one 3D model object.

[0002] In industrial safety technology, protected areas are secured in such a way that risks to people, machines, and the environment are minimized. Typically, this involves the use of separating protective devices such as fences in conjunction with non-contact protective devices such as light curtains, radar, or camera systems. For example, to ensure that production runs as efficiently as possible, it is necessary that required material can be brought into a protected area where, for safety reasons, people or other objects are prohibited.

[0003] DE 11 2021 005 409 T5 discloses a safety vision device and a safety vision system that brake or stop a robot without using an area sensor when a worker enters a work area of ​​the robot. This safety vision device comprises: a 3D human skeleton estimation model; a 3D robot skeleton estimation model; an input unit for inputting a two-dimensional image of a worker and a robot captured by an external camera, as well as a distance and inclination between the camera and the robot;an estimation unit that inputs the two-dimensional image and the distance and inclination between the camera and the robot into the 3D human skeleton estimation model and the 3D robot skeleton estimation model to estimate three-dimensional joint point data indicating the three-dimensional coordinate values ​​of the position of a joint point of the worker and the angles of a plurality of joint axes included in the robot; and an approach determination unit that calculates a worker area and a robot area based on the three-dimensional joint point data and the angles of the plurality of joint axes, and issues an instruction to decelerate or stop the robot depending on the degree of overlap of the worker area and the robot area.

[0004] WO 2020 / 125875 A1 discloses a control system for controlling a brake, a steering system, and / or a drive of an autonomous or automated vehicle. The control system comprises a detection device configured to detect an object from the vehicle's surroundings and to provide corresponding object data. An artificial intelligence module is configured to generate at least one first control command for the vehicle from the object data. A fallback device is configured to generate at least one second control command for the vehicle from the object data using a deterministic calculation rule. A safety device is configured to determine calculation values ​​based on the object data and to calculate a criticality value therefrom.A safety gate is configured to receive the first control command of the artificial intelligence module, the second control command of the fallback device, and the criticality value of the safety device, wherein the safety gate is configured to send a third control command to the vehicle based on the criticality value, wherein the third control command is one of or corresponds to the first control command of the artificial intelligence module and the second control command of the fallback device.

[0005] The task can be seen as providing a method that ensures efficient monitoring of a protected area that complies with safety requirements.

[0006] The object is achieved by a method according to claim 1. Dependent claims relate to advantageous embodiments of the invention.

[0007] The inventor investigated the control of object entry into a protected area based on automated detection using one or more 3D image sensors and recognized that with the development and improvement of 3D sensors, object classification can be carried out quickly and efficiently using these sensors. Methods for object classification have been known for a long time, are well established, and are used in image recognition systems for a wide variety of applications. These methods can, for example, distinguish objects from people using 3D images. What most of these methods have in common is the use of artificial intelligence, such as neural networks, and the necessary vast amounts of data and the use of significant computing power to train the artificial intelligence.

[0008] However, none of these methods is suitable for use in functional safety for monitoring a protected area. In particular, the IEC 61508 standard discourages the use of machine learning approaches in the safety function.

[0009] The method according to the invention for monitoring a protection area based on a model object data set with at least one 3D model object comprises the steps - Capturing a 3D data set of the protected area using a 3D image sensor, - Detecting at least one detection object in the acquired 3D data set, - Processing the 3D data set of the detected detection object by means of a training-based recognition algorithm, wherein the training-based recognition algorithm determines a geometric transformation for at least one of the 3D model objects in order to achieve congruence between the 3D model object and the detected detection object, - Application of the geometric transformation to the 3D model object, - Comparison of the transformed 3D model object with the detected detection object using a comparison algorithm, whereby a muting mode is triggered in case of a match and a safety mode is triggered in case of a deviation.

[0010] The aim of the method according to the invention is to detect the access of objects into the protection area, whereby a decision can be made by automatic detection as to whether detected objects are allowed to enter the protection area within the scope of the active protection function (so-called muting mode) or whether a safety-relevant situation exists that requires the triggering of a safety mode.

[0011] A protected area can be understood as a local boundary. The local boundary can encompass an area that is to be protected from the intrusion of unwanted objects. For example, a machine may be operating within the protected area, whereby intrusion into the protected area can pose a danger to both the machine and the object being introduced. For example, the introduction of material to be processed into the protected area can be intentional, while the intrusion of people, for example, can be unintentional.

[0012] A 3D model object is a virtual model of a known real object. This is a virtual representation of the real object, such as a cuboid, triangular, or round object. The 3D model object can be freely scalable, meaning the 3D model object can be freely rotated, continuously enlarged, or arbitrarily aligned without losing any information. The 3D model object can depict the external shape of the real object and / or other properties such as a surface texture. The 3D model object can be available, for example, as a CAD model.

[0013] The model object dataset comprises at least one 3D model object. The model object dataset can be stored in such a way that it is available and retrievable at any time. The model object dataset can comprise a collection of various 3D model objects. The model object dataset includes at least the 3D model object whose inclusion in the protected area is permitted.

[0014] A 3D image sensor is any sensor that can record spatial coordinates. The 3D image sensor can be, for example, a 3D camera, e.g., one based on the TOF (time-of-flight) measurement principle, a radar system, or a 3D LiDAR sensor. 3D image sensors of this type capture three-dimensional data in the form of a 3D data set. The captured data in the 3D data set preferably has three coordinates, each of which includes a height coordinate, a width coordinate, and a depth coordinate. The 3D image sensor can be arranged, for example, from a bird's-eye view, parallel to, or at a 45° angle to the protected area. A bird's-eye view of the 3D image sensor can be understood to mean that the 3D image sensor is arranged at the highest point of the protected area. A parallel arrangement of the 3D image sensor can be understood to mean that the 3D image sensor is arranged opposite a wall of the protected area.An arrangement at a 45° angle to the protection area can be understood to mean that the 3D image sensor forms an angle of approximately 45° with the floor of the protection area.

[0015] Detecting the acquisition object can be understood as the recognition of related structures within the 3D data set based on similar parameters, such as the intensity of the pixels or the contrast between the pixels. The acquisition object is the image of a real object captured by the 3D image sensor. A corresponding virtual 3D model of the acquisition object can be present in the model object data set. Correspondence in this regard will be investigated further.

[0016] The training-based recognition algorithm comprises at least one algorithm that has been at least partially trained. An at least partially trained algorithm can be understood as a computational process, wherein the schema underlying the computational process increases in precision through training with a training dataset. Preferably, the precision increases with each iteration of training, with the precision converging during the training of the algorithm. The training-based recognition algorithm can be constructed from a combination of different algorithms. The training-based recognition algorithm can be constructed in the form of one-dimensional or multi-dimensional layers. A layer consists of small individual units that can receive inputs from other units, perform processing, and generate an output as a result.The dimensions of layers can be understood as the number of layers in the training-based recognition algorithm. The architecture and number of layers can be chosen depending on the type and number of model objects to be distinguished.

[0017] Processing the 3D data set of the detected detection object using the training-based recognition algorithm involves passing the 3D data set of the detected detection object to the training-based recognition algorithm. The training-based recognition algorithm attempts to find the best possible association of a 3D model object with the 3D data set. If the model object data set comprises multiple 3D model objects, this may also include selecting a suitable 3D model object from among the 3D model objects contained in the model object data set. For at least one 3D model object, the training-based recognition algorithm determines a geometric transformation with which the 3D model object is adapted to the detection object. The geometric transformation leads to maximized congruence between the 3D model object and the detection object.Congruence defines the maximum overlap between the 3D model object and the detection object. The application of the geometric transformation to the 3D model object is advantageously not performed by the training-based recognition algorithm.

[0018] The result of processing using the training-based recognition algorithm is the geometric transformation and, if necessary, the selection of the best-fitting 3D model object. The geometric transformation allows the 3D model object to be converted into the transformed 3D model object, whereby the transformed 3D model object is congruent with the detected object.

[0019] The application of geometric transformation is an effective conversion of the 3D model object according to the parameters determined by the training-based recognition algorithm to provide a scaled, transformed 3D model object.

[0020] When comparing the transformed 3D model object with the acquisition object using a comparison algorithm, a check is performed to determine whether the determined geometric transformation results in congruence between the 3D model object and the acquisition object. The comparison algorithm performs a direct comparison of the values ​​of the transformed 3D model object with the values ​​of the acquisition object. The comparison algorithm can determine the degree of congruence between the transformed 3D model object and the acquisition object.

[0021] If the comparison algorithm's calculation results in congruence between the transformed 3D model object and the detection object, a muting mode is triggered. During muting mode, the detection object congruent with the transformed 3D model object is permitted to enter the protection zone.

[0022] If the match between the transformed 3D model object and the detection object fails, a safety mode is triggered. The safety mode initiates a safety action, such as establishing or controlling a safe state of the machine by shutting it down, and / or seals off the protection area in such a way that the detection object is prevented from entering the protection area, thus avoiding a hazardous situation for both the detection object and objects within the protection area.

[0023] The method according to the invention is characterized by efficient monitoring of the protected area, with the computational processes of the training-based detection algorithm being separated from the safety function. This allows powerful training-based algorithms of artificial intelligence or machine learning to be used in safety technology without having to accept their major disadvantage of a lack of verification or deterministic verifiability. This is because the comparison using the comparison algorithm allows for a separate review of the result before a decision is made as to whether a safety mode is triggered.

[0024] According to an advantageous embodiment of the invention, the training-based recognition algorithm comprises an artificial intelligence pre-trained on at least one 3D model object retrievable from the model object dataset. Artificial intelligence is understood to be the ability of a machine to imitate abilities such as logical reasoning, learning, planning, and creativity. Algorithms known in connection with artificial intelligence include, for example, artificial neural networks. Artificial neural networks are known for their use in automatic image recognition. A residual neural network (ResNet), for example, can be used for this purpose. ResNets are special types of neural networks used in image processing. They are characterized by their deep architectures, which can produce low error rates.

[0025] According to a particularly preferred development of the invention, the training-based recognition algorithm determines at least one parameter of the geometric transformation in the form of a scaling, a rotation, a distortion, and / or an alignment. Scaling of the data can be understood as a change in size. Rotation can be understood as a change around one axis. Alignment can be understood as a lateral shift. Distortion can be understood as an optical or perspective correction. Optical correction can be understood as a correction of a tunneling effect, aberrations, and the like. The training-based recognition algorithm can adapt the 3D model object to the detection object by scaling, rotation, and / or alignment.Preferably, the training-based recognition algorithm has already been trained on the model object dataset, and preferably the training has been completed.

[0026] According to a preferred embodiment of the invention, the comparison algorithm can perform a deterministic operation. A deterministic operation is understood to mean an algorithm that performs calculations without prior training. The comparison algorithm is based on an algorithm whose algorithm does not originate from artificial intelligence, but is based on deterministic operation.

[0027] According to a particularly advantageous embodiment of the invention, the training-based recognition algorithm can be independent of the comparison algorithm. The independence between the training-based recognition algorithm and the comparison algorithm can be understood to mean that there is no mathematical or process-related connection between these algorithms. Preferably, there is no interaction between the algorithms. Preferably, only the result of the determined geometric transformation in the form of the transformed 3D model object can be passed to the comparison algorithm for verification.

[0028] According to an advantageous embodiment of the invention, the comparison algorithm compares the detected object with a lower and upper tolerance limit of the transformed 3D model object. A tolerance limit can be understood as a threshold value used to calculate whether congruence exists. The tolerance limit refers, for example, to a geometric distance or a geometric boundary surface. If the object lies within the tolerance limits, congruence exists. If the object lies outside the tolerance limits, the geometric transformation has failed or there is no 3D model object corresponding to the object.

[0029] According to a preferred embodiment of the invention, the training-based recognition algorithm can determine a value that indicates which detection object is present. Particularly preferably, the determination of this value comprises a classification, such as using an image recognition algorithm. The classification can be determined independently of the geometric transformation. Within the scope of the classification, probability values ​​can be provided in the case of multiple model objects, with the probability quantifying which 3D model object is involved. The probability value that a specific detection object is present can, for example, be higher than 70%, preferably higher than 80%, and most preferably higher than 90%.

[0030] According to a particularly advantageous embodiment of the invention, the detection of at least one detection object in the acquired 3D data can be achieved by detecting contiguous 3D pixel sets in the acquired 3D data. A contiguous 3D pixel set can be defined, for example, by neighboring pixels correlating not only in one plane, i.e., having similar contrast or intensity values, but also in three-dimensional space.

[0031] According to an advantageous embodiment of the invention, the contiguous 3D pixel set has a minimum size. The minimum size can depend on the resolution of the 3D image sensor and on a known, predetermined size of the 3D model object.

[0032] In an advantageous embodiment of the invention, a structure of the surface of the detection model is determined by the training-based recognition algorithm and compared with the structure of the surface of the 3D model object by the comparison algorithm. Particularly advantageously, the detection model has a regular surface structure, as this can be particularly well recognized by the training-based recognition algorithm. The structure of the surface of the detection model can, for example, be smooth or rough. It is also possible for the surface structure to have a pattern, such as lines or geometric figures. If the detection object comprises bulk goods, soft materials, or liquids, the recognition of the surface structure may not be effective, as these structures are very irregular.

[0033] According to an advantageous embodiment of the invention, when comparing the transformed 3D model object with the detection object using a comparison algorithm, a spatial and / or temporal expected value is determined based on at least a length and / or an angle of the detection object and / or a speed and / or a distance of the detection object from the 3D image sensor. During the review of the subsequent 3D data set, the expected value of the previous 3D data set is retrieved in order to minimize computational effort during the comparison. For example, the position values ​​of two 3D data sets can be used to predict the value of the position of the detection object in the subsequent 3D data set in order to further reduce computational effort.

[0034] An embodiment of the invention is described below with reference to the drawings. These are not intended to limit the invention, but merely to illustrate it. The drawings show: Fig. 1a a plan view of a protection area with a detection object moving towards the protection area; Fig. 1b the arrangement from Fig. 1a along line AA; Fig. 2a a plan view of the protected area Fig. 1a with the detection object entering the protected area; Fig. 2b the arrangement from Fig. 2a along line AA; Fig. 3 a perspective view of a 3D model object, a transformed 3D model object and the detected detection object; Fig. 4 a perspective view of the transformed 3D model object and the detected acquisition object Fig. 3; Fig. 5 a perspective view of two different 3D model objects; Fig. 6 a diagram of procedural steps for monitoring a protected area.

[0035] An exemplary procedure for monitoring a protected area is described below using the Fig. 1a to Fig. 5 and in the scheme from Fig. 6 explained.

[0036] Fig. 1a shows an example of an arrangement of a 3D image sensor 20 on a wall 26 with a gate 28 (see Fig. 1b), wherein the 3D image sensor 20 monitors a protection area 22. The protection area 22 corresponds to the field of view of the 3D image sensor 20. A conveyor belt 24 is arranged in the direction of the wall 26 and the 3D image sensor 22 to transport an object into the protection area 22. By way of example, a detection object 10 is shown on the conveyor belt 24. The detection object 10 is outside the protection area 22 and cannot be detected by the 3D image sensor 20. The 3D image sensor 20 and the conveyor belt 24 are connected to a processing unit 50, here a computer, to control the conveyor belt 24. The goal is to allow known objects to pass through the gate 28, but to prevent unknown objects from entering. The known objects are stored as 3D model objects 30 in a 3D model object data set on the processing unit 50.

[0037] Fig. 2a shows the arrangement of Fig. 1a, wherein the detection object 10 is introduced into the protection area 22 by the conveyor belt 24. Fig. 2b shows a cross-section of the arrangement of Fig. 2a along line AA. The 3D image sensor captures a 3D data set, wherein the 3D data set includes spatial coordinates in the form of x, y, and z coordinates. An image of the detection object 10 is detected in the 3D data set. The detection of the detection object 10 in the captured 3D data is performed by recognizing contiguous 3D pixel sets in the captured 3D data and determining a detected detection object 12.

[0038] The detected detection object 12 is processed by a training-based recognition algorithm, see Fig. 3. The training-based recognition algorithm is an artificial intelligence pre-trained on the 3D model objects retrieved from the model object dataset. The training-based recognition algorithm determines a geometric transformation to achieve congruence between the 3D model object 30 and the detected detection object 12. The geometric transformation includes parameters in the form of scaling, rotation, distortion, and / or orientation.

[0039] The geometric transformation is applied to the 3D model object 30 to determine a transformed 3D model object 32. The 3D model object 30 is retrieved, and the transformation is applied to the 3D model object 30 to obtain a transformed 3D model object 32.

[0040] The comparison of the transformed 3D model object 32 with the detected detection object 12 is performed using a comparison algorithm. The comparison algorithm performs a deterministic operation, whereby the comparison algorithm is not based on any training. Based on the congruence between the detected detection object 12 and the transformed 3D model object 32, the comparison is performed using the comparison algorithm. To ensure the decoupling of the safety function from the artificial intelligence, the training-based detection algorithm is independent of the comparison algorithm.

[0041] As part of the comparison of the transformed 3D model object 32 with the detected acquisition object 12, the congruence of the detected acquisition object 12 is checked for compliance with tolerance limits 40, 42. The tolerance limits 40, 42 represent geometric boundaries, see Fig. 4. The comparison algorithm compares the detected detection object 12 with a lower and upper tolerance limit 40, 42 of the transformed 3D model object 32.

[0042] Within the context of several different possible 3D model objects 30, the training-based recognition algorithm determines a value that leads to the indication of which detection object 10 is present, see Fig. 5. The detected detection object 12 is compared with the known 3D model objects 30, here a box and a cylinder, to determine congruence. The training-based recognition algorithm determines the value that indicates the congruence between the detected detection object 12 and the 3D model object in percent. The detected detection object 12, here a section of a cylinder, achieves a congruence of 50% with the existing 3D model object of the box and a congruence of 90% with the existing 3D model object of the cylinder. Since the detected detection model 12 exhibits a higher congruence with the cylinder, the detection object 10 is classified as a cylinder.

[0043] If the comparison results in a match, a muting mode is initiated, while a discrepancy triggers a safety mode. During muting mode, the detection object 10 is permitted to pass through the protection area 22 and be transported through the gate 28. If the safety mode is triggered, the conveyor belt 24 is stopped to prevent the detection object 10 from entering the protection area 22 any further.

[0044] The following describes the procedure for monitoring the protected area using Fig.6. The acquisition of a 3D data set is carried out by means of the 3D image sensor 22. The detection of the detection object 10 in the 3D data set is carried out by means of image recognition and provides a detected detection object 12. The processing of the 3D data set using a training-based recognition algorithm, in this case an artificial intelligence, leads to the determination of a geometric transformation, wherein the geometric transformation determines parameters for maximizing congruence between the detected detection object and the 3D model object 30. The application of the geometric transformation to the 3D model object 30 results in a transformed 3D model object 32. The comparison of the transformed 3D model object 32 with the detected detection object 12 triggers a muting mode in the event of a match and a safety mode in the event of a deviation.In the muting mode, the detection object 10 is allowed to enter through the gate 28, while in the safety mode the conveyor belt 24 is stopped to prevent entry. List of reference symbols 10 Recording object 12 detected detection object 20 3D image sensor 22 Protection area 24 conveyor belt 26 Wall 28 Gate 30 3D model objects 32 transformed 3D model object 40 lower tolerance limit 42 upper tolerance limit 50 processing units

Claims

[1] Method for monitoring a protection area (22) based on a model object data set with at least one 3D model object (30), comprising the steps - capturing a 3D data set of the protected area by means of a 3D image sensor (20), - detecting at least one detection object (10) in the acquired 3D data set, - processing the 3D data set of the detected detection object (12) by means of a training-based recognition algorithm, wherein the training-based recognition algorithm determines a geometric transformation for at least one of the 3D model objects (30) in order to achieve congruence between the 3D model object (30) and the detected detection object (12), - Application of the geometric transformation to the 3D model object (30), - comparing the transformed 3D model object (32) with the detected detection object (12) by means of a comparison algorithm, wherein a muting mode is triggered in case of a match and a safety mode is triggered in case of a deviation. [2] Method according to claim 1, characterized by that the training-based recognition algorithm comprises an artificial intelligence pre-trained on at least one 3D model object (30) retrievable from the model object data set. [3] Method according to claim 1 or 2, characterized by that the training-based recognition algorithm determines at least one parameter of the geometric transformation in the form of a scaling, a rotation, a distortion and / or an orientation. [4] Method according to one of the preceding claims, characterized by that the comparison algorithm performs a deterministic operation. [5] Method according to one of the preceding claims, characterized bythat the training-based recognition algorithm is independent of the comparison algorithm. [6] Method according to one of the preceding claims, characterized by that the comparison algorithm compares the detected detection object (12) with a lower and upper tolerance limit (40, 42) of the transformed 3D model object (32). [7] Method according to one of the preceding claims, characterized by that the training-based recognition algorithm determines a value that indicates which detection object (10) is present. [8] Method according to one of the preceding claims, characterized by that the detection of at least one detection object (10) in the detected 3D data is carried out by detecting contiguous 3D pixel sets in the detected 3D data. [9] Method according to claim 8, characterized by that the contiguous 3D pixel set has a minimum size. [10] Method according to one of the preceding claims, characterized by that a structure of the surface of the detection model (10) is determined by the training-based recognition algorithm and compared with the structure of the surface of the 3D model object (30) by the comparison algorithm. [11] Method according to one of the preceding claims, characterized by that, when comparing the transformed 3D model object (32) with the detected detection object (12) by means of a comparison algorithm, a spatial and / or temporal expected value is determined based on at least a length and / or an angle of the detection object (10) and / or a speed and / or a distance of the detection object (10) to the 3D image sensor (20), wherein the expected value of the previous 3D data set is retrieved as part of the review of the subsequent 3D data set in order to minimize computational effort during the comparison.

Images