SECURING A MACHINE

Description

[0001] The invention relates to a method for securing a machine according to the preamble of claim 1 and a corresponding securing system.

[0002] The goal of safety is to protect people from sources of danger, such as those posed by machines in an industrial environment. The machine is monitored using sensors, and if a situation arises in which a person is at risk of getting dangerously close to the machine, appropriate safety measures are taken. Safety becomes particularly challenging in cases of cooperation between humans and machines or robots.

[0003] Traditionally, optoelectronic sensors such as light grids or laser scanners are primarily used for safety monitoring. More recently, cameras and even 3D cameras have been added. A common protection concept involves configuring protective fields that operating personnel are prohibited from entering while the machine is in operation. If the sensor detects an impermissible intrusion into the protective field, such as an operator's leg, it triggers a safety-related stop of the machine. Other intrusions into the protective field, for example, by static machine parts, can be taught in advance as permissible. Warning fields are often located upstream of the protective fields, where intrusions initially only lead to a warning in order to prevent the intrusion into the protective field and thus the protection in time, thus increasing the availability of the system.Alternatives to protective fields are also known, such as ensuring that a minimum distance is maintained between the machine and the person, depending on the relative movement ("speed and separation").

[0004] Sensors used in safety technology must operate with exceptional reliability and therefore meet stringent safety requirements, such as the EN 13849 standard for machinery safety and the device standard IEC 61496 or EN 61496 for electro-sensitive protective equipment (ESPE). To comply with these safety standards, a number of measures must be taken, such as reliable electronic evaluation using redundant, diverse electronics, functional monitoring, or specifically monitoring the contamination of optical components, particularly a front screen, and / or providing individual test targets with defined reflectances that must be detected at the corresponding scanning angles. Measures tailored to the specific sensor thus ensure that the sensor data is of sufficient quality to solve the detection task or, if not, detect a fault that would otherwise prevent this.

[0005] EP 3 651 458 A1 discloses a secure stereo camera that checks the functionality of its image sensors. To do so, it uses a reference depth map and statistically evaluates the differences between the current depth map and the reference depth map. This only works in a static environment.

[0006] DE 10 2018 117 274 A1 analyzes the pixel environments to determine when noise influences become too significant for reliable object detection. This ignores numerous factors that can prevent object detection.

[0007] In recent years, significant progress has been made in the field of artificial neural networks, entirely independent of security technology. This technology is reaching broad application maturity thanks to new neural network architectures ("deep learning") and the sharp increase in available computing power, particularly in the form of modern graphics processors. A key challenge in using neural networks for situation assessment and problem solving is the need to train the neural network with representative training data. Before the neural network can reliably perform the assigned task, it must be confronted with comparable situations and their predetermined evaluation. Based on these examples, the neural network learns the correct behavior. However, a neural network is only capable of generalization to a certain extent.

[0008] When using neural networks in safety technology, particular difficulties arise in proving detection capability and uncovering errors. If no safety-critical object is detected in the machine's vicinity, this does not necessarily mean that there is no danger. Rather, problems with the sensor or the scene may be causing an actually present object to go undetected. Sensor errors can possibly be intercepted using technical measures similar to conventional safe sensors without neural networks. This becomes difficult, for example, if the lighting fails in a scene or the scene is flooded with sunlight. In this case, a neural network will not detect an object, but will not disclose the cause, which can lead to dangerous misjudgments.

[0009] DE 10 2017 105 174 A1 describes a method for generating training data for monitoring a hazard source. A sensor that is considered safe by conventional standards evaluates the respective situation, which then provides annotated training data for training a neural network. However, this has nothing to do with the question of whether the image data to be evaluated will be of sufficient quality when the neural network is subsequently used.

[0010] EP 4 170 438 A1 discloses a safety control system with an image sensor unit that records the surroundings of a machine. The collected images are evaluated using a neural network to detect body parts and, if necessary, generate a safety signal.

[0011] EP 4 047 523 A1 presents a monitoring device for a system in which a safety sensor emits a switching signal upon detecting an object intrusion to trigger a safety function. Furthermore, an evaluation system is provided that evaluates measurement data from the safety sensor or another sensor with a neural system to generate warning messages and / or corrective interventions for the system's movement sequences based on this evaluation.

[0012] The paper by Begon, Jean-Michel, and Pierre Geurts, "Sample-free white-box out-of-distribution detection for deep learning," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, describes an approach for detecting so-called OOD (out-of-distribution) examples. This involves analyzing the internal activations of the neural network. This is complex and not feasible in practical security applications.

[0013] In a conference paper entitled "Building Trust in AI for Autonomous Vehicles" at the Nvidia GTC Developer Conference from March 20-23, 2023, Marco Pavone discusses the use of multiple deep neural networks and comparing their respective results. Despite the immense additional effort, this does not solve the problem for the security applications considered here, because a poor-quality image will lead to unreliable results even when used in multiple neural networks.

[0014] In a sense, the occurrence of unusable input data can be understood as anomaly detection, as investigated, for example, in the papers by Chalapathy, Raghavendra, and Sanjay Chawla, "Deep learning for anomaly detection: A survey," arXiv preprint arXiv:1901.03407 (2019) and Chalapathy, Raghavendra, Edward Toth, and Sanjay Chawla, "Group anomaly detection using deep generative models," Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part I 18, Springer International Publishing, 2019. A fundamental problem here is that there is usually a lack of sufficient training examples of the anomaly.

[0015] Kalyanasundaram, Girish, Puneet Pandey, and Manjit Hota, "A Pre-processing Assisted Neural Network for Dynamic Bad Pixel Detection in Bayer Images," Computer Vision and Image Processing: 5th International Conference, CVIP 2020, Prayagraj, India, December 4-6, 2020, Revised Selected Papers, Part II 5, Springer Singapore, 2021, describe how corrupt pixels can be detected using a neural network. However, this is only one of the possible errors, and the conventional approach aims to correct the corrupt pixels, whereas in a safety application, the goal is to determine whether the errors are critical in terms of accident prevention.

[0016] It is therefore an object of the invention to improve the verification of detection capability in a security application.

[0017] This object is achieved by a method and a safety system for protecting a machine according to claim 1 and 15, respectively. The machine is monitored by a sensor. Machine and sensor are mentioned only representatively in the singular; any complex safety applications with a multitude of sensors and / or machines are conceivable. The method is a computer-implemented method that runs, for example, in a processing unit of the sensor and / or a connected processing unit. The sensor preferably operates contactlessly and is, in particular, an optoelectronic sensor. The sensor and / or the hardware for evaluation are preferably designed to be safe.As used throughout this description, "safe" and "safety" mean that measures have been taken to control errors up to a specified safety level, or to comply with the requirements of a relevant safety standard for machinery safety or electro-sensitive protective devices, some of which are mentioned in the introduction. "Not safe" is the opposite of "safe"; therefore, for non-safe devices, transmission paths, evaluation systems, and the like, the specified fail-safety requirements are not met.

[0018] By evaluating the sensor data, it is determined whether a dangerous situation exists. If so, the machine is made safe. A dangerous situation exists in particular when an object or person is too close to the machine. Some concepts such as protective field monitoring or speed and separation monitoring are mentioned initially. Another example is safe object tracking with dangerous and non-dangerous trajectories of objects in the vicinity of the machine. An assessment of the sensor data using a machine learning method is also conceivable, as explained in more detail below. Depending on the safety application, the detected hazard, and other possible criteria, machine safeguarding can consist of slowing down, evading the machine, switching to another program, or, if necessary, stopping the machine.

[0019] The detection capability is also tested, i.e., whether an assessment of a dangerous situation is currently even possible. Otherwise, the machine is protected for this reason, analogous to the case of a detected dangerous situation and with the same possible safeguarding measures. This concerns a safety-relevant failure of the detection capability, not a possibly only very brief malfunction. The decisive factor is that the safeguarding must react quickly enough, particularly within the response time specified by the safety application. Conventional measures for this were discussed in the introduction and are also conceivable in the invention, but are insufficient in themselves, for example, a sensor failure, a lack of permeability of optical components, an image sensor test, and the like.

[0020] The invention is based on the fundamental idea that detection capability is tested using sensor data and a machine learning method. Machine learning, as is usually the opposite of traditional evaluation, refers to a data-driven approach in which evaluation strategies are not developed manually, but rather learned from examples. The result of the evaluation of the sensor data by the machine learning method is a quality score. This must meet a requirement that corresponds to a sufficient quality of the sensor data to allow the evaluation of the sensor data with regard to a dangerous situation.

[0021] The invention has the advantage that it can be reliably determined whether the sensor is functional and whether its detection range, i.e. the scenery around the machine, allows a risk assessment based on the sensor data. This allows for reliable protection, especially when a machine learning process is involved in evaluating the sensor data for the actual risk assessment. Human visual perception can be used as an analogy: In the dark, in heavy rain, or without glasses, a person would act much more cautiously. The invention enables the safety application to recognize that it is in a comparable exceptional situation. In contrast to some of the approaches mentioned in the introduction, the invention also copes with dynamic scenarios. The balance at which detection capability is no longer present can be adjusted very precisely.This is important because, although security is a priority, an overly cautious assessment can lead to significant losses in availability and productivity due to objectively unnecessary security measures.

[0022] The quality factor is preferably binary. It therefore directly distinguishes only between the two cases where an assessment of a dangerous situation is possible and not possible. Following the detection capability test of the machine learning process, no further evaluation is required; the binary decision or classification has already been made. It should be reiterated that a negative result of the detection capability test does not necessarily lead to immediate safeguarding. For example, with a camera as a sensor, the safeguarding decision can always only be made after evaluating n images or frames, in which case insufficient quality in a single image or in a few images may still be tolerable.

[0023] The quality factor preferably quantitatively evaluates at least one interference characteristic of the sensor data. Such a quality factor is, for example, a number in a certain value range such as [0, 1, ..., 10] or a percentage. It evaluates a specific interference characteristic or corruption modality of the sensor data, and from this, by comparison with predefined criteria, it can be very easily deduced whether the quality of the sensor data is sufficient for an assessment of the dangerous situation or not. At this point, it should be noted that there can be multiple quality factors and the quality factor can be multidimensional. The respective components can evaluate different interference characteristics. Even a quality factor that is both binary and quantitatively differentiating is therefore not a contradiction but is possible and then refers to different components.The quantitative components then enable, for example, a consistency check of the binary component or an analysis of how this assessment was arrived at. The latter is particularly interesting in terms of "explainable AI."

[0024] The machine learning method preferably includes a classifier. The assignment of a quality score can be considered a classification, both in the case of a binary quality score and a more differentiated, quantitatively evaluative quality score. Numerous machine learning methods exist for classification tasks, such as decision trees, support vector machines, K-nearest neighbors, or Bayesian classifiers. It is conceivable to use multiple classifiers or general machine learning methods, each responsible for one or more noise characteristics, in a splitting or (partially) overlapping manner. In this way, a machine learning method can be optimized specifically for certain noise characteristics.

[0025] The machine learning method preferably uses a neural network. A neural network, especially a deep neural network (deep learning) or convolutional neural network (CNN), is ideal for evaluating or specifically classifying even complex sensor data such as images or 3D point clouds.

[0026] The sensor is preferably a camera or a 3D sensor. These sensors provide a wealth of information about the machine's surroundings and can therefore solve a wide variety of safety applications. The sensor data is, accordingly, images or image data in the case of a conventional camera, or 3D point clouds or depth maps in the case of a 3D sensor, such as a 3D camera, LiDAR, laser scanner, or radar.

[0027] The evaluation of the sensor data preferably includes an object detector, which particularly detects foreign objects in the machine's environment. A camera or a 3D sensor is preferably used for this purpose. A foreign object is defined as an object in the machine's environment that is unknown or not expected. Depending on the safety application, each detected object is considered a foreign object, previously known objects are taught as a reference, or expected objects are dynamically recognized as such and differentiated from foreign objects. Whether a detected object indicates a dangerous situation also depends on the safety application.

[0028] Some possible criteria are the size, shape, position, speed or trajectory of the object.

[0029] The machine learning method is preferably trained using supervised learning, in which a large number of training examples from sensor data with a known corresponding quality factor are used as training data. Supervised training (supervised learning) means that the desired result—in this case, the quality factor corresponding to a set of sensor data from a training example—is specified externally. Training preferably takes place before the actual validation operation, but can also be refined or expanded during operation or during downtimes. During training, the machine learning method uses the training examples to learn the association of a quality factor with sensor data and, after training, is able to transfer this to sensor data unknown from the training.

[0030] The training examples are preferably modified with at least one perturbation property and at least one perturbation intensity to generate additional training examples. The initial training examples are thus deliberately perturbed or corrupted to obtain additional training examples. The perturbation can occur in various ways, i.e., with different perturbation properties or corruption modalities, and / or with varying degrees of severity, as expressed by a perturbation intensity. If the machine learning process later determines a quantitatively evaluative quality score, this can correspond to the perturbation intensity, but different scales are also conceivable.

[0031] The sensor data preferably comprises images, and at least one of the following interference characteristics is used: image that is at least partially too bright or too dark, image that is at least partially blurred, motion artifacts, static and / or dynamic image noise, image regions swapped, particularly due to address errors, image incomplete. This illustrates the previously rather abstract concept of interference characteristics using examples. A change in an interference characteristic could therefore consist of artificially brightening the image to a varying extent determined by the interference intensity, or in any desired combination for the other examples mentioned. Interference is therefore possible in two dimensions, which are determined on the one hand by the selection of the interference characteristic, particularly from a list as in the examples mentioned, and on the other hand by the extent of the interference or the interference intensity.The safety application becomes more robust the better the disturbance characteristics represent the possible disturbances or error cases in reality. Ideally, therefore, a list of considered disturbance characteristics is exhaustive. Since this is not possible in practice, the most important disturbance characteristics are preferably identified and considered instead. However, to a certain extent, the machine learning process can also generalize, so that not every scenario not considered in detail in advance and therefore not explicitly represented in the training data immediately leads to an overlooked hazard.

[0032] The training data is preferably evaluated, in particular with an object detector, to determine whether a dangerous situation is detected despite the interference characteristic. Depending on the result, a quality score is assigned to a training example. A prerequisite for supervised learning is the specification of the desired result, which is referred to as labeling or annotation and is usually an extremely laborious and time-consuming manual process. Particularly in the case of artificially generated or modified training data, as described in the previous paragraph, the appropriate label, i.e., the associated quality score, is initially missing, since it is not known a priori whether or not the assessment of a dangerous situation is still possible after a change. According to this embodiment, annotation is performed automatically through evaluation, i.e., risk assessment, based on the sensor data of the training example and, in particular, an object detector.The label of the original training example, which has not been altered or corrupted by perturbations, is known. Therefore, the evaluation for automatic annotation determines in particular whether this label is still being reproduced or whether the corruption was too severe.

[0033] The training data is preferably evaluated using the same method that is used to detect a dangerous situation in the machine's safety. "Evaluation" here refers to the automatic labeling or annotation of the preceding paragraph. In principle, this could be performed using any evaluation method or object detector. However, particular preference is given to using the evaluation method that is also used in actual operation. This ensures particularly good agreement in the evaluation of sensor data regarding the question of whether the quality of the sensor data is sufficient for the risk assessment. It is conceivable to use multiple evaluation methods or object detectors, whose labels are then combined.

[0034] The training data is preferably modified with increasing disturbance intensity and / or varying disturbance characteristics until an evaluation of the modified training data no longer detects a dangerous situation. When assessing whether a specific set of sensor data can still detect a dangerous situation, there is particular interest in the borderline cases where disturbances just barely allow this evaluation, especially object detection, or just no longer allow it. If these borderline cases are assessed correctly, this applies even more to simpler cases beyond the borderline.Therefore, this limit is advantageously tested by different disturbance properties and / or disturbance intensities, for example in an iterative procedure that gradually tries out different combinations of disturbance properties and increases their respective disturbance intensity or, as is implied by the term "increasing" here, systematically changes it in any other way.

[0035] To create a safety margin, a quality score corresponding to a no longer existing detection capability is preferably assigned to training examples in which the evaluation still identified a dangerous situation. This intentionally maintains a distance from the limit explained in the previous paragraph, and as a precaution, only those sensor data that at least slightly exceed the criteria are considered sufficient for the detection capability test. For example, if an interference characteristic is quantified in the range 0..10 and the limit of no longer existing detection capability is found to be 7, then a much higher requirement of a maximum interference characteristic of 5 is intentionally set, rather than testing accordingly at the limit of 7.This serves to better handle a potentially fatal error, where the detection capability test incorrectly considers sensor data to be sufficient for assessing a dangerous situation, but an actual hazard is overlooked during the actual evaluation or object detection due to insufficient sensor data quality. Thus, more false-negative detection capability results are accepted in order to prevent false-negative risk assessment results.

[0036] The evaluation of the sensor data for detecting a dangerous situation preferably also comprises a machine learning method. Until now, it remained unclear which method is used to detect whether a dangerous situation exists, except that it is preferably based on object detection, and a few examples are given in the introduction. With this embodiment, a machine learning method is now used not only for the detection capability test, but also for assessing a dangerous situation. The same machine learning methods mentioned above can be used, in particular a neural network. Particularly preferably, the machine learning method for the detection capability test is also used in a dual function for detecting a dangerous situation and is trained accordingly beforehand. Advantageously, a function, such as the detection capability test, is only subsequently trained.For example, there is an already trained neural network for object detection, and an additional output for the classification of the sensor data in terms of detection capability testing is added to it and then retrained.

[0037] The inventive safety system comprises at least one sensor for monitoring the machine and generating sensor data, and at least one control and evaluation unit. The term "control and evaluation unit" refers to any computing unit that can be part of the sensor, the machine, standalone, or a combination thereof. A method according to the invention is implemented therein in one of the described embodiments.

[0038] The invention will be explained in more detail below with regard to further features and advantages, using exemplary embodiments and with reference to the accompanying drawings. The figures of the drawing show: Fig. 1 shows an overview of an exemplary protection of a machine with a sensor; Fig. 2 shows a representation of the targeted disruption of training data and the automatic annotation of whether the respective disrupted training data still allows the detection of a dangerous situation; and Fig. 3 shows an exemplary flow diagram of the modification and annotation of training data and the resulting training of a neural network for a detection capability test.

[0039] Figure 1 shows an exemplary safety application in which a camera 10 monitors a robot 12. The camera 10 and the robot 12 are examples of a monitoring sensor and a machine to be monitored, respectively. The sensor data generated by the camera 10 is evaluated to determine whether a dangerous situation exists, in particular whether a person 14 is coming too close to the robot 12.

[0040] If a hazard is detected, a safety measure is initiated that slows down the robot 12, causes it to take evasive action, switches to a program with a different work area or a non-hazardous movement, or, if necessary, stops the robot 12 completely.

[0041] The evaluation of the sensor data is preceded by a detection capability test, which determines whether the sensor data even allows the assessment of a dangerous situation. For reasons that may lie particularly in the camera 10 or the scenery surrounding the robot 12, the quality of the sensor data is not always sufficient for this. The detection capability test uses a machine learning method and is later carried out, along with its training, with reference to the Figure 2 and 3 explained in more detail.

[0042] The evaluations are carried out in a computing unit, which can be at least one internal computing unit 16 of the camera 10, at least one connected external computing unit 18, or a combination of both. Examples of an internal computing unit are digital computing components such as a microprocessor or a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), a K1 processor, an NPU (Neural Processing Unit), a GPU (Graphics Processing Unit), a VPU (Video Processing Unit), or the like. An external computing unit can be a computer of any type, including notebooks, smartphones, tablets, a (security) controller, a local network, an edge device, or a cloud.There is also a wide selection of communication connections, such as I / O-Link, Bluetooth, WLAN, Wi-Fi, 3G / 4G / 5G and, in principle, any industrial standard.

[0043] The invention is described using a camera 10 as an example of a monitoring sensor; accordingly, the sensor data are exemplary images. Other sensors are conceivable, in particular a 3D sensor that generates a 3D point cloud or depth map as sensor data.

[0044] At least the detection capability test, and in a preferred development of the invention also the dual function of evaluating sensor data to determine whether a dangerous situation exists, is based on a machine learning method. A neural network is used below as an example and representative of any known machine learning method, particularly for classification. In a brief description, an input image is fed to the detection capability test, and the neural network provides feedback, referred to as a quality factor, indicating whether this input image can be used for another detection algorithm, namely the assessment of whether a dangerous situation exists. This detection algorithm can, as already mentioned, be a conventional image processing method or also a machine learning method.This can essentially be object recognition, but can also include other, particularly more advanced, evaluations such as object tracking, person recognition, pose determination, facial recognition, or code reading.

[0045] Based on the detection capability test and the subsequent assessment of whether a dangerous situation exists, at least three scenarios can be distinguished: Firstly, there may be no person or other object considered to be at risk, and this is correctly detected. Robot 12 can then work unhindered. Secondly, a person or other foreign object in a dangerous situation may have been correctly detected. Robot 12 must then be secured. Thirdly, the detection capability test may produce a negative result. In this case, it is no longer necessary to differentiate whether a dangerous situation is detected with the images identified as being of inadequate quality, because this statement would be unreliable anyway. The robot must still be secured at this point, although different measures can be taken than in the case of a detected dangerous situation.A currently missing detection capability is a potential threat and therefore less critical than a dangerous situation that has already been explicitly recognized.

[0046] Figure 2illustrates the training of the detection capability test and shows a representation of the targeted perturbation of training data and the automatic annotation of whether the respective perturbed training data still allows the detection of a dangerous situation. The starting point is undisturbed training images 20, which are preferably obtained from real sensor data and are not further manipulated. The undisturbed training images 20 should include both those in which a dangerous situation is detected, in particular an object or a person in a situation that is not permitted for safety reasons, as well as those without a dangerous situation or without a foreign object. The background should preferably be that of the safety application or at least sufficiently similar to it.For supervised learning, it is also preferable to know from the outset which of these two classes a training image 20 belongs to, although this can alternatively be determined automatically later on. Such training examples are very difficult to obtain and therefore are not available in sufficient numbers in practice. This applies especially to the various error cases and disturbances considered here.

[0047] To obtain sufficient training images that are distorted or corrupted in different ways and to varying degrees, the undisturbed training images 20 are subjected to at least one perturbation algorithm 20a-n. This follows a similar idea to data augmentation, with which a training data set is augmented through artificial modifications in a conventional manner. However, in contrast to conventional augmentation, the aim here is specifically to teach the machine learning method to evaluate such perturbations or changes during the detection capability test. A perturbation algorithm 22a-n manipulates the training image in at least one perturbation property or corruption modality at a definable perturbation intensity.

[0048] To enable reliable detection capability testing, the list of considered disturbance characteristics should be as complete as possible, i.e., it should reflect all relevant disturbances that could occur during operation. This is not always possible, but the generalization capability of a neural network helps to compensate for any gaps. For images, which are still used as an example of sensor data, the following list of disturbances can be given. This is not necessarily exhaustive, but is comprehensive enough for many security applications: The image is too bright or too dark, the image is blurred in at least some areas, the image contains motion artifacts or image noise, image regions are swapped, or the image is incomplete.

[0049] With the help of at least one perturbation algorithm 22a-22n, perturbed or corrupted training images 24a-n are created in a well-defined manner. Since there are multiple perturbation algorithms 22a-22n, and each can perform its perturbation with varying levels of intensity, many more perturbed training images 24a-n can be obtained than the initially available undisturbed training images. Furthermore, the perturbed training images 24a-n specifically contain the features to be trained.

[0050] To avoid having to evaluate the distorted training images 24a-n manually, which is not excluded, they are subjected to at least one detection algorithm 26a-m. This could be, for example, an object detector or another method for assessing whether a dangerous situation exists. Preferably, this is the method that will also be used later in operation to assess dangerous situations. This follows the heuristic that the criteria for assessing detection capability should then be the best fit. However, diversification in several directions is conceivable: During training and / or operation, several methods can be used, whereby they may or may not differ from one another between training and operation. This also supports the ability of the neural network trained in this way to generalize to unknown situations.

[0051] With the help of at least one detection algorithm 26a-m, the distorted training images 24a-n are now labeled or annotated. It is thus known whether each image is an example of an image that allows the assessment of a dangerous situation or whether the distortions are too strong for this. The distorted training images are sorted into two buckets, one for positive examples 28a and one for negative examples 28b.

[0052] Figure 3 shows in addition to Figure 2 An exemplary flow diagram of the modification and annotation of training data and the resulting training of a neural network for a detection capability test. In a first step S1, an undisturbed training image 20 is selected from the initially existing database of training examples, particularly from real data.

[0053] In a step S2, the training image is perturbed or corrupted. The corresponding perturbation algorithms 22a-22n have already been described with reference to Figure 2 The training image is modified with a noise feature or a combination of noise features, whereby most noise features do not binary complement a noise feature, but rather gradually change the training image with a noise intensity 22a-22n.

[0054] The respective resulting distorted training image 24a-n is evaluated in a step S3 to determine whether a dangerous situation is detected therein. Preferably, the evaluation that will also be used in operation is used for this purpose. The detection capability test is thus tested for a specific, namely the relevant detection algorithm 26a-m, wherein, as described above, Figure 2mentioned that several detection algorithms 26a-m can be used. It is also conceivable that the neural network for the detection capability test is pre-trained with general object detectors and then retrained application-specifically with the evaluation procedure intended for a specific safety application to assess a dangerous situation.

[0055] In step S4, it is determined whether a dangerous situation could be detected. This can be the result in itself. The training image is assigned a quality value as a label, indicating that the detection of a dangerous situation was possible, or that it is a positive example 28a. Accordingly, a dangerous situation not detected would be a negative example 28b. However, a more robust alternative is to know whether the original, undisturbed training images 20 belong to a dangerous situation or not. This must then be reproduced in step S4 for a positive example 28a; otherwise, it is a negative example 28b.Here, the meaning of positive example 28a and negative example 28b has shifted. A positive example 28a is no longer a recognized hazard, but rather the correct assessment of the hazard situation based on the initial knowledge of the undisturbed training image 20; accordingly, a negative example 28b represents the inability to reproduce this initial knowledge. The classification does not have to be limited to binary positive examples 28a and negative examples 28b. Quantitative quality scores for the various disturbance properties can be trained and later output, either to understand the binary decision (explainable AI) or to enable a subsequent binary decision based on the quality scores.

[0056] Steps S2-S4 can be iterated, with the perturbations being increased from step to step, either by adding a perturbing property or by increasing the perturbation intensity. This allows the limiting case to be determined as to how much perturbation the subsequent evaluation can tolerate, and thus also generates a sufficient number of training examples for images that do not meet detection capability. A further, outer iteration of steps S1 to S4 works through several or all of the undisturbed training images 20 in the database.

[0057] In step S5, the resulting labeled training examples are used to train a neural network for the detection capability test. It may be useful to use multiple neural networks that specialize in certain noise characteristics. To achieve this, the training dataset is divided according to the noise characteristics that influenced its creation. For example, there are noise characteristics that relate to local properties, such as blurred edges, and others that have a global effect, such as the swapping of the two image halves. Different network architectures may be advantageous for these.

[0058] It has been mentioned several times that the invention was explained using the example of camera 10 with its 2D images, but other sensors are also possible. In this case, other interference characteristics are preferably also taken into account. In the case of a 3D sensor, it is also conceivable to use different data for the detection capability test and its training, such as raw phase shifts from a phase-based time-of-flight method, than for the subsequent assessment of a hazard, which then works with 3D point clouds, for example.

[0059] The check during subsequent operation of the safety application to determine whether a dangerous situation exists can be performed using any evaluation method, such as object detection using conventional image processing, protective field evaluation, object tracking, and the like. However, the use of a neural network or other machine learning method is also conceivable at this point. Its training can even be based at least partially on the training data from the detection capability test. Furthermore, it is conceivable to retrain a neural network for object detection or other assessment of the dangerous situation using the detection capability test, giving it a dual function. This may require slightly modifying the architecture to obtain additional outputs for quality values.Some well-known neural networks, which are by no means exhaustive, and which can be suitable for detection capability testing, especially in connection with object detection, are Yolo (You only look once), MobileNet, ResNet and PoseNet.

Claims

1. A method for safeguarding a machine (12), in which a sensor (10) monitors the machine (12) and for this purpose generates sensor data that are evaluated so that a dangerous situation is recognized and, in the event of a dangerous situation, the machine (12) is safeguarded, wherein, in a detection capability check, it is checked whether an assessment of a dangerous situation is possible, and otherwise the machine (12) is safeguarded, characterized in that, in the detection capability check, the sensor data are assessed with at least one quality score in a machine learning method and an assessment of a dangerous situation is only considered possible with a sufficient quality score.

2. A method according to claim 1, wherein the quality score is binary or wherein the quality score quantitatively assesses at least one interference property of the sensor data.

3. A method according to one of the preceding claims, wherein the machine learning method has a classifier.

4. A method according to any one of the preceding claims, wherein the machine learning method has a neural network.

5. A method according to any one of the preceding claims, wherein the sensor (10) is a camera or a 3D sensor.

6. A method according to any one of the preceding claims, wherein the evaluation of the sensor data has an object detector which in particular recognizes foreign objects in the environment of the machine (12).

7. A method according to any one of the preceding claims, wherein the machine learning method is trained by a supervised learning in which a plurality of training examples (20, 28a-b) from sensor data with a known associated quality score are used as training data.

8. A method according to claim 7, wherein the training examples (20) are changed with at least one interference property to at least one interference intensity in order to generate further training examples (24a-m, 28a-b).

9. A method according to claim 8, wherein the sensor data comprise images and at least one of the following interference properties is used: image at least regionally too bright or too dark, image at least regionally blurred, motion artifacts, static and / or dynamic image noise, image regions swapped, image incomplete.

10. A method according to claim 8 or 9, wherein the training data (24a-n) are evaluated, in particular using an object detector, to determine whether a dangerous situation is recognized despite the interference property and, depending on the result, to assign a quality score to a training example (28a-b).

11. A method according to claim 10, wherein the training data (24a-b) are evaluated using the same method (26a-b) that is also used to recognize a dangerous situation in the safeguarding of the machine (12).

12. A method according to any one of the claims 8 to 11, wherein the training data (20) are changed with an increasing interference intensity and / or different interference properties until an evaluation of the changed training data (24a-n) no longer recognizes a dangerous situation.

13. A method according to any one of the claims 7 to 12, wherein, in order to provide a safety margin, training examples (28a-b) in which the evaluation still recognized a dangerous situation are already assigned a quality score corresponding to a no longer existing detection capability.

14. A method according to any one of the preceding claims, wherein the evaluation of the sensor data for recognizing a dangerous situation likewise comprises a machine learning method, in particular the machine learning method of the detection capability check in a dual function.

15. A safeguarding system (10, 16, 18) for safeguarding a machine (12), said safeguarding system (10, 16, 18) comprising at least one sensor (10) for monitoring the machine (12) and for generating sensor data and at least one control and evaluation unit (16, 18) in which a method according to any one of the preceding claims is implemented.