Method for training and using a neural network for estimating a confidence level of sensor measurements

Abstract

The invention refers to a method for training and use of a neural network (1) to assess the quality of measurement data (3b) delivered by a sensor (5), wherein the method for training comprises the following steps: providing an original measurement data set (3a), analyzing the original measurement data set or features extracted from the original measurement data set by an analyzing unit and / or an analyzing procedure (6) to generate an analyzing performance, - transforming the original measurement data set into one or more transformed data sets (4a, 4b, 4c, 4d) of different quality levels, - training the neural network by feeding transformed data sets and / or the features extracted from each transformed data sets by an extraction unit (30) on one hand, and for each transformed data set an assigned analyzing performance value on the other hand into the neural network. The invention further refers to a system and a machine with a trained neural network and to a method for operating a system.

Description

[0001] Method for training and using a neural network for estimating a confidence level of sensor measurements

[0002] The invention is in the field of sensor technology and in particular related to a measurement system with one or more sensors. Measurement systems can for example be designed as perception systems. Perception systems comprise perception sensors and perception methods that gather and process information about external conditions, for example parameter values or other pieces of information in the environment of the sensor or sensors. The sensors used in perception systems can include a great variety of sensors like, e.g. optical sensors as cameras, LiDAR, RADAR, and other imaging technologies. A perception system may, however also comprise or consist of other types of sensors like acoustic sensors. In principle, all kinds of sensors can be involved in a perception system.

[0003] The sensors of a system usually gather data and information about the surrounding environment. Sensor data may, for example, be processed by perception methods to identify and categorize objects, people, or other entities in the environment. Perception systems are used across a wide range of automated machines, such as self-driving cars, automated mobile machines, robotics, and surveillance systems.

[0004] Realizing a safe, and reliable perception system presents major challenges. Perception sensors often do not work perfectly in all situations, which means that perception systems always carry the risk of failing perceptions for the following reasons:

[0005] (a) Problems with a sensor: Perception systems rely on perception sensors to collect data about the environment. If the sensors are partially damaged or dirty, they may not be able to collect accurate data, which can result in incorrect perception.

[0006] (b) Problems with difficult external measurement conditions: Perception systems can struggle to perceive the environment in low-light conditions or during adverse weather such as heavy rain, snow, or fog. This can make it difficult for the sensors to gather enough or the correct information, for example to detect objects.

[0007] In the prior art, methods have been discussed to achieve a higher reliability of perception systems. For example, the US patent document US2022 / 0114805 Al describes an automated driving perception system which makes use of a trained multimodal perception method. The training is improved by generating, based on raw sensor data, augmented data by injecting noise, occlusions and misalignments. This way, the system shall be enabled to improve object detection, classification, tracking and sensor fusion. In the US patent document US2021 / 0117787 Al, a method is described for training a machine learning model for determining the quality grade of data sets from each of a plurality of sensors.

[0008] Taking into consideration the prior art, it is one goal of the current invention to improve the reliability of a perception system.

[0009] For achieving this goal, the invention is relating to a method for training a neural network for a system comprising the neural network and an intended function data processing unit to assess the quality of measurement data which are delivered by a sensor and processed by the neural network as well as by the intended function data processing unit, wherein the intended function data processing unit is arranged to execute an intended function by creating a trigger signal based on the measurement data delivered by the sensor, wherein the method for training comprises the following steps: providing an original measurement data set, analyzing the original measurement data set or features extracted by first extraction unit or extraction process from the original measurement data set by an analyzing unit and / or an analyzing procedure , which is arranged to execute the intended function or a part of the intended function to generate an analyzing performance,

[0010] -providing one or more transformed data sets by providing additional measurement data sets of different quality levels, that are used as transformed data sets and / or

[0011] -transforming the original measurement data set and / or additional measurement data sets into one or more transformed data sets of different quality levels, analyzing at least one of the transformed data sets and / or features, which are extracted by a first extraction unit or extraction process from each transformed data set by the analyzing unit and / or the analyzing procedure to generate an analyzing performance and comparing the analyzing performance of the transformed data sets and / or features extracted from each transformed data sets with the analyzing performance of the original measurement data set or the analyzing performance of the features, which are extracted by the first extraction unit from the original measurement data set to assign, based on the results of the comparison, analyzing performance values to the transformed data sets,

[0012] -training the neural network by feeding transformed data sets and / or the features extracted from each transformed data set by a second extraction unit or extraction process on one hand, and for each transformed data set the assigned analyzing performance value on the other hand into the neural network, wherein the first extraction unit or extraction process is arranged to extract different features than the second extraction unit or extraction process.

[0013] The feature extraction by the first and the second extraction unit or process may be, in one possible implementation, different in the following way:

[0014] The first extraction unit or process for example may be arranged to extract features which are also extracted for the execution of the intended function by the intended function data processing unit. The second extraction unit or process for example may be arranged to extract features which are rather reflecting, in the case of images, an overall image quality, which, e.g., shows up in major parts of the image. Hence, the first extraction unit may be arranged to extract features which are more local or concerning a smaller subset of the measurement data than the second extraction unit, which is arranged to extract more global features, or features, which concern a larger subset than the features which are extracted by the first extraction unit or process. For example, in the case of images, the first extraction unit may extract features which are restricted to smaller areas than the features extracted by the second extraction unit. Further, the first extraction unit may for example extract features which are geometrical shapes or concrete objects, whereas the second extraction unit extracts features which reflect geometrical sensor value distributions.

[0015] Therefore, the safety model and the neural network may work, in the case of images, with images showing objects as well as images showing no objects.

[0016] The trained neural network may in some cases be a convolutional neural network. This type of neural network is particularly appropriate for the analysis of image data, including feature extraction and classification of image content. Further details and advantages of convolutional networks are described below. The neural network itself may in one implementation be a computing device, for example a computer or a microprocessor unit. The method for training the neural network may also be implemented on a computer.

[0017] The original measurement data set can be provided by taking measurements by the sensor that delivers the measurement data the quality of which shall be assessed. The original measurement data set can also be a data set measured by a sensor of the same type or kind, for example, a color or black and white photographic sensor, an infrared photographic sensor or a microphone. The original measurement data set can also be a data set provided from another source like the internet or a data base. The original measurement data set should comprise as much information as possible reflecting data that have been recorded or measured. For example, if the sensor is a photographic sensor, the original measurement data set should be a photo taken with optimal weather conditions. By analyzing an original measurement data set and generating, based on these data, an analyzing performance, an analyzed data set and hence a reference analyzing performance is generated which can be measured by assigning to this performance a reference performance value.

[0018] Transformed data sets may in some cases be generated by a digital or electronic or analogue electronic degradation process or also by an optical degradation process or by electronically or digitally simulating such an optical degradation process. This procedure allows for the production of a huge number and variety of transformed data sets, for example more than 1000 or more than 10000 data sets, in a short time. On the other hand, transformed data sets may also be generated by providing additional measurement data sets of different quality levels. These additional measurement data sets may be data sets that have been measured under bad measurement conditions. In the case of image data, the additional measurement data may be pictures taken in bad, foggy or rainy weather or under low light conditions. Further, additional measurement data may be generated using purposely worsened device settings or device settings that are changed randomly. In case the additional measurement data are image data, they can for example be generated by taking photos by a camera choosing randomly changed / varied focus length of the lens, exposure time, aperture, sensitivity or other settings or by manipulating the quality of the images by applying water or dirt on the lens, moving objects in front of the lens, applying different lighting or moving the camera during exposures.

[0019] The transformed data sets may also be a mixture of digita I ly / e lectron ica I ly degraded data sets and data sets that are generated based on additional measurement data sets.

[0020] Before the original and / or transformed data sets are analyzed by the analyzing procedure, a first extraction unit may extract features which are selected to allow for example for an object detection. These features may for example comprise "local" features like geometrical shapes, sizes of objects or other features of geographic subset of images. The analyzing procedure is arranged to execute the intended function or a part of the intended function of an intended function data processing unit to generate an analyzing performance. Therein, executing a part of the intended function of an intended function data processing unit may mean to execute only some process steps or a certain part of the process that is defined by the intended function.

[0021] The intended function may for example comprise or consist of recognizing patterns, for example optical patterns in measurement data and creating trigger signals like alarm signals or control signals for a machine in the case that a predefined pattern is recognized. The intended function as well as the analyzing procedure may be executed by a computer or a microprocessor unit.

[0022] Hence, the analyzing unit and / or the intended function data processing unit may be or comprise or be a part of a microprocessor or a computing device. The analyzing procedure may in some cases for example comprise the detection and / or identification and / or classification of patterns in the original measurement data set, which may, e.g. be an optical image, but may as well be a set of acoustic data like a spectrum. Correspondingly, the analyzing procedure may be a detection of patterns in a data set representing for example an image or an acoustic spectrum. The patterns that can be detected may be linked to real world objects like vehicles, persons or other objects or they may be defined in an abstract way as defined features of an image or spectrum or generally features of a data set. The trigger signals generated by the intended function data processing unit may for example trigger an alarm, blocking a machine or a machine activity or any other activity or allowing an activity of a machine. Further, analyzing may, in the case of object detection, comprise to assign positions to the detected items in the environment of the sensors. The analysis and search for patterns may also comprise movements or data change trends, e.g. development of patterns in several subsequent images or measurement data sets. The analyzing procedure may not only be applied to measurement data but also to features which have been extracted from measurement data by the first extraction unit. Such features may for example be lines or edges detected in an image, shape features like angles, triangles, curves, circles or color features like the local distribution of colors, predominance of certain colors or clarity values, contrast values, or local contrast distribution. The extracted features may be, in the case of image data, a sketch of the original image or a black and white version or any other kind of a version with reduced data volume. But the extraction is not limited to the reduction of data and features that can be recognized by humans. The extracted features can in many cases be abstract like, in the case of image data or acoustical data, the distribution of intensities or a spectrum.

[0023] Such a feature extraction could also be chosen for analyzing image data by the trained neural network during its application after training, but it could also be chosen for a feature extraction, which is performed before providing extracted features as training data to the neural network. Therefore, a potential feature extraction which is performed before analyzing the data sets may be the same type of feature extraction, which may potentially be performed before providing the extracted features to the neural network for the training, or the two types of feature extraction may also be different from each other.

[0024] A process step or first extraction unit for extracting features from the original measurement data set and from the transformed data sets for the purpose to provide training data may be included in the analysis step or it may be performed before the step of analyzing the data for assigning analysis performance values for the training purposes. The feature extraction and the analysis may each be performed by a separate pre-trained neural network or they may be performed by one and the same neural network which can be a convolutional neural network. The first feature extraction process or unit which shall provide extracted features from the data sets to the neural network may also be implemented as a neural network or part of a neural network. The analyzing procedure as well as the intended function of the intended function data processing unit may comprise the recognition of patterns in the data sets or in the features, which have been extracted from the data sets by a first extraction unit. The analyzing unit or procedure may look in one case for predetermined patterns and count the number of one or more predetermined patterns that have been detected. In another case, the analyzing unit may also detect new patterns and compare them with patterns that have been found in other data sets. In the first case, the similarity of two data sets or features extracted from them, for example the original measurement data set and a transformed data set, and hence the quality or analyzing performance of a transformed data set, may be determined based on the type or number of corresponding patterns detected in the data sets which are compared. In the second case, the similarity of patterns which have been detected in two data sets or in features extracted from them may be determined based on well-known algorithms or on pre-trained neural networks. The similarity metrics that is applied depends on the type of patterns that is compared for the different data sets.

[0025] For example, if edges identified in two images are compared, their length, position and orientation may be measured and compared. If colors or color distributions in an image are compared, their distance in an RGB color space may be measured. This distance may be determined for a number of areas in the image. In the same way, the brightness values for pixels or sub-areas of two images may be compared to determine a similarity. Further, extracted features could also include the contrast or average contrast, brightness or average brightness of an image or parts of an image. These values may be measured or determined and the measured or determined values may be compared to assess the similarity of the features for two data sets, for example an original measurement data set and a transformed data set. On the basis of this kind of similarity, a quality or analyzing performance value of a data set may be determined.

[0026] For sound data, to measure a similarity of two acoustic events, for example the spectra or one or more sections of spectra may be compared or sound pressure curves for one or more frequencies or frequency bands may be compared. The same types of features as described above as being extracted before the analyzing process or in the course of the analyzing process may also be extracted from the data sets in order to provide the extracted features as training data directly to the neural network, which shall be trained.

[0027] The result of the analyzing procedure, i.e. the analyzing performance based on transformed, lower quality data sets (data sets with a lower analyzing performance value or with less similarity to the original measurement data) or on features extracted from these lower quality data sets is determined by comparison with the reference analyzing performance of the original measurement data set or features extracted from it, and an analyzing performance value is assigned to the analyzing results that are based on the transformed data sets.

[0028] Since the analyzing unit and / or an analyzing procedure is arranged to execute or at least simulate the intended function or a part of the intended function of an intended function data processing unit to generate an analyzing performance, the assigned performance values are valuable in view of the purpose the measurement data are used for. Hence, if a high performance value is assigned to certain measurement data, it is certain or at least highly probable that these measurement data may successfully be used by the intended function data processing unit.

[0029] The neural network is taught or trained which level of quality, represented by an analyzing performance value, each single transformed data set has compared to the reference performance value. For this purpose, the neural network is trained with the transformed measurement data sets and potentially also of the original measurement data set and / or features extracted from these data sets by a second extraction unit or step on one hand and on the other hand with the analyzing performance value of each of the transformed measurement data sets and potentially also of the original measurement data set. Examples for features that may be extracted for the analysis have been described above. Further features that can be extracted from the data sets and used for the analysis and for the training of the neural network may be for example subsets or a selection of the data of the measurement data sets and the transformed data sets like image data projected on channels of an RGB plane in the case of imaging data and in the case of acoustic data certain filtered spectra or a sound pressure or a function representing the changes of sound pressure or a sound intensity in a certain frequency over time. The trained neural network, after the training, is enabled to assign a performance value to each future real measurement data set that is captured by the sensor or to respective features that have been extracted from such captured measurement data sets.

[0030] It is a special and advantageous feature of one potential implementation of the invention that the assignment of a performance value and hence, the quality assessments of the original measurement data set and transformed measurement data sets are each based on analyzing procedure results, which are related to the results of an intended function, which is executed by an intended function data processing unit based on the measurement data. By this structure, the quality assessment has at least one focus on the aspects and properties of the measurement data sets, which have an importance for the neural network and the intended function data processing unit during runtime / application and actual use. The analyzing procedure for the training process of the neural network may be appropriately selected depending on the purpose for which the data measured by the sensor are used.

[0031] The focus of the training of the neural network is on the kind of quality that is delivered by the sensor and useful for the intended function data processing unit.

[0032] The invention may further provide that the transformation of the original measurement data set into transformed data sets comprises one or more of adding noise, filtering data, partially randomizing data, blurring data masking distorting data and / or the invention may provide that the transformation of the original measurement data set into transformed data sets in the case of image data comprises the simulation of optical defects. The transformation of the original measurement data set therefore may cause a change of the data, which is not reproducible and not reversible by the analyzing procedure and therefore, cannot be compensated. In the course of the transformation, information about the original measurement data set, for example the environment of the sensor or sensors, which was comprised or represented by the original measurement data set, is lost at least partially. This process can be described by a loss of quality of the data. The transformation can take place in several steps generating by a digital process or an elec- tronic / analog electronic process or any other data degrading process transformed data sets of different quality levels, i.e. data sets which comprise less and less of the original information about the original data, e.g., the environment of the sensor or sensors. The analyzing procedure usually generates less useful results from data sets with a lower quality. This fact allows for taking the result of the analyzing procedure as an indicator of the quality of the original measurement data set as well as of the transformed data sets.

[0033] In order to measure the quality of the data, a quality value can be assigned to the results of the analyzing procedure.

[0034] The different kinds of data transformation or data deterioration listed above can be implemented by electronic, in particular digital processes in a computer or a microprocessor device in order to be controllable and reproducible. These processes may simulate data changes that may also occur at the sensors in the real world. For this purpose, measurement data may first be changed by manipulating the sensors and then, the changes which occur may be analyzed and simulation algorithms may be based on the changes that have been observed. This way, in short time a large number of transformed data sets may be generated which is useful for the training process of a neural network. Some of the transformation mechanisms may simulate special environmental conditions. In the case that the sensors comprise imaging sensors, this may comprise low light conditions, a strong light beam hitting a sensor, rain, fog, raindrops on the surface of a sensor or full or partial occlusion of a sensor surface. In the case of acoustic data, the transformation may simulate noise, filtering of frequency bands or background sounds. Further, a method may provide, that the transformed data sets are one or more of resized, normalized and augmented.

[0035] These steps may be performed before the analyzing step of the data sets, before the extraction of features from the data sets by an extracting unit or process and before using the data for the training of the neural network. Resizing the data sets involves changing the size of the input data to a fixed size for computational efficiency or to ensure compatibility with the input size requirements for the analyzing procedure and / or the training of the network. Data normalization relates to scaling the input data to a common scale or range of values. This is done to make the input data more comparable and to ensure that the model or neural network can learn from the data effectively.

[0036] The data augmentation comprises the change of content of the original measurement data, loss of information, which was comprised in the original measurement data and partially the creation and addition of new information which is only partially or not at all linked to the information which was comprised in the original measurement data.

[0037] Finally, the analyzed dataset may be converted to a standardized format, e.g. a multi-channel image format, in the case of imaging data e.g. an RGB image format, where each channel is assigned useful information such as color, intensity, reflectivity, location, temperature, etc. For example, if a multi-dimensional point cloud is used as input, each 3-D or N- D point can be projected onto a 2D plane defined by the grid or generally speaking to a lower dimensional entity and pixel values can be assigned based on the local intensity.

[0038] These so converted or transformed data sets and / or features that have been extracted from the transformed data sets are used as training data sets for the training of the neural network or, in other words, for the safety model training. Extracted features may for example be certain patterns extracted from the transformed data sets, like, e.g. areas of high contrast, brightness or special colors or other qualities. It may also be provided that analyzing a data set or features extracted from a data set comprises determining the similarity of the data set or features extracted from the data set with the original measurement data set or features extracted from the original measurement data set, in particular identifying predefined patterns in the data set or in features extracted from the data set and comparing them with predefined patterns identified in the original measurement data set or in the features extracted from the original measurement data set.

[0039] Therein, the determination of the similarity of two data sets or features extracted from different data sets may include well defined metrics which have been defined in a deterministic way or algorithms that have been trained or are available as pre-trained modules, for example neural networks. Potential metrics for similarity have already been discussed above.

[0040] Analyzing a data set by the analyzing unit or procedure or by the intended function data processing unit generally may refer to analyzing generally a measurement data set or the original measurement data set or a transformed data set. For example, if the measurement data set is a set of imaging data from one or more sensors, recognizing patterns may include detecting / recog- nizing / identifying persons, objects, vehicles, numbers, letters, graphical icons, color spots of defined colors, intensive light spots, edges or peaks or other items which are part of the optical data measured by at least one of the optical sensors. Analyzing data sets may also refer to detecting predefined or not predefined classes of features in the data sets.

[0041] In the case of other sensors than imaging sensors, e.g. acoustic sensors, it may include recognizing specific noise or specific sounds or specific acoustic patterns which may indicate specific elements like a vehicle passing by or a damage of a motor running in the vicinity of a microphone which is acting as an acoustic sensor.

[0042] One implementation may also provide, that the analyzing performance value of a transformed data set is measured as a similarity of patterns that are identified by the analyzing unit or the analyzing procedure in the transformed data set or in the features extracted from each transformed data set on one hand and patterns identified in the original measurement data set or in features extracted from the original measurement data set on the other hand, or that the analyzing performance value of a transformed data set is measured as a number of predefined patterns that are identified by the analyzing unit or the analyzing procedure in the transformed data set or in the features extracted from each transformed data set as a share of the predefined patterns that are identified by the analyzing unit or the analyzing procedure in the original measurement data set or in the features extracted from the original measurement data set or that the analyzing performance value of a transformed data set is measured as the sum of confidence values of objects or predefined patterns that are detected by the analyzing unit or the analyzing procedure in the transformed data set or in the features extracted from each trans-formed data set.

[0043] Such a share of detected predefined patterns may refer to the number of patterns in one or more determined and / or predefined class(es) of patterns. The measurement of similarity of two patterns and possible metrics for the similarity have been explained above for a variety of potential patterns and features.

[0044] In the case of a system with imaging sensors, the analyzing procedure for training data as well as the intended function during application of the system may comprise detecting and in some cases, identifying a person or an object in the data sets. The analyzing may also include to localize the detected items and classifying them depending on the position where they have been detected. Some positions or certain parts of the space which is supervised by the sensors and is included in the image data may be specially classified, so that an item, which is localized in a classified place is assigned a special class or safety value. Therefore, certain areas of the images and corresponding subsets of the image data may be excluded from the analyzing procedure or included with higher or lower weight or the analyzing procedure may be restricted to certain areas.

[0045] For example, if items are detected in a special radius around a sensor or in a particular section of the environment and hence, in a particular section of the image of the environment, they (or the overall situation) may be classified as dangerous.

[0046] It may therefore further be provided, that a subset of the original measurement data set is selected for the training of the neural network or a subset is given a higher weight than the rest of the original measurement data set for the training of the neural network.

[0047] This measure may be taken, as explained above, if not all of the original measurement data are important for the surveillance or analysis during a later application of the trained network or some parts or sections of the training data are more important than others. In the case of imaging data, the subset of data may for example be related to certain directions relative to the sensors or certain sections of the images or of the environment of the sensors. A subset of the original measurement data set may also be reduced to features extracted from the original measurement data set, like, in the case of photographic data, a black and white image or an image with only reduced or selected colors where the original was a color image, or a reduction of an image to only edges of objects, or the reduction to a sketch showing the lines of highest contrast in the image or showing lines of a contrast above a certain value or a reduction to black and white without grey regions, that is, a contrast image with only white and black parts, wherein the white parts are those regions of the image where the brightness is above a defined level.

[0048] It may, for example, be provided that the subsets of imaging data which are provided for the training to the neural network or from which features are extracted to provide them for training to the neural network, is identical or not identical to or only overlapping with the subsets of image data that are analyzed in the analyzing procedure or from which features are extracted for performing the analyzing procedure to determine analyzing performance values. In the case of acoustic sensors, a subset of data provided to the neural network for training may for example be related to a certain part of the acoustic spectrum or sound volume or certain sound patterns over time, while the subset used for the determination of a quality may be different To elaborate further on this example, the whole spectrum may be provided to the neural network, but for the determination of the quality, the noise could be used and only the noise in a limited part of the spectrum could be extracted as a feature and further be analyzed.

[0049] In some cases, a subset of a data set may also be a set of features that have been extracted from the data set. A further implementation of the training method relates to a method for training a neural network and an intended function data processing unit which are arranged to work together in a system with at least one sensor wherein the system provides that the or each sensor takes measurements to generate measurement data, that the measurement data are sent to the neural network which assigns an analyzing performance value and / or, based on the analyzing performance value, a sensor confidence level to the measurement data and wherein the intended function data processing unit is arranged to trigger a signal, based on the measurement data as well as the analyzing performance value and / or the sensor confidence level, the method for training comprising: training the intended function data processing unit by feeding a multitude of measurement data of the or each sensor into the intended function data processing unit, wherein the measurement data are labeled with the target values of the intended function for each measurement data, and wherein the measurement data, which are used for training the intended function data processing unit are defined as the in-distribution measurement data and wherein measurement data, which have not been used for training the intended function data processing unit and differ in their feature distribution from the measurement data, which have been used for training the intended function data processing unit, are defined as out-of-distribution measurement data, training the neural network with one or more original measurement data sets and with transformed data sets according to one of training methods described above, wherein the one or more original measurement data sets are selected from the in-distribution measurement data.

[0050] Therein, the statement that out-of-distribution measurement data differ in their feature distribution from the measurement data, which have been used for training the intended function data processing unit, may mean that for a defined feature extraction process, a certain difference threshold in number and / or types of extracted features may be predefined and out-of-distribution measurement data are defined such that they lead to a difference of extracted features compared with the features extracted from in-distribution measurement data, that is above the predefined threshold.

[0051] By training the intended function data processing unit and the neural network in a coordinated way, it can be made sure that the performance value and hence, the confidence level which is determined by the neural network is significant for the functioning of the intended function data processing unit.

[0052] The intended function data processing unit may therefore be implemented as a learning system, for example, a second neural network, and may be trained for the intended function. Often, the intended function is to identify dangerous situations based on information concerning an environment of a vehicle or a machine or tool. Target values for the training of the intended function may therefore be the information, that a trigger signal shall be generated or not or that a dangerous object, pattern or situation has been identified or not. The information is collected from the measurement data of sensors and analyzed according to the intended function. The intended function data processing unit may be trained to create an alarm signal or a control signal if for example, a person or an object is approaching or appears at a certain position. This can be analyzed based on an image analysis or even a sound analysis with an algorithm which has been trained accordingly. A very similar or at least related algorithm may then be used for the analyzing unit or procedure which is used for labeling the training data of the neural network, which may also be named as the "quality assessment neural network".

[0053] A further implementation relates to a method for training a neural network and an intended function data processing unit which are arranged to work together in a system with at least one sensor wherein the intended function data processing unit is arranged to trigger a signal based on the measurement data, on the analyzing performance value and / or sensor confidence level and on the information if the measurement data are in-distribution data or out-of- distribution data, wherein the neural network is trained for distinguishing if the similarity of measurement data or feature patterns of measurement data with measurement data that have been used for training the intended function data processing unit or feature patterns of measurement data that have been used for training the intended function data processing unit is above a predefined threshold.

[0054] In some cases, but not necessarily, the training of the neural network for distinguishing if the similarity of measurement data with measurement data that have been used for training the intended function data processing unit is above a predefined threshold, that is, distinguishing if the measurement data are in-distribution measurement data, may be executed by feeding into the neural network on one hand randomly created artificial measurement data and on the other hand the information that the measurement data are not indistribution data or feeding in a clustering sub-model, in particular a dimensionality reduction sub-model of the neural network on one hand random data which do not belong to the identified clusters or random dimension-reduced data which differ in at least one dimension from the dimension-reduced data which are results of the clustering or the dimensionality reduction of the original measurement data set and the transformed data sets and on the other hand the information that the random dimension-reduced data do not reflect in-distribution data. A clustering sub-model may have the function to classify or distinguish data and assign them to different clusters or groups or classes. It may establish the groups or classes in an appropriate way so that differences between different clusters, classes or groups are as clear as possible. The clustering in some cases may be supported or enabled by a reduction of dimensions of the data space.

[0055] In addition to the training described above, the training of the neural network for distinguishing if the measurement data are in-distribution data or out-of- distribution data may provide to feed the neural network or a dimensionality reduction sub-model of the neural network on one hand with measurement data or dimension-reduced data which reflect a high level of similarity with indistribution measurement data and on the other hand with the information that the measurement data are in-distribution data.

[0056] By training the neural network (the quality assessment neural network) to distinguish if measurement data are in-distribution measurement data or not, i.e. out-of distribution measurement data, in addition to the training of the quality assessment as described above, it can be made sure that, during runtime, the intended function data processing unit only creates trigger signals based on measurement data which on one hand have a certain quality or confidence level and, on the other hand, have been used to train the intended function data processing unit. Hence, the appropriate working of the intended function data processing unit can be ensured, for measurement data which are out-of-distribution, i.e. the intended function data processing unit has not been trained with them, it cannot reliably be guaranteed that the intended function data processing unit works with the right results or reacts properly.

[0057] A further implementation of the invention relates to a data storage device comprising a plurality of stored instructions that, when executed on a processor circuitry of the computer controlled device, cause the computer controlled device to perform a method according to the description above.

[0058] A further implementation of the invention relates to a computer controlled device and data storage device comprising a plurality of stored instructions that, when executed on a processor circuitry of the computer controlled device, cause the computer controlled device to perform a method according to the description above.

[0059] In this case, the computer-controlled device may perform itself or control the original measurement by the sensors as well as the multiple transformation of the original measurement data sets into transformed data sets and the analyzing of the original measurement data sets and the transformed data sets. Further, the computer-controlled device may perform itself or control the assignment of performance values to the measurement data sets and the device may send the data sets as well as the performance values to the neural network to train it as described above. Further, the computer-controlled device may also perform the training of the intended function data processing unit by feeding it with the measurement data and the target values of the intended function, wherein the target values may consist of or comprise control signals or alarm signals.

[0060] The invention may further relate to a neural network that has been trained according to one of the methods explained above.

[0061] Further, the invention may relate to a neural network, in particular a convolutional neural network, with at least one clustering sub-model, in particular a dimensionality reduction sub- model and at least two fully connected layers, which are connected to the clustering sub-model, in particular a dimensionality reduction sub- model, wherein the neural network has been trained according to one of the methods described above.

[0062] The convolutional neural network may, for example comprise a "categorizer", which may be implemented as a support vector layer or sub-network and which acts as a dimensionality reduction sub- model. This kind of sub-model may find on its own certain clusters or point clouds in a two- or multi-dimensional space and may build categories based on the point clouds. A first fully connected layer may be connected to the dimensionality reduction submodel and may execute the performance value determination.

[0063] A second fully connected layer may be connected to the dimensionality reduction sub- model and may execute the determination if measurement data are in-distribution or out-of-distribution measurement data.

[0064] The invention may further relate to a system comprising a neural network and an intended function data processing unit wherein the neural network has been trained according to a method described above.

[0065] The invention may further relate to a system comprising a neural network and an intended function data processing unit which have been trained according to one of the methods described above.

[0066] The neural network may implement an architecture of a convolutional neural network (CNN) for classification. This can include several layers such as Convolutional layers, pooling layers, in particular MaxPooling layers, Dropout layers, and fully connected layers. The number of layers, filters, and neurons can vary depending on the size and complexity of the input data. The network may comprise more than two or more than 4 groups of convolutional layers and MaxPooling layers between the groups to form a deep convolutional network which allows for a highly complex analysis. The CNN is trained using the training dataset. During training, the weights of the CNN are adjusted iteratively with the original data set and the transformed data sets on one hand and with the performance values on the other hand to minimize the loss function between the predicted output (performance value) and the actual output of the network, when the input is one of the transformed data sets. The feature extraction and the analysis procedure of data sets may both be performed by the same CNN. CNNs have the special advantage that they allow for a reliable and efficient feature extraction on one hand and that they are appropriate for a categorization of features or patterns on the other hand. Further, by the insertion of MaxPool layers, the volume of data that are handled by the network can be reduced which provides for a high efficiency and capability of fast analysis of data. Fully connected layers are appropriate for an efficient classification of identified patterns.

[0067] Further, CNNs generally have the advantage to be very efficient in image analysis and classification. The proposed CNN can easily be designed to guarantee translational and rotational invariance. Of course, the neural network (the quality assessment neural network) may comprise at least one dimensionality reduction sub- model and at least two fully connected layers as described above.

[0068] The invention may further relate to a system with at least one sensor and a neural network and an intended function data processing unit which have been trained according to the method described above, wherein the intended function data processing unit is assigned to one or two or more than two sensors and is arranged to execute an intended function consisting of or including identifying predefined patterns in the measurement data or in features extracted from the measurement data delivered by the sensors and generating an alarm or trigger signal or starting or stopping a process of a machine only under the condition,

[0069] -that a predefined pattern has been identified based on measurement data of the sensor or sensors,

[0070] -that the confidence level of a selection of one or more sensors or all sensors or the system safety value of the group of sensors has been evaluated by the neural network to be above a predefined threshold and

[0071] -that the neural network has confirmed that the measurement data are in-dis- tribution data.

[0072] Therein, the intended function data processing unit for application can according to one implementation be arranged in the system relative to the trained neural network in such a way that the neural network after assignment of an analyzing performance value or a sensor confidence level to the measurement data set(s) either sends or allows to send the measurement data sets to the intended function data processing unit if the quality of the data is above a predefined confidence level and the measurement data are indistribution data or otherwise, it prevents the forwarding of the measurement data sets in case the quality of the data is below a predefined confidence level or if the measurement data are out-of-distribution data. In an alternative implementation, the measurement intended function data processing unit, wherein on one hand, the neural network assigns a confidence level to the data sets and determines if the measurement data are in-distribution data or out-of-distribution data and, on the other hand, the intended function data processing unit performs its intended function, for example an analysis / detection / identification process. After that, depending on the assigned confidence level and the confirmation if the measurement data are in-distribution data, the results of the further processing are forwarded or not forwarded to a next step in the system in order to decide about generating an alarm or trigger signal and / or controlling or blocking a machine which uses the system or is part of the system. The further data processing that is performed by a intended function data processing unit may include any step of extracting features or information from the measurement data set. The data processing by the intended function data processing unit may also potentially include a process that is identical or similar to the analyzing procedure that is performed for training the neural network.

[0073] The intended function data processing unit can in some implementations be focused on object detection and may be implemented as a learning system, in particular a neural network, for example a convolutional neural network and the feature extraction which is potentially performed before the further data processing may be different from the feature extraction that is performed with the measurement data before forwarding the extracted features to the trained neural network for quality assessment. The feature extraction which is potentially performed before the further data processing may comprise the extraction of localized features like edges, lines or curves or points or regions of high or low contrast or brightness. The system may be designed as a safety system, which checks the environment or vicinity of the sensors about present items or features, e.g. objects or persons by analyzing the measurement data delivered by one or more sensors by an intended function data processing unit. If a dangerous situation is detected and an object or person, that is identified as representing a danger approaches a sensor or features in the measurement data are detected that represent a dangerous situation or should trigger a process, an alarm, a control signal or trigger output is generated by the system as long as the confidence level of the measurement data and / or an overall safety level of the machine or system is satisfying / above a predefined level and the measurement data are identified as in-distribution measurement data . As soon as the confidence level of the measurement data is not satisfying / below a predefined level or the measurement data are identified as out-of-distribution measurement data, the system may be blocked or at least, no clearance signal or control signal may be generated by the system and the content of the measurement data may not be trusted.

[0074] The invention may also be implemented by a machine with a system as described above, wherein the machine is arranged to block, trigger, stop or limit at least one of its activities upon receipt of an alarm or trigger signal. Potential activities therein may be, e.g., a movement of the machine or the application and use of a tool.

[0075] In this case, either the system is part of the machine or the machine is a part of the system. In any case, the machine may be a vehicle, for example a caterpillar or a crane or another working machine, or a car for transport of humans and / or material. The system may control the machine in order to avoid dangerous situations and guarantee a defined safety level. For example, the sensors and the neural network may be mounted on the machine or vehicle or in the vicinity of a working location of the vehicle or machine.

[0076] Further, the invention relates to a method for operating a system with at least one sensor and a neural network and an intended function data processing unit wherein the neural network and the intended function data processing unit have been trained by a process according to a method described above, in particular a method for operating a system as described above, or for operating a machine as described above, wherein the or each sensor takes measurements to generate measurement data, the measurement data of the or each sensor are sent to the neural net-work which assigns to the measurement data on one hand a analyzing performance value and, based on the analyzing performance value a sensor confidence level and on the other hand the information if the measurement data are indistribution data or not and wherein the measurement data are further processed by the intended function data processing unit only if the sensor confidence level is above a predefined threshold and the measurement data are indistribution data.

[0077] The system by this procedure will produce a confidence level for sensor data after a measurement. The confidence level comprises the information if or to what extent the measurement result is reliable. Further, the neural network determines if the measurement data are in-distribution measurement data, i.e. if the intended function data processing unit has been trained with measurement data of the same kind. If one of these conditions is not fulfilled, the intended function data processing unit may not proceed with the measurement data. Otherwise, the output of the intended function data processing unit may be an alarm, for example an acoustical or optical alarm, a control signal or a trigger signal, for example a digital or electrical trigger signal, if the sensor and / or measurement is reliable, the measurement data are in-distri buton data and the intended function data processing unit detects a reason for creating a trigger signal. The alarm or trigger signal may be sent to an electrical circuitry, which initiates, blocks or modifies additional steps. The output of the system may also be a signal, which causes the sensor to repeat the last measurement or replace the last measurement data by other data or a signal which causes to replace the sensor by another sensor or replace the measurement data by measurement data of another sensor which has performed a measurement at the same time, if the sensor confidence level is too low or if the measurement data are out-of-distribution measurement data. Thereby, redundant or complimenting sensors may be controlled by the system.

[0078] The alarm or trigger signal may also comprise a signal, which initiates a safety routine for checking all available sensors and / or their measurement results or for cleaning or modifying the sensors. In this case, for checking the measure- merit results of the sensors, current or past measurement results of the sensors may be analyzed or analyzed again by the neural network. In any account, the further data processing of the measurement data may depend on the confidence level that is assigned to the measurement data by the neural network and on the determination, if the measurement data are in-distribu- tion data or not.

[0079] A further implementation may relate to a method for operating a system with at least one sensor and a neural network and an intended function data processing unit wherein the neural network and the intended function data processing unit have been trained by a process according to a method described above, in particular a method for operating a system as described above, or for operating a machine as described above, wherein the or each sensor takes measurements to generate measurement data, the measurement data of the or each sensor are sent to the neural network which assigns to the measurement data on one hand a analyzing performance value and, based on the analyzing performance value a sensor confidence level and on the other hand the information if the measurement data are indistribution data or not and wherein the intended function data processing unit creates a trigger signal if either

[0080] -the neural network confirms that he sensor confidence level is above a predefined threshold and the measurement data are in-distribution data and the intended function data processing unit is detecting a pattern on which it has been trained to create a trigger signal or

[0081] -if the sensor confidence level is below a predefined threshold or

[0082] -if the measurement data are not in-distribution data.

[0083] By the method which is described above for operating the system, it is ensured that a trigger signal, e.g. an alarm signal is given as soon as an uncertain situation occurs wherein the uncertain situation may be caused either by a sensor which is not working appropriately or at least measurement data which do not have the required confidence level or by measurement data which cannot be handled reliably by the intended function data processing unit, because the intended function data processing unit has not been trained with these measurement data or by a reliable determination by the intended function data processing unit that, based on reliable, high quality measurement data, a certain situation or pattern has been identified according to the intended function. The trigger signal, which e.g., may be generated if measurement data are out-of-distribution measurement data, may, among other potential consequences, also mean or have the consequence that a machine activity is continued in a secure mode, for example in a slow working mode.

[0084] A further implementation may relate to a method for operating a system as described above, wherein the measurement data that have been generated by measurements taken by the sensor or sensors, apart from the assignment of a sensor confidence level, are further processed by the intended function data processing unit which is identical or similar to the analyzing procedure or has an identical or similar function or structure as the analyzing procedure, that was applied during the training of the neural network for the transformed data sets and the original measurement data set and / or features extracted from the data sets.

[0085] If the measurement data that are generated by measurements taken by the sensor or sensors during operation / runtime of the system are further processed by a intended function data processing unit which is identical or to some extend similar to the analyzing procedure, that was applied during the training of the neural network, it is evident that the definition of the performance value which has been used for training the neural network and after the training determines the outcome of the analysis of any data set by the trained network, is appropriate for providing information about the usability and reliability of the analyzed measurement data for the further processing, since training data and measurement data during application are analyzed with the same or a similar focus.

[0086] A further implementation may relate to a method which provides that during operation / runtime of the system, a subset of the measurement data set is selected or a subset is given a higher weight than the rest of the measurement data set for the assignment of a performance value or safety level by the neural network. A subset of measurement data may mean for example a selection of data, which is part of the measurement data or features extracted from the measurement data.

[0087] This aspect may or may not be combined with a training mode of the neural network in which a subset or extracted features of the original measurement data set and corresponding subsets or extracted features of the transformed data sets are used for the training. In the case of combining training with a subset of data sets or extracted features on one hand and selection of a subset or extracted features of a measurement data set which is measured during application of the trained neural network on the other hand, the same subset of data may be selected during application as was used for the training. In this context, a subset of data may also mean features extracted from the data. For example, the subset of features may mean a subset of global features of the data.

[0088] In any account, by this measure, the effort of the neural network may be reduced, the assignment of confidence levels may be more sensitive in selected areas and the system is less prone to errors, which could be generated by the part of the measurement data, which is not part of the selected subset.

[0089] A further implementation of the method may provide, that the measurement data of at least one sensor are only provided for further use or data processing, if the sensor confidence level of the sensor is above a predefined safety threshold and the measurement data are confirmed to be in-distri buton measurement data.

[0090] It is one aspect of the invention that during use / runtime of the system comprising the trained neural network, it is not only secured that the system performs its function and, e.g. supervises the environment of the sensors by a data processing, but also meta-data are produced by the neural network which reflect the reliability or confidence level of the measured data and also in some cases, of the sensors. Based on the confidence level, the further use of the measurement data in the form of any data processing of the measurement data by a data processing unit may be stopped or blocked, prevented or modified to avoid errors and reduce the risk of malfunctions. In the same way, the further use of the measurement data may be blocked if the measurement data are determined to be out-of-distribution measurement data. In some of these cases, instead of or in addition to blocking a function, it may also be provided, that a trigger signal is generated for starting a process or an action.

[0091] The method may further provide, that, if the sensor confidence level of one sensor or measurement data provided by the sensor is below a predefined threshold, and / or the measurement data are determined to be out-of-distri- bution measurement data, the measurement data of the sensor are replaced by measurement data of at least one other, redundant sensor of the same kind or of a different kind. This redundancy control function has already been explained in detail above. The replacement measurement data can then be further processed by a data processing unit.

[0092] A further implementation of the method may provide, that assigned sensor confidence levels of two or more sensors are combined to generate a system safety value and that the measurement data are further processed by the intended function data processing unit only if the system safety value is above a predefined threshold and the measurement data are in-distribution data.

[0093] The system safety value can be generated based on the confidence level of just one sensor or more than one sensor. Therein, the system safety level can be generated by predefined rules. For example, the system safety level may be "unsafe", if the confidence level of one certain sensor or the confidence level of any sensor or the confidence level of all sensors or a subset of two or more sensors or an average confidence level or a combination of the above confidence levels is too low / critical.

[0094] Further, the method may provide, that at least one sensor is a camera, an optical scanning sensor, a LIDAR system, a RADAR system, an imaging sensor generating 2-dimensional or 3-dimensional images of an environment, or an acoustic sensor. These sensors may provide imaging data and they may deliver complimentary data. For example, if there are low light, foggy, or rainy conditions, RADAR sensors may still deliver more reliable or higher quality data than a camera with a sensor, which is sensitive in the wavelength range which is visible to a human. The sensors can also be combined with wavelength filters like color filters or the image data may be processed by projecting the data on certain color systems, like, e.g. the RGB color space. For example, selecting image data according to RGB or other color channels or also brightness channels can support the extraction of features from the imaging data. For example, the combination of a camera with a LIDAR system may be a good choice, because the two sensors may complement each other very well in different weather conditions.

[0095] The invention will be shown based on exemplary implementations in figures of a drawing and described in detail below.

[0096] Therein show:

[0097] Figure 1: a diagram illustrating the function of a perception method,

[0098] Figure 2: several data sets with varying quality,

[0099] Figure 3: a scheme illustrating the training method for the neural network,

[0100] Figure 4: the working process of a system with a trained neural network,

[0101] Figure 5: an alternative working process of a system with a trained neural network,

[0102] Figure 6: a diagram illustrating the scalability of the safety model,

[0103] Figure 7: a diagram showing the thresholds of confidence levels,

[0104] Figure 8: examples for different objects that can or cannot be detected,

[0105] Figure 9: a diagram with the classification of detections or nondetections and their contributions to the classification of confidence levels, and

[0106] Figure 10: a diagram explaining the training of the intended function data processing unit and the neural network, Figure 11: a diagram explaining the function of the neural network after training,

[0107] Figure 12: the internal structure of the neural network,

[0108] Figure 13: a diagram explaining a first training of the neural network,

[0109] Figure 14: a diagram explaining a second training of the neural network and

[0110] Figure 15: a schematic diagram explaining the collaboration of the neural network and the intended function data processing unit in a system.

[0111] Figure 1 shows an electronic or digital system, which has the function to detect certain objects 8a in a measurement data set. The system comprises a sensor 5 in the form of a camera. The measurement data set has / the measurement data have the reference sign 3b. The measurement data set 3b is delivered to a data processing unit in the form of a detection unit or procedure, which is in this application also referred to as intended function data processing unit 31. The intended function data processing unit 31 is arranged to detect certain patterns in the measurement data or measurement data sets. In the case the measurement data sets are imaging data sets captured by a camera, the predefined patterns 8a can be certain distinguishable representations of objects like for example geometrical features like straight lines or cor- ners / angles or human beings / persons or global features like a certain brightness or contrast level of an image.

[0112] On the right side of Figure 1, the reference sign 8a refers to a detected person in a frame in dotted lines.

[0113] Further, on the right side of Figure 1, three different measurement data sets in the form of images are shown with different qualities, wherein the upper box shows the best quality, the box in the middle a medium value quality of measured data, i.e. a blurred or foggy image, and the lower box a poor quality. The different quality levels may be caused by different environmental conditions like weather conditions or by a deterioration of the sensor / camera. Hence, the predefined pattern / person is distinguishable in the measurement data set represented in the upper box, while the detection / identification of the person in the middle box is uncertain and the detection in the lower box is impossible or highly uncertain. The problem is, that in many cases the intended function data processing unit 31 does not provide information about the certainty of its detection or analysis results. Various efforts may be made to improve the results of, for example, a detection mechanism, but even with improved results, a user of a detection system will not have more certainty about the reliability of the results, even if they become better from an objective perspective.

[0114] If this problem shall be solved, e.g., by a system with a trained algorithm or neural network, a huge quantity of training data is needed. These data should preferably comprise measurement data of varying quality because the system needs to be trained with data of different levels of quality.

[0115] Figure 2 shows as an example including three groups of data sets: on the left side, one or more original measurement data sets 3a and on the right side in the box which is defined by a dotted line, transformed data sets 4a, 4b of a lower quality. The original measurement data sets can be generated by measurements taken by a sensor with optimal measurement conditions. The transformed data sets are generated based on an original measurement data set with a good quality by various transformation activities, which partially change or eliminate the data of the original measurement data set and conserve only a part or one or more aspects of the original data.

[0116] By the transformation, various effects of data changes can be imitated or simulated which may occur in a similar way in the real world with the measurement data, like blurring of a picture, low light conditions, glaring of light, fog, rain or a raindrop on the lens of a camera. Similar transformations may be performed with other kinds of measurement data like, e.g. acoustic data. These transformations may include filtering of frequencies or frequency bands or adding noise, adding background sounds or distorting sounds.

[0117] The advantage of the production of transformed data sets in a different way than just measuring under different conditions is that a high number of measurement data of different quality may be produced by an automatable technical process in a reproducible or controllable way. This is important because for the training of a neural network, extremely high numbers or quantities of training data are desirable. In Figure 3, the training process of a safety model, also referred to as neural network 1 or quality assessment neural network, is shown. The neural network may be designed as a physical network that may be implemented as a computing device or as an electronic circuitry, but also as an algorithm, which is implemented as a software on a computer and may be stored on a storage device. According to Figure 3, the sensor 5 in the form of a camera takes pictures of the environment, for example, pictures in the wavelength spectrum, which is visible for a human eye. The sensor 5 may also be implemented as a combination of two or more imaging sensors, for example, as a combination of a camera with a LIDAR system. These pictures, which serve as a basis for training data, are referred to as original measurement data set 3a. The original measurement data set(s) is (are) sent to a processing unit which transforms and changes in step 32 the original measurement data set(s) and generates transformed data sets. The original measured data set 3a and transformed data sets 4a, 4b are sent in step 33 to the analyzing procedure or unit 6 which may be arranged as a trained neural network and / or as a software process or algorithm running on a computing unit or microprocessor, which can detect and / or identify patterns in the data set(s). For this purpose, the analyzing unit or procedure 6 may perform a similar or the same function as the intended function data processing unit 31, but in some cases, the functions of the analyzing unit 6 and the function of the intended function processing unit 31 may also differ to some extent. For example, the analysing unit 6 as a neural network may differ from the intended function processing unit, which may also be implemented as a neural network, with regard to the fully connected output layer. The analysing unit 6 and the intended function data processing unit may be trained in a different manner. In one implementation, the analysing unit 6 and the intended function data processing unit may be trained in a similar or in the same way and they may be identical with regard to some layers, but the analysing unit 6 may have some layers less than the network which forms the intended function data processing unit. Further, the analyzing unit or procedure 6 may comprise a first feature extraction unit 30a for feature extraction or may receive extracted data from a first feature extraction unit 30a, which may be positioned before the analyzing unit 6 at its data input side. The processing unit 31 and the potential second feature extraction 30b that may be combined with it is described in detail with reference to Fig. 4. The processing unit 31 is used for the further processing of measurement data when the trained neural network 1 is in use. The first feature extraction unit 30a is different from the second feature extraction unit 30 as already explained above.

[0118] Both analyzing units 6 and 31 may be designed as convolutional neural networks, which may rely on extracted features or perform a feature extraction function as well as, subsequently, a classification function of detected patterns. The feature extraction of analysing unit 6 and intended function data processing unit 31 may be different, that is, the features which are extracted and provided to the analyzing unit 6 may be different from the features that are extracted and provided to the intended function data processing unit 31 as discussed further below.

[0119] In Fig. 3, an optional feature extraction unit 30a is shown before the analyzing procedure 6. This unit 30a comprises the extraction of features from the original measurement data set as well as the transformed data sets. The feature extraction may comprise the reduction of data to some aspects and hence, the reduction of data which have to be analysed in the analyzing procedure. The feature extraction unit 30a may extract global features like an overall contrast, sharpness or brightness of an image or an overall blurred effect.

[0120] The features which are extracted by the extraction unit 30 and provided directly to the neural network 1 for training purposes may in some cases be different from the features that are extracted by the extraction unit or process 30a and further used for the determination of an analyzing performance., since the goal of the analysing performance is different from the assessment of a quality to the measurement data.

[0121] In the case that the original measurement data are image data, the features extracted by the feature extraction unit 30 may be global features like an overall or average contrast, brightness or sharpness of an image.

[0122] The features extracted by the second feature extraction unit 30 and forwarded directly to the neural network 1 for the training purposes may be the same features that are extracted from the measurement data during use of the neural network and that are during measurement by the sensors and processing of the measurement data provided to the neural network for the quality assessment as shown in figs 4 and 5 and described further below. In the case of sound measurements, features like certain frequency bands or time windows of an acoustic event may be extracted.

[0123] If the optional feature extraction 30a in Fig. 3 is not implemented, the data sets 3a, 4a, 4b are sent directly to the analyzing unit / procedure 6. The steps of feature extraction 30a and 30 are not obligatory, however, it serves to focus the subsequent data processing and analyzing steps to certain aspects and to reduce the need of resources.

[0124] The analyzing procedure or unit 6 analyses the data sets 3a, 4a, 4b or optionally, the extracted features and performs its analyzing / detection function. For the different transformed data sets with different quality, the performance of the analyzing procedure or unit 6 may be different. Because the transformed data sets are derived from the original measurement data set, the performance of the analyzing / detection function may be compared for the different data sets and analyzing performance values corresponding to quality values may be assigned, wherein the analyzing performance value of an original measurement data set may serve as a reference.

[0125] One concrete example of an implementation of the determination of performance values may be described as follows: If in the original measurement data set, measured under intended triggering conditions of the intended function data processing unit (31), a certain number x of patterns or objects, e.g., persons in an image or edges of objects are detected or identified, it can be assumed that a performance value of detecting x objects or x edges is a performance value of 100%. If in a transformed data set only a certain share, e.g., 50% of the x patterns / objects / edges can be identified, the performance value of the analyzing procedure or unit 6 for this transformed data set may be considered as 50%, in any account less than 100%.

[0126] Another example of an implementation of the determination of performance values may be the measurement of the average contrast of an image taking the average contrast of the original measurement image data as a reference, e.g., 100% and measure the contrast of other image data sets as a share of the reference contrast.

[0127] The analyzing performance may be measured by assigning to each data set a number of identified patterns / objects or a share of 100% performance. The analyzing performance values assigned to the data are represented in Fig. 3 by the reference sign 7. The analyzing performance values serve as training data and are sent directly to the neural network 1 that shall be trained, in a step indicated by the arrow 12. At the same time, the corresponding data sets to which the individual analyzing performance values refer, or features extracted from these data sets by the feature extracting unit 30 are also sent to the neural network 1 in a step indicated by the arrow 13. Hence, a large number of data sets along with a performance value for each single data set can be fed into the neutral network 1 as training data. The neural network 1 is trained to assign to any future measurement data set, which is provided by a measurement by a sensor, an analyzing performance value for a further data processing, for example a detection / identification or other process. The training of the neural network may in some cases include back-propagation methods. The analyzing performance value reflects a quality or reliability of the measurement data provided by the sensor or the sensors, which can be considered as a condition for the further processing of the measured data.

[0128] Figure 4 shows the structure of a system with a sensor, a processing of sensor measurement data and an appropriately trained neural network 1 for detecting the quality or confidence level of measurement data.

[0129] The system comprises one or more sensors like a camera 5 or an optical scanning sensor. More than one camera or scanning sensor may also be provided to offer more information, e.g. in additional wavelength ranges, for example infrared, which enable the system to perceive details from the environment at low light conditions. Two or more optical cameras may also be provided with a distance between the cameras to provide different angles of view and allow for a sensor fusion and calculation of 3d- images. The measurement data sets 3b or fused data delivered by the sensors are sent to a trained network 1, which assigns a quality value or analyzing performance value 7 (conf. Fig. 3) or confidence level to each measurement data set, which is then interpreted as a confidence level of the sensor for a given point of time.

[0130] Optionally, a second feature extraction unit or step 30 may be provided. In this case, features are extracted from the measurement data and the extracted features are sent to the trained neural network 1. The second feature extraction unit 30 and the analysis may be combined in one neural network 1, e.g. if this is designed as a convolutional neural network. Convolutional neural networks are particularly appropriate or the extraction of features from graphical data sets, like, e.g. photos.

[0131] Convolutional neural networks (CNNs) comprise convolutional layers to filter inputs for useful information. A convolution operation is combining input data (feature map) with a convolution kernel (filter) to form a transformed feature map. The filters in the convolutional layers are modified based on learned parameters to extract the most useful information for a specific task. A CNN may, e.g. filter information about the shape of an object in an image for the purpose of object recognition but could as well extract other features depending on the task.

[0132] Convolutional neural networks may be applied with advantage in image recognition or image classification, as well as speech recognition, or natural language processing.

[0133] Convolutional networks may comprise an input layer, an output layer, and one or more hidden layers. The layers of a convolutional network are arranged in three dimensions which allows for transforming an input volume in three dimensions to an output volume. The hidden layers are a combination of convolution layers, pooling layers, normalization layers, and fully connected layers. CNNs use multiple convolutional layers to filter input volumes to greater levels of abstraction.

[0134] Based on the quality value or confidence level of each measurement data set or the extracted features, the neural network 1 may send the measurement data sets to the intended function data processing unit 31 or not. For distinguishing between different outputs like "sufficiently high level of confidence" or "Low level of confidence", the neural network comprises a dimensionality reduction layer which categorizes the results of its analysis. The neural network may, in addition, be able to distinguish also between "in-distribution measurement data" and "out-of-distribution measurement data", as discussed further below. The unit 31 (the intended function data processing unit) may serve as an object detection unit which detects predefined or not predefined objects in the measurement data sets and triggers actions under certain conditions, which depend on the detected objects and it may have a similar structure as the network 1 or and be implemented as a learning system, e.g. as a second neural network. The intended function data processing unit 31 may also comprise a convolutional neural network with a structure similar to that of the neural network 1. If the neural network 1 decides to send a measurement data set to the intended function data processing unit 31, for example, an object detection unit, or to let the intended function data processing unit execute its function on a measurement data set it may be considered certain that the data processing unit 31 can generate a good detection / identifica- tion result based on the data set because during training of the neural network 1, it has been trained to recognize the quality and reliability of data delivered by a sensor which may be of the same kind or similar as the camera 5. Further, the analyzing procedure 6 which has been used for the analysis of original measurement data sets and transformed data sets during the training phase of the neural network may be chosen to be of the same or a similar structure to the processing which is performed by the processing unit 31. However, analyzing procedure 6 may use different extracted features than intended function processing unit 31. In a data set with a high performance value, an object 8a may reliably be detected by the subsequent intended function data processing step 31 as a predefined pattern.

[0135] In Fig. 4, inside the box representing the trained neural network 1, two sections 9a and 9b are shown which represent data sets of different analyzing performance value or quality. The potential performance values of the measurement data 3b after feature extraction by the unit 30 are classified as high quality values 9a or low quality values 9b depending on their position on the vertical quality axis of the diagram shown in Fig. 4 inside the box 1 and if they are below or above a defined quality threshold 18.

[0136] Unreliable detection results with a performance value which is below the threshold 18 may also be registered and forwarded to a unit 14. At the same time, the neural network 1 may prevent the forwarding of these detection re- sults / measurement data to the intended function data processing unit 31 for further use. This is shown in Figure 4 by the symbolic switch 40 which is activated by the neural network through an activating mechanism 40a either to forward the detection results to the unit 31 or to prevent the transmission of the detection results to the unit 31. The unit 14 may collect unreliable performance value results and if a certain absolute number is reached or a certain number or share of the detection results / measurement data sets is considered as unreliable or after a certain number of delivered measurement data sets, still no or too few reliable data sets have been registered, an alarm or trigger signal 15 may be generated which may cause an alarm unit 16 to block a machine 17. Further, the machine 17 may also be blocked as soon as a reliable measurement data set has been provided to the intended function data processing unit 31 and an object 8a or a pattern that may be dangerous for the operation of the machine 17 has been detected and / or identified by the unit 31. Such a dangerous pattern may also comprise a combination of several subsequent measurement data sets, which may represent a movement of an object over time. In the case of detection of a dangerous pattern, an alarm or trigger signal is generated by the data processing unit 31 and sent to the alarm unit 16. Next to the intended function data processing unit 31, the feature extraction unit 30 is positioned to extract features and to provide them for further analysis to the unit 31.

[0137] Figure 5 shows an alternative structure of a system with an appropriately trained neural network 1.

[0138] As according to the system shown in Figure 4, a sensor / camera 5 provides measurement data sets 3b, which are, in one implementation, forwarded directly to the trained neural network 1 and in parallel to the data processing unit 31, which may again be an object detection unit.

[0139] In one possible implementation, the measurement data sets 3b may first be sent to a feature extraction unit 30 represented by a dotted line and only the extracted features are forwarded to the trained neural network 1. Also in this implementation, it may be provided that the feature extraction 30 and the analysis are combined in one neural network, e.g. a convolutional neural network.

[0140] At the same time, the measurement data sets 3b may be sent to the intended function data processing unit 31, for example for an object detection. Before reaching the processing unit 31, the measurement data may be subject to a feature extraction by the feature extraction unit 30b. In this case, only features extracted by the feature extraction unit 30b are further processed by the data processing 31.

[0141] The second feature extraction unit 30 is different from the feature extraction unit 30b, which may be similar or identical to the first feature extraction unit 30a, since the tasks of neural network 1 and the intended function data processing unit 31 are different. The feature extraction unit 30 may for example extract global features from image data.

[0142] The intended function data processing unit 31 performs its detection / identifi- cation function with a positive or negative result, which may in both cases be a good or poor (reliable or not reliable) result, depending on the quality of the measurement data set. For deciding, if the result is good or poor, reliable or not, the trained neural network 1, which may be designed as a convolutional neural network, assigns to each measurement data set or to the extracted features a performance value which indicates, if the performance value or confidence level of each individual measurement data set is sufficient and lies above a predefined threshold 18 and hence, if a processing result, e.g., a detection / identification of an object for each individual measurement data set by the data processing unit 31 is reliable. The threshold 18 shown in the box 1 in Fig. 5 therefore divides the performance range or the range of possible confidence levels into two classes: the acceptable performance values / confidence levels 19 and inacceptable performance values / confidence levels 20.

[0143] The feature extraction 30 may in one potential implementation be included in the trained neural network 1, in particular if this is designed as a convolutional neural network.

[0144] The processing results, e.g., detection / identification results of the intended function data processing unit 31 and the corresponding assigned performance values / confidence levels 19, 20 are then combined in unit 14. On the right side of Fig. 5 as well as on the right side of Fig. 4, the detected object 8a, a person, is surrounded by box, which shows that the object / person has been identified by the processing unit 31.

[0145] Only processing / detection / identification results of measurement data sets with a sufficient confidence level are considered for generating an alarm or trigger signal 15 which is sent to a unit 16 which acts as a control unit or blocking unit for a machine which is not shown in Figure 5. Another trigger may be generated if one or more confidence levels of measurement data sets from one or more sensors are below a defined threshold, because in this case, sensors may be damaged or weather conditions may be generally bad and any measurement may be unreliable and the status and control of the system is uncertain.

[0146] Figure 6 shows a diagram, which shall illustrate how several safety models in the form of neural networks 1, la, which are assigned to different sensors or groups of sensors 5, 5a, may be implemented as trained neural networks 1, la and can be combined to an integrated safety model for a system. The single neural networks 1, la, will assign individual confidence values or confidence classes to each measurement data set of one or more sensors. If a sufficient number of sensors at a point of time are forwarding measurement data sets of a sufficiently high confidence level, those measurement data sets may be fused in an integration unit 23 to generate fused sensor data 21 which may be further processed in an intended function data processing unit 31 (conf. Fig. 5) to generate a comfortable and enriched detection / identification result. At the same time, an overall confidence level 22 of the measurement data for the whole system may be generated by combining individual confidence values of sensors or measurement data to confirm the reliability of the detection / identification result.

[0147] Figure 7 is showing a diagram with different measurement data sets 3a, 4a, 4b, 4c, 4d, of different quality. The vertical axis of the diagram is related to the performance value which is assigned to the data sets, while the horizontal axis is related to the quality of data which is influenced in the training phase by the applied transformation and in the working phase by the environmental conditions and intrinsic problems of the sensors or other elements of the system. (Generally, low data quality leads to a low performance value but the two values are not necessarily proportional, since the analyzing procedure in some ranges of low quality may still be able to compensate for deficiencies of quality.)

[0148] 3a refers to the original measurement data set with the highest quality of data captured under optimal conditions. From this measurement data set, the analyzing procedure / unit 6 (conf. Fig. 3) can detect the maximum number of objects or patterns and hence, the performance value of the analyzing proce- dure / detection / identification function for this data set is at its maximum. The different data sets are divided in classes according to their assigned performance value or performance value classes. The range of performance values is subdivided by two threshold values 24, 25 into three performance value classes or confidence level classes: Highly safe 26, moderately safe 27 and unsafe 28.

[0149] The measurement data sets belonging to classes 27, 28 may in one potential implementation not be used for the data processing 31 / detection / identifica- tion of objects (conf. Fig. 5) or respective detection / identification results will not be confirmed. In another implementation with a lower safety level, measurement data sets belonging to the classes 26 and 27 may be used for the data processing 31 / detection / identification of objects or respective detection / identification results will be confirmed.

[0150] Figure 8 is showing a potential measurement data set with a number of detected objects 8a, 8b, and other items 29a, 29b, 29c, 29d, 29e.

[0151] The objects 8a, 8b correspond to predefined patterns which can be detected by an analyzing unit or procedure 6. Such an analyzing unit or procedure 6 according for each of the implementation examples shown in Figures 1 to 7 may, for example, be realized as a trained neural network. This neural network may also be combined with the neural network 1, which performs the function of a performance value assignment or one single neural network may perform both functions of assigning a performance value and analyzing data sets in the training phase. The neural network for detecting / identifying objects has to be trained appropriately.

[0152] The items 29a, 29b, 29c, 29d, 29e will in most cases not or not correctly be detected and / or identified by the analyzing unit or procedure 6 and by the intended function data processing unit 31 for different reasons: The image of item 29a is too blurred and therefore cannot be identified as the image of a person. Image / item 29b shows just a part or section of a perceivable object and therefore is not identified. 29c is the image of an item which is not expected and possibly does not cause any danger and therefore is not identified, because it is not in a target group of identifiable objects. Image 29d is too small and therefore cannot be detected and / or identified.

[0153] Item 29e is a non- item, which means that the measurement data set has the property that this place in the data set does not show anything. Detecting this property in the measurement data set would mean to confirm a "true negative" (TN), which is not a helpful analysis result of the data set, because it means that at a specific place of the data set, no object is detected and in reality, there is no object.

[0154] In one implementation of the invention, the training of the neural network 1, the trained network, the system and the method for operating the system refer to a detection by the neural network of "true positives" (TP) only. That means, the number of positively detected objects in a data set is counted and compared for the evaluation of a performance value to the "true" objects which are present and can be detected in the perfect original measurement data set by the analyzing unit / procedure. The share of true positives compared to the total detectable number of objects in percent may be used as a performance value.

[0155] In another implementation of the invention, as shown in Figure 9, the training of the neural network 1, the trained network 1, the system and the method for operating the system may, in contrast, take into consideration a detection of true positives (TP), "false negatives" (FN), which means that an object, which is at a certain place of the data set, is not detected and "false positives" (FP) which means, that objects are detected at places of the data set, where in reality there is no object.

[0156] If TPs, FNs and FPs are taken into consideration for the training of the neural network and the assignment of a performance value to the data sets, a performance value F may be calculated as proportional to

[0157] F=Performance =2x PxR / (P+R)

[0158] Wherein

[0159] P=Precision= TP / (TP+FP) and

[0160] R=Recall= TP / (TP+FN). Therein, in the formulas TP means the number of detected True Positives, FP means the number of False Positives, FN means the number of False Negatives. These numbers are, in the training phase, determined by comparing the analyzing results of the transformed data sets with the analyzing results of an original measurement data set. Another possible and applicable metric for a performance value would be AP, also known as "average precision", which can be calculated as

[0161] AP= / >( / ?) dR

[0162] Wherein P and R are defined as above.

[0163] Figure 10 shows the training of the intended function data processing unit 31 and of the neural network 1. Measurement data 50 are fed to the intended function data processing unit 31, as shown by the arrow 51. At the same time, labels of these measurement data, which have been determined in an analysing unit 52, are fed to the intended function data processing unit 31 as target values for training, as shown by the arrow 53. The potential measurement data 50 which are appropriate for the training of the intended function data processing unit 31 are located in the circle 50, even if not all of these data are actually used for the training. These measurement data have certain similar structures and a pattern detection will detect similar patters among them. The measurement data 50 are defined as the in-distribution data and a pattern detection unit will detect that the similarity among them is above a certain threshold.

[0164] Among the measurement data 50, a selection 50a is chosen for the training of the neural network for the task to analyse the performance value and the confidence level of measurement data. Measurement data of the selection 50a and, in addition, transformed measurement data based on these data, are fed into the neural network 1 as shown by the arrow 54. These measurement data are labelled by an analysing unit 6 which assigns a performance value to the measurement data as already described above with reference to the Figure 6.

[0165] In addition, the neural network may also be trained to decide / distinguish during runtime / application of the system, if measurement data are in-distribution measurement data or out-of-distribution measurement data. For this purpose, the neural network may be fed with measurement data from within the circle 50, and data from outside the circle 50, but inside the area 55. These data are not appropriate for the training of the intended function data processing unit 31 and are not used for this purpose. Their structure and patterns inside these data are not similar to the structure of data from inside the circle 50 (in-distribution measurement data).

[0166] Both elements, the intended function data processing unit 31 and the neural network 1 are trained for working together during use / runtime in a highly reliable way.

[0167] Figure 11 is illustrating the function of the neural network 1 during runtime / application. Both measurement data from both in-distribution 50 and out-of-distribution 55 may be delivered by sensors to the neural network 1 as represented by the arrows 56, 57. The neural network on one hand is trained to distinguish between sufficiently high performance values and confidence levels ("safe" measurement data) and not sufficient performance levels / confi- dence levels ("unsafe" measurement data).

[0168] On the other hand, the neural network is trained to distinguish between indistribution measurement data and out-of-distribution measurement data.

[0169] A processing of measurement data by an intended function data processing unit 31 may be decided to be reliable only if the performance value / confi- dence level is above a predefined level (areas "a" and "d" in Figure 11, meaning "safe") and at the same time, the measurement data can be identified as in-distribution measurement data (measurement data may only be either indistribution measurement data or out-of-distribution measurement data. The two labels are excluding each other). Hence, only measurement data in the area "a" of Figure 11 shall be further processed by the intended function data processing unit 31.

[0170] Figure 12 shows a potential internal structure of the neural network 1. Measurement data 58 are sent to the neural network and the first layer or one of the first layers 59 of the neural network is a feature extraction layer which extracts certain features from the data. After some internal layers of the network, at the output side of the neural network, one of the last layers is a dimensionality reduction layer 60 which reduces some dimensions of the data or projects the data on a lower dimensional subspace. This dimensionality reduction layer 60 may also be called a categorizer. Each of two fully connected layers 61, 62, which are in contact to the dimensionality reduction layer 60, may generate in independent output. Layer 61 may be provided and trained to determine if the measurement data are safe or unsafe, that is, if the level of confidence is sufficient. Layer 62 may be provided and trained to determine if the measurement data are in-distribution measurement data or out-of-dis- tribution data.

[0171] Figure 13 illustrates how the fully connected layer 61 is working. The diagram shows that the dimensionality reduction layer 60 clearly divides the incoming measurement data in two point clouds which are clearly separate in the remaining dimensions. The fully connected layer 61 determines if a given measurement data belongs to the "safe" or "unsafe" point cloud and assigns the confidence level accordingly.

[0172] Figure 14 illustrates, how the fully connected layer 62 may be working. For example, measurement data that are out-of-distribution data will be positioned outside of both point clouds "safe" and "unsafe" by the dimensionality reduction layer 60. Examples of such out-of-distribution measurement data are shown by the squares 63, 64, 65. This enables the fully connected layer 62 to recognize the out-of-distribution measurement data, in-distribution measurement data may for example be positioned by the dimensionality reduction layer 60 in one of the point clouds "safe" or "unsafe" or in the immediate environment of these two point clouds.

[0173] Figure 15 is showing the working of a system with an intended function data processing unit 31 and a neural network 1 during runtime, when the system is applied after training. The intended function data processing unit 31 may comprise a data extraction unit 30b as shown in Figure 5 or a data extraction unit 30b may be positioned on the input side of the unit 31. In the same way, the neural network 1 may comprise a data extraction unit 30 as shown in Figure 5 or a data extraction unit 30 may be positioned on the input side of the neural network 1.

[0174] Measurement data 66 are delivered by one or more sensors to the system. The trained intended function data processing unit 31may execute its function and detect, for example, certain patterns that may lead to triggering a signal. The measurement data 66 are also delivered to the neural network which may at the same time, decide, if the measurement data are safe or unsafe and if the data are in-distribution data or not.

[0175] If the measurement data are safe and in-distribution data, the intended function data processing unit 31 may be allowed to process the measurement data. If the measurement data are either not safe or not in-distribution data, the intended function data processing unit 31 may be allowed to process the measurement data. The trigger signal output module 67 may collect the results of the unit 31 and of the neural network 1 and decide about the further proceeding.

[0176] The invention as described above, allows for a very resource-efficient control of a system and a machine based on measurements and respective measurement data sets with a selective consideration of the reliability and confidence level of the measured data and a safety level of the system as well as a consideration if the measurement data are appropriate for being processed by an intended function data processing unit.

Claims

Claims1. Method for training a neural network (1) for a system comprising the neural network (1) and an intended function data processing unit (31) to assess the quality of measurement data (3b) which are delivered by a sensor (5) and processed by the neural network as well as by the intended function data processing unit, wherein the intended function data processing unit is arranged to execute an intended function by creating a trigger signal based on the measurement data delivered by the sensor, wherein the method for training comprises the following steps: providing an original measurement data set (3a), analyzing the original measurement data set or features extracted by a first extraction unit or extraction process (30a) from the original measurement data set by an analyzing unit and / or an analyzing procedure (6), which is arranged to execute the intended function or a part of the intended function to generate an analyzing performance,-providing one or more transformed data sets by providing additional measurement data sets of different quality levels, that are used as transformed data sets (4a, 4b, 4c, 4d) and / or-transforming the original measurement data set and / or additional measurement data sets into one or more transformed data sets (4a, 4b, 4c, 4d) of different quality levels, analyzing at least one of the transformed data sets and / or features, which are extracted by a first extraction unit or extraction process (30a) from each transformed data set by the analyzing unit and / or the analyzing procedure (6) to generate an analyzing performance and comparing the analyzing performance of the transformed data setsand / or features extracted from each transformed data sets with the analyzing performance of the original measurement data set or the analyzing performance of the features, which are extracted by the first extraction unit or extraction process (30a) from the original measurement data set to assign, based on the results of the comparison, analyzing performance values (7) to the transformed data sets,-training the neural network by feeding transformed data sets and / or the features extracted from each transformed data set by a second extraction unit or extraction process (30) on one hand, and for each transformed data set the assigned analyzing performance value on the other hand into the neural network, wherein the first extraction unit or extraction process (30a) is arranged to extract different features than the second extraction unit or extraction process (30).

2. Method according to claim 1 characterized in that the transformation of the original measurement data set (3a) into transformed data sets (4a, 4b, 4c, 4d) comprises one or more of adding noise, filtering data, partially randomizing data, blurring data masking distorting data and / or characterized in that the transformation of the original measurement data set (3a) into transformed data sets (4a, 4b, 4c, 4d) in the case of image data comprises the simulation of optical defects, and / or wherein the transformed data sets are one or more of resized, normalized and augmented.

3. Method according to claim 1 or 2, characterized in that the analyzing performance value (7) of a transformed data set (4a, 4b, 4c, 4d) is measured as a similarity of patterns (8a, 8b) that are identified by the analyzing unit or the analyzing procedure (6) in the transformed data set or in the features extracted from each transformed data set on one hand and patterns identified by the analyzing unit or the analyzing procedure (6) in the original measurement data set or in features extracted from the original measurement data set on the other hand, or that the analyzing performance value (7) of a transformed data set (4a, 4b, 4c, 4d) is measured as a number of predefined patterns (8a, 8b) that are identified by the analyzing unit or the analyzing procedure (6) in the transformed data set or in the features extracted from each transformed data set as a share of the predefined patterns that are identified by the analyzing unit or the analyzing procedure (6) in the original measurement data set (3a) or in the features extracted from the original measurement data set or that the analyzing performance value (7) of a transformed data set (4a, 4b, 4c, 4d) is measured as the sum of confidence values of objects or predefined patterns that are detected by the analyzing unit or the analyzing procedure (6) in the transformed data set or in the features extracted from each transformed data set.

4. Method according to one of claims 1 to 3, characterized in that a subset of the original measurement data set (3a) is selected for the training of the neural network (1) or a subset is given a higher weight than the rest of the original measurement data set for the training of the neural network.

5. Method for training a neural network (1) and an intended function data processing unit (31) which are arranged to work together in a system with at least one sensor (5) wherein the system provides that the or each sensor takes measurements to generate measurement data (3b),that the measurement data are sent to the neural network which assigns an analyzing performance value (7) and / or, based on the analyzing performance value, a sensor confidence level to the measurement data and wherein the intended function data processing unit (31) is arranged to trigger a signal, based on the measurement data as well as the analyzing performance value and / or the sensor confidence level, the method for training comprising: training the intended function data processing unit by feeding a multitude of measurement data (50) of the or each sensor into the intended function data processing unit, wherein the measurement data are labeled with the target values of the intended function for each measurement data, and wherein the measurement data (50), which are used for training the intended function data processing unit are defined as the in-distribution measurement data and wherein measurement data, which have not been used for training the intended function data processing unit and differ in their feature distribution from the measurement data, which have been used for training the intended function data processing unit, are defined as out-of-distribu- tion measurement data ; training the neural network with one or more original measurement data sets (3a) and with transformed data sets (4a, 4b, 4c, 4d) according to one of claims 1 to 4, wherein the one or more original measurement data sets are selected from the in-distribution measurement data.

6. Method according to claim 5, wherein the intended function data processing unit (31) is arranged to trigger a signal based on the measurement data, on the analyzing performance value and / or sensor confidence level and on the information if the measurement data are in-dis- tribution data or out-of-distribution data,characterized in that the neural network is trained for distinguishing if the similarity of measurement data or feature patterns of measurement data with measurement data that have been used for training the intended function data processing unit or feature patterns of measurement data that have been used for training the intended function data processing unit is above a predefined threshold, wherein the training in particular provides feeding into the neural network on one hand randomly created artificial measurement data and on the other hand the information that the measurement data are not in-distribu- tion data or feeding in a clustering sub-model, in particular a dimensionality reduction sub-model of the neural network on one hand random data which do not belong to the identified clusters or random dimension-reduced data which differ in at least one dimension from the dimension-reduced data which are results of the clustering or the dimensionality reduction of the original measurement data set and the transformed data sets and on the other hand the information that the random dimension-reduced data do not reflect in-distribution data.

7. Data storage device comprising a plurality of stored instructions that, when executed on a processor circuitry of a computer controlled device, cause the computer controlled device to perform a method according to one of claims 1 to 6.

8. Neural network (1) which has been trained according to one of claims 1 to 4.

9. Neural network (1), in particular a convolutional neural network, with at least one clustering sub-model, in particular a dimensionality reduction sub-model and at least two fully connected layers, which are connected to the clustering sub-model, in particular a dimensionality reduction submodel, wherein the neural network has been trained according to one of claims 1 to 6.

10. System comprising a neural network (1) and an intended function data processing unit (31) wherein the neural network has been trained according to one of claims 1 to 4.

11. System comprising a neural network (1) and an intended function data processing unit (31) which have been trained according to claim 5 or 6.

12. System according to claim 10 or 11 wherein the intended function data processing unit (31) is assigned to one or two or more than two sensors and is arranged to execute an intended function consisting of or including identifying predefined patterns (8a, 8b) in the measurement data (3b) or in features extracted from the measurement data delivered by the sensors and generating an alarm or trigger signal or starting or stopping a process of a machine only under the condition-that a predefined pattern has been identified based on measurement data of the sensor or sensors,-that the confidence level of a selection of one or more sensors or all sensors or the system safety value of the group of sensors has been evaluated by the neural network (1) to be above a predefined threshold (24, 25) and-that the neural network has confirmed that the measurement data are in-distribution data.

13. Machine with a system according to claims 10, 11 or 12, wherein the machine (17) is arranged to block, trigger, stop, allow or limit at least one of its activities upon receipt of a trigger signal of the intended function data processing unit (31).

14. Method for operating a system with at least one sensor (5) and a neural network (1) and an intended function data processing unit (31) wherein the neural network (1) and the intended function data processing unit have been trained by a process according to claim 5 or 6, in particular a method for operating a system according to claim 10 or 11, or for operating a machine according to claim 12,wherein the or each sensor takes measurements to generate measurement data (3b), the measurement data of the or each sensor are sent to the neural network which assigns to the measurement data on one hand an analyzing performance value (7) and, based on the analyzing performance value a sensor confidence level and on the other hand the information if the measurement data are in-distribution data or not and wherein the measurement data (3b) are further processed by the intended function data processing unit (31) only if the sensor confidence level is above a predefined threshold and the measurement data are in-distribution data.

15. Method for operating a system with at least one sensor (5) and a neural network (1) and an intended function data processing unit (31) wherein the neural network (1) and the intended function data processing unit have been trained by a process according to claim 5 or 6, in particular a method for operating a system according to claim 10 or 11, or for operating a machine according to claim 12, wherein the or each sensor takes measurements to generate measurement data (3b), the measurement data of the or each sensor are sent to the neural network which assigns to the measurement data on one hand an analyzing performance value (7) and, based on the analyzing performance value a sensor confidence level and on the other hand the information if the measurement data are in-distribution data or not and wherein the intended function data processing unit (31) creates a trigger signal if either- the neural network confirms that the sensor confidence level is above a predefined threshold and the measurement data are in-distribution data and the intended function data processing unit is detecting a pattern on which it has been trained to create a trigger signal or- if the sensor confidence level is below a predefined threshold or- if the measurement data are not in-distribution data.

16. Method according to claim 14 or 15, characterized in that assigned sensor confidence levels of two or more sensors (5) are combined to generate a system safety value and that the measurement data (3b) are further processed by the intended function data processing unit (31) only if the system safety value is above a predefined threshold and the measurement data are in-distribution data, and / or characterized in that at least one sensor (5) is a camera, an optical scanning sensor, a LIDAR system, a RADAR system, an imaging sensor generating 2-dimensional or 3-dimensional images of an environment, or an acoustic sensor.

Images