Computer program product for analyzing data originating from at least one device

Abstract

The invention relates to a computer program product (100, 400) for analyzing data originating from at least one device (360), the computer program product (100, 400) comprising: an ML workpiece (120, 500) comprising functionality of an ML model; and monitoring the workpiece (130, 140, 600), where the ML model has been trained and / or set up with a machine learning method using training data, or can be trained and / or set up, where the ML workpiece (120, 500) is designed and set up to generate ML results (122, 590) as a response to inputting the input data (102, 402, 552) into the ML workpiece (120, 500) based on data originating from the device (360), where the workpiece (130, 140, 600) is monitored. The computer program product (100, 400) is designed and arranged to generate ML result reliability information (132, 142, 690) regarding the ML result (122, 590) as a response to inputting the input data (102, 402, 552) into the monitored workpiece (130, 140, 600), where the training data or a portion of the training data is used to train the monitored workpiece (130, 140, 600), and where the computer program product (100, 400) is further designed and arranged, when executed by the computer (300, 400), to generate the ML result reliability information (132, 142, 690) based on the ML result (122, 590). A qualified ML result (190, 702, 704, 706) is output based on the ML result reliability information (132, 142, 690) and the ML result reliability information (132, 142, 690).

Description

Technical Field

[0001] The present invention relates to a computer program product and a computer-implemented method for generating a qualified output of an ML model (ML: machine learning) based on input data originating from a device, wherein the ML model has been trained and / or set or can be trained and / or set using machine learning methods. Background Art

[0002] Such computer program products and methods are known in the art.

[0003] For example, U.S. Pat. No. 10,504,020 B2 discloses a neural network-based classifier, such as for identifying parts and assigning corresponding part numbers thereto. In addition, the U.S. Pat. No. 10,504,020 B2 discloses a confidence value that can be assigned to each prediction of the classifier, such as the part number predicted by the matching engine classifier in the above use case.

[0004] A disadvantage of the above-mentioned prior art is that no method or systematic approach is disclosed for generating confidence or reliability information about the results of the neural network analysis. Summary of the invention

[0005] It is therefore an object of the present invention to provide a system and / or method for systematically and / or efficiently generating confidence, resilience and / or reliability information about results created by artificial intelligence methods.

[0006] This object is achieved by a computer program product having the features of claim 1 .

[0007] The computer program product is designed and arranged for analyzing data originating from at least one device, wherein the computer program product comprises:

[0008] -ML artifacts, including the functionality of the ML model, and

[0009] - monitoring of workpieces,

[0010] The ML model has been trained and / or set using a machine learning method, or can be trained and / or set using a machine learning method.

[0011] The ML artifact is designed and arranged to produce an ML result in response to input data entered into the ML artifact, wherein the input data is based on data originating from at least one device,

[0012] And the monitoring artifact is designed and arranged to generate ML result reliability information about the ML result in response to inputting input data into the monitoring artifact.

[0013] The computer program product is also designed and arranged to output qualified ML results based on the ML results and the ML result reliability information when executed by the computer.

[0014] The present invention integrates monitoring artifacts with ML artifacts (ML: machine learning) in one computer program product, resulting in a single computer program product for ML artifact reasoning or ML model reasoning and monitoring such reasoning. In a preferred embodiment, two or more monitoring artifacts according to the present specification can be integrated with ML artifacts in one computer program product, resulting in a single computer program product for ML artifact reasoning or ML model reasoning and monitoring such reasoning using two or more monitoring artifacts.

[0015] The computer program product includes the operations necessary to compute the output of the ML artifact or ML model. The architecture of the ML model or ML artifact is extended by additional data processing flows, in parallel with the original ML artifact or ML model architecture. These parallel flows have no impact on the training or inference results, considering the original ML task processed. Therefore, the core components for predicting the values ​​required by the business logic (inference results) can remain unchanged. However, by integrating the data processing flows for monitoring into the computer program product, the relevant aspects of monitoring (input data, internal data flows and intermediate data representations within the original ML model or ML artifact, and inference results) can be accessed.

[0016] Integrating the monitoring artifact together with the ML artifact into the computer program product solves the above mentioned tasks for the following reasons:

[0017] - Monitoring data streams, meaning that for example monitoring artifacts can be directly configured and / or trained based on the data on which the ML model or ML artifact to be monitored was trained. Thus, when an ML model or ML artifact is updated by retraining it with new data, the monitoring component, for example the monitoring artifact, can also be updated.

[0018] - No additional code is required for monitoring. Only qualified ML results are considered for further processing, e.g. for communication with other systems. Thus, consistency between different environments is also guaranteed (which is often required for successful deployment and operation if the consistency of the model behavior itself is guaranteed).

[0019] New monitoring methods can be written into the computer program product in the form of further data processing flows (eg further monitoring artifacts) and can thus be efficiently incorporated into the respective library.

[0020] A computer program product may be, for example, any type of computer software including instructions, computer software codes, computer software instructions, program instructions and / or program codes. A computer program product may, for example, include so-called source code and / or include so-called "binaries", where a binary may be designed and configured as a code generated by compiling the corresponding source code.

[0021] The computer program product may be any implementation of computer software stored on any type of electronic device or other device designed and arranged or storing computer software, computer software code and / or computer software instructions. The device for storing the computer program product may be, for example, an electronic memory, such as a so-called read-only memory (ROM), a so-called random access memory (RAM) and / or other similar electronic memories, such as EPROM, etc. Such memories may be semiconductor-based, or designed and arranged as magnetic devices, such as hard disks, or other memories known in the art for storing electronic data.

[0022] The computer program product may, for example, include a ML artifact and a monitoring artifact, including all their components.The computer program product may, for example, include components of the ML artifact necessary for reasoning and components of the monitoring artifact.

[0023] The computer program product may, for example, include only those components of the ML artifact necessary for reasoning and only those components of the monitoring artifact necessary for reasoning, respectively.

[0024] The computer program product may also include the ML artifact and the monitoring artifact in such a way that the ML artifact and the monitoring artifact are combined using a compilation, optimization and / or combination process to obtain the computer program product. Such optimization, compilation and / or combination of the monitoring artifact and the ML artifact may be designed and arranged according to the present specification. Such optimization, compilation and / or combination of the monitoring artifact and the ML artifact may, for example, be designed and arranged in such a way that the computer program product includes components of the ML artifact and the monitoring artifact that are necessary for reasoning. In its preferred embodiment, the computer program product may, for example, include only those components of the ML artifact and the monitoring artifact that are necessary for reasoning. Such optimization, compilation and / or combination of the monitoring artifact and the ML artifact may, for example, also be designed and arranged in such a way that the computer program product includes components of the ML artifact and the monitoring artifact that are necessary for reasoning, training and / or setting the ML artifact and the monitoring artifact.

[0025] The device may be any type of electronic device designed and configured to transmit data to other electronic devices. It may, for example, be designed as a sensor for detecting physical, chemical and / or other properties and / or conditions. The device may also, for example, include sensors for detecting physical, chemical and / or other properties and / or conditions.

[0026] The device may, for example, be designed and configured as a temperature sensor, a pressure sensor, an optical sensor, a camera, a position sensor and / or other similar sensors, or may include such sensors.

[0027] A device may, for example, include a sensor according to the present description and a communication component for transmitting data from and / or to the device via a wired and / or wireless communication connection.

[0028] The input data may be, for example, data originating from at least one device. The input data may be, for example, data directly originating from at least one device. The input data may also be designed and arranged as data generated by one or more processing steps performed on data originating from at least one device.

[0029] Such one or more processing steps may for example comprise so-called pre-processing for preparing data originating from at least one device as input data for input into an ML artifact and / or a monitoring artifact. Such pre-processing may comprise for example removal of unusable data, normalization of data, denoising of data, completion of data, reconstruction of data and / or similar processing steps. In case such data originating from at least one device are image data, pre-processing may comprise all types of image processing. This may also comprise for example modification of the image resolution or color scheme, or vectorization of the image data.

[0030] The input data may also be other types of data based on data originating from at least one device, such as data obtained through any type of transformation, database, Internet search, etc.

[0031] Artifacts as used in this specification are to be understood as so-called software artifacts. Software artifacts are used, for example, in software development and may be, for example, code elements that are used to create software or used during the creation of software, code elements that are at least partially part of the software and / or code elements that generate special functionality when run on a computer. Such functionality may be integrated into the software by using code elements during the creation of the software.

[0032] Software artifacts are typically created during the software development process and can refer to specific methods or processes in software development. For example, a software build may contain the developer's code and a series of different artifacts. Some of these artifacts may explain how the software works, while other artifacts may enable the software to run or represent one or more functions of the software. For example, artifacts of code may include dependencies, project source code, or resource lists. These artifacts can be stored in a repository so that they can be organized and retrieved as needed.

[0033] Software artifacts according to this specification may include, for example, the following items: source code, prototypes (fully functional versions of software that help build a basic functional version of its project), data models, design documents, scripts, Unified Modeling Language (UML), and work products. Software artifacts according to this specification may also be, for example, so-called "code-related artifacts" or "code artifacts". Such code-related artifacts may serve as the basis for the software and / or enable developers to test the software before launching it. Code-related artifacts may include, for example, source code, compiled code, setup scripts, test suites, generated objects, and logs generated during testing and quality assurance.

[0034] An artifact in the Unified Modeling Language (UML) is a specification of a physical piece of information used or produced by the software development process or by the deployment and operation of a system. Examples of artifacts include model files, source code files, scripts and binary executables, tables in a database system, development deliverables, word processing documents, or email messages. An artifact is a physical entity deployed on a node (i.e., device and execution environment). Other UML elements such as classes and components are first represented as artifacts, and then instances of these artifacts are deployed. Artifacts can also be composed of other artifacts.

[0035] Best practices around software artifacts include using a repository to store software artifacts, and managing and maintaining it appropriately.

[0036] ML artifacts may be designed and configured as ML functionality, ML software applications and / or ML software components. ML artifacts may also be designed and configured to implement ML functionality when executed by a computer. Such ML functionality may, for example, include reasoning and / or training and / or configuration functionality.

[0037] An ML artifact (ML: Machine Learning) may be designed and arranged as a software code, such as a source code or a binary code, which is at least a part of a computer program product or is used during the creation of a computer program product.

[0038] An ML artifact comprising the functionality of an ML model may also be designed and arranged as: the ML artifact is designed and arranged as software code, such as source code or binary code, which when executed by a computer represents, among other things, at least the functionality of a trained or untrained ML model.

[0039] The ML artifact may be designed and configured, for example, to include an ML model.

[0040] A computer program product comprising an ML artifact may be designed and arranged such that, for example, the computer program product comprises at least one or more portions of the source code or binary code of the ML artifact. It may also be designed and arranged such that the computer program product contains a compiled and / or optimized version of at least one or more portions of the source code or binary code of the ML artifact. A computer program product comprising an ML artifact may be designed and arranged such that the ML artifact is used to create a computer program product such that the computer program product comprises the functionality of a trained or untrained ML model.

[0041] The computer program product comprising the ML artifact may be designed and arranged such that, for example, the computer program product comprises a compiled and / or optimized version of at least one or more parts of the ML artifact. The compilation and / or optimization may be designed and arranged such that the compiled or uncompiled parts and / or functionality of the ML artifact contained in the computer program product are optimized for reasoning. This may be done by compilation and / or optimization as known in the prior art, for example by omitting parts of the ML artifact that are not used for reasoning. Such parts may, for example, be parts that are only used for training and / or setting up the ML artifact or ML model. The compilation and / or optimization may also be designed and arranged such that the compiled or uncompiled parts and / or functionality of the ML artifact contained in the computer program product are optimized for reasoning and training.

[0042] The ML artifact may be a software code in any software computer language or controller programming language, such as Python, C, C+, C++, a programming language according to standard IEC 61131, etc., or a compiled version of such a software code.

[0043] The ML artifact may be designed and arranged, for example, as a so-called “computation graph.” This may be designed and arranged, for example, such that the ML artifact includes or consists of software code representing the structure of the computation graph.

[0044] Developing AI through ML models often relies on static extraction of historical data from the environment that the ML model should later leverage. The underlying assumption is that future data will be the same as past data. However, this will lead to erroneous predictions when data may change over time in an unforeseen way, potentially harming, for example, an industrial production process that uses the results of the ML model. Changes in data may be due to many reasons, such as degradation of physical sensors, maintenance work, dust on camera lenses, changes in lighting, changes in previous production steps, or some other aspects.

[0045] Therefore, when the ML model is deployed to operation, e.g. as an ML artifact or computer program product according to the present specification, it is advantageous to add a monitoring artifact. This component is designed and arranged to derive meaningful metrics, e.g. from the input data, the behavior of the ML model or ML artifact and / or the inference results or ML results, so that an indication can be given whether the current prediction of the ML model is still reliable or no longer reliable.

[0046] The monitoring artifact may be designed and arranged such that it is, for example, able to detect differences in the data used to train the ML model or ML artifact and the input data (i.e., the data used for inference). The monitoring artifact may be designed and arranged such that it can detect, for example, whether the input data "deviates" from the training data (e.g., due to changes in the performance of the sensors used), and / or that, for example, the ML model or ML artifact has not been trained for a use case covered by the input data.

[0047] The monitoring artifact may, for example, be designed and configured such that it is capable of detecting statistical or other differences between data used for training and data used for inference (e.g., input data according to the present specification). Such statistical or other differences may, for example, be related to the range of values ​​covered by the training data compared to the input data (e.g., statistical information about the brightness or brightness range of a picture), information about noise within such data (e.g., about the signal-to-noise ratio), information about the size, form, and / or design of a component, person, or other thing to be detected by the ML model or ML artifact, or other similar differences between the training data and data used as input data for inference.

[0048] Such differences can, for example, be detected using the input data, such as by comparing information related to the input data with information related to the training data of the ML model or ML artifact (such as statistical analysis results and / or ML analysis results regarding the input data and the training data, or the input data and the training data themselves).

[0049] Such differences can also be detected, for example, by using the ML results, such as whether the ML results allow for a clear and relatively unambiguous conclusion, or whether the ML results are rather unclear, possibly even ambiguous. This can be achieved, for example, in the case of an ML model where the expected output is 1 or 0, reliable results are results close to 1 and 0 (e.g., results greater than 0.9 or 0.75, or less than 0.1 or 0.25), and less reliable results can be results that are clearly between 0 and 1 (e.g., about 0.5 or, for example, in the range between 0.25 and 0.75).

[0050] The monitoring artifact may be designed and arranged to monitor functionality and / or monitor software components. The monitoring artifact may also be designed and arranged to implement monitoring functionality when executed by a computer.

[0051] The monitoring artifact may be designed and arranged as software code, such as source code or binary code, which is at least a part of a computer program product or is used during creation of a computer program product.

[0052] The monitoring artifact may, for example, include the functionality of a monitoring ML model (ML model with monitoring function) or a statistical or other analytical model. This may be designed and arranged such that the monitoring artifact is designed and arranged as software code, such as source code or binary code, which, when executed by a computer, among other things, represents at least the functionality of a trained or untrained monitoring ML model and / or a statistical or other analytical model.

[0053] The statistical or other analysis model may represent functionality of, for example, statistical analysis of input data, image analysis of input data, big data analysis of input data, and / or any other evaluation of input data.

[0054] In the present specification, the phrase "training" of a monitoring artifact is used to train and / or set up a monitoring artifact independently of the setting and / or structure of the monitoring artifact. This means that the phrase "training" of a monitoring artifact is used for a monitoring artifact using, for example, an ML model, as well as a monitoring artifact using a statistical model or other analysis model or another concept. The phrase "training" of a monitoring artifact is used, for example, to train a corresponding monitoring ML model according to the present specification, as well as for setting up a statistical model or other analysis model according to the present specification used by a monitoring artifact.

[0055] A computer program product comprising a monitoring artifact may be designed and arranged such that, for example, the computer program product comprises at least one or more portions of the source code or binary code of the monitoring artifact. It may also be designed and arranged such that the computer program product comprises a compiled and / or optimized version of at least one or more portions of the source code or binary code of the monitoring artifact. A computer program product comprising a monitoring artifact may be designed and arranged such that the monitoring artifact is used to create a computer program product such that the computer program product comprises the functionality of a trained or untrained monitoring ML model and / or a statistical or other analytical model.

[0056] A computer program product comprising a monitoring artifact may be designed and arranged such that, for example, the computer program product comprises a compiled and / or optimized version of at least one or more portions of the monitoring artifact. Compilation and / or optimization may be designed and arranged such that compiled or uncompiled portions and / or functionality of the monitoring artifact contained in the computer program product are optimized for reasoning. This may be accomplished by compilation and / or optimization as known in the prior art, for example by omitting portions of the monitoring artifact that are not used for reasoning. Such portions may, for example, be portions that are only used for training and / or setting up the monitoring artifact, such as trained or untrained monitoring ML models and / or statistical or other analytical models used for training and / or setting up the monitoring artifact. Compilation and / or optimization may also be designed and arranged such that compiled or uncompiled portions and / or functionality of the monitoring artifact contained in the computer program product are optimized for reasoning and training.

[0057] The monitoring artifact may be a software code in any software computer language or controller programming language, such as Python, C, C+, C++, a programming language according to standard IEC 61131, etc., or a compiled version of such a software code.

[0058] The monitoring artifact can be designed and configured, for example, as a so-called “computation graph.” This can be designed and configured, for example, such that the monitoring artifact includes or consists of software code representing the structure of the calculation graph.

[0059] In addition to the monitoring artifact, the computer program product may also include at least one further monitoring artifact, wherein each of the at least one further monitoring artifact may be designed and configured as a monitoring artifact according to the present description.

[0060] The monitoring artifact may be designed and arranged (or may be integrated in a computer program product) so that the monitoring artifact is automatically updated or trained once the ML model or ML artifact is updated or trained. The monitoring artifact may be further designed and arranged (or may be integrated in a computer program product) in such a way that the monitoring artifact is automatically updated or trained once the ML model or ML artifact is updated or trained.

[0061] Machine learning methods are to be understood as, for example, automated ("machine") methods, which do not generate results by means of predetermined rules, but rather identify patterns from many examples (usually automatically) by means of machine learning algorithms or learning methods and then generate statements about the data to be analyzed based on the patterns.

[0062] Such machine learning methods can be trained and configured, for example, as supervised learning methods, partially supervised learning methods, unsupervised learning processes or empowered learning methods (“reinforcement learning”).

[0063] Examples of machine learning methods are, for example, regression algorithms (e.g. linear regression algorithms), generation or optimization of decision trees (so-called “decision trees”), learning methods for neural networks, clustering methods (e.g. so-called “k-means clustering”), learning methods or generation of support vector machines (SVM), learning methods or generation of sequential decision models, or learning methods or creation of Bayesian models or networks.

[0064] The result of applying such a machine learning algorithm or learning method to certain data, such as according to the present specification, is referred to as a "machine learning model" or "ML model." Such an ML model represents the digitally stored or storable result of applying a machine learning algorithm or learning method to the analyzed data or training data.

[0065] The generation of ML models can be designed and configured so that the ML models are reformulated by applying machine learning methods. The generation of ML models can also be designed and configured so that an existing ML model is changed or adjusted by applying machine learning methods.

[0066] Examples of such ML models are the result of a regression algorithm (e.g. a linear regression algorithm), a neural network, a decision tree, the result of a clustering method (including, for example, the obtained clusters or cluster categories, definitions and / or parameters), a support vector machine (SVM), a sequential decision model, or a Bayesian model or network.

[0067] The neural network can be, for example, a so-called “deep neural network”, “feedforward neural network”, “recurrent neural network”, “convolutional neural network” or “autoencoder neural network”. Applying corresponding machine learning methods to the neural network is often referred to as “training” the corresponding neural network.

[0068] The decision tree can be formed and arranged, for example, as a so-called “3rd generation iterative binarizer” (ID3), a classification or regression tree (CART), or a so-called “random forest”.

[0069] At least in conjunction with the present description, a neural network is to be understood as an electronic device, a software code and / or a software code stored in an electronic memory on another storage component or device, comprising a network of so-called nodes, whereby each node is usually connected to several other nodes. Nodes are also referred to as neurons or units, for example. Each node has at least one input and one output connection. As input nodes of a neural network, such nodes are to be understood as nodes that can receive signals (data, stimuli, patterns, etc.) from the outside world. Output nodes of a neural network are those nodes that can send signals, data, etc. to the outside world. So-called "hidden nodes" are those nodes in a neural network that are neither formed as input nodes nor as output nodes.

[0070] For example, the neural network can be formed as a so-called deep neural network (DNN). Such a "deep neural network" is a neural network in which the network nodes are arranged in layers (whereby the layers themselves can be one-dimensional, two-dimensional or even higher-dimensional). A deep neural network comprises at least one or two so-called hidden layers, which only comprise nodes that are neither input nodes nor output nodes. That is, the hidden layers have no connections to the input signals or the output signals.

[0071] So-called "deep learning" is to be understood as a class of machine learning techniques that use multiple layers of nonlinear information processing for supervised or unsupervised feature extraction and transformation, as well as pattern analysis and classification. In the context of such "deep learning", so-called "deep neural networks" are often used, for example according to the present specification.

[0072] The neural network can also have a so-called autoencoder structure, for example. Such an autoencoder structure can be suitable, for example, for reducing the dimensionality of data and, for example, for identifying similarities and similarities.

[0073] Neural networks can also be formed, for example, as so-called classification networks, which are particularly suitable for classifying data into categories. Such classification networks are used, for example, in conjunction with handwriting recognition or object or character recognition.

[0074] Another possible structure of the neural network can be, for example, a design as a so-called “deep belief network”.

[0075] The neural network may also have a combination of the above structures. For example, the architecture of the neural network may include an autoencoder structure to reduce the dimensionality of the input data, which autoencoder structure may then be further combined with another network structure, for example, to detect features and / or anomalies within the reduced dimensional data, or to classify or cluster the reduced dimensional data.

[0076] The values ​​describing the individual nodes and their connections, including other values ​​describing a particular neural network, can be stored, for example, in a set of values ​​describing the neural network. Such a set of values ​​then represents, for example, the design of the neural network. If such a set of values ​​is stored, for example, after training the neural network, the design of the trained neural network is stored. For example, it is possible to train the neural network with corresponding training data in a first computer system, then store a corresponding set of values ​​assigned to the neural network, and transfer it to a second system as the design of the trained neural network, and then, for example, apply the trained neural network there.

[0077] A neural network or a general ML model may be implemented as software code stored on a computer, on a controller, in a computer network or in the cloud, such as an ML artifact according to the present specification. It may also be stored in an electronic memory or semiconductor memory or other suitable storage device, such as a hard disk, etc.

[0078] Neural networks can generally be trained by inputting input data into the neural network and analyzing corresponding output data from the neural network to determine parameter values ​​for individual nodes or their connections using various known learning methods. In this way, a neural network can be trained in a manner known in the art using known data, patterns, stimuli or signals so that a neural network trained in this manner can be used, for example, to analyze other data.

[0079] Generally, training of a neural network is understood to mean processing data used to train the neural network in the neural network using one or more training algorithms in order to calculate parameters or transfer functions of individual nodes of the neural network or of a connection between two nodes within the neural network.

[0080] For training a neural network, for example, according to the present description, one of the methods of so-called "supervised learning" can be used, for example. Here, by training with corresponding training data, the results or skills assigned to the data are trained into the network. In addition, so-called unsupervised learning methods can also be used to train a neural network. For example, such an algorithm generates a model for a given set of inputs, which describes these inputs and enables predictions based on them. For example, there are clustering methods with which the data can be divided into different categories, for example by characteristic patterns, if they differ from one another.

[0081] When training a neural network, it is also possible to combine supervised and unsupervised learning methods if, for example, parts of the data are associated with trainable properties or abilities, while other parts of the data are not.

[0082] In addition, so-called reinforcement learning methods can also be used to train neural networks, at least in other aspects.

[0083] For example, training that requires relatively high computing power of a corresponding computer can be done on a high-performance system, while other work or data analysis using the trained neural network can be performed on a lower-performance system. Such other work and / or data analysis using the trained neural network can be performed, for example, on auxiliary systems and / or controllers, control devices, programmable logic controllers or modular programmable logic controllers or other corresponding devices according to the present specification.

[0084] Machine learning, and / or monitoring of machine learning systems, works in two main phases: training and inference.

[0085] Inference is the process of generating results for an ML artifact, ML model, or monitoring artifact by inputting new data into those artifacts / models to generate results, where the new data has not been used to train and / or set up the ML artifact, ML model, or monitoring artifact. For example, machine learning inference is the ability of a machine learning system to make predictions based on new data. Machine learning or monitoring inference requires three key components: a data source, a machine learning or monitoring system that processes the data, and a data destination.

[0086] During the training phase, a machine learning model or monitoring model is fed a curated dataset so that it can “learn” everything it needs to about the type of data it will analyze. Then, during the inference phase, the model can make predictions.

[0087] Training refers to the process of building a model using a machine learning algorithm. Training involves the use of deep learning frameworks (such as the software library "TensorFlow") and training data sets. IoT data provides a source of training data that data scientists and engineers can use to train machine learning models for a variety of use cases ranging from fault detection to consumer intelligence.

[0088] More specifically, a trained neural network uses what it has learned to work in the digital world, for example, to recognize images, spoken words, blood diseases, or to suggest shoes someone might buy next, or similar use cases, such as in a streamlined form of neural network applications. A faster and more efficient version of this neural network infers things about new data it is presented with based on its training. In the AI ​​lexicon, this is called "inference." Without training, inference cannot happen.

[0089] Inference refers to the process of making predictions using a trained machine learning algorithm. IoT data can be used as input to a trained machine learning model, resulting in predictions that can guide decision logic on the device, at the edge gateway, or elsewhere in the IoT system.

[0090] In many use cases for machine learning systems, such as in industrial settings or controllers, you need to be able to keep learning and quickly apply it to data that has never been seen. Here, compilation and / or optimization for inference can be used to enable an inference-optimized machine learning system to take smaller batches of real-world data and return the same answer quickly.

[0091] While this is a relatively new field in computer science, there are two main approaches to taking this massive neural network and modifying it for speed and improved latency on other devices, computers, handheld devices, and / or in applications running across other networks.

[0092] The first method looks at the parts of the neural network that are not activated after training. These parts are not needed and can be "pruned". The second method seeks ways to fuse multiple layers of the neural network into a single computational step.

[0093] An ML result may be designed and set to be any result produced by an ML model and / or ML artifact as an answer to input data into the ML model and / or ML artifact. It may be or include, for example, various digital information such as numbers, alphanumeric data, vectors, tensors of any dimension, or similar outputs.

[0094] The ML result reliability information is designed and arranged to contain an indication and / or meaningful measure whether the ML result should still be reliable or no longer be reliable.

[0095] Methods for creating ML result reliability information can be:

[0096] - static, i.e. a predefined sequence of mathematical operations is performed on the input data to obtain the respective monitoring value, or

[0097] - Trained, ie consisting of operations configured / parameterized based on data provided in a training phase, wherein trainable methods may also include static operations.

[0098] ML result reliability information about ML results can be based on:

[0099] - Input data,

[0100] - Internal data flows and intermediate data representations within the original ML model or ML artifact, and / or

[0101] -ML results.

[0102] The ML result reliability information may, for example, contain information that the ML result is reliable, unreliable, critical and / or must be observed, or similar information. The ML result reliability information may also contain all types of reliability metrics, such as reliability scores within a given numerical range. Such a numerical range may be, for example, a reliability value within a range between 0 and 10, where 0, for example, means unreliable and 10 means completely reliable. The ML result reliability information may also include several different types of information, such as the above information and / or similar information.

[0103] ML result reliability information can be generated, for example, based on input data, internal data flows within the original ML model or artifact, and / or ML results, by using various statistical or other mathematical or big data analysis methods. ML result reliability information can also be generated as an inference result of inputting input data, internal data flow data within the ML model or artifact, and / or ML results into the monitoring artifact. In this case, the monitoring artifact can be designed and set, for example, as an ML model, such as a neural network, a statistical model, and / or any other mathematical model.

[0104] A qualified ML result may include, for example, an ML result and / or ML result reliability information. A qualified ML result may also include, for example, information based on the ML result and / or information based on the ML result reliability information.

[0105] A qualified ML result may, for example, be designed and arranged so that, in the case where the ML result reliability information gives a hint that the input data may be different in one or more properties or types from the data used to train or set up the ML model or ML artifact, the qualified ML result includes information about such differences and / or information about the input data (e.g., requiring special attention). A qualified ML result may, for example, only include such a warning, or include such a warning including the ML result. A qualified ML result may, for example, be designed and arranged so that, in the case where there is a small difference between the training data and the input data, the qualified ML result includes a warning and the ML result, while in the case where there is a large difference between the training data and the input data, the qualified ML result only includes the corresponding warning.

[0106] A computational graph is a directed graph comprising a plurality of nodes and directed edges. Each directed edge connects a corresponding first node to a corresponding second node, and the second node represents an operation that receives the output of an operation represented by a corresponding first node or the numerical value of a first node or the numerical value of a variable represented by a first node as input. Nodes correspond to operations or variables, wherein variables also include parameters and numerical values. Variables can feed their values ​​into operations, and operations can feed their outputs into other operations. In this way, each node in the graph defines a function of a variable. The values ​​fed into and out of a node are usually tensors, which essentially means multidimensional arrays. Therefore, it contains scalars, vectors, and matrices, as well as higher-order tensors.

[0107] In some implementations, the operations represented in the computation graph are mathematical operations and / or logical operations. The operations may be, for example, neural network operations or operations for different types of machine learning models.

[0108] For example, the operations represented by the computational graph may be operations necessary for a neural network or machine learning model to compute inference (i.e., process inputs through the layers of the neural network) or the corresponding structure of the machine learning model to generate outputs for the inputs. As another example, the operations represented by the computational graph may be operations necessary to train a neural network or machine learning model by executing a training program to adjust parameter values ​​of the neural network or machine learning model (e.g., determine training values ​​of parameters from initial values ​​of the parameters). In some cases, such as during training of a neural network or machine learning model, the operations represented by the computational graph may include operations performed by multiple copies of the neural network or machine learning model.

[0109] Representing a neural network or other machine learning model as a computational graph provides a flexible and sophisticated way to implement neural networks efficiently, especially when the computation of the neural network is distributed across multiple devices with different hardware profiles.

[0110] Implementing a mathematical and / or logical process as a computational graph, such as implementing a neural network or another machine learning model as a computational graph, can have one or more of the following advantages. Typically, operations, such as mathematical and / or logical operations, can be represented as a computational graph of nodes and directed edges as described above. As an example, an operation that generates inferences from an input, i.e., an input to a neural network or many other machine learning models, can be represented as a computational graph of nodes and directed edges as described above.

[0111] The advantages of using mathematical and / or logical software or workflow implementations as computational graphs are described in more detail below using neural networks or other machine learning models as examples. A computing system or computing network can process such a computational graph representation to efficiently perform operations of a neural network or machine learning model. The system achieves this efficiency because the computational graph has less abstraction than a traditional neural network or other machine learning models represented as a sequence of layers, a tree structure, etc. In particular, the computational graph can be more easily divided for parallel operations than a traditional neural network representation, such as on different hardware devices, processors, processor cores, CPUs, GPUs, etc. As an illustration, a subgraph of a computational graph can be assigned to a unique device, such as a hardware device, a CPU core, a graphics processing unit (GPU), etc. For example, each subgraph can be assigned to a different device from other subgraphs, and each device performs operations in the corresponding subgraph to reduce the total time required to perform operations of a neural network or machine learning model.

[0112] Other general advantages of computation graphs are:

[0113] - Node-link structures: Different node-link structures of computational graphs can store a lot of information. Network-based relational problems can often be represented in this format in an advantageous way.

[0114] -Distributed computing: Huge problems with billions of nodes / elements cannot be handled by a single core or system. Distributed computing can be implemented directly in the computational graph, saving a lot of computation and reducing time complexity.

[0115] -Relational problems: Many relational problems can only be solved by computing graphs.

[0116] The computation graph can be transferred or exported to other devices and / or environments relatively easily. In special development environments, in other applications or environments, such as mobile applications, embedded devices and backend servers, computation graphs made using the software library "TensorFlow" can be used, for example. This is mainly because, for example, "TensorFlow" uses computation graphs as a format for saving models.

[0117] The computation graph is also easy to optimize to make the software workflow more efficient and, for example, allow it to run on less powerful hardware. Such optimizations, which may be performed by a corresponding optimizer and / or compiler, may for example:

[0118] - reason about the values ​​of tensors statically by folding constant nodes in your computations ("constant folding");

[0119] - Separate independent computational subsections and split them between threads or devices;

[0120] - Simplify arithmetic operations by eliminating common subexpressions.

[0121] To actually implement neural networks and other machine learning models as computational graphs, software libraries such as TensorFlow (https: / / www.tensorflow.org / ), Torch (http: / / torch.ch / ), Theano (https: / / pypi.org / project / Theano / ), and / or Pytorch (https: / / pytorch.org / ) are popular examples of deep learning or machine learning libraries. All of these libraries use computational graphs to represent their machine learning and deep learning models and flow their data and computations through the graph. When defining and / or setting up, for example, a neural network or machine learning model in, for example, Tensor-flow, Torch, Theano, or Pytorch, the network or ML model is created as a computational graph, which can then be executed on the desired hardware.

[0122] In a preferred embodiment, the computer program product is designed and arranged such that the ML artifact is designed and arranged for training an ML model based on training data, or for enabling training of an ML model based on training data,

[0123] And / or the monitoring artifact is designed and configured to train itself based on the training data, or to be able to train itself based on the training data.

[0124] An advantage of an ML artifact and / or monitoring artifact that is designed and arranged to train itself is that it is easier to train such models and / or artifacts or to retrain such models and / or artifacts.

[0125] In a preferred embodiment, the computer program product may be constructed such that the ML artifact and / or the monitoring artifact are trained using corresponding training data respectively according to the present specification, and then the computer program product is created by combining the trained ML artifact and the trained monitoring artifact.

[0126] This combination of ML artifacts and monitoring artifacts may be performed, for example, by directly combining the two components.

[0127] This combination can also be performed by using a compilation and / or optimization process for combining ML artifacts and monitoring artifacts. Such compilation and / or optimization can be designed and configured according to the present specification. It can, for example, include a process or step for removing parts that are not necessary for reasoning (e.g., only parts necessary for training ML artifacts and / or monitoring artifacts).

[0128] If the ML artifact and / or the monitoring artifact must be retrained or retrained with new training data, the original ML artifact and / or the original monitoring artifact (both of which include corresponding components, respectively used to train themselves) are used for such retraining. After the retraining, the computer program product is again assembled or compiled according to the present description, for example according to the previous paragraphs, using the retrained ML artifact and / or the retrained monitoring artifact.

[0129] In a preferred embodiment, the training data used to train the monitoring artifacts is the same as the data used to train the ML model, or at least includes the data or a portion of the training data.

[0130] In another preferred embodiment, the training data used to train the ML model is based on the training data used to train the monitoring artifact. In a further preferred embodiment, the training data used to train the monitoring artifact is based on the training data used to train the ML model.

[0131] The training data based on other training data can be designed and arranged so that the training data is a part of the other training data, includes the other training data or corresponds to the other training data. In addition, it can be designed and arranged so that the training data is a preprocessed or processed version of the other training data, wherein the preprocessing or processing can be designed and arranged according to the present specification.

[0132] An advantage of these preferred embodiments is that the monitoring artifact can be trained based on the training of the ML artifact. This makes it easier to detect whether the input data used for reasoning using the ML artifact is the type of data that the ML model is trained on, or whether there are differences between the input data and the data that the ML model is trained on. The latter situation is a situation that can or should be observed or should be noted.

[0133] The training data may be, for example, so-called “labeled data” or so-called “unlabeled data”.

[0134] So-called labeled data is used, for example, in supervised learning algorithms and typically consists of data that is put into a system (e.g., a machine learning system or a monitoring system of a machine learning system) and an expected result is assigned to each input data set, which is the expected output of the machine learning system and / or monitoring system after the input data set is input.

[0135] Unlabeled data is often used in unsupervised learning. This unlabeled data consists only of data that is input to the machine learning system or the monitoring system of the machine learning system.

[0136] The training data may be designed and arranged so that it can be input into the ML artifact and / or the monitoring artifact without being changed. The training data may also be designed and arranged so that it is input into the ML artifact and / or the monitoring artifact after one or more preprocessing steps or optimization steps according to the present specification.

[0137] The computer program product may further be designed and arranged such that, after the ML artifact and the monitoring artifact are trained or arranged based on the training data, the trained ML artifact and the trained or modified monitoring artifact are combined into the computer program product.

[0138] As an advantage of this embodiment, ML artifacts and monitoring artifacts can be processed, managed, configured, trained and / or improved separately, and the computer program products generated therefrom can be optimized, for example, for specific devices and / or reasoning, etc.

[0139] This further enables systematic and / or efficient generation of confidence, resilience, and / or reliability information regarding results created by machine learning methods.

[0140] Combining the ML artifact and the monitoring artifact to obtain a computer program product can, for example, be designed and arranged according to the present description.

[0141] The step of combining the ML artifact and the monitoring artifact into a computer program product can, for example, be designed and arranged to connect the ML artifact and the monitoring artifact to a common entry point, entry layer or entry artifact and / or to a common output point, output layer or output artifact. Such a common entry point, entry layer or entry artifact can, for example, be designed and arranged as a pre-processing artifact according to the present specification. Furthermore, such a common output point, output layer or output artifact can be designed and arranged as a post-processing artifact according to the present specification.

[0142] The process of combining the trained ML artifacts and the trained or modified monitoring artifacts into a computer program product can be designed and arranged as an optimization, compilation and / or combination process according to the present description, for example.

[0143] In a preferred embodiment, the computer program product contains only elements of the ML artifacts and monitoring artifacts required for reasoning. The combination of the ML artifacts and the monitoring artifacts into the computer program product can be designed and arranged such that only the components of the ML artifacts and the monitoring artifacts required for reasoning are transferred to the computer program product. This can be achieved, for example, by removing all parts that are only required for training and / or setting during the combination of the ML artifacts and the monitoring artifacts into the computer program product. An advantage of this embodiment is that the computer program product has improved performance because it contains fewer components than a direct combination of the ML artifacts and the monitoring artifacts. This can enable, for example, the computer program product to be deployed on less powerful hardware, then it is necessary to run the complete ML artifacts and / or the complete monitoring artifacts.

[0144] In another preferred embodiment, the computer program product contains elements of the ML artifacts and monitoring artifacts required for training and required for reasoning. The combination of the ML artifacts and monitoring artifacts into a computer program product can be further designed and arranged so that the components of the ML artifacts and monitoring artifacts required for training and reasoning are transferred to the computer program product. An advantage of this embodiment is that retraining or further training of the ML artifacts or ML models and / or monitoring artifacts can be performed by directly using the computer program product. This enables, for example, retraining or further training of the system directly in the deployed version of the computer program product.

[0145] If the computer program product is run on a computer, the computer program product may be designed and arranged such that the ML artifact and the monitoring artifact are executed in a single program run or cycle.

[0146] When executed by a computer, a computer program product executed in a single program run or loop may be designed and arranged such that, after the execution of the computer program product is initiated on the computer, the functionality of the ML artifact and the functionality of the monitoring artifact are executed and / or qualified ML results are output before the execution of the computer program product ends. If the execution of the computer program product is organized in loops, the execution of the computer program product is executed in a single program loop and may be designed and arranged such that, after the program loop starts, the functionality of the ML artifact and the functionality of the monitoring artifact are executed and / or qualified ML results are output before the loop ends.

[0147] When executed by a computer, a computer program product executed in a single program run or loop may further be designed and arranged such that, after a single start command of the computer program product, or for one loop of the computer program product, execution of the computer program product does not stop or end before the functionality of the ML artifacts and the functionality of the monitoring artifacts have been executed and / or qualified ML results have been output.

[0148] For example, execution of the computer program product ends if execution of the computer program product stops due to corresponding instructions within the computer program product, if new input data is to be used with the computer program product, or a user reaction to a qualified ML result is required for further processing. If new input data is to be used with the computer program product, execution of the computer program product ends or stops, and a new cycle or run of the computer program product is started for the new input data.

[0149] An advantage of this embodiment is that qualified ML results can be generated with reduced effort, since, for example, only one computer program product has to be deployed on the respective system, and no separate tools have to be deployed for creating the ML results and for creating the corresponding reliability information.

[0150] The computer program product may further be designed and / or arranged as a single workpiece.

[0151] The computer program product may further be designed and arranged such that,

[0152] The computer program product comprises a computation graph including both an ML artifact and a monitoring artifact,

[0153] or,

[0154] It is possible to implement a computation graph including both ML artifacts and monitoring artifacts, wherein the computation graph is part of a computer program product.

[0155] In the preferred embodiment, monitoring artifacts are integrated together with ML artifacts in a computation graph, resulting in a single computation graph for, for example, ML artifact or ML model inference and monitoring of the inference.

[0156] The computation graph includes the operations necessary to compute the output of the ML artifact or ML model. Thus, the architecture of the ML model or ML artifact is extended by additional data processing flows for monitoring, in parallel with the original ML artifact or ML model architecture. These parallel flows do not affect the training or inference results, considering the original ML task being processed. Thus, the core components for predicting the values ​​required by the business logic (e.g., inference results) remain unchanged. However, by integrating the data processing flows for monitoring into the computation graph, all relevant aspects of monitoring (input data, internal data flows and intermediate data representations within the original ML model or ML artifact, and inference results) are accessible.

[0157] Computer program products, ML artifacts and monitoring artifacts as well as computation graphs may be designed and configured, for example, according to the present description.

[0158] Integrating monitoring artifacts into the computation graph together with ML artifacts solves the above tasks, for example, for the following reasons:

[0159] - Monitoring data streams, meaning for example monitoring artifacts, can be directly configured and / or trained based on the data used to train the ML model or ML artifact to be monitored. Thus, when an ML model or ML artifact is updated by retraining it with new data, the monitoring artifact can also be updated.

[0160] - No additional code is required for monitoring. Only qualified ML results are considered for further processing, such as communication with other systems. Therefore, consistency between different environments is also guaranteed (which is usually required for successful deployment and operation if the consistency of the model behavior itself is guaranteed).

[0161] - New monitoring methods can be written into the computation graph in the form of additional data processing flows (e.g. additional monitoring artifacts) and can thus be efficiently incorporated into the corresponding library.

[0162] The advantage of structuring a computer program product as a computation graph is that program sequences are bundled and / or structured such that related program sequences are bundled in the same structure of the computation graph, so that the program sequences can be technically run more efficiently in a computer. In particular, for example, the processing of a program can be distributed more easily and efficiently to several computing cores, processor cores, processors, computers, computer cores or other hardware.

[0163] In a preferred embodiment, the computer program product is designed and arranged as a single computational graph that includes both ML artifacts and monitoring artifacts.

[0164] In a further preferred embodiment, the ML artifact is designed and configured as a computation graph, and / or the monitoring artifact is designed and configured as a computation graph. In this embodiment, the computer program product can be created, for example, by connecting the ML artifact computation graph and the monitoring artifact computation graph. Such a connection can be, for example, a direct connection of the two computation graphs, or a connection of the two computation graphs with a common input node or structure and / or a common output node or structure. Such an input node or structure can, for example, be designed and configured as a pre-processing node or structure for data originating from at least one device, for example to create input data. Such an output node or structure can, for example, be designed and configured as a post-processing node or structure, for example to create a qualified ML result.

[0165] In a further preferred embodiment, an optimization and / or compilation process is used to combine the ML artifact computation graph and the monitoring artifact computation graph into a single computation graph included in or implementing a computer program product. Such optimization and / or compilation process can be designed and configured according to the present specification.

[0166] The ML artifacts and monitoring artifacts as part of the computation graph may be designed and arranged such that the computation graph consists of a fully connected graph structure, wherein the fully connected graph structure contains elements implementing the ML artifact functionality and contains elements implementing the monitoring artifact functionality. Such ML artifact functionality and / or monitoring artifact functionality may be, for example, corresponding reasoning functionality and / or corresponding training and / or setting functionality. The fully connected graph may be designed and arranged, for example, such that the graph cannot be separated into separate unconnected subgraphs without cutting at least one edge of the graph, such as an edge between two nodes.

[0167] The computer program product may further be designed and arranged such that,

[0168] The computer program product further comprises a pre-processing workpiece to convert data from at least one device into input data,

[0169] and / or

[0170] The computer program product also includes post-processing artifacts to generate qualified ML results.

[0171] Pre-processing artifacts, data from at least one device, and input data can be designed and arranged, for example, according to the present specification. In addition, post-processing artifacts and qualified ML results can be designed and arranged according to the present specification.

[0172] The computer program product may be further designed and arranged such that the computer program product comprises a computation graph comprising both the ML artifact and the monitoring artifact, and

[0173] The computation graph also includes pre-processing artifacts and / or post-processing artifacts.

[0174] The computation graph and the integration of pre-processing artifacts and / or post-processing artifacts into the computation graph may be designed and configured according to the present description.

[0175] The above mentioned task is also solved by a computer implemented method for producing a qualified output of an ML model based on input data originating from a device,

[0176] The method comprises the following steps:

[0177] - Feed input data into a functional ML artifact including an ML model to produce an ML result,

[0178] - Feed input data into monitoring artifacts to generate ML result reliability information,

[0179] -Output qualified ML results based on the ML results and ML result reliability information.

[0180] Thus, the method steps of inputting input data into an ML artifact, inputting input data into a monitoring artifact, and outputting qualified ML results based on the ML results and ML result reliability information are performed in a single run of the computer program product when executed on a computer.

[0181] Since both the functionality of running the ML artifact and monitoring the ML model inference by monitoring the artifact are run in the same run during the execution of the computer program product on the computer, a method is created to systematically and efficiently generate confidence, resilience space and / or reliability information about results created by artificial intelligence methods.

[0182] When running on a computer, the ML model, ML artifacts, input data, ML results, monitoring artifacts, ML result reliability information, qualified ML results, outputting qualified ML results, and performing the method steps in a single run of the computer program product can be designed and arranged according to the present specification.

[0183] When running on a computer, the method steps of inputting input data into an ML artifact, inputting input data into a monitoring artifact, and outputting a qualified ML result, which are performed in a single run of the computer program product, can be designed and arranged so that, after starting the run of the computer program product and / or the computer-implemented method on the computer, before the end of the execution of the run of the computer program product, the inputting input data into the ML artifact, inputting input data into the monitoring artifact, executing the functionality of the ML artifact, executing the functionality of the monitoring artifact, and outputting a qualified ML result are performed. If the execution of the computer program product and / or the computer-implemented method is organized in cycles, the fact that the method steps of inputting input data into an ML artifact, inputting input data into a monitoring artifact, and outputting a qualified ML result are performed in a single run of the computer program product, when running on a computer, can be designed and arranged so that, after starting a new cycle of the execution of the computer program product and / or the computer-implemented method on the computer, before the end of the execution of the cycle of the computer program product, the inputting input data into the ML artifact, inputting input data into the monitoring artifact, executing the functionality of the ML artifact, executing the functionality of the monitoring artifact, and outputting a qualified ML result are performed.

[0184] When executed by a computer, the computer program product and / or the computer implemented method executed in a single program run or loop may further be designed and arranged such that, after starting the computer program product or starting a new loop of the computer program product, the execution of the computer program product or the loop of the computer program product is not stopped or ended before the functionality of the ML artifacts and the functionality of the monitoring artifacts have been executed and / or qualified ML results have been output.

[0185] The execution of the computer program product ends, for example, if the execution of the computer program product is stopped due to a corresponding instruction within the computer program product, if new input data are to be used with the computer program product, or a user reaction to a qualified ML result is required for further processing. The execution of the computer program product also ends, for example, if a new start command is required to restart the computer-implemented method. The execution of the computer program product also ends, for example, before a new cycle of execution of the computer program product or a new cycle of the computer-implemented method is to be started. If new input data are to be used with the computer program product, the execution of the computer program product ends or stops, and a new cycle or run of the computer program product is started for the new input data.

[0186] The above mentioned object is also solved by a computer-implemented method for generating a trained ML model and a trained or modified monitoring artifact based on training data,

[0187] The method comprises the following steps:

[0188] - inputting the training data into a ML artifact comprising the functionality of the ML model to produce a trained ML artifact,

[0189] - inputting the training data into the monitoring artifact to produce a trained or modified monitoring artifact, wherein the method steps of inputting the training data into the ML artifact and inputting the training data into the monitoring artifact, and producing the trained ML artifact (120, 500) and the trained or modified monitoring artifact (130, 140, 600) are performed in a single run of the computer program product when run on a computer.

[0190] A particularly efficient method for generating reliability information about artificial intelligence methods is created by such a method of parallel training of ML artifacts and corresponding monitoring artifacts. By carrying out such training of both artifacts in one program run, the ML artifacts and the monitoring artifacts are trained and / or set up simultaneously on the same data basis, so that the monitoring of the ML results is always correctly and efficiently adapted to the trained ML artifacts or ML models.

[0191] By additionally performing the training and / or configuration of the ML artifacts and the monitoring artifacts in the same program run, such training and / or configuration is performed very efficiently, since, for example, no separate training and / or configuration processes have to be triggered and managed.

[0192] When running on a computer, the ML model, training data, ML artifacts, trained ML artifacts, monitoring artifacts, trained or modified monitoring artifacts, and the execution of the method steps in a single execution of the computer program product can be designed and configured according to the present description.

[0193] In a preferred embodiment of the computer-implemented method, the method may be designed and arranged such that, after training and / or modifying the ML artifact and the monitoring artifact based on the training data, the trained ML artifact and the trained or modified monitoring artifact are combined into a computer program product.

[0194] Modified ML artifacts and / or monitoring artifacts may be designed and configured such that, for example, the ML artifacts and / or monitoring artifacts are trained or configured using input data, parameters, or other similar data to generate corresponding modified ML artifacts and / or monitoring artifacts.

[0195] Combining trained and / or modified ML artifacts and monitoring artifacts into a computer program product may be designed and arranged according to the present specification.

[0196] The combination of trained and / or modified ML artifacts and monitoring artifacts may, for example, be designed and arranged such that the trained and / or modified ML artifacts and monitoring artifacts may be combined as such and, for example, may be further combined with other software or program elements to produce a computer program product.

[0197] The combination of trained and / or modified ML artifacts and monitoring artifacts can be further designed and arranged so that, for example, after removing parts of the ML artifacts and / or monitoring artifacts that are not needed for reasoning, the trained and / or modified ML artifacts and monitoring artifacts are combined into a computer program product. Combining these modified ML artifacts and / or monitoring artifacts into a computer program product can be performed in addition with other software or program elements.

[0198] The combination of trained and / or modified ML artifacts and monitoring artifacts can also be designed and configured so that the compilation and / or optimization process steps for compiling and / or optimizing the ML artifacts and / or monitoring artifacts are performed to produce a computer program product. Likewise, the ML artifacts and monitoring artifacts are also combined into a computer program product, which includes such compilation and / or optimization processes that can be performed using other software or program elements.

[0199] Such compilation and / or optimization processes and removal of portions of ML artifacts and / or monitoring artifacts not required for reasoning may be further designed and configured according to the present specification.

[0200] The computer-implemented method may be further designed and arranged such that the computer program product comprises the ML artifact and the monitoring artifact as a single computational graph,

[0201] Alternatively, the computer program product is implemented as a single workpiece.

[0202] Thus, the ML artifact can be designed and configured as, for example, a trained ML artifact according to the present specification or an untrained ML artifact according to the present specification. In addition, the monitoring artifact can be designed and configured as, for example, a trained or modified monitoring artifact according to the present specification or an untrained or unmodified monitoring artifact.

[0203] The ML artifacts and monitoring artifacts as part of the computer program product can be further designed and arranged as a single computation graph according to the present specification. The computer program product can also be designed and arranged as a single computation graph or include a computation graph, which itself includes the ML artifacts and the monitoring artifacts or includes the functionality of the ML model or artifacts and the monitoring artifacts.

[0204] The computer-implemented method may further be designed and arranged as follows: a computer program product is designed and arranged according to the present description.

[0205] Further preferred embodiments can be found in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0206] The present invention is explained in more detail below by way of examples with reference to the accompanying drawings.

[0207] The accompanying drawings show:

[0208] Figure 1 : Example of the architecture of an ML system with integrated monitoring;

[0209] Figure 2 : An example of the process steps for setting up an ML-inference system, including monitoring the corresponding ML system;

[0210] Figure 3 : Example of a production system that includes a quality inspection component;

[0211] Figure 4 :according to Figure 3 The structure of the quality inspection component;

[0212] Figure 5 :according to Figure 4 An example of a quality check component of , wherein the quality check component is implemented as a computational graph;

[0213] Figure 6a: According to Figure 5 Detailed example of preprocessing subgraphs;

[0214] Figure 6b: Setting the grounds for inference Figure 5 Detailed example of a neural network computation subgraph;

[0215] Figure 6c: Setting the grounds for inference Figure 5 A detailed example of the monitoring computation subgraph;

[0216] Figure 6d: According to Figure 5 A detailed example of the post-processing computation subgraph;

[0217] Figure 7a: Setting up the ground truth for inference and training Figure 5 Detailed example of a neural network computation subgraph;

[0218] Figure 7b: Setting up the ground truth for inference and training Figure 5 A detailed example of monitoring computation subgraphs. DETAILED DESCRIPTION

[0219] Figure 1 An example of a system including a neural network 120 having two monitoring components 130 , 140 for monitoring the performance of the neural network 120 is shown. Figure 1The system shown is implemented as a computation graph 100, which includes an input layer 110 for preprocessing input data 102 input into the computation graph 100. The preprocessed input data 102 is then forwarded from the input layer 110 to the neural network 120, the trained monitoring component 130, and the static monitoring component 140, or further processed in the corresponding components 120, 130, 140.

[0220] After the data generated by the input layer 110 is input into the neural network 120, an inference result 122 is generated. To this end, the neural network has been trained using the corresponding training data. The trained monitoring component 130 is also composed of a machine learning system that is designed and configured to detect anomalies in the input data. The trained monitoring component is designed and configured so that it detects input data types that are not suitable for the input data types that have been used to train the neural network.

[0221] To this end, the trained monitoring component 130 has been trained using the same training data that has also been used to train the neural network 120. After the data from the input layer 110 is input into the trained monitoring component 130, the trained monitoring component 130 outputs a monitoring value 132 containing information whether the data from the input layer 110 corresponds to the type of data used to train the neural network.

[0222] The data from the input layer 110 is also fed to a static monitoring component 140, which is designed and arranged to calculate statistical parameters of the data fed to the component 140. As a result of feeding the data from the input layer 110 into the static monitoring component 140, the static monitoring component 140 outputs one or more monitoring values ​​142, which contain information on whether the statistical parameters calculated by the static monitoring component 140 are within preset boundaries.

[0223] Then, the inference results 122, the monitoring values ​​132 from the trained monitoring components, and the monitoring values ​​142 from the static monitoring components are fed into the output layer 150 to prepare qualified inference results 190. The result data 190 is output from the computation graph 100 to a device such as a computer, HMI (human-machine interface), and / or a user.

[0224] exist Figure 1 In the example shown, the input layer 110 , the neural network 120 , the trained monitoring component 130 , the static monitoring component 140 , and the output layer 150 are designed and arranged as computational subgraphs of the computational graph 100 .

[0225] exist Figure 1In the illustrated example, computation graph 100 is an example of a computer program product according to the present specification, wherein neural network 120 is an example of an ML artifact according to the present specification. Trained monitoring component 130 and static monitoring component 140 are examples of monitoring artifacts according to the present specification.

[0226] Figure 1 The dotted arrows shown pointing from the neural network 120 to the trained monitoring component 130 and the static monitoring component 140 symbolically indicate that, in a preferred embodiment, the inference results 122 of the neural network 120 can also be input into the trained monitoring component 130 and / or the static monitoring component 140 for advanced analysis of the data.

[0227] Figure 2 It shows that by training respectively Figure 1 The neural network 120 and according to Figure 1 The trained monitoring component 130 and the static monitoring component 140 are used to train and deploy Figure 1 The process of computing graph 100.

[0228] In a first process step 210 , the training data are input into a corresponding training system.

[0229] In a second process step 212, the training data is validated to avoid using inconsistent or incomplete training data to train the above-mentioned components 120, 130, 140.

[0230] In the next process step 214 , the training data is preprocessed by normalizing the data to a numerical range between 0 and 1.

[0231] Then, in a next step 216 , the data is labeled by assigning a planned output result of the neural network 120 , wherein the planned output result is the result that the neural network 120 should output when the corresponding training data is input into the neural network 120 .

[0232] Then, in a next step 218, this labeled data is characterized to assign corresponding features to this data.

[0233] Thereafter, in a next step 220, the neural network 120 is trained using the training data created by the process. In a parallel process step 230, the trained monitoring component 130 and the static monitoring component 140 are also trained and / or configured using the created training data.

[0234] Thereafter, in a model merging step 240 , the trained neural network 120 and the trained and / or configured trained monitoring component 130 and static monitoring component 140 are integrated into the computation graph 100 together with the input layer 110 and the output layer 150 .

[0235] Then, in a conversion step 250, the computation graph 100 is compiled into a deployable version suitable for the dedicated hardware on which the system is planned to be deployed. The conversion step 250 can be designed and arranged, for example, as a compilation step or an optimization step according to the present specification.

[0236] In a final process step 252, the compiled model is then deployed on the corresponding hardware.

[0237] and Figure 3 The production system shown is related to Figure 3 Figure 7b shows a further example of an embodiment of the present invention.

[0238] The production system consists of a robot 350 that is controlled by a controller 300 to manipulate and process products. The controller 300 includes a CPU component 302 that is designed and arranged to run a control program for running manipulation and production processes using the robot 350. Control commands from the control program in the CPU component 302 are forwarded from the CPU component 302 to the robot 350 via the internal bus 301 of the controller 300, the I / O component 304 of the controller 300, and the fieldbus connection 352. Then, sensors and feedback signals from the robot 350 are also transmitted from the robot 350 back to the controller 300 via the fieldbus connection 352.

[0239] The production process is supervised and controlled by a user interface 340 running on a computer 340 which is connected to the CPU component 302 of the controller 300 via an OPC UA communication connection 342 .

[0240] The controller 300, for example, may be designed and configured as a modular programmable logic controller 300 having a CPU module 302, an I / O module 304, and a quality inspection module 400 connected via a backplane 301. In another embodiment, the controller 300 may be implemented as a software controller, wherein the CPU component 302, the I / O component 304, and the quality inspection component 400 are designed as software applications or Apps connected via an internal bus 301. The internal bus 301 may also be designed as a corresponding software application or App, or as a software or hardware bus.

[0241] Figure 3 The system shown also consists of a camera 364 that takes images of the product produced by the robot 350 to control its quality. The images are transmitted from the camera 360 to the quality inspection component 400 or quality inspection module 400 of the controller 300 via an Ethernet connection 362.

[0242] The quality inspection component 400 includes a neural network that is or has been trained to distinguish correctly produced products from products with production errors. The quality inspection component 400 is further designed and arranged so that it can check whether the images captured by the camera 360 and transmitted to the quality inspection component 400 are suitable images for use as training data to train the neural network in the quality inspection component. In this way, the quality inspection component 400 can determine whether the results of the neural network analysis of the product pictures captured by the camera 360 should be reliable or whether additional attention or inspection is required.

[0243] This arrangement and functionality of the quality inspection component 400 is discussed in more detail in the following figures.

[0244] Figure 4 It shows that Figure 3 A more detailed structure of the quality inspection component 400 is shown. Figure 4 In the embodiments shown in the following figures, the quality check component 400 is designed and configured as a software application. The software quality check component 400 can be designed and configured as an application 400 for the software controller 300, or can be deployed on the corresponding hardware 400 of the hardware controller 300.

[0245] like Figure 4 The quality inspection component 400 shown includes a device for pre-processing the Figure 3 A pre-processing component 410 of image data 402 of camera 360 is shown. The pre-processing component normalizes and vectorizes the input data 402, which is explained in more detail in subsequent figures.

[0246] The quality inspection component 400 includes a neural network 500 for image recognition and a monitoring component 600 for monitoring the average brightness of a corresponding image 402 captured by the camera 360. The preprocessed data generated by the preprocessing component 410 is then input into the neural network 500 and into the monitoring component 600 in parallel.

[0247] Image data 402 from camera 360 is an example of data originating from at least one device according to the present Description, wherein data produced by pre-processing component 410 is an example of input data according to the present Description.

[0248] The pre-processed image 402 is then fed into a neural network 500 that is trained and configured to recognize whether the product shown on the corresponding image 402 is correctly produced. The neural network 500 then outputs an output vector 590 containing this information.

[0249] In parallel, the pre-processed image 402 is fed into the monitoring component 600. The monitoring component 600 is designed and arranged to calculate the average brightness of the image 402 and to check whether the average brightness of the image 402 input to the monitoring component 600 is within the brightness range of the image used to train the neural network 500. Then, the result of the check is output by the monitoring component 600 as a monitoring result 690.

[0250] A possible output 690 of the monitoring component 600 may be that the image 402 used as a training image with brightness completely within the brightness range within the neural network 500. Another possible output 690 of the monitoring component 600 may be that the brightness of the image 402 is still within the extended range of brightness of the training images used to train the neural network 500. A further possible output 690 of the monitoring component 600 is that the average brightness of the image 402 is outside the brightness range of the images used to train the neural network 500.

[0251] If the average brightness of image 402 used with neural network 500 is outside the average brightness range of pictures used to train neural network 500, the probability that the inference results of neural network 500 are incorrect increases, and the probability that, for example, incorrectly produced products are not properly identified by neural network 500 increases.

[0252] The quality check component 400 also includes a post-processing component 490 for combining the results of the neural network 500 and the monitoring results produced by the monitoring component 600. The combination produces a qualified result 700, 702, 704, 706 of the neural network 500, including additional information about the reliability of the result. The reliability information is based on the monitoring result 690 of the monitoring component 600.

[0253] If the average brightness of the image 402 analyzed by the neural network component 500 is within the average brightness range of the images used to train the neural network 500, the content of the reliability information 690 will be, for example, that the output 590 of the neural network 500 is reliable. The content of the reliability information 690 will also be, for example, that the output of the neural network is still reliable, but it must be observed if the average brightness of the image 402 is within the brightness extension range of the images used to train the neural network 500. If the average brightness of the image 402 is outside the brightness extension range of the images used to train the neural network 500, the content of the reliability information 690 will be, for example, that the classification result of the neural network 500 is unreliable.

[0254] The post-processing component creates the so-called qualified results 700, 702, 704, 706 of the neural network 500, which are output to the HMI 340, such as Figure 3As shown. In the case of a reliable result, as explained above, the "OK result" 702 only includes the output of the neural network 500. In the case that the result of the neural network 500 must be observed according to the above, the corresponding "observation result" 704 includes the output result of the neural network 500 and a warning message that the image information should be observed. In the case that the result of the neural network 500 is unreliable according to the above, only the warning message 706 is output to the HMI 340 as the "warning result" 706. Such a warning message may, for example, include information that the image quality is too poor to perform reliable image recognition using the neural network 500.

[0255] In this example, the output vector 590 generated by the neural network 500 is an example of an ML result according to the present specification, the monitoring result 690 generated by the monitoring component 600 is an example of ML result reliability information according to the present specification, and the qualified results 700, 702, 704, 706 generated by the post-processing component 490 are examples of qualified ML results according to the present specification.

[0256] exist Figure 5 , a single computation graph 400 is shown according to Figure 4 The corresponding subgraphs of the computation graph 400 are the pre-processing subgraph 410, the neural network subgraph 500, the monitoring subgraph 600, and the post-processing subgraph 490. These subgraphs 410, 500, 600, 490 are computation graphs themselves, which are connected to each other to construct the quality check component computation graph 400 into a single computation graph according to the present specification.

[0257] To better understand the functionality of the quality check component computation graph 400 , the size of the data flowing through the computation graph 400 is shown next to most of the arrows connecting the different elements of the quality check computation graph 400 .

[0258] A grayscale image 402 having a size of 28×28 pixels is passed from the camera 360 to the quality inspection component computation graph 400. The grayscale image 402 is passed to a pre-processing computation graph 410 of the quality inspection component computation graph 400, where the values ​​of the 28×28 pixels are passed to a range between 0 and 1 to produce a normalized image 402. Further details of the pre-processing computation graph 110 are shown in FIG6a.

[0259] Then, the pre-processed image 402 is fed into the neural network computation graph 500. The neural network 500 consists of three layers 510, 520, 530, each of which is implemented as Figure 5After the normalized image 402 is input into the first layer 510 of the neural network 500 and then processed by the second layer 520 and the third layer 530, an output vector 590 is generated as a result of the processing in the third layer 530.

[0260] The neural network 500 is designed and configured so that it can identify the correct and incorrect versions of 5 different products, respectively, generating an output vector of size 10.

[0261] In the first layer computation graph 510 of the neural network 500, the image 402 of size 28×28 pixels is vectorized into a vector of dimension 784×1. In the second layer computation graph 520 of the neural network 500 consisting of 32 neural network nodes, an output vector of dimension 32×1 is generated and passed to the third layer computation graph 530 consisting of 10 neural network nodes. The output of the third layer computation graph 530 is the said output vector 590 of size 10×1, which is then fed into the post-processing computation graph 490. The neural network computation graph 500 is explained and described in more detail in conjunction with FIG. 6b for inference and FIG. 7a for training purposes.

[0262] The quality check component computation graph 400 also includes a monitoring component 600 as a monitoring component computation graph 600 or a monitoring computation graph 600. The monitoring computation graph 600 also receives the normalized image 402 and computes the average brightness value ABR of 28×28 pixels in a “compute average brightness” computation graph 610.

[0263] The monitoring calculation graph 600 also includes a standard brightness component 620, which includes a mean brightness (Mu or μ) and a standard deviation (Sigma or σ) with respect to a standard brightness distribution, a brightness probability density function, or a Gaussian distribution function:

[0264]

[0265] The mean brightness value fitted to the image used to train the neural network 500.

[0266] In the reliability information creation calculation subgraph 630 of the monitoring calculation graph 600, it is checked whether the average brightness of the normalized image 402 calculated is

[0267] - within two standard deviations (2xSigma or 2σ) around the mean (Mu or μ) of the standard brightness distribution,

[0268] - within the 3xSigma (or 3σ) extension of the standard brightness distribution around Mu (or μ),

[0269] - or outside the 3Sigma deviation of the standard brightness distribution around Mu.

[0270] The first case produces a "reliable result" according to the present specification, the second case corresponds to a "result that is still reliable but must be observed" according to the present specification, and the third case corresponds to an "unreliable result" according to the present specification. These respective results are then passed to the post-processing calculation graph 490 as monitoring results 690.

[0271] The post-processing computation graph then produces qualified results 700, 702, 704, 706 as previously described.

[0272] The monitoring computation graph 600 is described in more detail in conjunction with Figure 6c regarding inference and Figure 7b regarding training. The post-processing computation graph 490 is described in more detail in conjunction with Figure 6d.

[0273] FIG. 6a shows in more detail the Figure 5 The pre-processed sub-graph 410 is shown in FIG.

[0274] The preprocessing calculation graph 41 is a normalization calculation graph 410 for normalizing the pixel values ​​of the input image 402. The input image 402 has 28×28 pixels, and each pixel is represented by the numerical value of the corresponding grayscale tone of the pixel. In the first node 412 of the normalization calculation graph 410, the maximum value MAX of all pixel values ​​of the image 402 is estimated. In parallel, the input image 402 is also input in the second node 414, so as to determine the minimum pixel value MIN of all pixels of the image 402.

[0275] Then, the first node 412 and the second node 414 and the input image 402 itself are connected to the third node 416 of the normalized computation graph 410, and a normalized grayscale value is calculated for each pixel of the input image 402. Then, the output of the normalized computation graph 410 or the preprocessed computation graph 410 is also a 28×28 image 402, but with normalized grayscale values ​​between 0 and 1.

[0276] The normalized or pre-processed input image 402 is then passed to the neural network computation graph 500 and the monitoring computation graph 600, which are shown in more detail in Figures 6b and 6c, respectively.

[0277] Before explaining the neural network computation graph 500 shown in FIG. 6 b in more detail, some general remarks on representing a neural network as a computation graph are first made below.

[0278] A neural network layer that receives input from a previous layer can use a parameter matrix to perform a matrix multiplication between the parameter matrix and the input. In the conventional case, this matrix multiplication is a representation of multiple neural network nodes in a computational graph. For example, a matrix multiplication can be broken down into multiple multiplication and addition operations, and each operation can again be represented by a different node in the computational graph. The operation represented by each node can generate a corresponding output, which flows to subsequent nodes on directed edges. After the operation represented by the final node generates the result of the matrix multiplication, the result flows to another node on a directed edge. The result is equivalent to the output of the neural network layer that performed the matrix multiplication.

[0279] In some other cases, the matrix multiplication is represented as a node in the graph. The operation represented by the node can receive as input an input tensor on a first directed edge and a weight tensor (e.g., a parameter matrix) on a second directed edge. The node can process, for example, perform a matrix multiplication of the input tensor and the weight tensor to output an output tensor on a third directed edge, which is equivalent to the output of the neural network layer.

[0280] Other neural network operations that can be represented by nodes in the computational graph include other mathematical operations such as subtraction, division, and gradient calculations; array operations such as concatenation, splicing, splitting, or sorting; and neural network building block operations such as SoftMax, Sigmoid, ReLU, or convolution.

[0281] Computational graphs make it easy to train neural networks using the gradient descent algorithm, making it many times faster than traditional neural network implementations. Computational graphs are also used in weather forecasting by reducing the associated computation time. The advantage is that derivatives are calculated quickly. It is also called "reverse mode derivation" or "backpropagation".

[0282] FIG. 6 b shows a neural network computation graph 500 consisting of three neural network layers 510 , 520 , 530 in more detail.

[0283] The first neural network layer 510 is a so-called “flattened” layer 510 that converts the 28×28 pre-processed input image 402 into a vector of size 784×1, which is then passed to the layer 2 computation subgraph 520 of the neural network computation graph 500.

[0284] This vector is then passed to a second neural network layer 520 consisting of 32 neural network nodes. Here, in this example, the computation graph, in particular the neural network computation graph, will not be explained in detail. The computation graph, in particular the neural network computation graph, is designed and arranged according to the prior art known from the literature (e.g. "Make Your Own Neural Network", Tariq Rashid, CreateSpace Independent Publishing Platform, 2016, ISBN-13: 978-1530826605). In the example shown here, a simplified version of the neural network layer is used, in which the bias values ​​of the different nodes of the neural network layer are not taken into account. This simplification obtains still correct results and makes the corresponding drawings less complicated.

[0285] The second layer subgraph 520 shown in FIG6b includes a weight matrix 524 of size 784×32, which includes weights assigned to the second layer computation subgraph 520. In the matrix multiplication node 522 of the second layer computation graph, the input vector from the first layer is multiplied by the weight matrix 524 in the corresponding multiplication node 522 to generate an output vector of dimension 32×1 and enter the activation node 526 of the second layer computation graph 520. Here, the vector output from the matrix multiplication node 522 is multiplied by the sigmoid function S(x)=1 / (1+e (-x) ) are multiplied.

[0286] After applying the activation function in the corresponding node 526 of the second layer calculation graph 520, the output vector ASL2 of the second layer calculation graph 520 of dimension 32×1 is passed to the third layer calculation graph 530. The third layer 530 of the neural network calculation graph 500 includes 10 neural network nodes. The third layer calculation graph 530 includes a weight matrix node 534, which contains a weight matrix of dimension 32×10. In the matrix multiplication node 532, the input vector ASL2 of the third layer calculation graph is multiplied by the weight matrix from the weight node 534 to produce a vector of dimension 10×1.

[0287] This vector is then passed to the activation node 536 of the layer 3 computation graph 530 to multiply the vector with the sigmoid activation function described above. The result of this multiplication is the output vector ASL3 590 of the neural network computation graph 500, which is then further passed to the post-processing or output preparation computation subgraph 490, which is further explained in detail in FIG6d.

[0288] The output vector ASL3 590 of the neural network computation graph 500 is a vector of dimension 10×1. The neural network computation graph 500 or neural network 500 is trained to detect 5 different products, each having a correctly produced version and an incorrectly produced version. The 10 different product versions that the neural network 500 can detect correspond to positions within the output vector 590. The product versions identified by the neural network 500 then correspond to values ​​close to 1 in the corresponding positions of the output vector 590. In the example shown in FIG. 6 b , the product version according to the third position in the output vector 590 is identified by the neural network 500. This can, for example, be associated with a correctly produced version of product number 2. Product versions that are not identified by the neural network 500 correspond to values ​​close to 0 in the neural network output vector 590.

[0289] FIG6 c shows a detailed version of the monitoring calculation subgraph 600 .

[0290] The pre-processed input image 402 from the pre-processing computation subgraph 410 is input into the “Average Luminance Calculator” node 610 of the monitoring computation subgraph 600, which calculates the average luminance value of the normalized input image 402. This can be performed, for example, by calculating the average of all 28×28 grayscale values ​​of the normalized input image 402. This average luminance value is then forwarded to the “Reliability Information Creation” computation subgraph 630 for creating reliability information “M” about the input image 402 and / or the corresponding output vector 590 estimated by the neural network 500.

[0291] Typically, the reliability information creation calculation subgraph 630 works in such a way that a normal distribution function is fitted to the average brightness value of each preprocessed normalized training image used to train the neural network calculation graph 500. This normal distribution function (also called Gaussian distribution function) (see above) has a mean value Mu (=μ) 622 and a standard deviation Sigma (=σ) 624. After the monitoring calculation graph 600 is set up using the training images used to train the neural network calculation graph 600, the determined Mu (=μ) 622 and Sigma (=σ) 624 are stored in the normal distribution node 620 of the monitoring calculation subgraph 600 together with the complete set of all brightness values ​​(={N}) 626 used to determine the normal distribution for monitoring.

[0292] The first case M=0 produces a "reliable result" according to this specification, the second case M=1 corresponds to a "result that is still reliable but must be observed" according to this specification, and the third case M=2 corresponds to an "unreliable result" according to this specification.

[0293] If the average brightness value of the normalized input image 402 calculated by the average brightness calculation node 610 is within the range between Mu-2xSigma and Mu+2xSigma, the brightness value of the normalized input image 402 is considered to be well within the brightness value range of the image used to train the neural network 500, and the result generated by the neural network 500 after the corresponding normalized image 402 is input into the neural network 500 is considered to be reliable. In this case, a value of 0 is assigned to the quality indicator variable "M" 690.

[0294] If the average brightness value of the normalized input picture 402 is within the range between mu-2xSigma and mu-3xSigma or mu+2xSigma and mu+3xSigma, the result produced by the neural network 500 after the normalized image 402 is input to the neural network 500 is considered to still be reliable but must be observed. In this case, a value of 1 is assigned to the quality indicator variable "M" 690.

[0295] If the average brightness value of the average brightness value of the normalized input image 402 is greater than mu+3xSigma or less than mu-3xSigma, the average brightness of the image 402 is considered to be outside the average brightness range of the image used to train the neural network 500, and the output result 590 after the normalized image 402 is input into the neural network 500 is considered to be unreliable. In this case, a value of 2 is assigned to the quality indicator "M" 690.

[0296] As shown in Figure 6c, these assignment steps of assigning the numbers 0, 1 or 2 to the quality indicator variable "M" 690 are completed in two nodes 632, 634 within the reliability information creation calculation subgraph 630 of the monitoring calculation graph 600. The value of the quality indicator variable "M" 690 is then output to the post-processing or output preparation calculation subgraph 490 of the quality check component calculation graph 400.

[0297] FIG. 6 d shows a detailed view of a post-processing or output preparation computation subgraph 490 of the quality check component computation graph 400 .

[0298] From the neural network computation graph 500, the output vector 590 is passed to the post-processing computation graph 490 and fed to the result derivation node 492 of the post-processing computation subgraph 490. The result derivation node 492 is designed and arranged to decode the output vector 590 to receive the identified product information "R" 493, such as the product #2 of the correct production version that has been identified in the current example. Then, the identified product information "R" 493 is forwarded to the first decision node 494.

[0299] The quality indicator variable "M" 690 is also passed from the monitoring calculation subgraph 600 to the post-processing subgraph 490 and is also input to the first decision node 494. The first decision node 494 is designed and arranged so that if the quality indicator variable "M" 690 has a value of 0, an "OK qualified result" 702 containing the identified product information "R" 493 is output to the HMI 340.

[0300] If the quality indicator variable “M” 690 contains a value of 1 or 2, “M” 690 is forwarded to the second decision node 496 of the post-processing calculation subgraph 490. The second decision node 496 is designed and arranged so that, if the quality indicator variable “M” 690 has a value of 1, an “observation quality result” 704 containing the recognized product information “R” 493 is output to the HMI 340 together with a warning message that the image information should be observed. The second decision node 496 is further designed and arranged so that, if the quality indicator variable “M” 690 has a value of 2, a “warning qualified result” 706 containing a warning message that the output of the neural network 500 is unreliable is output to the HMI 340.

[0301] FIG7a shows a version of a neural network computation graph 500 including inference and training components. The components 510, 520, 530 of the neural network computation graph 500 necessary for inference have been explained in conjunction with FIG6b. These components of the neural network computation graph 500 are the layer 1 computation graph 510, the layer 2 computation graph 520, and the layer 3 computation graph 530.

[0302] The neural network computation graph 500 shown in FIG7a further includes a computation subgraph 515, which includes components for training the neural network 500 by adding training functionality to the neural network computation graph 500. In order to train the neural network 500, a normalized training image 552 is input into the neural network computation graph 500, and the training input image 552 is processed by the first layer 510, the second layer 520, and the third layer 530 to create an inference result output vector 590. The normalized training image 552 is thus constructed in the same manner as the normalized image 402 used for inference in the embodiment shown in FIG6b.

[0303] The training process is briefly described below, but not explained in detail. The training is performed according to principles known in the prior art, such as described in "Make Your Own Neural Network", Tariq Rashid, CreateSpace Independent Publishing Platform, 2016, ISBN-13:9781530826605 or similar documents.

[0304] The training procedure for training the neural network 500 is so-called supervised learning using labeled training data. Thus, a target output vector 551 is assigned to the training image 552, wherein the target vector 551 shows the planned result that should be achieved after the training image 552 is input into the neural network computation graph 500.

[0305] The training image 552 and the corresponding planning result 551 are examples of training data according to the present specification.

[0306] As already explained, after the normalized training image 552 is input into the inference components 510, 520, 530 of the neural network computation graph, an output vector 590 as shown in FIG7a is created. This output vector 590 is then input to the output error calculation node 553 together with the target vector 551 assigned to the training image 552. The output error value OE is calculated by the output error calculation node 553 by subtracting the output vector 590 from the target vector 551 and multiplying the difference with itself.

[0307] The output error OE is then passed to the 3rd layer Delta weight node 554 for calculating the change ΔWL3 of the 3rd layer weight due to the current training step. To this end, the output vector ASL3590 and the learning rate Alpha (=α) 555 are also input into the 3rd layer Delta weight node 554. Then, the change ΔWL3554 of the 3rd layer weight is calculated according to the above-mentioned prior art. In creating a new 3rd layer weight node 556, a new 3rd layer weight WL3 534 is calculated from the Delta weight ΔWL3 554 and the previous 3rd layer weight WL3 534.

[0308] In order to calculate the change of the second layer weight WL2, the output error OE 553 is also forwarded to the second layer error node 558 to calculate the second layer error L2E 558 by additionally using the third layer weight WL3 534. In the next step, by additionally using the output of the second layer calculation graph ASL2 520, the output of the first layer calculation graph L1O 510 and the learning rate Alpha (= α), the second layer error L2E558 is forwarded to the calculation second layer weight change node ΔWL2 560 for calculating the change of the second layer weight in the current training step. The calculation is also completed according to the above-mentioned document. In creating a new second layer weight node 562, a new second layer weight WL2 524 is calculated by adding the previous second layer weight WL2 524 and the second layer weight change ΔWL2562.

[0309] To receive a trained neural network computation graph 500 , the procedure is typically performed using several training images 552 and correspondingly assigned target vectors 551 .

[0310] FIG7 b shows a monitoring computation graph 600 with reasoning and training components. The reasoning components 610, 620, 630 have been explained with reference to FIG6 c. In order to make the monitoring computation graph 600 also suitable for the training picture 552 used to train the neural network computation graph 500, the training picture 552 is also input into the monitoring computation graph 600 by inputting it into the average brightness calculation node 610 to calculate the average brightness ABR of the training picture 552.

[0311] In order to adapt the monitoring calculation graph 600 accordingly to the training picture 552, a new standard brightness distribution is created using the fitting node 650 (see above).

[0312] The fitting node 650 uses the average luminance ABR of the training picture 552 and the luminance set {N} 626 used to estimate the current standard luminance distribution 622 to fit a new standard luminance distribution (see above) to the newly gathered luminance data (ABR+{N}).

[0313] After fitting the new standard brightness distribution 620 , the newly estimated mean value Mu (=μ) 622 and standard deviation Sigma (=σ) 624 are stored in the standard brightness distribution node 620 .

[0314] In Figures 7a and 7b, the data flows for inference of the neural network 500 and monitoring 600, respectively, are shown by dashed arrows.

[0315] The quality inspection component 400 may, for example, be composed of a pre-processing computation graph 410 according to FIG. 6 a and a post-processing computation graph 490 according to FIG. 6 d , as well as a neural network computation graph 500 according to FIG. 7 a and a monitoring computation graph 600 according to FIG. 7 b . In this case, such a quality inspection component computation graph 400 may, for example, be deployed and / or used in a corresponding controller 300 .

[0316] The advantage of this embodiment is that the corresponding quality check component 400 can be used directly for retraining, for example with new training data, because it also contains the training component of the neural network computation graph 500 and the training component of the monitoring computation graph 600. The disadvantage of this embodiment is that such a quality check component computation graph 400 may require improved hardware performance also for inference, for example due to the additional training component.

[0317] In an alternative embodiment, the quality inspection component 400 may, for example, include a preprocessing computation graph 410 according to FIG. 6a and a post-processing computation graph 490 according to FIG. 6d, as well as a neural network computation graph 500 according to FIG. 6b and a monitoring computation graph 600 according to FIG. 6c, the latter two consisting only of the corresponding components necessary for reasoning. The advantage of this embodiment is that it requires less powerful hardware than the hardware required for the aforementioned embodiment. However, its disadvantage is that the retraining of the neural network computation graph 500 and the monitoring computation graph 600 must be performed using separate versions of the neural network computation graph 500 and / or the monitoring computation graph 600. Such a separate version of the neural network computation graph 500 can be designed and arranged, for example, according to FIG. 7a, and such a separate version of the monitoring computation graph 600 can be designed and arranged, for example, according to FIG. 7b.

[0318] In this second embodiment, the deployment or construction of the quality inspection component 400 may be performed, for example, in the manner described below.

[0319] To train the neural network computation graph 500, the neural network computation graph 500 according to FIG. 7a is used. To train the monitoring computation graph 600, the monitoring computation graph according to FIG. 7b is used. Then, in the following compilation and / or optimization process according to the present specification, only the components of the neural network computation graph 500 required for training 550 are removed from the neural network computation graph 500, and only the components required for training the new probability density function 650 for fitting the monitoring computation graph 600 are removed from the monitoring computation graph 600.

[0320] Then, the neural network computation graph 500 and the monitoring computation graph 600 are connected to the pre-processing computation graph 410 for input data and the post-processing computation graph 490 for output data. In this way, the quality check component computation graph 400 is constructed as a single-connection computation graph optimized for reasoning. Then, such a quality check component computation graph 400 can be deployed and / or used in the corresponding controller 300.

[0321] If the quality check component computation graph 400 must be retrained using new training data or training images 402, the process must then retrain the previously used neural network computation graph 500 according to FIG. 7 a and the monitoring computation graph 600 according to FIG. 7 b again using new training data 552. Then, the quality check computation graph 400 suitable for reasoning is constructed again according to the previous description.

[0322] The aforementioned steps of creating the quality check computation graph 400 from the training neural network computation graph 500 and the training monitoring computation graph 600 can generally be applied independently, regardless of whether the resulting quality check computation graph 400 is optimized for inference and / or still has training capabilities. The construction or building process can additionally include optimization steps to further optimize the performance of the deployed quality check computation graph 400. Such optimization can also, for example, include the distribution of the quality check computation graph 400 on two or more hardware devices, processors or processor cores, or prepare the computation graph 400 for such distribution.

[0323] All types of quality check computation graphs 400 described in the embodiments described in this specification are examples of computer program products according to the present specification. All embodiments of the quality check computation graphs 400 described in this specification are single-connected and / or fully-connected computation graphs according to the present specification.

Claims

1. A computer program product (100, 400) for analyzing data originating from at least one device (360), the computer program product (100, 400) comprising: - a ML artifact (120, 500) comprising the functionality of the ML model, and - Monitoring workpieces (130, 140, 600), wherein the ML model has been trained and / or configured, or is capable of being trained and / or configured, using a machine learning method using training data, wherein the ML artifact (120, 500) is designed and arranged to generate an ML result (122, 590) based on data originating from the at least one device (360) in response to inputting input data (102, 402, 552) into the ML artifact (120, 500), wherein the monitoring artifact (130, 140, 600) is designed and arranged to generate ML result reliability information (132, 142, 690) regarding the ML result (122, 590) in response to inputting the input data (102, 402, 552) into the monitoring artifact (130, 140, 600), wherein the training data or a portion of the training data is used to train the monitoring artifact (130, 140, 600), and wherein the computer program product (100, 400) is further designed and arranged to output a qualified ML result (190, 702, 704, 706) based on the ML result (122, 590) and the ML result reliability information (132, 142, 690) when executed by a computer (300, 400).

2. The computer program product according to claim 1, characterized in that The ML artifact (120, 500) is designed and arranged for training the ML model based on training data (551, 552), or for enabling training the ML model based on the training data (551, 552), And / or the monitoring workpiece (130, 140, 600) is designed and arranged to train itself based on the training data (551, 552), or to be able to train itself based on the training data (551, 552).

3. The computer program product according to claim 2, characterized in that After the ML artifact (120, 500) and the monitoring artifact (130, 140, 600) are trained or set based on the training data (551, 552), the trained ML artifact (120, 500) and the trained or modified monitoring artifact (130, 140, 600) are combined into the computer program product (100, 400).

4. The computer program product according to any one of the preceding claims, characterized in that If the computer program product (100, 400) is run on a computer (300, 400), the ML artifact (120, 500) and the monitoring artifact (130, 140, 600) are executed in a single program run or cycle.

5. The computer program product according to any one of the preceding claims, characterized in that The computer program product (100, 400) is designed or arranged as a single workpiece (400).

6. The computer program product according to any one of the preceding claims, characterized in that The computer program product (100, 400) comprises a computation graph (400) comprising both the ML artifact (120, 500) and the monitoring artifact (130, 140, 600), or, A single computation graph (400) including both the ML artifact (120, 500) and the monitoring artifact (130, 140, 600) can be implemented, wherein the single computation graph (400) is part of the computer program product.

7. The computer program product according to any one of the preceding claims, characterized in that The computer program product (100, 400) further comprises a pre-processing artifact (110, 410) to convert data (102, 402, 552) from the at least one device into the input data, and / or The computer program product also includes a post-processing artifact (150, 490) to generate the qualified ML result (190, 702, 704, 706).

8. The computer program product according to claim 6 or 7, characterized in that The single computation graph (400) includes the pre-processing artifact (110, 410) and / or the post-processing artifact (150, 490).

9. A computer-implemented method for generating qualified output (190, 702, 704, 706) of an ML model based on data originating from at least one device (360), comprising the steps of: - based on data originating from the at least one device (360), inputting input data (102, 402, 552) into a ML artifact (120, 500) comprising the functionality of the ML model to produce an ML result (122, 590), - inputting the input data (102, 402, 552) into a monitoring artifact (130, 140, 600) to generate ML result reliability information (132, 142, 690), - outputting a qualified ML result (190, 702, 704, 706) based on the ML result (122, 590) and the ML result reliability information (132, 142, 690), Wherein, when the computer program product (100, 400) is run on a computer (300, 400), the following method steps are performed in a single run of the computer program product: inputting input data (102, 402, 552) into the ML artifact (120, 500), inputting the input data (102, 402, 552) into the monitoring artifact (130, 140, 600), and outputting a qualified ML result (190, 702, 704, 706) based on the ML result (122, 590) and the ML result reliability information (132, 142, 690).

10. A computer-implemented method for generating a trained ML model (120, 500) and / or ML artifact (120, 500) and a trained or modified monitoring artifact (130, 140, 600) based on training data (551, 552), comprising the steps of: - inputting the training data (551, 552) into a ML artifact (120, 500) comprising the functionality of the ML model to produce a trained ML artifact (120, 500), - inputting the training data (551, 552) into a monitoring artifact (130, 140, 600) to generate a trained or modified monitoring artifact (130, 140, 600), wherein when the computer program product (100, 400) is run on a computer (300, 400), the following method steps are performed in a single run of the computer program product: inputting the training data (551, 552) into the ML artifact (120, 500) and inputting the training data (551, 552) into the monitoring artifact (130, 140, 600), and generating the trained ML artifact (120, 500) and the trained or modified monitoring artifact (130, 140, 600).

11. The computer-implemented method of claim 10, wherein: After training and / or modifying the ML artifact (120, 500) and the monitoring artifact (130, 140, 600) based on the training data, the trained ML artifact (120, 500) and the trained or modified monitoring artifact (130, 140, 600) are combined into the computer program product (100, 400).

12. The computer-implemented method according to any one of claims 9 to 11, characterized in that: The computer program product (100, 400) comprises the ML artifact (120, 500) and the monitoring artifact (130, 140, 600) as a single computation graph (400), Alternatively, the computer program product (100, 400) is implemented as a single workpiece (400).

13. The computer-implemented method according to any one of claims 9 to 12, characterized in that: The computer program product (100, 400) is designed and configured as a computer program product (100, 400) according to any one of claims 1 to 8.

Images