Method for generating a control program for a programmable logic controller, and automation system

Abstract

The invention relates to a method for generating a control program for a programmable logic controller in an automation system on the basis of an AI pipeline which contains at least one AI model optionally with an input data preprocessing function and / or an output data post-processing function, wherein the following steps are carried out: providing a latency model for predicting the computing time for running an AI pipeline on the basis of hardware configurations and software configurations and a compatibility model for mapping AI pipeline functions to software configurations; detecting a hardware configuration and a software configuration of the programmable logic controller; generating a set of AI pipeline candidates on the basis of the compatibility model; selecting an AI pipeline from the set of AI pipeline candidates by evaluating the performance of the AI pipeline candidates after training the AI pipeline candidates while taking into account a prediction of the computing time on the basis of the latency model; and generating a source code for the control program with the selected AI pipeline to be run on the programmable logic controller in the automation system.

Description

[0001] Method for generating a control program of a programmable logic controller

[0002] Control and automation system

[0003] Description.

[0004] The present invention relates to a method for generating a control program of a programmable logic controller (PLC) in an automation system. The invention further relates to a method for operating an automation system and to an automation system.

[0005] The patent application claims priority from German patent application 10 2024 110 997.1, the disclosure of which is hereby incorporated by reference.

[0006] Machines or systems in an automation system are often controlled using programmable logic controllers (PLCs). The PLC can be an external device or a software PLC. To control or regulate the actuators and sensors of the machine or system, the PLC usually uses a communication interface in the form of a fieldbus system. The actuators and sensors of the machine or system can then be networked with each other via the fieldbus system.

[0007] By reading the measurement data from the sensors and / or the current actual data from the actuators connected to the inputs of the programmable logic controller, the PLC receives information about the status of the machine or system. The actuators are connected to the outputs of the programmable logic controller and enable the control of the machine or system. To dynamically control the actuators, the PLC generates the output data for the actuators based on the actual data and / or the measurement data from the sensors. This data can be individual values ​​or groups of values. The actuators can also be controlled based on specifications, for example, based on motion profiles.

[0008] To ensure the desired operation of the machine or system, the PLC's control task determines which output data generated by the PLC, based on the corresponding input data, is fed to the actuators. Data processing by the PLC is usually cyclical and involves three steps: providing current input data (e.g., actual data from the actuators and / or measured data from the sensors), processing the input data into output data, and outputting the output data for motion control (e.g., target positions, etc.).

[0009] Industrial PLCs are deterministic real-time systems. Determinism here means that an event or calculation is completed within a fixed time interval. These time intervals are defined by the specific applications of the respective automation system. Programmable logic controllers generally require hard real-time capability, which means that deadlines must be consistently met and never exceeded. In addition, a valid result must be available for the actuators at the time of the deadline. This is particularly relevant if exceeding the deadline could lead to personal injury or property damage, for example, if a robot arm is not decelerated in time.

[0010] In some automation system applications, algorithms based on artificial intelligence (AI) are also executed on the corresponding PLCs, also referred to as AI models. The term AI model primarily refers to neural networks, but the term also encompasses statistical models such as linear models, support vector machines, decision trees, or random forests. AI can improve the efficiency of the PLC, especially in the area of ​​sensor data processing, for example, in image processing, or even enable certain functions in the first place.

[0011] Given the above-mentioned aspect of the PLC's real-time capability, it is necessary to integrate the used AI models into the PLC's runtime environment so that both latency and determinism requirements can be met. A guarantee of the operability of a selected AI model in the context of the given deterministic environment and prior to its implementation is therefore desirable, but not known in the state of the art.

[0012] It is an object of the invention to provide an alternative or improved method for generating a control program of a programmable logic controller (PLC) in an automation system or for operating an automation system, as well as a corresponding automation system. The object of the invention is achieved by the independent claims. Advantageous developments, additional features, and / or advantages of the invention emerge from the dependent claims and the following description.

[0013] It should be noted that all features mentioned in connection with the disclosed method can also be embodiments of the disclosed automation system, and vice versa.

[0014] To generate a control program of a programmable logic controller in an automation system based on a Kl pipeline, which contains at least one Kl model with optional preprocessing of input data and / or postprocessing of output data, the following steps are carried out: Providing a latency model for predicting a computing time for an execution of a Kl pipeline based on hardware configurations and software configurations and a compatibility model for mapping Kl pipeline functions to software configurations; Creating a set of Kl pipeline candidates based on the compatibility model; Capturing a hardware configuration and a software configuration of the programmable logic controller;Selecting a Kl pipeline from the set of Kl pipeline candidates by evaluating the performance of the Kl pipeline candidates after training the Kl pipeline candidates, taking into account a prediction of a computing time based on the latency model; Creating a source code of the control program with the selected Kl pipeline for execution on the programmable logic controller in the automation system;

[0015] When integrating AI models or AI pipelines into the controller, the latency of the model must be taken into account during model training for a given hardware specification. Furthermore, the AI ​​model must be integrated into the PLC runtime environment to ensure that both latency and determinism requirements can be met.

[0016] The latency model can determine a latency in the form of a numerical value resulting from the sum of the runtimes of the executed KL pipeline functions. KL pipeline functions are typically operators or sequences of operators. The latency model is capable of predicting the latency of KL pipelines. The latency model considers not only the hardware configuration but also the software configuration of the target system to predict the execution time.

[0017] The compatibility model can be used to provide a data structure in which Kl pipeline functions are mapped to equivalent functions in the software configuration. The compatibility model ensures that no Kl pipeline candidates are generated that are not executable on a specific target system.

[0018] When selecting and training AI model candidates, it is typically not checked whether the AI ​​model is executable on the target system. While hardware specifications are taken into account, this alone is insufficient to certify actual operability. With the compatibility model, which considers both the hardware and software configuration of the target system, operability is inherently guaranteed.

[0019] The guaranteed operability also enables automation of the inference code generation. Since the KL pipeline also optimizes pre- and / or post-processing steps, the KL model is augmented.

[0020] The set of AI pipeline candidates can be created using at least one application-specific training dataset.

[0021] This involves considering combinatorial possibilities of sequences of operators, using only operators that are compatible with the target system according to the compatibility model. This results in a search graph with various possible paths.

[0022] The compatibility of the AI ​​pipeline candidates is already ensured in this step. During generation, not only different AI models are considered, but also pre- and / or post-processing steps. Here, too, the compatibility model ensures the executability of the operators under consideration.

[0023] A one-shot neural architecture search method can be used to create the set of AI pipeline candidates. When selecting the AI ​​pipeline from the set of AI pipeline candidates, the generalization capability of the AI ​​pipeline can be considered.

[0024] A Pareto optimality method can be used to select the Kl-pipeline from the set of Kl-pipeline candidates.

[0025] The selected KL pipeline can be retrained with a training dataset to improve the generalization ability.

[0026] When selecting the KL pipeline from the set of KL pipeline candidates, if no pipeline candidate meets a given latency requirement, an adjustment of the hardware configuration and / or the software configuration can be performed.

[0027] The Kl models for the Kl pipeline candidates can be selected from the group of neural networks, statistical models, support vector machines, decision trees, and / or random forests.

[0028] To operate an automation system, the source code of the control program is uploaded to the programmable logic controller. The control program can then be executed to perform an automation task. An automation system can be operated in this way to perform an automation task.

[0029] The invention is explained in more detail below using exemplary embodiments with reference to the attached schematic and not-to-scale drawings.

[0030] In the merely exemplary figures (Fig.) of the drawings show:

[0031] Fig. 1 schematically shows an embodiment of an automation system;

[0032] Fig. 2 schematically shows an embodiment of a method for generating a control program of a programmable logic controller of the automation system;

[0033] Fig. 3 schematically shows a neural network model; Fig. 4 schematically shows a search graph for a candidate AI pipeline;

[0034] Fig. 5 schematically shows a Kl pipeline candidate;

[0035] Fig. 6 schematically shows the performance of different Kl pipeline candidates.

[0036] The following definitions are used below:

[0037] KL pipeline: Consists of a KL model and optional pre-processing of the input data or optional post-processing of the output data.

[0038] Training: Optimization of a KL model consisting of a set of adjustable parameters. Using a suitable method, it determines the parameter combinations that minimize a cost function (loss function) for a given dataset. This involves differentiable KL models (usually artificial neural networks) that are optimized using gradient descent methods (especially backpropagation) on supervised training data (features and labels).

[0039] Latency: The time delay between requesting a prediction from a model and receiving the prediction. This includes, among other things, the computation of the model itself, but also any overhead caused by communication with the execution environment, memory access, and communication between individual execution units.

[0040] Real-time capability: Requirement that an event or calculation is completed within a fixed time interval.

[0041] Operability: Executability of an AI model or AI pipeline on a target system, which is defined by the mapping of all operators of the AI ​​model or AI pipeline to a semantically equivalent function on the target system.

[0042] Compatibility: If a Kl model or Kl pipeline is operable on a target system, then the Kl model or Kl pipeline and target system are compatible with each other.

[0043] Generalization capability: The ability of an AI model or pipeline to make accurate (i.e., as close as possible to the ground truth) predictions on previously unseen data. Once a AI model has been trained on training data, the AI ​​model should function as flawlessly as possible in a subsequent application on newly generated, previously unseen data by learning fundamental concepts that can be generalized to new data and reflect the underlying relationships between features and labels.

[0044] The following schematically describes the structure and operation of an automation system with a programmable logic controller (PLC), as well as a method for generating a control program of a programmable logic controller (PLC) in the automation system and a method for operating the automation system. Corresponding reference numerals are used for corresponding features.

[0045] Figure 1 shows an exemplary automation system 10, which, in the configuration shown, is part of a packaging machine. The automation system 10 comprises a conveyor belt 1, on which packaging units 2 are transported in the direction of arrow 3, and a programmable logic controller (PLC) 20, which controls processes in the automation system 10 to be described below. A control program is installed on the PLC 20 for this purpose. The PLC 20 is technically described by a hardware configuration and a software configuration.

[0046] The hardware configuration includes, for example, components such as the motherboard, microprocessors (CPU), memory modules (RAM), graphics processors (GPU), and so on. The software configuration of the PLC 20 includes information such as the type and version of the operating system used, the type and version of the control program, for example, the TwinCAT automation software, specifications for required and / or available software libraries for performing various tasks of the PLC 20, and other software-specific features. In this disclosure, in connection with the creation of a control program for the PLC 20, the hardware and software specifications together are also referred to as the target system.

[0047] The automation system 10 further comprises a sensor device 4, which comprises, for example, a camera and / or a scale, and an ejection station 5. The sensor device 4 and the ejection station 5 are parts of a device for inspecting the packaging units 2. The inspection checks whether each produced packaging unit 2 meets certain specified quality requirements. For the purposes of this description, it is assumed that each of the packaging units 2 is classified in a binary manner by the inspection, i.e., assigned to one of the categories OK or not OK, based on criteria such as visually detectable damage, dimensions to be maintained, the presence or absence of labels, a prescribed weight, etc.If it is determined that a packaging unit 2 is not OK, it is sorted out of the system at the discharge station 5, for example, removed from conveyor belt 1. While the packaging units 2 that fall into the OK category remain on conveyor belt 1 and are further processed in a manner not shown. The binary classification serves only as a simplified example. The classification could also be significantly more complex and, for example, have a large number of different quality levels or even include a hierarchical structure with main classes and subclasses.

[0048] The packaging machine processes two packaging units 2 at a time in a specific first time period AT 1 , which is indicated in Figure 1 as the distance between the packaging units 2. The first time period AT 1 can be, for example, 200 ms. A maximum permissible decision time period AT2, in which a decision must be made as to whether the individual packaging unit 2 is OK or not OK, has a value that is generally somewhat shorter than the first time period AT 1 and depends, among other things, on the structural distance between the sensor device 4, for example the camera, and the discharge station 5. The maximum permissible decision time period AT2 can be, for example, 150 ms.After the maximum permissible decision time AT2 has elapsed, the result of the classification of the individual packaging unit 2 detected by the camera 4 must be securely available in the PLC 20 so that the PLC 20 can supply the discharge station 5 with control commands accordingly.

[0049] As already mentioned at the beginning, the PLC 20 is a deterministic real-time system. The maximum permissible decision time AT2 therefore defines a time interval that must be consistently adhered to and must never be exceeded, as otherwise downstream processes of the continuously or quasi-continuously operating automation system 10 would be impaired. In the context of the example shown in Figure 1, a process for classifying packaging unit 2 that can reliably provide the result within the maximum permissible decision time AT2 is also referred to as real-time capable.

[0050] The sensor device 4 is connected to the PLC 20 via a suitable interface 21, which is configured to interpret the sensor signals or data transmitted by the sensor device 4. In this context, interpreting means performing the classification described above based on the sensor data.

[0051] For this purpose, artificial intelligence (AI) methods are used, which are implemented as algorithms or software modules or programs on the PLC 20 and are also referred to as AI models in the context of the present disclosure. With the aid of the AI ​​models, the PLC 20 is able to perform tasks such as the classification of the packaging units 2 in such a way that random changes in the boundary conditions that are insignificant for the task, such as, for example - during an optical inspection - a change in the ambient light conditions or a variable orientation of a packaging unit 2 on the conveyor belt 1, do not influence the classification result. The result output by the AI ​​model, its output, is also referred to below as a prediction.

[0052] A large number of Kl models are known in the prior art, each of which can be based on different mathematical models (for example, on artificial neural networks) and place different demands on the hardware and / or software configuration of the PLC 20 or, with certain hardware and / or software configurations, provide different performance, i.e., provide predictions at different speeds and / or with different accuracy.

[0053] In its function within the PLC 20, the Kl model is also integrated in software into a so-called Kl pipeline, which is designated as Kl pipeline 22 in Figure 1. The Kl pipeline comprises the Kl model used as well as any necessary pre-processing steps for input data and / or post-processing steps for output data. In the embodiment shown in Figure 1, the Kl pipeline 22 comprises the Kl model 24 as well as a pre-processing step 23 for the sensor data provided by the sensor device 4 and a post-processing step 25 for the output output by the Kl model 24. Neural networks are primarily used as the Kl model. However, the Kl model can also be another (classical) statistical model, such as a linear model, support vector machine, decision tree, random forest, etc.

[0054] If the classification of the packaging units 2 is based on image analysis, the Kl model 24 can be implemented as an image classifier. The preprocessing step 23 can then be, for example, that the input images provided by the sensor device 4 are preprocessed in a specific manner. In the example of an image classifier, a preprocessing step can be, for example, changing the image size of an input image or adjusting the saturation of the input image. The postprocessing step 24 can then be, for example, converting the predictions of the Kl model 24 into a different representation. In the example of an image classifier, a postprocessing step can be, for example, selecting the class with the highest probability or determining the modal value of the probability distribution after the output from the Kl model.

[0055] With regard to the automation system 10 and the creation of a control program of the PLC 20, one challenge is to select the Kl pipeline 22 for the respective task in the automation system 10, in the present example for the classification of the packaging units 2, and for a given target system of the PLC 20 and to integrate it into the runtime environment of the PLC 20 in such a way that an operability guarantee and a guarantee of the real-time capability of the control program can be given in advance.

[0056] The operability guarantee here means that even before the integration of an AI pipeline into a PLC, it is guaranteed that the selected AI model can be executed by the target system of the PLC.

[0057] It is desired to provide a control program optimized for the respective automation system or the specific PLC.

[0058] The method disclosed below for generating a control program for PLC 20, which is schematically illustrated in Figure 2, is executed at least partially as a computer-implemented method on automation system 10 itself or on an external computer. In a first step S1, a latency model and a compatibility model are provided. The latency model is understood as a function that maps the specific AI pipeline and the target system to a numerical value, the latency. Latency refers to the computing time required to execute the AI ​​pipeline on the target system under the given hardware and software requirements. The latency model can be used to predict the latency of AI pipelines that are not specifically known at the time the latency model is created.

[0059] The KL models of KL pipelines can be described as sequences of operators. Operators are elementary mathematical operations, such as convolution operators or activation functions, which are known to be used to specify KL models. Operators form the basic building blocks of KL models and are used in a wide range of KL models. By selecting from the multitude of operators or combining them in different ways, previously unknown KL models can be assembled.

[0060] For example, an artificial neural network 30 can be constructed as a special form of a Kl model, as shown schematically in Figure 3. In the representation of Figure 3, an input vector x, which is represented by the values ​​xi, Xj, x n is represented, is processed schematically from left to right in the neural network 30.

[0061] The neural network 30 shown in Fig. 3 is an MLP (multilayer perceptron)-type neural network, which, in a known manner, consists of three layers: an input layer 31, a hidden layer 32, and an output layer 33. The hidden layer 33 can in turn be composed of multiple layers. Each neuron in one layer is connected to every neuron in the next layer, with each connection having a numerical value called a weight. The MLP network has a feedforward architecture in which information flows through the network in only one direction, without feedback loops or cyclic connections.

[0062] MLP networks are used primarily in prediction applications. Alternatively, RBF (radial basis function)-type neural networks can also be used.

[0063] Each of the three layers, the input layer 31, the hidden layer 32, and the output layer 33 forms an operator, whereby the input layer 31 and the hidden layer in the example each represent a matrix multiplication with a first bias 311 and a second bias 321, and the output layer 33 represents a matrix multiplication without bias.

[0064] The runtime for calculating such operators in an artificial neural network 30 can vary greatly from target system to target system, depending on the specific hardware and software configuration. Therefore, the runtime of the respective operators is determined depending on the hardware and software configuration acquired in the second step S2.

[0065] The latency of the AI ​​pipeline is then determined as the sum of the runtimes of the operators of the AI ​​model, plus the runtimes of pre- and post-processing. It is also possible to describe pre- and post-processing with operators and then determine their runtimes.

[0066] In the concrete example of Figure 1, the latency of the neural network 30 shown in Figure 3, which together with post- and pre-processing defines the Kl pipeline 22, is determined on the given target system (ZS) by the following calculation rule:

[0067] Latency(AI — Pipeline, ZS) = Runtime(MatMultBias, ZS) + Runtime(MatMultBias, ZS) + Runtime(MatMult, ZS) + Runtime(Pre — Processing, ZS) + Runtime(Post — Processing, ZS) Equation 1)

[0068] Latency (Kl-Pipeline, ZS) refers to the latency of the considered Kl-Pipeline on the considered target system ZS. The abbreviation MatMult stands for the matrix multiplication operation described above, and MatMultBias stands for matrix multiplication with bias.

[0069] In order to determine the latency on a target system, the runtimes of the operators, for example MatMultBias, are previously measured on a number of different target systems and stored in a database.

[0070] The hardware configuration of a target system can, for example, include a specific type of personal computer (PC), a specific microprocessor used in the PC, for example, an Intel Core i7 CPU, and a specific memory configuration, such as 16 GB DDR4 RAM. The software configuration of a target system includes a version of the operating system running on the PLC 20, for example, a Microsoft Windows operating system version, a version of the software running on the PLC 20, for example, a basic TwinCAT system, and a version of certain software libraries executed by the PLC 20, such as a TwinCAT Vision version. Furthermore, such a description can also include the parameterization of a specific execution mode, for example, with or without multithreading, use of a KL accelerator, etc.

[0071] Table 1 below shows, line by line, the runtimes for the MatMultBias and MatMult operators for selected hardware and software configurations. In this example, the hardware configuration is a C6030 PLC with a Core i7-11850HE CPU. For the TwinCAT version 3.2.7 software configuration, the runtime for the MatMultBias operator is 0.003s (line 1), and the MatMult runtime for the MatMult operator is 0.002s (line 2). For the TwinCAT version 3.2.1 software configuration, the runtime for the MatMultBias operator is 0.006s (line 3).

[0072] Table 1

[0073] Using such a database, the latencies of AI models can now be calculated based on the individual operators of the corresponding AI models, taking the target system into account. For operators not contained in the database for the given target system, runtimes can be interpolated based on similarities in certain attributes.

[0074] The runtimes of pre-processing and post-processing, if they can be described with operators, can also be determined in this way. Furthermore, a latency model can also consider contextualizations of operator sequences, something that allows a target system to combine the consecutive execution of two MatMult operations and thus execute them particularly efficiently. This allows even more complex latency relationships to be modeled compared to a summation of operators alone (see Equation 1).

[0075] Instead of a database-based determination, where a previously measured runtime is determined for each operator or operator sequence of the AI ​​model, which is then accumulated to form the latency, (learned) AI models (such as neural networks) can also be used to determine the latency model. Here, too, the AI ​​models can map more complex latency dependencies between sequences of operators, so that, as in the aforementioned example, two MatMult operators can be combined by a AI accelerator, and the resulting latency on the selected target system is then smaller than the sum of the two individual latencies.

[0076] The latency model can be used to predict the expected latency for each AI pipeline for executing an automation task in the automation system, taking into account the software and hardware configuration of the target system.

[0077] As mentioned above, the first step S1 also provides a compatibility model of the automation system. This is advantageous because known AI models often use operators that are not necessarily supported on every target system. For example, the latest AI operators, such as the transpose of a matrix, are only supported with some delay in software platforms such as newer automation software versions, so a AI model that uses such an operator would simply not be executable on certain target systems, regardless of aspects such as the runtime of the operator or the latency of the AI ​​pipeline. This also applies accordingly to pre-processing and post-processing if these are described with operators.

[0078] To avoid selecting a CI model or CI pipeline that imposes unfulfillable requirements on the target system, the compatibility model ensures that no candidates for the CI model or CI pipeline are generated that are not executable on the specific target system. The compatibility model is a rule-based model that maps functions of the CI model or CI pipeline to functionally equivalent functions of the target system. Here, too, the contextualization of operator sequences can potentially be considered, for example, that an individual function of a CI pipeline has no equivalent to a function of the target system, but does when combined sequentially with another operator.

[0079] This can be understood as a type of database, the structure of which is illustrated by way of example in Table 2 below. Table 2 shows the corresponding equivalent function for the MatMultBias, MatMult, and imageResize (changing an image size) operators, line by line, for the hardware and software configuration of the target system. In the example, the hardware configuration of the target systems is PLC type C6030 with a Core i7-11850HE CPU. The software configuration of the target systems is at least TwinCAT version 3.2 or at least TwinCAT version 3.2.1. The MatMultBias operator is assigned the target function TwinCAT MatMultBias (line 1), the MatMult operator is assigned the target function TwinCAT MatMult (line 2), and the imageResize operator is assigned the target function TwinCAT imageResize (line 3).

[0080] Table 2

[0081] If an operator does not have a corresponding target function on a target system, the corresponding field can, for example, remain empty or contain the entry "not present." The compatibility model, a rule-based model that checks a CI model or CI pipeline and a description of the target system to determine whether the CI model is executable or operable on the target system, can be used to determine the compatibility between the CI model and the target system. For example, the compatibility model checks whether the software libraries in the target system, e.g., TwinCAT Vision, support certain operators for real-time execution.

[0082] Instead of a database-based determination, however, (learned) Kl models (such as neural networks or large language models) can also be used to determine the compatibility model.

[0083] In a second step S2 of the method for generating a control program for the PLC 20, which is schematically illustrated in Figure 2, the specific hardware configuration or software configuration of the automation system 10 is then recorded. This can be done automatically via software or via a user interface (human-machine interface, HMI) and interaction with a user.

[0084] Based on the captured hardware or software configuration, a set of suitable AI pipeline models is then created using the compatibility model in a third step S3. For this purpose, a large number of different AI pipelines are generated as candidate solutions. Various combinatorial possibilities of operator sequences are generated, using only those operators that are compatible with the captured hardware or software configuration of the target system according to the compatibility model.

[0085] This process can be represented as a search graph 40, as shown schematically in Figure 4, whose operators are connected via various path options represented as dashed arrows. The search graph 40 shown in Figure 4 refers to the automation system 10 shown in Figure 1, which is part of a packaging machine.

[0086] A first layer 41 of the search graph 40 contains pre-processing steps. With respect to an image of a packaging unit 2 transmitted by the sensor device 4, this can, as already indicated above, be, for example, an adjustment of an image size 411 (resize), as well as an increase 412 or decrease 413 of an image saturation value. A second layer 42 and a third layer 43 are arranged between a model input 44, which includes the input vector of the Kl model, and a model output 45 (the output). These each contain various operators required to execute the task to be solved, for example, the classification task of the automation system in Figure 1, such as matrix multiplication 421, 431, matrix multiplication with bias 422, 432, and identity operators 423, 433.

[0087] Not shown in Figure 4, the search graph 40 can include additional operators and layers, for example, a layer containing post-processing steps. A concrete path through the search graph 40 results in a Kl pipeline, which, according to the compatibility model, represents a possible Kl pipeline candidate compatible with the target system.

[0088] To ensure that potential AI pipeline candidates are suitable AI pipeline candidates, the AI ​​models of the potential AI pipeline candidates are trained using a task-specific training dataset and then evaluated on a validation dataset previously separated from the training dataset.

[0089] In the example of Figure 1, a training data set for a visual inspection can, for example, comprise images of packaging 2, with an annotation (labels) that identify the imaged packaging 2 in the sense of a ground truth as belonging to one of the categories OK and not OK.

[0090] The suitable AI models themselves are determined using an AI algorithm, for example, a graphical neural network model. The creation of AI model candidates, which are then considered in the AI ​​pipeline candidates, can be done using libraries. When generating the AI ​​pipeline candidates, not only various AI models are considered, but also the (dataset-specific) pre- and post-processing steps.

[0091] While in principle any AI model and / or AI pipeline could be generated as candidates, the generation can be limited to a one-shot neural architecture search (see, for example, Zichao Guo et al.: Single Path One-Shot Neural Architecture Search with Uniform Sampling, in https: / / arxiv.org / abs / 1904.00420). For this purpose, a search space in the form of a directed acyclic graph (DAG) can be selected, which contains only operators compatible with the target system. Then, in each training iteration, which corresponds to a specific adaptation of the model based on the training data, paths in the DAG are sampled, which are then optimized using backpropagation with the aid of classical cost functions. Some of the blocks selected during sampling can appear in multiple candidates, so that their optimization implicitly influences the optimization of other candidates.As a result, optimized parameters or weights of the model are shared, resulting in what is known as weight sharing. A task-specific dataset or a combination of these is used for training. After this initial training, the entire optimized search space, in the form of a DAG, in which each path represents a specific neural network, represents the set of candidates compatible with the target system.

[0092] Figure 5 shows an example of a generated Kl pipeline candidate 50, which was found using the search graph 40 shown in Figure 4 and a graphical neural network. A pre-processing layer 51 contains the operators for adjusting the image size 511 (resize) and increasing the image saturation value 512. The model input 54 corresponds to the input vector of the Kl model, as described in connection with Figure 4. The Kl model 52 itself contains the first operator 521 of matrix multiplication with bias and the second operator 522 of matrix multiplication without bias. Furthermore, the Kl pipeline candidate 22 includes a model output 55. Not shown here is a possible post-processing layer that would directly follow the model output 55.

[0093] With the AI ​​pipeline candidate generation, a number of different AI pipelines are identified that could serve as solution candidates, with all potential model candidates already having been comprehensively trained. The compatibility model ensures that only AI pipeline candidates are generated that are fundamentally capable of running on the selected target system.

[0094] In a fourth step S4, a Kl-pipeline is now selected from the set of suitable Kl-pipeline candidates for use in the control program of the PLC 20, which is described below with reference to Figure 6.

[0095] The AI ​​models of the AI ​​pipeline candidates are trained using additional, particularly large, task-specific training datasets. Training is carried out using suitable feedback loops and algorithms, as is well known in the technical field of AI, using, for example, residuals and cost functions. By training the AI ​​models, their generalization capability can be improved. Their latency remains unchanged, since only the model's weights are changed, not the model architecture.

[0096] Each AI pipeline candidate 50, represented as a cross in diagram 60 of Figure 6, exhibits a latency, i.e., the resulting computing time required for the execution of the AI ​​pipeline on the target system under the given hardware and software requirements. The latency is plotted on the abscissa (x) in diagram 60. As already described above, this latency can be calculated as the sum of the runtimes of the individual operators used in the AI ​​pipeline using the latency model.

[0097] Each of the AI ​​pipeline candidates 50 also exhibits a generalization capability, which is plotted on the ordinate (y) in diagram 60. Generally speaking, generalization capability can be understood as the ability of an AI model to make accurate (i.e., as close as possible to the ground truth) predictions on previously unseen data. Therefore, if an AI pipeline candidate 50 is trained on training data, this model should also function as flawlessly as possible in an application on newly generated, previously unseen data by having learned fundamental concepts that can be generalized to new data and reflect the underlying relationships between features (in the example, packaging units 2) and labels (in the example, OK or not OK).With respect to the diagram, the generalization capability can also be interpreted as the accuracy of the predictions made by the respective Kl pipeline candidate 50 regarding whether packaging units 2 are OK or not. Corresponding metrics for generalization capability are known in the art.

[0098] The horizontal and vertical position of the individual Kl pipeline candidates in diagram 60 reflects their performance in terms of their latency (the further to the left in the diagram, the better, as the latency is lower) and their generalization ability (the further up in the diagram, the better, as the generalization ability is greater).

[0099] To evaluate the latency of the individual Kl pipeline candidates, a latency requirement resulting from the real-time boundary conditions of the automation system 10, represented in diagram 60 by a value 65 and a corresponding dashed vertical line 66, is compared with the expected latencies contained in the latency model of all considered Kl pipeline candidates.

[0100] The latency requirement 65 for the selected AI pipeline 22, in terms of the maximum time available for generating a prediction, corresponds in the example of Figure 1 to the maximum permissible decision time AT2 of 150 ms, reduced by a time period for other processes that must also run during this time period, such as status checks of the discharge station 5, control of the conveyor belt 1, and the like. In this example, 50 ms of computing time is assumed for these tasks, so the latency requirement for the AI ​​pipeline 22 is 100 ms (150 ms - 50 ms = 100 ms).

[0101] All Kl pipeline candidates that have a higher latency than the latency requirement 65, i.e., those shown in diagram 60 to the right of the dashed vertical line 66, are discarded in the fourth step S4 and are not considered for use in the control program of the PLC 20. From the remaining Kl pipeline candidates, so-called Pareto-optimal candidates can first be sought. In this context, a Pareto-optimal candidate is defined as one whose generalization capability and latency properties are not surpassed by any other candidate in terms of a simultaneous improvement in generalization capability and latency, thus improving both quality characteristics simultaneously.

[0102] The set of Pareto-optimal candidates is arranged along the Pareto front 61 shown in diagram 60. From this set of Kl-pipeline candidates, the Kl-pipeline 22 that exhibits the highest generalization capability with a latency lower than the latency requirement 65 is then selected for use in the control program of the PLC 20. In diagram 60, this Kl-pipeline candidate is marked by the circled cross 63.

[0103] The selection of the AI ​​pipeline 22 from the set of AI pipeline candidates in the fourth step S4 can alternatively also be performed by a user. This can be done, for example, by providing the user with certain metrics and details of the AI ​​pipeline candidates on an HMI during the execution of the method, such as the latencies of the AI ​​pipeline candidates, a metric for their generalization capability, the type of AI model used for each AI pipeline candidate, its energy efficiency, and / or the software libraries used. The user then has the opportunity to manually set certain weightings in the selection algorithm via the HMI interface or to select an AI pipeline candidate themselves.

[0104] In target system configurations where no AI pipeline model meeting the latency requirement is found during candidate selection in the fourth step S4, the hardware and / or software specifications of the target system are adapted in a fifth step S5. A suggestion for this adaptation can be made to a user based on a database containing various hardware and / or software specifications with corresponding performance data.

[0105] For example, if it is determined in the fourth step S4 that the Kl pipeline candidates created in the third step S3 are very complex and their execution is therefore too slow on a given CPU, the fifth step S5 can switch to a computer with a faster processor and faster graphics card as hardware, and to a more powerful software library.

[0106] Therefore, in this case, as indicated in Figure 2, after making the changes to the hardware and / or software configuration, the method is run through again in a loop starting from the second step S2 by detecting the changed hardware and software configuration in the second step S2, creating a set of suitable Kl pipeline models on the basis of the compatibility model in the third step S3, and selecting a Kl pipeline for use in the control program of the PLC 20 from the set of suitable Kl pipeline candidates in the fourth step S4.

[0107] Furthermore, the AI ​​pipeline 22 selected in the fourth step S4 can be retrained in a sixth step S6 using application-specific data sets to improve its generalization capability. Here, too, the latency remains unchanged. The reason for this step is that the AI ​​pipeline 22 selected in the fourth step S4 may not have been sampled often enough during training during the joint optimization, so there is still optimization potential with regard to the generalization capability of the AI ​​pipeline 22. In a seventh step S7, a source code of the control program of the PLC 20 of the automation system 10 is created using the AI ​​pipeline 22 selected in the fourth step S4. In this process, the AI ​​pipeline 22 is exported from the AI ​​training environment into target system-compatible PLC source code and saved. The PLC source code is integrated into the PLC environment of the PLC 20.

[0108] The PLC source code is generated in such a way that operators for the KL pipeline execution, such as preprocessing steps, are mapped to software-specific library functions, such as TwinCAT library functions. This represents a step whose feasibility is guaranteed by the compatibility model. The PLC 20's control program, with its deterministic cyclic execution, thus ensures that calculation results are available at the expected time.

[0109] The PLC code generated in the seventh step S7 is executed on the PLC 20 during operation of the automation system 10 so that the PLC can reliably predict the task to be solved using the Kl pipeline 22, in the example the classification of the packaging units 2, within the specified maximum latency.

[0110] The presented method is also capable of optimizing the pre- and post-processing steps. The compatibility model allows such optimized operators to be mapped to specific software functions, which then allows the PLC code to be generated automatically in the seventh step, S7.

[0111] Furthermore, as already explained above, using the latency model allows for details of the execution environment (such as an ONNX (Open Neural Network Exchange) runtime and its configuration, a TwinCAT version, an operating system, etc.) to be taken into account, since the PLC code generated in the seventh step (S7) can be provided to the user with a specific software configuration, which can ultimately be delivered as an image for the target system. This also implicitly enables the optimization of an execution environment, including software specifications, for a given model.

[0112] By using the latency model, the real-time capability of the generated control program is always guaranteed. The compatibility model, which takes into account both the hardware and software specifications of the target system, further ensures operability. Operability is therefore inherently guaranteed, and a situation in which the source code of the PLC control program created in the fifth step S5 cannot be executed in the selected configuration of the target system due to hardware incompatibility can be ruled out. The invention is not limited to the described and illustrated embodiments. Rather, it also encompasses all expert developments within the scope of the invention defined by the patent claims. In addition to the described and illustrated embodiments, further embodiments are conceivable, which may include further modifications and combinations of features.

[0113] Reference symbol

[0114] AT1 First period

[0115] AT2 Decision time

[0116] 1 conveyor belt

[0117] 2 packaging units

[0118] 3 Arrow

[0119] 4 Sensor device

[0120] 5 Discharge station

[0121] 10 Automation system

[0122] 20 Programmable Logic Controller (PLC)

[0123] 21 Interface

[0124] 22 Kl pipeline

[0125] 23 Pre-processing

[0126] 24 class model

[0127] 25 Post-Processing

[0128] 30 Neural Network

[0129] 31 input layers;

[0130] 311 first bias

[0131] 32 hidden layers;

[0132] 321 second bias

[0133] 33 Output Layer

[0134] 40 search graph

[0135] 41 Pre-processing

[0136] 42 Decision Block 1

[0137] 43 Decision Block 2

[0138] 44 Model input

[0139] 45 model edition

[0140] 50 Kl pipeline candidate

[0141] 51 pre-processing layers

[0142] 52 Kl-Model

[0143] 54 Model input

[0144] 55 model edition

[0145] 60 diagram

[0146] 61 Pareto Front

[0147] 65 Latency requirement

[0148] 67 Dashed vertical line First step Second step Third step Fourth step Fifth step Sixth step Seventh step

Claims

Claims 1. A method for generating a control program of a programmable logic controller (20) in an automation system (10) based on a Kl pipeline containing at least one Kl model with optional preprocessing of input data and / or postprocessing of output data, comprising: providing a latency model for predicting a computing time for an execution of a Kl pipeline based on hardware configurations and software configurations and a compatibility model for mapping Kl pipeline functions to software configurations; Detecting a hardware configuration and a software configuration of the programmable logic controller (20); creating a set of AI pipeline candidates (50) based on the compatibility model; Selecting a Kl-pipeline (22) from the set of Kl-pipeline candidates (50) by evaluating a performance of the Kl-pipeline candidates after training the Kl-pipeline candidates, taking into account a prediction of a computing time based on the latency model; and Creating a source code of the control program with the selected Kl pipeline (22) for execution on the programmable logic controller (20) in the automation system (10).

2. The method according to claim 1, wherein the latency model determines a latency in the form of a numerical value resulting from a sum of the runtimes of the executed Kl pipeline functions.

3. The method of claim 1 or 2, wherein the compatibility model creates a data structure in which Kl pipeline functions are mapped to equivalent functions in the software configuration.

4. The method according to one of claims 1 to 3, wherein the set of Kl pipeline candidates (50) is created using at least one application-specific training data set.

5. The method according to claim 4, wherein a one-shot neural architecture search method is used to create and train the set of Kl pipeline candidates (50).

6. The method according to one of claims 1 to 5, wherein a generalization capability of the Kl pipeline is taken into account when selecting the Kl pipeline (22) from the set of Kl pipeline candidates (50).

7. The method according to claim 6, wherein the Pareto optimality of the Kl pipeline (22) is taken into account for selecting the Kl pipeline (22) from the set of Kl pipeline candidates (50).

8. The method according to any one of claims 1 to 7, wherein, when selecting the KL pipeline (22), retraining is carried out with a training data set in order to improve the generalization capability.

9. The method according to one of claims 1 to 8, wherein, when selecting the Kl pipeline (22) from the set of Kl pipeline candidates (50), if no pipeline candidate meets a predetermined latency requirement, an adaptation of the hardware configuration and / or the software configuration is carried out.

10. The method according to one of claims 1 to 9, wherein Kl models for the Kl pipeline candidates (50) are selected from the group of neural networks, statistical models, support vector machines, decision trees, and / or random forests.

11. A method for operating an automation system (10), comprising the steps: Generation of a control program of a programmable logic controller (20) of the automation system using the method according to one of claims 1 to 10, Uploading the source code of the control program to the programmable logic controller (20), Running the control program to perform an automation task.

12. Automation system (10) for executing an automation task, which is operated with the method according to claim 11.

Images