Method for generating a control program of a programmable logic controller and automation system

Abstract

To generate a control program of a programmable logic controller in an automation system based on an AI pipeline that contains at least one AI model with optional preprocessing of input data and / or postprocessing of output data, the following are carried out: providing a latency model for predicting a computing time for an execution of an AI pipeline based on hardware configurations and software configurations and a compatibility model for mapping AI pipeline functions to software configurations; detecting a hardware configuration and a software configuration of the programmable logic controller; creating a set of AI pipeline candidates based on the compatibility model;Selecting an AI pipeline from the set of AI pipeline candidates by evaluating the performance of the AI ​​pipeline candidates after training the AI ​​pipeline candidates, taking into account a prediction of a computing time based on the latency model; and creating a source code of the control program with the selected AI pipeline for execution on the programmable logic controller in the automation system.

Description

[0001] The present invention relates to a method for generating a control program for a programmable logic controller (PLC) in an automation system. The invention further relates to a method for operating an automation system and to an automation system itself.

[0002] Machines or systems within an automation system are often controlled using programmable logic controllers (PLCs). The PLC can be an external device or a software PLC. To control or regulate the actuators and sensors of the machine or system, the PLC typically uses a communication interface in the form of a fieldbus system. The actuators and sensors of the machine or system can then be networked with each other via the fieldbus system.

[0003] By reading the measurement data from the sensors and / or the current actual data from the actuators connected to the inputs of the programmable logic controller (PLC), the PLC receives information about the status of the machine or system. The actuators are connected to the outputs of the PLC and enable the control of the machine or system. For dynamic control of the actuators, the PLC generates output data for the actuators based on the actual data and / or the measurement data from the sensors. This data can consist of individual values ​​or groups of values. Actuators can also be controlled based on predefined parameters, such as motion profiles.

[0004] To ensure the desired operating mode of the machine or system, the PLC's control task defines which output data, generated by the PLC based on corresponding input data, is supplied to the actuators. The PLC typically processes the data cyclically, comprising three steps: providing current input data (e.g., actual data from the actuators and / or measurement data from the sensors), processing the input data into output data, and outputting the output data for motion control (e.g., target positions, etc.).

[0005] Industrial PLCs represent deterministic real-time systems. Determinism here refers to the requirement that an event or calculation must be completed within a precisely defined time interval. These time intervals are defined by the specific applications of the respective automation system. For programmable logic controllers, hard real-time capability is generally required, meaning that deadlines must be strictly adhered to and never exceeded. Additionally, a valid result for the actuators must be available at the time of the deadline. This is particularly relevant if exceeding the deadline could lead to personal injury or property damage, for example, if a robot arm is not braked in time.

[0006] In some automation system applications, algorithms based on artificial intelligence (AI), also known as AI models, are executed on the corresponding PLCs. Here, the term AI model primarily refers to neural networks, but it should also encompass statistical models such as linear models, support vector machines, decision trees, and random forests. AI can improve the efficiency of PLCs, particularly in the processing of sensor data, for example in image processing, or even enable certain functions in the first place.

[0007] From the perspective of the PLC's real-time capability mentioned above, it is necessary to integrate the AI ​​models used into the PLC's runtime environment to meet the requirements for both latency and determinism. Therefore, a guarantee of the operability of a chosen AI model within the context of the given deterministic environment and prior to its implementation is desirable, but not known in the current state of the art.

[0008] It is an object of the invention to provide an alternative or improved method for generating a control program for a programmable logic controller (PLC) in an automation system or for operating an automation system, as well as a corresponding automation system.

[0009] The object of the invention is achieved by the independent claims. Advantageous further developments, additional features and / or benefits of the invention will become apparent from the dependent claims and the following description.

[0010] It should be noted that all features mentioned in connection with the disclosed method can also be configurations of the disclosed automation system, and vice versa.

[0011] To generate a control program for a programmable logic controller (PLC) in an automation system based on an AI pipeline containing at least one AI model with optional preprocessing of input data and / or postprocessing of output data, the following steps are performed: Providing a latency model to predict the computation time for executing an AI pipeline based on hardware and software configurations, and a compatibility model to map AI pipeline functions to software configurations; creating a set of AI pipeline candidates based on the compatibility model; and capturing the hardware and software configurations of the programmable logic controller.Selecting an AI pipeline from the set of AI pipeline candidates by evaluating the performance of the AI ​​pipeline candidates after training them, taking into account a prediction of the computation time based on the latency model; creating source code for the control program with the selected AI pipeline for execution on the programmable logic controller in the automation system.

[0012] When integrating AI models or AI pipelines into the control system, the model's latency must be considered during model training for a given hardware specification. Furthermore, it is necessary to integrate the AI ​​model into the PLC's runtime environment to meet both latency and determinism requirements.

[0013] The latency model can determine a latency in the form of a numerical value, which results from the sum of the runtimes of the executed AI pipeline functions. AI pipeline functions are typically operators or sequences of operators.

[0014] The latency model is capable of predicting the latency of AI pipelines. In predicting execution time, the latency model considers not only the hardware configuration but also the software configuration of the target system.

[0015] The compatibility model provides a data structure in which AI pipeline functions are mapped to equivalent functions in the software configuration. This model ensures that no AI pipeline candidates are generated that are not executable on a specific target system.

[0016] When selecting and training AI model candidates, it is not usually verified whether the AI ​​model is executable on the target system. While hardware specifications are considered, this alone is insufficient to guarantee actual operability. With the compatibility model, which considers both the hardware and software configuration of the target system, operability is inherently guaranteed.

[0017] The guaranteed operability also enables the automation of inference code generation. Since pre- and / or post-processing steps are also optimized with the AI ​​pipeline, the AI ​​model is augmented.

[0018] The creation of the set of AI pipeline candidates can be done using at least one application-specific training dataset.

[0019] This involves considering combinatorial possibilities of operator sequences, using only operators that are compatible with the target system according to the compatibility model. This results in a search graph with various possible paths.

[0020] The compatibility of the AI ​​pipeline candidates is already ensured at this stage. During generation, not only are different AI models considered, but also preparatory and / or post-processing steps. Here, too, the compatibility model ensures the executability of the operators under consideration.

[0021] A one-shot neural architecture search method can be used to create the set of AI pipeline candidates.

[0022] When selecting the AI ​​pipeline, the generalizability of the AI ​​pipeline can be taken into account from the set of AI pipeline candidates.

[0023] A Pareto optimality method can be used to select the AI ​​pipeline from the set of AI pipeline candidates.

[0024] The selected AI pipeline can be retrained with a training dataset to improve its generalization capability.

[0025] If, when selecting the AI ​​pipeline from the set of AI pipeline candidates, no pipeline candidate meets a predefined latency requirement, an adjustment of the hardware configuration and / or the software configuration can be performed.

[0026] The AI ​​models for the AI ​​pipeline candidates can be selected from the group of neural networks, statistical models, support vector machines, decision trees, and / or random forests.

[0027] To operate an automation system, the source code of the control program is uploaded to the programmable logic controller (PLC). The control program can then be executed to perform an automation task. An automation system can thus be operated to perform an automation task.

[0028] The invention is explained in more detail below with reference to exemplary embodiments and the accompanying schematic and not to-scale drawings.

[0029] The figures (Fig.) in the drawings are merely examples and show: Fig. 1 schematically an embodiment of an automation system; Fig. 2 schematically an embodiment of a method for generating a control program of a programmable logic controller of the automation system; Fig. 3. Schematic representation of a neural network model; Fig. 4 schematically illustrates a search graph for an AI pipeline candidate; Fig. 5 schematically represents an AI pipeline candidate; Fig. Figure 6 schematically shows the performance of various AI pipeline candidates.

[0030] The following definitions will be used in the following: AI pipeline: Consists of an AI model and optional pre-processing of the input data or optional post-processing of the output data. Training: Optimization of an AI model, consisting of a set of adjustable parameters, which, using a suitable method, determines the parameter combinations that minimize a loss function for a given dataset. This involves considering differentiable AI models (usually artificial neural networks) that are optimized on supervised training data (features and labels) using gradient descent methods (specifically backpropagation). Latency: The time delay between requesting a prediction to a model and receiving the prediction. This includes, among other things, the computation of the model itself, but also any overhead due to communication with the execution environment, memory accesses, and communication between individual execution units. Real-time capability: The requirement that an event or calculation completes within a fixed time interval. Operability: The ability of an AI model or AI pipeline to be executed on a target system, which is defined by the mapping of all operators of the AI ​​model or AI pipeline to a semantically equivalent function on the target system. Compatibility: If an AI model or AI pipeline is operable on a target system, then the AI ​​model or AI pipeline and the target system are compatible with each other. Generalization capability: The ability of an AI model or AI pipeline to make accurate (i.e., as close as possible to the underlying truth) predictions on previously unseen data. If an AI model is trained on training data, it should function as flawlessly as possible later in an application on newly generated, previously unseen data by having learned fundamental concepts that can be generalized to new data and that reflect the underlying relationships between features and labels.

[0031] The following schematically describes the structure and operation of an automation system with a programmable logic controller (PLC), a method for generating a control program for a PLC in the automation system, and a method for operating the automation system. Corresponding reference symbols are used for corresponding features.

[0032] Fig. Figure 1 shows an exemplary automation system 10, which in the configuration shown is part of a packaging machine. The automation system 10 comprises a conveyor belt 1, on which packaging units 2 are transported in the direction of arrow 3, and a programmable logic controller (PLC) 20, which controls processes within the automation system 10 that will be described below. A control program is installed on the PLC 20 for this purpose. The PLC 20 is technically described by a hardware configuration and a software configuration.

[0033] The hardware configuration includes components such as the motherboard, microprocessors (CPU), memory modules (RAM), graphics processing units (GPU), and so on. The software configuration of the PLC 20 includes information such as the type and version of the operating system used, the type and version of the control program, for example, the TwinCAT automation software, specifications of required and / or available software libraries for performing various tasks of the PLC 20, and other software-specific features. For the purposes of this disclosure, in the context of creating a control program for the PLC 20, the hardware and software specifications together are also referred to as the target system.

[0034] The automation system 10 further comprises a sensor device 4, which may include, for example, a camera and / or a scale, and a rejection station 5. The sensor device 4 and the rejection station 5 are parts of a device for inspecting the packaging units 2. The inspection verifies whether each individual packaging unit 2 meets certain predefined quality requirements.

[0035] For the purposes of this description, it is assumed that each of the packaging units 2 is classified binary by inspection, meaning it is assigned to one of the categories OK or not OK, based on criteria such as visually detectable damage, required dimensions, presence or absence of labels, a prescribed weight, etc. If a packaging unit 2 is found to be not OK, it is rejected from the system at the reject station 5, for example, removed from conveyor belt 1, while the packaging units 2 that fall into the OK category remain on conveyor belt 1 and are processed further in a manner not shown. The binary classification serves here only as a simplified example.The classification could also be significantly more complex and, for example, have a large number of different quality levels or include a hierarchical structure with main classes and subclasses.

[0036] The packaging machine processes two packaging units 2 at a time within a specific initial time period ΔT1, which is in Fig. The distance between the packaging units 2 is indicated by 1. The first time period ΔT1 can be, for example, 200 ms. A maximum permissible decision time period ΔT2, in which a decision must be made as to whether the individual packaging unit 2 is OK or not OK, has a value that is generally somewhat shorter than the first time period ΔT1 and depends, among other things, on the physical distance between the sensor device 4, for example, the camera, and the rejection station 5. The maximum permissible decision time period ΔT2 can be, for example, 150 ms. After the maximum permissible decision time period ΔT2 has elapsed, the result of the classification of the individual packaging unit 2 detected by the camera 4 must be stored in the PLC 20 so that the PLC 20 can supply the rejection station 5 with corresponding control commands.

[0037] As mentioned at the outset, the PLC 20 is a deterministic real-time system. The maximum permissible decision time ΔT2 therefore defines a time interval that must be strictly adhered to and never exceeded, as otherwise downstream processes of the continuously or quasi-continuously operating automation system 10 would be impaired. In connection with the in Fig. In the example shown, a process for classifying the packaging unit 2, which can reliably provide the result within the maximum permissible decision time ΔT2, is also referred to as real-time capable.

[0038] The sensor device 4 is connected to the PLC 20 via a suitable interface 21, which is configured to interpret the sensor signals or data transmitted by the sensor device 4.

[0039] In this context, interpreting means performing the classification described above based on the sensor data.

[0040] For this purpose, artificial intelligence (AI) methods are used, which are implemented as algorithms or software modules or programs on the PLC 20 and are also referred to as AI models within the scope of this disclosure. With the help of the AI ​​models, the PLC 20 is able to perform tasks such as the classification of packaging units 2 in such a way that insignificant, random changes in the boundary conditions, such as – in the case of an optical inspection – a change in the ambient light conditions or a variable orientation of a packaging unit 2 on the conveyor belt 1, do not affect the result of the classification. The result output by the AI ​​model is hereinafter also referred to as a prediction.

[0041] In the state of the art, a large number of AI models are known, each of which can be based on different mathematical models (for example, on artificial neural networks) and place different demands on the hardware and / or software configuration of the PLC 20, or deliver different performance with certain hardware and / or software configurations, i.e., provide predictions at different speeds and / or with different accuracy.

[0042] Within its function as part of the PLC 20, the AI ​​model is also integrated into a so-called AI pipeline, which is located in Fig. 1 is designated as AI pipeline 22. The AI ​​pipeline comprises the AI ​​model used as well as any necessary pre-processing steps of input data and / or post-processing steps of output data. In the embodiment in Fig. 1 The AI ​​pipeline 22 includes the AI ​​model 24 as well as a preprocessing step 23 of the sensor data provided by the sensor device 4 and a postprocessing step 25 of the output issued by the AI ​​model 24.

[0043] The primary AI model used is neural networks. However, the AI ​​model can also be another (classical) statistical model, such as a linear model, support vector machine, decision tree, random forest, etc.

[0044] If the classification of the packaging units 2 is based on image analysis, the AI ​​model 24 can be implemented as an image classifier. The preprocessing step 23 can then, for example, involve preprocessing the input images provided by the sensor device 4 in a specific way. In the image classifier example, a preprocessing step could be, for example, resize an input image or adjust its saturation. The postprocessing step 24 can then, for example, involve converting the predictions of the AI ​​model 24 into another representation. In the image classifier example, a postprocessing step could be, for example, selecting the class with the highest probability or determining the mode of the probability distribution after the output from the AI ​​model.

[0045] With regard to the automation system 10 and the creation of a control program for the PLC 20, a challenge lies in selecting the AI ​​pipeline 22 for the respective task in the automation system 10, in this example for the classification of the packaging units 2, and for a given target system of the PLC 20, and integrating it into the runtime environment of the PLC 20 in such a way that an operability guarantee as well as a guarantee of the real-time capability of the control program can be given in advance.

[0046] Operability guarantee here means that, even before the integration of an AI pipeline into a PLC, it is guaranteed that the selected AI model can be executed by the target system of the PLC.

[0047] The aim is to provide a control program optimized for the respective automation system or the specific PLC.

[0048] The following disclosed method for generating a control program for the PLC 20, which is schematically represented in Fig. 2 is at least partially executed as a computer-implemented procedure on the automation system 10 itself or on an external computer.

[0049] In the first step, S1, a latency model and a compatibility model are provided. The latency model is understood as a function that maps the specific AI pipeline and the target system to a numerical value, the latency. Latency refers to the computation time required to execute the AI ​​pipeline on the target system under the given hardware and software conditions. The latency model can be used to predict the latency of AI pipelines that are not specifically known at the time the latency model is created.

[0050] The AI ​​models of AI pipelines can be described as sequences of operators. Operators are fundamental mathematical operations, such as convolution operators or activation functions, which are known to be used for specifying AI models. These operators form the basic building blocks of AI models and are used in a wide variety of them. By selecting from the multitude of operators or combining them in different ways, previously unknown AI models can be assembled.

[0051] For example, an artificial neural network 30, as a special form of an AI model, can be used, as in Fig. 3. The structure is shown schematically. In the representation of the Fig. 3. An input vector x is defined by the values ​​x1, x i , x nis represented in the neural network 30, which is processed schematically from left to right.

[0052] The in Fig. The neural network shown in Figure 30 is an MLP (Multilayer Perceptron)-type neural network, consisting of three layers: an input layer (Figure 31), a hidden layer (Figure 32), and an output layer (Figure 33). The hidden layer (Figure 33) can itself be composed of multiple layers. Each neuron in one layer is connected to every neuron in the next layer, with each connection having a numerical value called a weight. The MLP network has a feedforward architecture, meaning that information flows through the network in only one direction, without feedback loops or cyclic connections.

[0053] MLP networks are used particularly in predictive applications. Alternatively, RBF (radial basis function)-type neural networks can also be used, for example.

[0054] Each of the three layers, the Input Layer 31, the Hidden Layer 32, and the Output Layer 33, forms an operator, where the Input Layer 31 and the Hidden Layer in the example each represent a matrix multiplication with a first bias 311 and a second bias 321, and the Output Layer 33 a matrix multiplication without bias.

[0055] The runtime of calculating such operators in an artificial neural network 30 can vary significantly from target system to target system, due to the specific hardware and software configuration. Therefore, the runtime of the respective operators is determined based on the hardware and software configuration identified in the second step S2.

[0056] The latency of the AI ​​pipeline is then determined as the sum of the runtimes of the AI ​​model's operators, plus the runtimes of pre- and post-processing. It is also possible to describe pre- and post-processing using operators and then determine their runtimes.

[0057] In the specific example of the Fig. 1 will be the latency of the in Fig. The neural network 30 shown in Figure 3, which together with post- and pre-processing defines the AI ​​pipeline 22, is determined on the given target system (ZS) by the following calculation rule: Latency(AI-Pipeline,ZS)=Runtime(MatMultBias,ZS)+Runtime(MatMultBias,ZS)+Runtime(MatMult,ZS)+Runtime(Pre-Processing,ZS)+Runtime(Post-Processing,ZS)

[0058] Latency (AI pipeline, ZS) should be interpreted here as the latency of the AI ​​pipeline under consideration on the target system ZS. The abbreviation MatMult stands for the matrix multiplication operation described above, and MatMultBias for matrix multiplication with bias.

[0059] In order to determine the latency on a target system, the runtimes of the operators, for example MatMultBias, are first measured on a plurality of different target systems and stored in a database.

[0060] The hardware configuration of a target system can include, for example, a specific type of personal computer (PC), a specific microprocessor used in the PC, such as an Intel Core i7 CPU, and a specific memory configuration, such as 16 GB of DDR4 RAM. The software configuration of a target system includes the version of the operating system running on the PLC 20, such as a Microsoft Windows version, the version of the software running on the PLC 20, such as a basic TwinCAT system, and the version of specific software libraries executed by the PLC 20, such as a TwinCAT Vision version. Furthermore, such a description can also include the parameterization of a specific execution mode, such as with or without multithreading, use of an AI accelerator, etc.

[0061] Table 1 below shows, row by row, the runtimes for the operators MatMultBias and MatMult for selected hardware and software configurations. The hardware configuration in this example is PLC type C6030 with a Core i7-11850HE CPU. For the software configuration TwinCAT version 3.2.7, the runtime for the operator MatMultBias is 0.003s (row 1), and for MatMult, the runtime is 0.002s (row 2). For the software configuration TwinCAT version 3.2.1, the runtime for the operator MatMultBias is 0.006s (row 3). Table 1 Zielsystem Operator Laufzeit [in s] SPS-Typ CPU TwinCATVersion ... C6030 Core i7-11850HE 3.2.7 ... MatMultBias 0,003 C6030 Core i7-11850HE 3.2.7 ... MatMult 0, 002 C6030 Core i7-11850HE 3.2.1 ... MatMultBias 0,006

[0062] Using such a database, the latencies of AI models can now be calculated according to the individual operators of the corresponding AI models, taking the target system into account. For operators that are not included in the database for the given target system, runtimes can be interpolated based on similarities of certain attributes.

[0063] The runtimes of pre-processing and post-processing, if they can be described with operators, can also be determined in this way.

[0064] Furthermore, a latency model can also take into account contextualizations of operator sequences, such as a target system combining the sequential execution of two MatMult operations and thus executing them particularly efficiently. This allows for the modeling of more complex latency relationships compared to a simple summation of operators (see Equation 1).

[0065] Instead of a database-driven approach, where a previously measured runtime is determined for each operator or operator sequence of the AI ​​model and then accumulated to calculate the latency, (learned) AI models (such as neural networks) can also be used to determine the latency model. Here, too, the AI ​​models can represent more complex latency dependencies between sequences of operators, so that, as in the previously mentioned example, two MatMult operators can be combined by an AI accelerator, and the resulting latency on the chosen target system is then lower than the sum of the two individual latencies.

[0066] The latency model can thus predict the expected latency for each AI pipeline for carrying out an automation task in the automation system, taking into account the software and hardware configuration of the target system.

[0067] As mentioned above, the first step, S1, also provides a compatibility model of the automation system. This is advantageous because well-known AI models often use operators that are not necessarily supported on every target system. For example, the latest AI operators, such as matrix transposition, are only supported in software platforms like newer automation software versions after a certain time lag. Therefore, an AI model using such an operator would simply not be executable on certain target systems, regardless of factors like the operator's runtime or the latency of the AI ​​pipeline. This also applies to pre-processing and post-processing when these are described using operators.

[0068] To prevent the selection of an AI model or pipeline that places unfulfillable demands on the target system, the compatibility model ensures that no candidates are generated that are not executable on the specific target system. The compatibility model is a rule-based model that maps functions of the AI ​​model or pipeline to functionally equivalent functions of the target system. Here, too, the contextualization of operator sequences can potentially be considered, for example, that a single function of an AI pipeline may not have an equivalent to a function of the target system, but may have one when sequentially combined with another operator.

[0069] This can be understood as a type of database, the structure of which is exemplified in Table 2 below. Table 2 shows, row by row, the corresponding equivalent function for the operators MatMultBias, MatMult, and imageResize (changing an image size) for the hardware and software configuration of the target system. In this example, the hardware configuration of the target systems is PLC type C6030 with a Core i7-11850HE CPU. The software configuration of the target systems is at least TwinCAT version 3.2 or at least TwinCAT version 3.2.1. The operator MatMultBias is assigned the target function TwinCAT MatMultBias (row 1), the operator MatMult the target function TwinCAT MatMult (row 2), and the operator imageResize the target function TwinCAT imageResize (row 3). Table 2 Zielsystem Operator SPS-Typ CPU TwinCATVersion ... Zielfunktion C6030 Core i7-11850HE >= 3.2 ... MatMultBias TwinCATMatMultBias C6030 Core i7-11850HE >= 3.2 ... MatMult TwinCATMatMult C6030 Core i7-11850HE >= 3.2.1 ... ImageResize TwinCATVisionResize

[0070] If an operator on a target system does not have a corresponding target function, the corresponding field may, for example, remain empty or contain the entry "not found".

[0071] The compatibility model, a rule-based model that checks whether an AI model or AI pipeline and a description of the target system are executable or operable on the target system, can determine the compatibility between the AI ​​model and the target system. For example, the compatibility model verifies whether the software libraries in the target system, such as TwinCAT Vision, support specific operators for real-time execution.

[0072] Instead of a database-based determination, (learned) AI models (such as neural networks or large language models) can also be used in determining the compatibility model.

[0073] In a second step S2 of the procedure for generating a control program for the PLC 20, which is schematically in Fig. As shown in Figure 2, the specific hardware and / or software configuration of the automation system 10 is then recorded. This can be done automatically via software or through a user interface (Human-Machine Interface, HMI) and interaction with a user.

[0074] Based on the recorded hardware or software configuration, a set of suitable AI pipeline models is then created in a third step (S3) using the compatibility model. For this purpose, a large number of different AI pipelines are generated that could be considered as solution candidates. Various combinatorial possibilities of operator sequences are created, using only those operators that are compatible with the recorded hardware or software configuration of the target system according to the compatibility model.

[0075] This process can, as in Fig. Figure 4 shows schematically, and the search graph 40 is represented, whose operators are connected via various path options shown as dashed arrows. The one in Fig. The search graph 40 shown in Figure 4 refers to the one in Fig. 1 Automation system 10 shown, which is part of a packaging machine.

[0076] In the first layer 41 of the search graph 40, pre-processing steps are included. With regard to an image of a packaging unit 2 transmitted by the sensor device 4, this can, as already indicated above, involve, for example, adjusting the image size 411 (resize), as well as increasing 412 or decreasing 413 an image saturation value. Between a model input 44, which comprises the input vector of the AI ​​model, and a model output 45 (the output), a second layer 42 and a third layer 43 are arranged. These each contain various operators that are necessary for executing the task to be solved, for example, the classification task of the automation system in Fig. 1, are needed, for example matrix multiplication 421, 431, matrix multiplication with bias 422, 432, and identity operator 423, 433.

[0077] In Fig. While not shown in Figure 4, the search graph can include 40 additional operators and layers, such as a layer containing post-processing steps. A specific path through the search graph 40 yields an AI pipeline that, according to the compatibility model, represents a potential AI pipeline candidate compatible with the target system.

[0078] To ensure that potential AI pipeline candidates are suitable AI pipeline candidates, the AI ​​models of the potential AI pipeline candidates are trained using a task-specific training dataset and then evaluated on a validation dataset previously separated from the training dataset.

[0079] In the example of the Fig. 1. A training dataset for a visual inspection can, for example, include images of packaging 2, with an annotation (labels) that identifies the depicted packaging 2 as belonging to one of the categories OK and not OK in the sense of a ground truth.

[0080] The appropriate AI models themselves are determined using an AI algorithm, such as a graphical neural network model. The creation of AI model candidates, which are then considered within the AI ​​pipeline candidates, can be accomplished using libraries. The generation of AI pipeline candidates takes into account not only various AI models but also the (dataset-specific) pre- and post-processing steps.

[0081] While in principle any AI model and / or AI pipeline could be generated as candidates, the generation can be restricted to a one-shot neural architecture search (see, for example, Zichao Guo et al.: Single Path One-Shot Neural Architecture Search with Uniform Sampling, in https: / / arxiv.org / abs / 1904.00420). For this purpose, a search space in the form of a directed acyclic graph (DAG) can be selected, containing only operators compatible with the target system. Then, in a training iteration, which corresponds to a concrete adaptation of the model based on the training data, paths in the DAG are sampled and subsequently optimized using backpropagation with the aid of classical cost functions. Some of the blocks selected during sampling may appear in multiple candidates, so their optimization implicitly influences the optimization of other candidates.Consequently, optimized parameters or weights of the model are shared, resulting in so-called weight sharing. A task-specific dataset or a combination thereof is used for training. After this initial training, the entire optimized search space, in the form of the DAG (Differential Analysis Group), where each path represents a specific neural network, constitutes the set of candidates compatible with the target system.

[0082] Fig. Figure 5 shows an example of a generated AI pipeline candidate 50, which was created using the in Fig. The search graphs shown in Figure 40 were found using a graphical neural network. A pre-processing layer 51 contains the operators Resize 511 and Increase Image Saturation 512. The model input 54 corresponds to the information provided in the context of Fig. Figure 4 describes the input vector of the AI ​​model. The AI ​​model 52 itself contains the first operator 521 of matrix multiplication with bias and the second operator 522 of matrix multiplication without bias. Furthermore, the AI ​​pipeline candidate 22 includes a model output 55. Not shown here is a possible post-processing layer that would directly follow the model output 55.

[0083] The AI ​​pipeline candidate generation process identifies a multitude of different AI pipelines that are potential solution candidates, having already undergone comprehensive training of all potential model candidates. The compatibility model ensures that only AI pipeline candidates that are fundamentally compatible with the selected target system are generated.

[0084] In a fourth step S4, an AI pipeline is selected from the set of suitable AI pipeline candidates for use in the control program of the PLC 20, which is described below with reference to Fig. 6 is described.

[0085] The AI ​​models of the AI ​​pipeline candidates are trained using further, particularly extensive, task-specific training datasets. This training is carried out using suitable feedback loops and algorithms, employing, for example, residuals and cost functions, as is well known in the technical field of AI. Training the AI ​​models improves their generalization ability. Their latency remains unchanged, as only the model's weights are modified, not the model architecture itself.

[0086] Each AI pipeline candidate 50, in the diagram 60 of the Fig. The cross represented by number 6 indicates latency, which is the resulting computation time required to execute the AI ​​pipeline on the target system under the given hardware and software conditions. The latency is plotted on the x-axis in diagram 60. As described above, this latency can be calculated as the sum of the runtimes of the individual operators used in the AI ​​pipeline, using the latency model.

[0087] Each of the 50 AI pipeline candidates also exhibits a generalization capability, which is plotted on the ordinate (y) in diagram 60. In general terms, generalization capability can be understood as the ability of an AI model to make accurate (i.e., as close as possible to the underlying truth) predictions on previously unseen data. Thus, if an AI pipeline candidate 50 is trained on training data, this model should also function as flawlessly as possible in an application on newly generated, previously unseen data by having learned fundamental concepts that can be generalized to new data and reflect the underlying relationships between attributes (in the example, packaging units 2) and labels (in the example, OK or not OK).The generalization capability, as depicted in the diagram, can also be interpreted as the accuracy of the predictions made by each AI pipeline candidate 50 regarding whether packaging units 2 are OK or not OK. Corresponding metrics for generalization capability are known in the prior art.

[0088] The horizontal and vertical position of the individual AI pipeline candidates in diagram 60 thus reflects their performance with regard to their latency (the further left in the diagram, the better, due to lower latency) and their generalization ability (the further up in the diagram, the better, due to greater generalization ability).

[0089] To evaluate the latency of the individual AI pipeline candidates, a latency requirement resulting from the real-time boundary conditions of the automation system 10, represented in diagram 60 by a value 65 and a corresponding dashed vertical line 66, is compared with the expected latencies of all considered AI pipeline candidates contained in the latency model.

[0090] The latency requirement 65 for the selected AI pipeline 22, in the sense of a maximum period available for generating a prediction, corresponds in the example to the Fig. The maximum permissible decision time ΔT2 of 150ms is reduced by the time required for other processes that must also run within this period, such as condition checks of the discharge station 5, control of conveyor belt 1, and similar tasks. In this example, 50ms of computation time is assumed for these tasks, resulting in a latency requirement of 100ms for the AI ​​pipeline 22 (150ms - 50ms = 100ms).

[0091] All AI pipeline candidates with a latency higher than the latency requirement 65, i.e., those shown to the right of the dashed vertical line 66 in diagram 60, are rejected in the fourth step S4 and not considered for use in the PLC 20 control program. From the remaining AI pipeline candidates, so-called Pareto-optimal candidates can first be sought. In this context, a candidate is considered Pareto-optimal if, with regard to its generalization capability and latency properties, no other candidate surpasses it in terms of simultaneously improving both generalization capability and latency; that is, a candidate that would improve both quality characteristics at the same time.

[0092] The set of Pareto-optimal candidates is arranged along the Pareto front 61 shown in diagram 60. From this set of AI pipeline candidates, the one that exhibits the highest generalization capability with a latency smaller than the latency requirement 65 is typically selected as the AI ​​pipeline 22 for use in the PLC control program 20. In diagram 60, this AI pipeline candidate is marked by the circled cross 63.

[0093] The selection of AI pipeline 22 from the set of AI pipeline candidates in step S4 can alternatively be performed by a user. This can be done, for example, by displaying specific metrics and details of the AI ​​pipeline candidates to the user on a human-machine interface (HMI) during the procedure. These metrics might include the latency of the AI ​​pipeline candidates, a metric for their generalization ability, the type of AI model used for each candidate, its energy efficiency, and / or the software libraries used. The user then has the opportunity to manually set specific weights in the selection algorithm via the HMI interface or to select an AI pipeline candidate themselves.

[0094] In configurations of the target system where, in the fourth step S4 of the candidate selection, no AI pipeline model is found that meets the latency requirement, the hardware and / or software specification of the target system is adjusted in a fifth step S5. A user can be presented with a suggestion for this adjustment based on a database containing various hardware and / or software specifications with corresponding performance data.

[0095] For example, if in the fourth step S4 it is determined that the AI ​​pipeline candidates created in the third step S3 are very complex and their execution is therefore too slow on a given CPU, in the fifth step S5, for example, a switch can be made to a computer with a faster processor and faster graphics card as hardware, and to a more powerful software library.

[0096] Therefore, in this case, as in Fig. As indicated in step 2, after making changes to the hardware and / or software configuration, the procedure is repeated in a loop starting from the second step S2, by recording the changed hardware and software configuration in the second step S2, creating a set of suitable AI pipeline models based on the compatibility model in the third step S3, and selecting an AI pipeline from the set of suitable AI pipeline candidates for use in the control program of the PLC 20 in the fourth step S4.

[0097] Furthermore, the AI ​​pipeline 22 selected in step four (S4) can be retrained in step six (S6) using application-specific datasets to improve its generalization capability. The latency remains unchanged in this step as well. The rationale for this step is that the AI ​​pipeline 22 selected in step four (S4) may not have been sampled often enough during the joint optimization process, thus offering further optimization potential regarding the generalization capability of the AI ​​pipeline 22.

[0098] In a seventh step (S7), source code for the control program of PLC 20 of automation system 10 is created, using the AI ​​pipeline 22 selected in the fourth step (S4). The AI ​​pipeline 22 is exported from the AI ​​training environment into target system-compatible PLC source code and saved. This PLC source code is then integrated into the PLC environment of PLC 20.

[0099] The PLC source code is generated in such a way that operators for AI pipeline execution, such as preprocessing steps, are mapped to software-specific library functions, such as TwinCAT library functions. This is a step whose feasibility is guaranteed by the compatibility model. The PLC 20 control program, with its deterministic cyclic execution, thus ensures that calculation results are available at the expected time.

[0100] The PLC code generated in step seven S7 is executed on the PLC 20 during the operation of the automation system 10, so that it can reliably predict the task to be solved using the AI ​​pipeline 22, in this example the classification of the packaging units 2, within the specified maximum latency.

[0101] The presented method is capable of optimizing both pre- and post-processing steps. Furthermore, the compatibility model allows these optimized operators to be mapped to specific software functions, enabling the PLC code to be automatically generated in the seventh step, S7.

[0102] Furthermore, as explained above, the latency model allows for the consideration of details of the execution environment (such as an ONNX runtime (Open Neural Network Exchange) and its configuration, a TwinCAT version, an operating system, etc.), since the PLC code generated in step seven, S7, can be provided to the user with a concrete software configuration that can ultimately be delivered as an image for the target system. This also implicitly enables the optimization of an execution environment, including software specifications, for a given model.

[0103] The use of the latency model guarantees the real-time capability of the generated control program at all times. Furthermore, the compatibility model, which considers both the hardware and software specifications of the target system, ensures operability. Operability is therefore inherently guaranteed, and a situation in which the PLC control program source code generated in step S5 is not executable in the selected target system configuration due to hardware incompatibility can be ruled out.

[0104] The invention is not limited to the described and illustrated embodiments. Rather, it also encompasses all further developments by skilled craftsmen within the scope of the invention defined by the claims. In addition to the described and illustrated embodiments, further embodiments are conceivable, which may include further modifications and combinations of features. Reference sign ΔT1 First time period ΔT2 Decision time 1 conveyor belt 2 packaging units 3 Arrow 4 Sensor device 5 Exit station 10 Automation systems 20 Programmable Logic Controllers (PLCs) 21 Interface 22 AI pipeline 23 Pre-Processing 24 AI model 25 Post-Processing 30 Neural network 31 Input Layer; 311 first bias 32 Hidden Layers; 321 second bias 33 Output Layer 40 Search graph 41 Pre-Processing 42 Decision block 1 43 Decision block 2 44 Model input 45 model edition 50 AI pipeline candidates 51 Pre-Processing Layer 52 AI model 54 Model input 55 model edition 60 Diagram 61 Pareto front 65 Latency requirement 67 Dashed vertical line S1 First Step S2 Second Step S3 Third Step S4 Fourth Step S5 Fifth Step S6 Sixth Step S7 Seventh Step QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Zichao Guo et. al: Single Path One-Shot Neural Architecture Search with Uniform Sampling, in https: / / arxiv.org / abs / 1904.00420

[0081]

Claims

[1] Method for generating a control program of a programmable logic controller (20) in an automation system (10) based on an AI pipeline which includes at least one AI model with optional preprocessing of input data and / or postprocessing of output data, comprising: Providing a latency model to predict computation time for the execution of an AI pipeline based on hardware and software configurations, and a compatibility model to map AI pipeline functions to software configurations; Capturing a hardware configuration and a software configuration of the programmable logic controller (20); creating a set of AI pipeline candidates (50) based on the compatibility model; Selecting an AI pipeline (22) from the set of AI pipeline candidates (50) by evaluating the performance of the AI ​​pipeline candidates after training them, taking into account a prediction of computation time based on the latency model; and Creating source code of the control program with the selected AI pipeline (22) for execution on the programmable logic controller (20) in the automation system (10). [2] Method according to claim 1, wherein the latency model determines a latency in the form of a numerical value resulting from a sum of the runtimes of the executed AI pipeline functions. [3] Method according to claim 1 or 2, wherein the compatibility model creates a data structure in which AI pipeline functions are mapped to equivalent functions in the software configuration. [4] Method according to any one of claims 1 to 3, wherein the creation of the set of AI pipeline candidates (50) is carried out using at least one application-specific training data set. [5] Method according to claim 4, wherein a one-shot neural architecture search method is used to create and train the set of AI pipeline candidates (50). [6] Method according to any one of claims 1 to 5, wherein when selecting the AI ​​pipeline (22) from the set of AI pipeline candidates (50) the generalizability of the AI ​​pipeline is taken into account. [7] Method according to claim 6, wherein the Pareto optimality of the AI ​​pipeline (22) is taken into account when selecting the AI ​​pipeline (22) from the set of AI pipeline candidates (50). [8] Method according to any one of claims 1 to 7, wherein when selecting the AI ​​pipeline (22) retraining with a training data set to improve the generalization capability. [9] Method according to any one of claims 1 to 8, wherein, when selecting the AI ​​pipeline (22) from the set of AI pipeline candidates (50), no pipeline candidate meets a predetermined latency requirement, an adjustment of the hardware configuration and / or the software configuration is performed. [10] Method according to any one of claims 1 to 9, wherein AI models for the AI ​​pipeline candidates (50) are selected from the group consisting of neural networks, statistical models, support vector machines, decision trees, and / or random forests. [11] Method for operating an automation system (10), comprising the steps: Generation of a control program of a programmable logic controller (20) of the automation system using the method according to one of claims 1 to 10, Uploading the source code of the control program to the programmable logic controller (20), Executing the control program to perform an automation task. [12] Automation system (10) for performing an automation task, which is operated using the method according to claim 11.

Images