Method of generating control programs for programmable logic controllers and automation systems

Abstract

To generate control programs for programmable logic controllers (PLCs) in an automation system based on an AI pipeline—the AI ​​pipeline comprising at least one AI model with optional input data preprocessing and / or output data postprocessing—the following steps are performed: providing a latency model for predicting the computation time for executing the AI ​​pipeline based on hardware and software configurations, and a compatibility model for mapping AI pipeline functionality to the software configuration; detecting the hardware and software configurations of the PLC; generating a set of AI pipeline candidate schemes based on the compatibility model; selecting an AI pipeline from the set of AI pipeline candidate schemes by evaluating the performance of the AI ​​pipeline candidate schemes after training them, taking into account the predicted computation time based on the latency model; and using the selected AI pipeline to generate source code for a control program to be executed on the PLC in the automation system.

Description

Technical Field

[0001] This invention relates to a method for generating control programs for a programmable logic controller (PLC) in an automation system. The invention further relates to a method for operating an automation system, and to the automation system itself.

[0002] This patent application claims priority to German Patent Application 10 2024 110 997.1, the disclosure of which is hereby incorporated by reference. Background Technology

[0003] Machines or systems in automation systems often rely on programmable logic controllers (PLCs). A PLC can be an external device or a software PLC. To control or regulate the actuators and sensors of a machine or system, a PLC typically uses a communication interface in the form of a fieldbus system. The actuators and sensors of the machine or system can then be networked together via the fieldbus system.

[0004] The PLC receives information about the state of the machine or system by reading measurement data from sensors and / or current actual data from the actuators connected to the programmable logic controller (PLC). The actuators are connected to the outputs of the PLC and allow control of the machine or system. For dynamic control of the actuators, the PLC generates the actuator output data based on the actual data and / or measurement data from the sensors, where the aforementioned data can be individual values ​​or groups of values. The actuators can also be controlled based on specifications, for example, using movement curves.

[0005] To provide the desired operating mode of a machine or system, the PLC's control task depends on the corresponding input data to determine which output data generated by the PLC is fed to the actuator. The PLC typically processes data in a three-step cycle: providing current input data (e.g., actual data from the actuator and / or measurement data from sensors); processing the input data into output data; and outputting output data for motion control (e.g., target position, etc.).

[0006] Industrial PLCs are deterministic real-time systems. Here, determinism refers to the requirement that an event or calculation must be completed within a defined time interval. These time intervals are defined by the specific application of the corresponding automation system. Programmable logic controllers typically require hard real-time capability, meaning that deadline requirements must be met consistently and never exceeded. Furthermore, valid results must be provided to actuators before the deadline. This is especially important if exceeding the deadline could lead to personal injury or property damage, such as if a robotic arm fails to brake in time.

[0007] In some applications of automation systems, artificial intelligence (AI) based algorithms (also known as AI models) are also executed on the corresponding PLC. In this context, the term AI model primarily refers to neural networks, but the term is also intended to include statistical models such as linear models, support vector machines, decision trees, or random forests. AI can improve the efficiency of PLCs, especially in the field of sensor data processing, such as in image processing, or in the first place, enable certain functions.

[0008] Given the aforementioned aspects of the real-time capabilities of PLCs, it is necessary to integrate the AI ​​model used into the PLC's runtime environment to meet both latency and deterministic requirements. Therefore, it is desirable to guarantee the operability of the selected AI model within the context of a given deterministic environment and prior to its implementation, but this is unknown in the prior art. Summary of the Invention

[0009] The purpose of this invention is to specify a control program for generating a programmable logic controller (PLC) in an automation system or an alternative or improved method for operating an automation system, as well as a corresponding automation system.

[0010] The objective of this invention is addressed by the independent claims. Advantageous further developments, additional features, and / or advantages of this invention arise from the dependent claims and the following description.

[0011] It should be noted that all features mentioned in the context of the disclosed methods can also be implementations of the disclosed automated systems, and vice versa.

[0012] To generate a control program for a programmable logic controller (PLC) in an automation system based on an AI pipeline containing at least one AI model with optional input data preprocessing and / or output data postprocessing, the following steps are performed: providing a latency model for predicting the computation time for executing the AI ​​pipeline based on hardware and software configurations, and a compatibility model for mapping AI pipeline functionality to the software configurations; generating a set of AI pipeline candidate schemes based on the compatibility model; detecting the hardware and software configurations of the PLC; selecting an AI pipeline from the set of AI pipeline candidate schemes by evaluating the performance of the AI ​​pipeline candidate schemes after training them, taking into account the predicted computation time based on the latency model; and generating the source code of the control program using the selected AI pipeline for execution on the PLC in the automation system.

[0013] When integrating an AI model or AI pipeline into the control system, the latency of the model with given hardware specifications must be considered during model training. Furthermore, it is necessary to integrate the AI ​​model into the PLC's runtime environment to meet both latency and deterministic requirements.

[0014] The latency model can determine the latency in numerical form, which is generated by the sum of the runtime of the executed AI pipeline functions. AI pipeline functions are typically operators or sequences of operators.

[0015] Latency models can predict the latency of AI pipelines. These models consider not only the hardware configuration but also the software configuration of the target system to predict execution time.

[0016] A compatibility model can be used to provide a data structure in which AI pipeline functionality is assigned equivalent functionality in a software configuration. The compatibility model ensures that no AI pipeline candidate solutions are generated that cannot be executed on a specific target system.

[0017] When selecting and training AI model candidates, it's common practice not to check whether the AI ​​model can run on the target system. Hardware specifications are considered, but this is insufficient to verify practical operability. By utilizing a compatibility model that takes into account the hardware and software configurations of the target system, operability is inherently guaranteed.

[0018] Guaranteed operability also allows for the automation of inference code generation. The AI ​​pipeline further enhances the AI ​​model by optimizing preprocessing and / or post-processing steps.

[0019] A set of AI pipeline candidate schemes can be generated using at least one application-specific training dataset.

[0020] Combinatorial possibilities of operator sequences are considered, with only operators compatible with the target system according to a compatibility model used. This results in a search graph with a wide variety of possible paths.

[0021] In this step, the compatibility of AI pipeline candidate solutions has been ensured. During the generation process, not only different AI models are considered, but also preprocessing and / or post-processing steps. In this context, the compatibility model also ensures the executability of the considered operators.

[0022] A one-time neural architecture search method can be used to generate a set of AI pipeline candidate solutions.

[0023] When selecting an AI pipeline, consider its generalization capabilities from a set of AI pipeline candidates.

[0024] The Pareto optimization method can be used to select an AI pipeline from a set of AI pipeline candidates.

[0025] The selected AI pipeline can be retrained using the training dataset to improve its generalization ability.

[0026] When selecting an AI pipeline from a set of AI pipeline candidates, if no pipeline candidate meets the specified latency requirements, the hardware and / or software configurations can be adjusted.

[0027] The AI ​​model for the AI ​​pipeline candidate solution can be selected from a group of neural networks, statistical models, support vector machines, decision trees, and / or random forests.

[0028] To operate the automated system, the source code of the control program is further loaded onto the programmable logic controller (PLC). The executable control program then performs the automated tasks. Thus, the automated system for performing automated tasks becomes operational. Attached Figure Description

[0029] The invention is described in more detail below with reference to the accompanying illustrative and non-to-scale drawings.

[0030] The accompanying diagram is merely an example. Figure 1 It shows:

[0031] Figure 1 This is a schematic diagram of an implementation plan for an automation system;

[0032] Figure 2 This is a schematic diagram of an implementation scheme for a method of generating control programs for programmable logic controllers in an automation system;

[0033] Figure 3 This is a schematic diagram of a neural network model;

[0034] Figure 4 This is a schematic diagram of the search graph for candidate solutions in an AI pipeline.

[0035] Figure 5 This is a schematic diagram of candidate solutions for an AI pipeline;

[0036] Figure 6 This is a schematic diagram illustrating the performance of various AI pipeline candidate solutions. Detailed Implementation

[0037] The following definitions are used below:

[0038] AI pipeline: consists of an AI model and optional preprocessing of input data or optional postprocessing of output data.

[0039] Training: Optimization of an AI model consisting of a set of adjustable parameters, which uses appropriate methods to determine the parameter combination to minimize the loss function of a given dataset. Differentiable AI models (typically artificial neural networks) are considered, which are optimized on supervised training data (features and labels) using gradient descent methods (particularly backpropagation).

[0040] Latency: The time delay between requesting a prediction from the model and receiving the prediction. This includes the model's own computation, but also any overhead incurred due to communication with the execution environment, memory access, and communication between individual execution units.

[0041] Real-time capability: The requirement that an event or computation be completed within a defined time interval.

[0042] Operability: The ability to execute an AI model or AI pipeline on a target system, defined by the ability of all operators of the AI ​​model or AI pipeline to map semantically equivalent functions on the target system.

[0043] Compatibility: If the AI ​​model or AI pipeline is operable on the target system, then the AI ​​model or AI pipeline and the target system are compatible with each other.

[0044] Generalization ability: The ability of an AI model or AI pipeline to make accurate predictions (i.e., predictions that are as close as possible to the underlying truth) on previously unseen data. If an AI model is trained on training data, it should learn basic concepts that can be applied to new, previously unseen data in later applications with as few errors as possible, reflecting the underlying relationships between features and labels.

[0045] The following text schematically describes the structure and function of an automation system with a programmable logic controller (PLC), as well as a method for generating a control program for the PLC in the automation system and a method for operating the automation system. Corresponding reference numerals are used for corresponding features.

[0046] Figure 1 An exemplary automated system 10, which is part of a packaging machine in the illustrated configuration, is shown. The automated system 10 includes a conveyor belt 1 and a programmable logic controller (PLC) 20, on which packaging units 2 are conveyed in the direction of arrow 3. The PLC controls the processes within the automated system 10, as will be described below. For this purpose, the control program is mounted on the PLC 20. The PLC 20 is technically described by its hardware and software configurations.

[0047] The hardware configuration includes components such as a motherboard, microprocessor (CPU), memory module (RAM), and graphics processing unit (GPU). The software configuration of the PLC 20 includes information such as the type and version of the operating system used, the type and version of the control program (e.g., TwinCAT automation software), the specifications of the required and / or available software libraries for performing various tasks of the PLC 20, and other software-specific characteristics. In the context of this disclosure, the joint hardware and software specifications, in conjunction with the creation of the PLC 20's control program, are also referred to as the target system.

[0048] The automation system 10 also includes a sensor device 4 and an outlet station 5, the sensor device including, for example, a camera and / or a scale. The sensor device 4 and the outlet station 5 are part of a device for inspecting the packaging units 2. This inspection checks whether the individual packaging units 2 produced meet certain specified quality requirements.

[0049] For the purposes of this description, it is assumed that each of the packaging units 2 is classified into binary categories upon inspection, i.e., assigned to one of the acceptable or unacceptable categories based on criteria such as visually detectable damage, required dimensions, presence or absence of labels, and specified weight. If a packaging unit 2 is found to be unacceptable, it is sorted out of the system at the discharge station 5, for example, removed from conveyor belt 1, while packaging units 2 classified as acceptable remain on conveyor belt 1 and are further processed in a manner not shown. The binary classification method is only a simplified example here. Classification can also be much more complex and, for example, involve a large number of different quality levels or a hierarchical structure with main categories and subcategories.

[0050] The packaging machine processes two packaging units 2 within a specific first time period ΔT1, where ΔT1 is... Figure 1 The distance between packaging units 2 is represented in the figure. The first time period ΔT1 can be, for example, 200 ms. The maximum permissible decision time ΔT2 has a value that is generally slightly smaller than the first time period ΔT1, and depends particularly on the structural distance between the sensor device 4 (e.g., a camera) and the discharge station 5. Within the maximum permissible decision time ΔT2, a decision must be made regarding whether a single packaging unit 2 is qualified. The maximum permissible decision time ΔT2 can be, for example, 150 ms. After the maximum permissible decision time ΔT2 has elapsed, the classification result of the individual packaging unit 2 detected by the camera 4 must be safely available in the PLC 20 so that the PLC 20 can supply corresponding control commands to the discharge station 5.

[0051] As mentioned earlier, PLC 20 is a deterministic real-time system. Therefore, the maximum permissible decision time ΔT2 must be consistently adhered to and must never be exceeded; otherwise, downstream processes of the continuously or quasi-continuously operating automation system 10 will be compromised. In conjunction with... Figure 1The example shown illustrates a process for classifying packaging unit 2 that reliably provides results within the maximum permissible decision time ΔT2, also known as real-time capability.

[0052] Sensor device 4 is connected to PLC 20 via a suitable interface 21, which is used to interpret sensor signals or data transmitted by sensor device 4. In this context, interpretation means performing the above-mentioned classification based on sensor data.

[0053] For this purpose, an artificial intelligence (AI) approach is used, which is implemented as an algorithm, software module, or program on PLC 20 and is also referred to as an AI model in the context of this disclosure. With the assistance of the AI ​​model, PLC 20 is able to perform tasks such as classifying packaging unit 2, such that random changes in boundary conditions unrelated to the task—such as changes in ambient lighting conditions or variable orientation of packaging unit 2 on conveyor belt 1 in the case of optical inspection—do not affect the classification results. The result output by the AI ​​model (i.e., its output) is also referred to as a prediction.

[0054] In the prior art, a large number of AI models are known, each of which may be based on different mathematical models (e.g., artificial neural networks) and impose different requirements on the hardware and / or software configuration of the PLC 20, or provide different performance using certain hardware and / or software configurations, i.e., provide predictions at different speeds and / or with different accuracies.

[0055] Within the functionality of the PLC 20, the AI ​​model is also integrated as software into a so-called AI pipeline, which... Figure 1 This is labeled AI Pipeline 22. An AI pipeline includes the IA model used, and, depending on the situation, preprocessing steps for the input data and / or postprocessing steps for the output data. Figure 1 In the illustrated implementation, the AI ​​pipeline 22 includes an AI model 24, a preprocessing step 23 for sensor data provided by the sensor device 4, and a postprocessing step 25 for the output published by the AI ​​model 24.

[0056] Neural networks are primarily used as AI models. However, AI models can also be other (classical) statistical models, such as linear models, support vector machines, decision trees, random forests, etc.

[0057] If the classification of packaging unit 2 is based on image evaluation, then AI model 24 can be implemented as an image classifier. Preprocessing step 23 can then be, for example, preprocessing the input image provided by sensor device 4 in some way. In the example of an image classifier, the preprocessing step could be, for example, changing the image size of the input image (re-sizing the image) or adjusting the saturation of the input image. Postprocessing step 24 can then be, for example, transforming the predictions of AI model 24 into different representations. In the example of an image classifier, the postprocessing step could be, for example, selecting the class with the highest probability or determining the modality value of the probability distribution after the output from the AI ​​model.

[0058] Regarding the creation of the automation system 10 and the control program for the PLC 20, one challenge is to select the AI ​​pipeline 22 for the corresponding task in the automation system 10, in this example for the classification of the packaging unit 2, and for the given target system of the PLC 20, and integrate it into the runtime environment of the PLC 20 so that the operability guarantee and real-time capability guarantee of the control program can be given in advance.

[0059] Operability assurance means that even before the AI ​​pipeline is integrated into the PLC, the selected AI model is guaranteed to be executable by the PLC's target system.

[0060] The goal is to provide a control program optimized for a specific automation system or PLC.

[0061] The method for generating control programs for PLC 20 disclosed below (in...) Figure 2 (Illustrated in the diagram) is executed at least in part as a computer implementation on the automation system 10 itself or on an external computer.

[0062] In the first step S1, a latency model and a compatibility model are provided. The latency model is understood as a function that maps a specific AI pipeline and the target system to a numerical value (i.e., latency). Latency refers to the computation time required to execute an AI pipeline on the target system under given hardware and software conditions. The latency model can be used to predict the latency of the AI ​​pipeline, which is unknown at the time the latency model is generated.

[0063] AI models in an AI pipeline can be described as sequences of operators. Operators are fundamental mathematical operations, such as convolution operators or activation functions, whose specifications are known for use in AI models. Operators form the basic building blocks of AI models and are widely used in them. Previously unknown AI models can be assembled by selecting from a multitude of operators or combining them in different ways.

[0064] For example, as a special form of AI model, artificial neural networks 30 can be like Figure 3 The structure is illustrated in the diagram. Figure 3 In the representation, the input vector x, represented by the values ​​x1, xi, xn, is schematically processed from left to right in the neural network 30.

[0065] Figure 3 The neural network 30 shown is an MLP (Multilayer Perceptron) type neural network, which consists of three layers in a known manner: an input layer 31, a hidden layer 32, and an output layer 33. The hidden layer 33 can also consist of multiple layers. Each neuron in one layer is connected to every neuron in the next layer, with each connection having a numerical value called a weight. MLP networks have a feedforward architecture, where information flows unidirectionally within the network, without feedback loops or recurrent connections.

[0066] MLP networks are particularly useful for prediction applications. Alternatively, for example, RBF (Radial Basic Function) type neural networks can also be used.

[0067] Each of the input layer 31, hidden layer 32, and output layer 33 forms an operator, where in the example, the input layer 31 and the hidden layer each represent matrix multiplication with a first bias 311 and a second bias 321, and the output layer 33 represents matrix multiplication without bias.

[0068] Depending on the specific hardware and software configurations, the computation time of such operators in the artificial neural network 30 can vary significantly between different target systems. Therefore, in the second step S2, the computation time of the corresponding operator is determined based on the recorded hardware and software configurations.

[0069] The latency of the AI ​​pipeline is then determined as the sum of the runtime of the operators in the AI ​​model and the runtime of preprocessing and postprocessing. Preprocessing and postprocessing can also be described using operators, and their runtimes can then be determined.

[0070] exist Figure 1 In a specific example, Figure 3 The latency of the neural network 30 shown, together with post-processing and pre-processing, defines the AI ​​pipeline 22 and is determined on a given target system (ZS) by the following calculation rules:

[0071]

[0072] (Equation 1)

[0073] Latency (AI pipeline, ZS) is understood here as the latency of the AI ​​pipeline considered on the target system ZS. The abbreviation MatMult represents the matrix multiplication operation described above, and MatMultBias represents biased matrix multiplication.

[0074] To determine the latency on the target system, the runtime of the operator is first measured on several different target systems, such as MatMultBias, and the runtime is stored in a database.

[0075] The hardware configuration of the target system may include, for example, a specific type of personal computer (PC), a specific microprocessor used in the PC, such as an Intel Core i7 CPU, and a specific memory configuration, such as 16 GB of DDR4 memory. The software configuration of the target system includes the version of the operating system running on the PLC 20, such as a version of Microsoft Windows, the version of the software running on the PLC 20, such as the basic TwinCAT system, and versions of certain software libraries executed by the PLC 20, such as a version of TwinCAT Vision. Additionally, this description may include parameterization of specific execution modes, such as using or not using multithreading, using an AI accelerator, etc.

[0076] Table 1 below shows examples of the runtime for MatMultBias and the MatMult operator for the selected hardware and software configurations. The hardware configuration in the examples is a C6030 PLC with a Core i7-11850HE CPU. For the TwinCAT version 3.2.7 software configuration, the runtime for the MatMultBias operator is 0.003s (row 1), and the runtime for the MatMult operator is 0.002s (row 2). For the TwinCAT version 3.2.1 software configuration, the runtime for the MatMultBias operator is 0.006s (row 3).

[0077]

[0078] Table 1

[0079] With the help of a database generated in this way, the latency of an AI model can now be calculated based on individual operators of the associated AI model, while taking the target system into account. For operators not included in the database for a given target system, runtime can be interpolated based on the similarity of certain attributes.

[0080] If operators can be used to describe the process, the runtime of preprocessing and postprocessing can also be determined in this way.

[0081] Furthermore, the time delay model can also take into account the context of the operator sequence, which the target system can summarize by performing two MatMult operations consecutively, thus executing them particularly efficiently. This allows for the simulation of more complex time delay relationships compared to simple summation operators (see Equation 1).

[0082] Instead of database-based determination, where the runtime of previously measured operations is determined for each operator or sequence of operators in the AI ​​model, and then said runtime is accumulated into a latency, the (learned) AI model (such as a neural network) can also be used for latency model determination. Here, the AI ​​model can also map more complex latency dependencies between operator sequences; therefore, as in the example mentioned above, two MatMult operators can be combined by an AI accelerator, and the resulting latency on the selected target system is less than the sum of the two individual latencyes.

[0083] Therefore, taking into account the software and hardware configuration of the target system, the latency model can be used to predict the expected latency of each AI pipeline used to automate tasks in an automated system.

[0084] As mentioned above, the first step S1 also provides a compatibility model for the automation system. This is advantageous because known AI models often use operators that are not necessarily supported on every target system. For example, the latest AI operators (such as transpose matrices) are only supported in software platforms (such as newer versions of automation software) with a slight delay, meaning that AI models using such operators will not be able to execute on some target systems, regardless of factors such as operator runtime or AI pipeline latency. This also applies to preprocessing and postprocessing when describing with the help of operators.

[0085] To avoid selecting AI models or pipelines that impose unrealizable requirements on the target system in this sense, compatibility models ensure that candidate solutions for AI models or pipelines that cannot be executed on a specific target system are not generated. Compatibility models are rule-based models that map the functionality of an AI model or pipeline to functionally equivalent functions of the target system. Contextualization of operator sequences can also be considered here; for example, a single function of an AI pipeline may not be equivalent to a function of the target system, but may be equivalent when sequentially combined with another operator.

[0086] This can be understood as a type of database, the structure of which is shown in Table 2 below. Table 2 shows the corresponding equivalent functions of the operators MatMultBias, MatMult, and imageResize (changing image size) for the hardware and software configuration of the target system, row by row. The hardware configuration of the target system in the example is a C6030 PLC, with a Core i7-11850HE CPU. The software configuration of the target system is at least TwinCAT version 3.2 or TwinCAT version 3.2.1. The operator MatMultBias is assigned to TwinCAT MatMultBias (row 1) as its target function, the operator MatMult is assigned to TwinCAT MatMult (row 2) as its target function, and the operator imageResize is assigned to TwinCAT imageResize (row 3) as its target function.

[0087]

[0088] Table 2

[0089] If the operator does not have a corresponding target function on the target system, the corresponding field can, for example, remain empty or contain an entry "unavailable".

[0090] Compatibility models are rule-based models that examine whether the description of the AI ​​model or AI pipeline and the target system can be executed or operated on the target system, thereby determining the compatibility between the AI ​​model and the target system. For example, a compatibility model is used to check whether software libraries in the target system (such as TwinCAT Vision for real-time execution) support certain operators.

[0091] However, instead of database-based determination, (learned) AI models (such as neural networks or large language models) can also be used in compatibility model determination.

[0092] In Figure 2 In the second step S2 of the process for generating the control program for PLC 20, as illustrated schematically, the specific hardware or software configuration of the automation system 10 is then recorded. This can be done automatically via software or through a user interface (HMI) and interaction with the user.

[0093] Based on the recorded hardware and software configurations, a suitable set of AI pipeline models is generated in the third step, S3, using a compatibility model. For this purpose, a large number of different AI pipelines are generated, suitable as candidate solutions. Various combinations of operator sequences are generated, where, according to the compatibility model, only those operators compatible with the recorded hardware or software configuration of the target system are used.

[0094] like Figure 4 As illustrated in the diagram, the process can be represented as a search graph 40, in which operators are connected via various path options, as indicated by dashed arrows. Figure 4 The search diagram 40 shown refers to Figure 1 The automation system 10 shown is part of the packaging machine.

[0095] The first layer 41 of the search diagram 40 contains a preprocessing step. Regarding the image of the packaging unit 2 transmitted by the sensor device 4, this could, for example, be image resizing 411 (re-sizing), as indicated above, and increasing or decreasing the image saturation value 412 or 413. The second layer 42 and the third layer 43 are arranged between the model input 44 and the model output 45 (output), the model input including the input vector of the AI ​​model. Each of these contains various operators required to perform the task to be solved, for example, Figure 1 The classification tasks of automated systems include matrix multiplication 421, 431, biased matrix multiplication 422, 432, and identity operators 423, 433.

[0096] Figure 4 As not shown, search graph 40 may include additional operators and layers, such as layers containing post-processing steps. An AI pipeline is generated through specific paths in search graph 40, representing possible AI pipeline candidates compatible with the target system, according to a compatibility model.

[0097] To ensure that potential AI pipeline candidates are suitable AI pipeline candidates, AI models of potential AI pipeline candidates are trained using a task-specific training dataset, and then the AI ​​models are evaluated on a validation dataset previously separated from the training dataset.

[0098] exist Figure 1 In the example, the training dataset used for visual inspection may include, for example, images of package 2 with annotations (labels) indicating that package 2 is depicted as belonging to one of the qualified and unqualified categories in the sense of basic facts.

[0099] The appropriate AI model itself is determined using AI algorithms, such as graph neural network models. Libraries can be used to generate candidate AI models, which are then considered as part of the AI ​​pipeline candidate list. When generating AI pipeline candidate lists, not only different AI models are considered, but also (dataset-specific) preprocessing and post-processing steps are taken into account.

[0100] While any AI model and / or AI pipeline can theoretically be generated as candidate solutions, generation can be limited to a one-off neural architecture search (e.g., see Zichao Guo et al.: One-off Neural Architecture Search with Uniform Sampling, https: / / arxiv.org / ABS / 1904.00420). For this purpose, a search space in the form of a directed acyclic graph (DAG) can be chosen, containing only operators compatible with the target system. Then, in each training iteration corresponding to a specific adjustment of the model based on the training data, paths in the DAG are sampled and optimized using backpropagation with the aid of a classical cost function. Some blocks selected during sampling may appear in multiple candidate solutions, such that their optimization implicitly influences the optimization of other candidate solutions. As a result, the optimization parameters or weights of the model are shared, leading to so-called weight sharing. Task-specific datasets or combinations of such datasets are used for training. After the initial training is completed, the entire optimization search space in the form of a DAG (where each path represents a specific neural network) represents a set of candidate solutions compatible with the target system.

[0101] Figure 5 Showing the use Figure 4 The example shown is the search graph 40 and the generated AI pipeline candidate scheme 50 found by the graph neural network. The preprocessing layer 51 contains operators for image resizing 511 (re-sizing) and image saturation value increase 512. (Combined with...) Figure 4 The model input 54 corresponds to the input vector of the AI ​​model. The AI ​​model 52 itself contains a first operator 521 of biased matrix multiplication and a second operator 522 of unbiased matrix multiplication. Furthermore, the AI ​​pipeline candidate scheme 22 includes a model output 55. Possible post-processing layers, which will immediately follow the model output 55, are not shown here.

[0102] AI pipeline candidate generation identifies multiple different AI pipelines suitable as solution candidates, with comprehensive training of all potential model candidates completed. A compatibility model ensures that only AI pipeline candidates that can substantially run on the selected target system are generated.

[0103] In step S4, an AI pipeline is now selected from a set of suitable AI pipeline candidates used in the PLC 20 control program, which will be referred to below. Figure 6 Describe it.

[0104] AI models for AI pipeline candidates are trained using additional, particularly extensive task-specific training datasets. As is well known in the field of AI, appropriate feedback loops and algorithms are used for training, such as residual and cost features. Training the AI ​​model improves its generalization ability. Their latency remains unchanged because only the model weights are altered, not the model architecture.

[0105] Depend on Figure 6 The cross-plot 60 represents each AI pipeline candidate 50, including latency, which is the computation time required to execute the AI ​​pipeline on the target system under given hardware and software conditions. Latency is plotted on the x-axis of Figure 60. As described above, the latency can be calculated as the sum of the runtimes of the individual operators used in the AI ​​pipeline using a latency model.

[0106] The AI ​​pipeline candidate 50 also possesses generalization ability, which is plotted on the y-axis in Figure 60. Generally, generalization ability can be understood as the ability of an AI model to make accurate (i.e., as close as possible to the basic facts) predictions on previously unseen data. Therefore, if the AI ​​pipeline candidate 50 is trained on training data, the model should also operate as error-free as possible in applications to newly generated, previously unseen data by learning basic concepts that can be applied in a generalized manner to new data and reflect the fundamental relationship between features (in the example, packaging unit 2) and labels (in the example, pass or fail). Regarding the graph, generalization ability can also be interpreted as the accuracy of the predictions made by the corresponding AI pipeline candidate 50 regarding whether packaging unit 2 is pass or fail. The corresponding metric for generalization ability is known in the prior art.

[0107] Therefore, the horizontal and vertical positions of individual AI pipeline candidate solutions in Figure 60 reflect their performance in terms of waiting time (the further left in the figure, the lower the latency and the better the performance) and generalization ability (the higher in the figure, the higher the generalization ability).

[0108] To evaluate the latency of a single AI pipeline candidate, the latency requirement generated by the real-time constraints of the automated system 10, represented by value 65 and the corresponding vertical dashed line 66 in Figure 60, is compared with the expected latency of all considered AI pipeline candidates contained in the latency model.

[0109] exist Figure 1In the example, given the maximum time available to generate the prediction, the latency requirement 65 of the selected AI pipeline 22 corresponds to a maximum permissible decision time ΔT2 of 150 ms, minus the time periods of other processes that must also run during the said cycle, such as the status checks of unloading station 5, the control of conveyor belt 1, etc. In this example, a computation time of 50 ms is assumed for these tasks, therefore the latency requirement of AI pipeline 22 is 100 ms (150 ms - 50 ms = 100 ms).

[0110] All AI pipeline candidates with a delay higher than the delay requirement 65 (i.e., those shown to the right of the vertical dashed line 66 in Figure 60) are rejected in step S4 and are not considered for use in the control program of PLC 20. From the remaining AI pipeline candidates, a so-called Pareto optimal candidate can be sought first. In this context, a candidate is described as Pareto optimal if, in terms of its generalization ability and delay characteristics, no other candidate surpasses it in terms of simultaneous improvement of both generalization ability and delay; that is, if it will improve both quality characteristics simultaneously.

[0111] A set of Pareto optimal candidate schemes is arranged along the Pareto front 61 shown in Figure 60. Then, from this set of AI pipeline candidate schemes, the one with the highest generalization ability and a delay below the delay requirement 65 is typically selected as the AI ​​pipeline 22 for use in the control program of the PLC 20. In Figure 60, this AI pipeline candidate scheme is marked by a circled cross 63.

[0112] Alternatively, in step S4, the selection of the AI ​​pipeline 22 from a set of AI pipeline candidate solutions can also be performed by the user. This can be accomplished, for example, by presenting the user with certain metrics and details of the AI ​​pipeline candidate solutions on the HMI during program execution, such as the latency of the AI ​​pipeline candidate solutions, a metric for their generalization ability, the type of AI model used for each AI pipeline candidate solution, their energy efficiency, and / or the software libraries used. The user then has the opportunity to manually set certain weights in the selection algorithm via the HMI interface, or to select the AI ​​pipeline candidate solutions themselves.

[0113] During the candidate solution selection process, if no AI pipeline model configuration meeting the latency requirements is found in step S4, the hardware and / or software specifications of the target system are adjusted in step S5. Such adjustments can be suggested to the user based on a database storing various hardware and / or software specifications with corresponding performance data.

[0114] For example, if in step S4 it is determined that the AI ​​pipeline candidate scheme generated in step S3 is too complex and therefore too slow to run on a given CPU, then step S5 may involve switching to a computer with a faster processor and a faster graphics card as hardware, and switching to a more powerful software library.

[0115] Therefore, in this situation, such as Figure 2 As instructed, after making changes to the hardware and / or software configuration, the process is repeated in a loop starting from the second step S2 by recording the changed hardware and software configuration in the second step S2, generating a set of suitable AI pipeline models based on the compatibility model in the third step S3, and selecting an AI pipeline from the set of suitable AI pipeline candidates in the fourth step S4 for use in the control program of PLC 20.

[0116] Furthermore, the AI ​​pipeline 22 selected in step S4 can be retrained in step S6 using an application-specific dataset to improve its generalization ability. Here, latency remains unchanged. The reason for this step is that the AI ​​pipeline 22 selected in step S4 may not have been sampled frequently enough during training in the joint optimization phase, thus there is still potential for optimization in terms of the generalization ability of the AI ​​pipeline 22.

[0117] In step S7, the AI ​​pipeline 22 selected in step S4 is used to generate the source code for the control program of the PLC 20 of the automation system 10. During this process, the AI ​​pipeline 22 is exported from the AI ​​training environment to the target system-compatible PLC source code and saved. The PLC source code is then integrated into the PLC environment of the PLC 20.

[0118] The PLC source code is generated in such a way that operators for AI pipeline execution (such as preprocessing steps) are mapped to software-specific library functions, such as TwinCAT library functions, the feasibility of which is guaranteed by a compatibility model. Through its deterministic cyclic execution, the PLC 20's control program ensures that calculation results are available at the expected time.

[0119] During the operation of the automation system 10, the PLC code generated in step S7 is executed on PLC 20, enabling the PLC to reliably predict the task to be solved by the AI ​​pipeline 22 within a specified maximum delay, which in the example is the classification of packaging unit 2.

[0120] The proposed method can also optimize the preprocessing and postprocessing steps. The compatibility model allows these optimized operators to be mapped to specific software functions, and then allows for the automatic generation of PLC code in the seventh step S7.

[0121] Furthermore, as mentioned above, a delay model can be used to account for details of the execution environment (e.g., the ONNX runtime (Open Neural Network Exchange) and its configuration, TwinCAT version, operating system, etc.), because the PLC code generated in step S7 can be provided to users with specific software configurations, which can ultimately be delivered as an image of the target system. This also implicitly allows for optimization of the execution environment, including the software specifications of a given model.

[0122] The use of a time-delay model always guarantees the real-time capability of the generated control program. Simultaneously, a compatibility model considering the hardware and software specifications of the target system ensures operability. Therefore, operability is inherently guaranteed, and it eliminates the possibility that the source code of the PLC control program generated in step S5 may not be executable in the selected configuration of the target system due to hardware incompatibility.

[0123] This invention is not limited to the embodiments described and illustrated. Rather, it encompasses all technical developments within the scope of the invention as defined by the patent claims. Further embodiments are contemplated in addition to the described and illustrated embodiments, which may include further modifications and combinations of features.

[0124] List of reference numerals

[0125] ΔT1 First time period

[0126] ΔT2 decision time period

[0127] 1 Conveyor Belt

[0128] 2 Packaging Units

[0129] 3 arrows

[0130] 4 sensor devices

[0131] 5 Unloading Station

[0132] 10 Automation Systems

[0133] 20. Programmable Logic Controllers (PLCs)

[0134] 21 interface

[0135] 22AI pipeline

[0136] 23 Preprocessing

[0137] 24 AI models

[0138] 25 Post-processing

[0139] 30 Neural Networks

[0140] 31. Input layer;

[0141] 311 First Bias

[0142] 32 hidden layers;

[0143] 321 Second Bias

[0144] 33 Output Layer

[0145] 40 search graphs

[0146] 41 Preprocessing

[0147] 42 Decision Block 1

[0148] 43 Decision Block 2

[0149] 44 Model Inputs

[0150] 45 Model Output

[0151] 50 AI pipeline candidate solutions

[0152] 51 Preprocessing Layer

[0153] 52 AI models

[0154] 54 model inputs

[0155] 55 Model Output

[0156] 60 pictures

[0157] 61 Pareto Front

[0158] 65 Delay Request

[0159] 67 Vertical dashed line

[0160] S1 First Step

[0161] S2 Second Step

[0162] S3 Third Step

[0163] S4 Step 4

[0164] S5 Step 5

[0165] S6 Step 6

[0166] S7, the seventh step.

Claims

1. A method for generating control programs for a programmable logic controller (20) in an automated system (10) based on an AI pipeline, said AI pipeline containing at least one AI model with optional input data preprocessing and / or output data postprocessing, said method comprising: Provides a latency model for predicting the computation time for executing an AI pipeline based on hardware and software configurations, and a compatibility model for mapping AI pipeline functionality to software configurations. The hardware and software configurations of the programmable logic controller (20) are detected; a set of AI pipeline candidate schemes (50) are generated based on the compatibility model; By evaluating the performance of the AI ​​pipeline candidate schemes after training them, and considering the predicted computation time based on the latency model, an AI pipeline (22) is selected from the set of AI pipeline candidate schemes (50); and The source code of the control program is generated using the selected AI pipeline (22) for execution on the programmable logic controller (20) in the automation system (10).

2. The method of claim 1, wherein the latency model determines the latency in numerical form, and the latency is generated by the sum of the runtimes of the executed AI pipeline functions.

3. The method according to claim 1 or 2, wherein the compatibility model generates a data structure in which the AI ​​pipeline functionality is given an equivalent function in the software configuration.

4. The method according to any one of claims 1 to 3, wherein the set of AI pipeline candidates (50) is generated by means of at least one application-specific training dataset.

5. The method of claim 4, wherein the set of AI pipeline candidate schemes (50) is generated and trained using a one-time neural architecture search method.

6. The method according to any one of claims 1 to 5, wherein when selecting the AI ​​pipeline (22) from the set of AI pipeline candidate schemes (50), the generalization capability of the AI ​​pipeline is taken into consideration.

7. The method of claim 6, wherein when selecting the AI ​​pipeline (22) from the set of AI pipeline candidate schemes (50), the Pareto optimality of the AI ​​pipeline is taken into consideration.

8. The method according to any one of claims 1 to 7, wherein when the AI ​​pipeline (22) with the training dataset is selected, retraining is performed to improve the generalization ability.

9. The method according to any one of claims 1 to 8, wherein if no pipeline candidate meets the specified latency requirement when the AI ​​pipeline (22) is selected from the set of AI pipeline candidate schemes (50), the adjustment of the hardware configuration and / or the software configuration is performed.

10. The method according to any one of claims 1 to 9, wherein the AI ​​model used for the AI ​​pipeline candidate scheme (50) is selected from a group of neural networks, statistical models, support vector machines, decision trees and / or random forests.

11. A method for operating an automated system (10), comprising the following steps: The control program for the programmable logic controller (20) of the automation system is generated using the method according to any one of claims 1 to 10; The source code of the control program is loaded onto the programmable logic controller (20); The control program is executed to perform automated tasks.

12. An automation system (10) for performing automated tasks, which operates using the method according to claim 11.

Images