METHOD FOR INTEGRATING A FUNCTIONAL BLOCK WITH ARTIFICIAL INTELLIGENCE

Description

[0001] The present invention relates to a method for integrating at least one AI functional block comprising artificial intelligence into a controller for an industrial machine.

[0002] The use of so-called artificial intelligence, i.e., machine learning methods, neural networks, support vector machines (SVMs), and the like, in industrial processes is generally known. However, such use is not standard.

[0003] Therefore, the use of artificial intelligence in industrial control systems often faces significant hurdles. Many different architectures exist for such artificial intelligence, which can be implemented in different programming languages and with different interfaces. Due to these many variations, incompatibilities can easily arise when integrating and / or executing artificial intelligence in an industrial control system. The integration of artificial intelligence therefore typically requires considerable effort.

[0004] WO 2021 / 076093 A1 discloses a method according to the preamble of claim 1. WO 2020 / 040721 A1 describes an acceleration module for artificial intelligence. EP 3 671 571 A1 relates to a method for generating an AI model. None of the aforementioned documents relates to an IEC 61499 runtime environment.

[0005] The object underlying the invention is to provide a method which simplifies and accelerates the integration of artificial intelligence into the control of an industrial machine.

[0006] This object is achieved by a method according to claim 1.

[0007] In the method according to the invention, an AI model is selected for execution in the AI function block, e.g., from a plurality of AI models. Furthermore, the AI function block is linked, at least partially automatically, to other function blocks of a control software of the industrial machine. Finally, the AI function block is executed, at least partially automatically, with the AI function block preferably being executed using an IEC 61499 runtime environment.

[0008] The invention thus provides a largely automated method in which an operator only needs to specify, for example, that an AI function block is required, which AI model is to be executed in the AI function block, and how the AI function block is to be connected to the other function blocks. The operator thus only needs a few clicks until the AI function block is functionally integrated into the control software of the industrial machine and can be executed. In particular, the operator does not require any knowledge of the software libraries (AI inference engines) required to execute the AI model. Furthermore, they do not have to deal with configuring the communication between the AI model and the control software.

[0009] Control tasks that can be improved or simplified by artificial intelligence can thus be integrated into the control software with minimal effort. The time required to become functional is significantly reduced, allowing modifications to existing machines, for example, to be implemented much more quickly, which in turn reduces production downtime. The construction of new industrial machines is also shortened, thus reducing the time to market.

[0010] The control system of an industrial machine can, for example, consist of several computing devices (PLCs, computers, etc.) connected via a data network, which together control the industrial machine. The various computing devices can execute the functions specified by the function blocks.

[0011] The industrial machine can comprise a multitude of field devices, such as sensors and actuators. The field devices and the computing devices can be connected to each other via the data network, which can preferably be a fieldbus (e.g., Sercos III, EtherCAT, EIP, and the like). In particular, the interaction of the computing devices is based on IEC 61499, as explained in more detail later. The function blocks are preferably implemented on distributed computing devices. Furthermore, the function blocks can communicate with each other in an event-based manner.

[0012] The control software can be divided into functional blocks, with the connections or links between the functional blocks being defined during the configuration of the control software, resulting in a desired control behavior of the control software. At least one of the functional blocks can be designed as an AI functional block designed to execute the AI model. The AI model can, for example, comprise a neural network, a machine learning algorithm, a support vector machine (SVM), and the like. Furthermore, the AI model can also comprise AI variants outside the machine learning domain, such as rule-based AI systems, AI systems based on logic reasoning, planning and optimization algorithms, search algorithms, etc.The variety of AI models that can be used is an advantage of the approach described here, since the necessary runtime environment is appropriately initialized and linked to the control software.

[0013] The AI model can, for example, perform a classification task or perform image processing. For example, the AI model can be designed to check parts manufactured in the industrial process for their correct shape and / or arrangement and output a corresponding result. The AI function block can therefore, for example, be linked to other function blocks that pass the images to be evaluated to the AI function block and further process the calculation result of the AI function block.

[0014] The AI model can, in particular, be an already trained model that is ready for productive use.

[0015] Linking the function blocks can, for example, involve graphically connecting the function blocks in a graphical user interface, for example, using graphical elements that specifically indicate a connection between an output of one function block and an input of another function block. If such a connection is created graphically by an operator, the communication interfaces can be created automatically to then establish an actual data connection between the respective function blocks.

[0016] Linking can also involve linking at the code level, particularly by setting input and output variables, for example, in source code. The AI function block is thus integrated into the control system of the industrial machine. Generally speaking, the AI function block can receive and evaluate sensor data (measured values, image data, and the like).

[0017] Furthermore, the calculation results generated by the AI function block can be used to control the actuators of the industrial process.

[0018] By linking the function blocks, it is defined, in particular, via which data interfaces the function blocks exchange data, events, and the like. Furthermore, it can be defined which function blocks communicate with which other function blocks. The function blocks can collectively form at least part of the control software of the industrial machine.

[0019] As will be explained in more detail later, it is possible, for example, to automatically detect which inputs and outputs the AI model and thus the AI function block requires, which then allows the connection to the other function blocks to be (automatically) adjusted accordingly.

[0020] The at least partially automated execution of the AI function block can comprise a deployment. As will also be explained in more detail later, the executing computing device (i.e., the compute host) can be determined automatically, in particular, and the AI function block is then executed on the executing computing device. According to the invention, the AI function block is executed using an IEC 61499 runtime environment or in an IEC 61499 runtime environment (RTE).

[0021] The runtime environment according to IEC 61499 specifically includes the ability to distribute an application (i.e., the AI function block and / or the other function blocks or a network of function blocks) across multiple computing devices (compute hosts). Each compute host runs its own IEC 61499 runtime environment. The necessary communication ("cross-communication") between the runtime environments is preferably generated automatically by the runtime environments according to IEC 61499.

[0022] For example, this means that an IEC 61499 runtime environment is initialized on the most suitable compute host. In this runtime environment, only a single function block, in particular the AI function block, is executed. In addition to the IEC 61499 runtime environment, a runtime environment for the AI model is also set up on this compute host, in particular automatically. The automatic setup of the runtime environment for the AI model ensures that the runtime environment is suitable for the AI model. For example, TensorFlow or ONNX can be used for neural networks, or an mlpack library for machine learning models. The two runtime environments on the same compute host can then exchange data with each other very efficiently, among other things to meet real-time requirements.

[0023] IEC 61499 is an event-driven programming language. An IEC 61499 runtime environment allows the execution of control software on various distributed hardware resources, such as PLCs and / or Linux and / or Windows devices. For each function block in the IEC 61499 runtime environment, the executing hardware and / or computing device can be individually specified. This allows some function blocks to be executed on a Linux system, while others can be executed on a Windows system.

[0024] Through the inventive use of the IEC 61499 runtime environment, the AI function block can be flexibly integrated into a distributed architecture of computing devices, whereby the AI function block can thus be easily executed on a suitable computing device (e.g., with sufficient computing power). The IEC 61499 runtime environment also ensures that communication between the AI function block and the other function blocks of the control software (possibly on other computing devices) is automatically and correctly established. Communication can take place via the aforementioned data network.

[0025] Further embodiments of the invention can be found in the description, the subclaims and the drawings.

[0026] According to a first embodiment, the AI function block and / or the other function blocks are function blocks according to IEC 61499. According to the invention, the function blocks can be activated by events and can also generate events themselves to activate other function blocks. The function blocks, and in particular the AI function block, appear at least externally as IEC 61499 function blocks. The internal structure of the AI function block is described in more detail below.

[0027] According to another embodiment, the AI functional block comprises a runtime container that provides the IEC 61499 runtime environment and an AI container in which the artificial intelligence is executed, with the runtime container and the AI container communicating with each other. Generally speaking, the AI functional block can thus be executed in a virtualized environment.

[0028] The runtime container preferably provides data interfaces to the other function blocks. Communication between the runtime container and the AI container preferably enables the execution of the AI functions. The AI model is executed in the AI container. This has the particular advantage that, thanks to the AI container's communication with the IEC 61499 runtime environment, communication with the other function blocks can automatically take place in a data format that is also suitable for the other function blocks (which are designed, for example, according to IEC 61499). By providing the IEC 61499 functionality through the runtime container, the functionality of the AI model is integrated directly at the control code level, i.e., directly into the control software.

[0029] The runtime container preferably runs a virtual IEC 61499 runtime environment. Depending on the selected AI model, the required AI container can be determined, preferably automatically. The AI container provides the inference environment for the AI model. In particular, different AI containers may be required depending on whether the AI model was created in PyTorch, ONNX, KERAS, Tensorflow, and the like. The AI container and / or the runtime container can be, for example, a Docker container, an LXC container, a containerd container, and the like.

[0030] The required AI container can first be preselected based on the AI model. For example, for an AI model that is a neural network, an AI container with a suitable framework (e.g., Tensorflow, PyTorch, or ONNX) can be preselected. For an AI model with a machine learning model (e.g., SVM), an AI container with a framework based on "mlpack" can be preselected instead. In a second step, an AI container with the appropriate version of the framework (e.g., Tensorflow 1 or Tensorflow 2) can be selected and used.

[0031] In particular, a collection of AI containers is available that preferentially encapsulate the corresponding runtime environment, with the most suitable AI container being selected based on the AI model. This makes it possible to automatically execute a wide variety of AI models. A restriction to, for example, neural networks in Tensorflow 1 can be eliminated.

[0032] According to the invention, the control software is executed on a plurality of different computing devices connected by a data network, wherein one of the computing devices is automatically selected for executing the AI function block, for example based on the available computing power of the computing device. According to the method, it is therefore possible to check which of the computing devices is most suitable for executing the AI function block. In addition to the available computing power of the computing device, the presence of special hardware configured for executing machine learning methods in the computing device can alternatively or additionally be decisive. After the computing device (i.e. the compute host) has been selected, initialization takes place on the selected computing device. In this case, a so-called POD can be created, which comprises a plurality of containers.In particular, the AI container and the runtime container can be initialized together on the computing device.

[0033] According to a further embodiment, the runtime container and the AI container are automatically initialized on the selected computing device. Preferably, the runtime container and the AI container are on the same computing device.

[0034] According to a further embodiment, after initializing the AI container, a deployment of the selected AI model into the AI container takes place, preferably automatically. Alternatively or additionally, after initializing the runtime container, a deployment, preferably an instance of the runtime container, takes place, particularly automatically, to the selected computing device. Deployment (software distribution) refers in particular to the semi- or fully automated installation and configuration of software (here, the AI container and / or the runtime container).

[0035] According to a further embodiment, the AI container and the runtime container are executed on the same hardware and, in particular, communicate directly via the shared hardware, for example, via shared memory areas. Communication on the same hardware can thus be very fast and low-latency, which can accelerate the functioning of the artificial intelligence.

[0036] According to a further embodiment, the communication interfaces between the AI container and the runtime container are configured automatically. In particular, for the configuration of the communication interfaces between the AI container and the runtime container, the inputs and outputs of the AI model are determined, preferably automatically, and the communication interfaces between the AI container and the runtime container are adapted to the inputs and outputs. For example, if it is determined that the AI model requires several different inputs with different data types, but the AI model only provides one output of a single data type, the runtime container can provide communication interfaces (i.e., data interfaces) via which the AI model is provided with precisely the required inputs and via which the result determined by the AI model can be retrieved at the output.

[0037] Furthermore, the runtime container can provide the inputs and outputs required by the AI model externally, i.e., for the other function blocks. In this example, the AI function block would then comprise several inputs with different data types and the one specified output. The inputs and outputs of the AI function block are then available for linking to the other function blocks. In particular, the automatically determined inputs and outputs can be displayed graphically to an operator in a graphical user interface. For example, the inputs and outputs of the AI function block can be determined based on information, in particular based on metadata, of the AI model. For neural networks, the metadata can, for example, include the "input shapes" and "output shapes," i.e., the dimensions of the tensors / matrices of the inputs and outputs.

[0038] According to a further embodiment, the method is initiated by an operator, whereby the operator first selects an AI function block and then the AI model, in particular in the form of an executable file or a binary file, for the AI function block. The operator can then additionally initiate deployment. Overall, it may therefore only be necessary to perform three steps (selecting the AI function block, selecting the AI model, and initiating deployment). The operator can be guided through these three steps graphically in a graphical user interface. In the first step, the operator can then graphically select the AI function block. In the second step, the operator can select the AI model, for example by selecting it from a dropdown list or by specifying a file containing the AI model.By selecting an executable file and / or a binary file, the AI model can be executed directly in the AI function block, without, for example, the need for compilation. In the final step, the operator can then initiate deployment, whereupon the computing device is automatically selected and the function block is executed on the computing device. The aforementioned steps can be displayed to the operator, in particular, by an engineering system. The engineering system can be software and / or hardware for configuring and programming the computing devices of the industrial machine. The engineering system can configure the computing devices in accordance with IEC 61499.

[0039] Another subject of the invention is an industrial machine according to claim 11. The industrial machine is designed, among other things, to enable the selection of an AI model for an AI function block, wherein the AI model is executed in the AI function block, to link the AI function block at least partially automatically with further function blocks of a control software of the industrial machine, to execute the AI function block at least partially automatically, wherein the execution of the AI function block takes place by means of an IEC 61499 runtime environment.

[0040] The aforementioned functions of the industrial machine can be provided, for example, by an engineering system that may be part of the industrial machine. In particular, the engineering system may comprise a computing device that is at least temporarily connected to the industrial machine via a data network.

[0041] According to one embodiment of the industrial machine, the industrial machine comprises multiple computing devices, wherein different computing devices execute different functional blocks and the functional blocks communicate with each other according to IEC 61499. The functions of the industrial machine are therefore preferably controlled by means of distributed hardware.

[0042] Finally, the invention also relates to a computer program product according to claim 13.

[0043] The statements regarding the method according to the invention apply accordingly to the industrial machine according to the invention and the computer program product according to the invention. This applies in particular with regard to advantages and preferred embodiments. It is understood that all embodiments and features mentioned herein can be combined with one another, unless explicitly stated otherwise.

[0044] The invention is described below purely by way of example with reference to the drawings. They show: Fig. 1 a schematic view of an industrial machine with multiple computing devices; Fig. 2 interconnected function blocks for controlling the industrial machine; Fig. 3 Steps to integrate artificial intelligence into a functional block; Fig. 4 schematic of the execution of an AI function block on a computing device.

[0045] Fig. 1 shows an industrial machine 10. The industrial machine 10 comprises three computing devices 12, 14, 16. The computing devices 12, 14 can be designed, for example, as programmable logic controllers (PLCs), whereas the computing device 16 can be designed as a Windows device with an engineering system.

[0046] The computing devices 12, 14, 16 are connected to each other via a data network 18 in the form of a fieldbus. The industrial machine 10 also includes a camera 20, a gripper 22, and a motor 24 of a conveyor belt 26. The field devices 20, 22, 24 are also connected to the data network 18. The conveyor belt 26 can be used to transport workpieces 28 to be processed by the industrial machine 10.

[0047] The workpieces 28 are recorded by the camera 20. Based on the images recorded by the camera 20, defective workpieces 28 are to be detected and sorted out using the gripper 22.

[0048] An engineering system is executed on the computing device 16, which allows the programming and configuration of the field devices 20, 22, 24. The engineering system also determines which of the computing devices 12, 14, 16 executes which parts of the programming and configuration.

[0049] In Fig. 2 A schematic view is shown, similar to the one displayed to an operator of the engineering system. The view shows various function blocks FB1-FB5. The function blocks FB1-FB5 are connected to each other by graphical connectors 30. The graphical connectors 30 indicate how the outputs of one function block are linked to the inputs of another function block.

[0050] The function blocks can perform different tasks. For example, function block FB1 can control motor 24 and thus control the movement of workpieces 28. Using graphical connectors 30, an operator can specify that the outputs of a function block forward information to other function blocks. For example, after moving a workpiece 28, function block FB1 can inform function block FB3 that a workpiece 28 is now within the range of camera 20. Function block FB3 can then instruct camera 20 to take an image of workpiece 28. The image from camera 20 can then be transferred to function block FB4.

[0051] In the function block FB4, an artificial intelligence ("AI" in Fig. 2 and Fig. 4 ) to evaluate the images generated in the function block FB3.

[0052] Fig. 3now shows the steps an operator can perform to configure the FB4 function block into an AI function block and begin executing the AI function block. In the first step, the operator selects an AI function block. By clicking on "Configure," a binary file containing the AI model can then be selected in the second step. In this example, a file for the classification of workpieces 28 is selected ("Workpiece_classification_model.onnx"). After the selection, an automatic adjustment 32 takes place, in which the inputs and outputs of the AI model are determined and the inputs and outputs of the AI function block are adjusted accordingly. Finally, in the third step, the operator can execute the AI function block by clicking on "Deploy."

[0053] This results in, as in Fig. 4For example, a pod consisting of two containers, namely a runtime container 34 and an AI container 36, is created on the computing device 12. The containers 34, 36 communicate via shared memory areas 38.

[0054] Additional containers can also be executed in another pod on the computing device 12. The runtime container 34 communicates with other runtime environments via the data network 18. Both the runtime container 34 and the other runtime environments that execute the additional function blocks FB1, FB2, FB3, and FB5 operate according to IEC 61499.

[0055] During operation, the additional function blocks FB1, FB2, FB3, and FB5 can, for example, transmit image data from camera 20 to the AI function block FB4. There, the runtime container 34 receives the data and transmits it to the AI container 36 via the shared memory area. In the AI container 36, the AI model performs a classification. Once the classification is complete, the classification result is transferred via the runtime container 34 to the additional function blocks FB1, FB2, FB3, and FB5. The additional function blocks can then use the classification result to determine, for example, whether the gripper 22 should sort out a workpiece 28.

[0056] Overall, it is possible for an operator to integrate a complex AI model into the control system of the industrial machine 10 with just a few clicks. List of reference symbols

[0057] 10 industrial machine 12 computing device 14 computing device 16 computing device 18 data network 20 camera 22 gripper 24 motor 26 conveyor belt 28 workpieces 30 graphical connectors 32 automatic adaptation 34 runtime container 36 AI container 38 shared memory area FB1Function block FB2Function block FB3Function block FB4KI-Function block FB5Function block

Claims

1. A method of integrating at least one AI function block (FB4), which comprises artificial intelligence, into a controller for an industrial machine (10), wherein, in the method, - for the AI function block (FB4), an AI model is selected for execution in the AI function block (FB4), - the AI function block (FB4) is linked in an at least partly automated manner to further function blocks (FB1-FB3, FB5) of control software of the industrial machine (10), - the AI function block (FB4) is brought to execution in an at least partly automated manner, characterized in that the execution of the AI function block (FB4) takes place by means of an IEC 61499 runtime environment, wherein the function blocks (FB1-FB5) are activated by events and also generate events themselves to activate other function blocks (FB1-FB5), wherein the control software is executed on a plurality of different computing devices (12-16) connected by means of a data network, wherein one of the computing devices (12-16) is selected in an automated manner for the execution of the AI function block (FB4).

2. A method according to claim 1, wherein the AI function block (FB4) and the further function blocks (FB1-FB3, FB5) are function blocks according to IEC 61499.

3. A method according to claim 1 or 2, wherein the AI function block (FB4) comprises a runtime container (34), which provides the IEC 61499 runtime environment, and an AI container (36) in which the artificial intelligence is executed, wherein the runtime container (34) and the AI container (36) communicate with one another.

4. A method according to any one of the preceding claims, wherein one of the computing devices (12-16) is selected in an automated manner for the execution of the AI function block (FB4) based on the available computing power of the computing device (12-16).

5. A method according to claim 3 and 4, wherein the runtime container (34) and the AI container (36) are automatically initialized on the selected computing device (12-16).

6. A method according to claim 5, wherein, after the initialization of the AI container (36), a deployment of the selected AI model into the AI container (36) takes place and / or, after the initialization of the runtime container (34), a deployment, in particular an automatic deployment, preferably of an instance, of the runtime container (34) to the selected computing device (12-16) takes place.

7. A method according to at least one of the claims 3 to 6, wherein the AI container (36) and the runtime container (34) are executed on the same hardware and communicate directly via the common hardware, for example via jointly used memory areas.

8. A method according to at least one of the claims 3 to 7, wherein the communication interfaces between the AI container (36) and the runtime container (34) are automatically configured.

9. A method according to claim 8, wherein, for the configuration of the communication interfaces between the AI container (36) and the runtime container (34), the inputs and the outputs of the AI model are determined, preferably automatically determined, and the communication interfaces between the AI container (36) and the runtime container (34) are adapted to the inputs and outputs.

10. A method according to any one of the preceding claims, wherein the method is initiated by an operator, wherein the operator first selects an AI function block (FB4) and then selects the AI model, in particular in the form of an executable file or a binary file, for the AI function block (FB4).

11. An industrial machine (10) which is configured - to enable a selection of an AI model for an AI function block (FB4), wherein the AI model is executed in the AI function block (FB4), - to link the AI function block (FB4) in an at least partly automated manner to further function blocks (FB1-FB3, FB5) of control software of the industrial machine (10), - to bring the AI function block (FB4) to execution in an at least partly automated manner, characterized in that the execution of the AI function block (FB4) takes place by means of an IEC 61499 runtime environment, wherein the function blocks (FB1-FB5) can be activated by events and also configured themselves to generate events in order to activate other function blocks (FB1-FB5), - the industrial machine (10) is further configured to execute the control software on a plurality of different computing devices (12-16) connected by means of a data network, wherein one of the computing devices (12-16) is selected in an automated manner for the execution of the AI function block (FB4).

12. An industrial machine (10) according to claim 11, wherein the industrial machine (10) comprises a plurality of computing devices (12-16), wherein different computing devices (12-16) execute different ones of the further function blocks (FB1-FB3, FB5) and the function blocks communicate with one another according to IEC 61499.

13. A computer program product comprising commands which, on the execution of the program by a computer, cause the computer to perform the method according to at least one of the claims 1 to 10.