Industrial control device, industrial control system, and method of operating the same

Abstract

Industrial control device, industrial control system and method of operation thereof. An industrial control device executes a plurality of containerized control applications with a constant period and comprises queuing procedures associated with respective network interfaces that manage incoming or outgoing network packets in a queue and block network packets with a timestamp not older than a barrier timestamp, and agents that first execute a maintenance process in each period and then execute control applications within the remaining part of the period. The maintenance process comprises I / O processing with technical devices on behalf of control applications and transmission of a move command to each of the queuing procedures, the move command indicating an advancement of the barrier timestamp and comprising a timestamp indicating a start time of the respective period. The queuing procedures adjust their barrier timestamps based on the received start time. Time deterministic behavior is achieved in an efficient way.

Description

Technical Field

[0001] This invention relates to the field of app-based industrial control systems with real-time requirements. More particularly, this invention relates to industrial control equipment, industrial control systems, methods of operation thereof, and computer program products. Background Technology

[0002] In industrial control systems with real-time requirements, multiple control applications installed on interconnected industrial control devices periodically perform input and output operations with corresponding technical devices connected to the respective industrial control devices. Furthermore, the control applications communicate with each other by exchanging network packets according to, for example, the Common Datagram Protocol (UDP). In this context, the sender-receiver relationship is advantageously transparent to the control applications. That is, the control applications may not be aware of who their communication partners are, how many there are, or what their Internet Protocol addresses are. The control applications transmit UDP packets with predefined identifiers to dummy addresses or similar, and a proxy handles network address translation, multicasting, and the like to route the UDP network packets to one or more desired receiving control applications based on the predefined identifiers in the UDP network packets and according to the configuration plan of the industrial control system.

[0003] During the reconfiguration of industrial control systems, changes such as the commissioning and decommissioning of technical equipment are reflected through corresponding reconfigurations of control applications. That is, the configuration plan of the industrial control system upon which the routing of network packets is based is changed. In this paper, the following problem arises: network packets may be in progress while the changed configuration plan is being implemented. Therefore, it is unclear whether the ongoing network packets will be routed according to the old rules before reconfiguration or according to the changed rules after reconfiguration. However, in industrial control systems, deterministic behavior is required during reconfiguration.

[0004] In this paper, time-deterministic behavior means, in one respect, that network packets transmitted during a given period should reliably arrive at the receiver within a well-defined period, which can be exactly one of the same period, the next period, or the next-after-next cycle (time-deterministic delivery of network packets). Furthermore, time-deterministic behavior means that reconfiguration can be performed such that a control application put into service begins receiving network packets from the period prior to its entry into service, and no network packets are routed to a control application that was taken out of service from the period in which it was taken out of service (time-deterministic rerouting of network packets in the case of reconfiguration).

[0005] A common approach to achieving time-deterministic behavior in real-time industrial control systems is to subdivide the cycle into smaller time periods, reserving time slices for communication, reconfiguration, and processing. However, this leads to overconfiguration, longer cycles, and wasted valuable CPU time.

[0006] WO2022 / 053335 discloses a message bus that enables serialization of publish / subscribe topic-based communication to achieve precise timing of switching between control apps that are added to and removed from services.

[0007] Further background information is provided in the bibliography at the end of the specification. Summary of the Invention

[0008] Therefore, one object of the present invention is to provide an apparatus for achieving time-deterministic behavior in an industrial control system in an efficient manner.

[0009] According to a first aspect, an industrial control device is proposed for an industrial control system comprising multiple industrial control devices. The proposed industrial control device is configured to execute multiple control applications at a constant cycle and includes: multiple containers, each containerizing a corresponding one of the control applications; multiple network interfaces, including physical network interfaces of the industrial control devices and multiple virtual network interfaces of the corresponding containers; and multiple queuing disciplines, each queuing discipline associated with at least one of the network interfaces and configured to manage incoming or outgoing network packets in the queue via the corresponding network interface, and to hold packets in the queue. (back) A network packet having a timestamp no older than the barrier timestamp maintained by the corresponding queuing procedure; a real-time container agent configured to perform maintenance processing in each cycle and to cause execution of each in the control application only after the maintenance processing has been completed, wherein the maintenance processing includes: performing output and input processing on behalf of the control application using technical devices connected to the industrial control equipment, and transmitting a move command to each in the queuing procedure, the move command instructing the corresponding queuing procedure to advance its barrier timestamp, wherein the move command includes a timestamp indicating a logical execution time, which is the start time of the corresponding cycle, and the corresponding queuing procedure is configured to adjust its barrier timestamp in response to receiving the move command, based on the logical execution time indicated by the timestamp included in the move command.

[0010] By utilizing the proposed industrial control equipment, both for network packet switching and for reconfiguration, time-deterministic behavior can be achieved in industrial control networks, while maximizing the use of available processing time per cycle on each industrial control device. Therefore, time-deterministic behavior can be achieved in an efficient manner.

[0011] More specifically, maintenance processing for real-time containers and control applications can be executed continuously in a single-threaded manner, thus promoting time-deterministic behavior. That is, input / output operations utilizing the technology device are performed while the control application is idle (interrupted and / or not being executed). Furthermore, reconfiguration can be performed by the real-time container agent as part of the maintenance process when the control application is idle. On the other hand, execution of the control application can begin once the maintenance process is complete, advantageously eliminating the need for fixed time slices for maintenance processing and control application execution, thus advantageously avoiding over-provisioning. However, queuing procedures and the operating system and its components (such as the network stack) of industrial control equipment specifically allow parallel operation with the real-time container agent and containerized control apps. Therefore, network packets can propagate in the background without requiring reserved time slices for network packet propagation and are not even periodically limited. On the other hand, queuing procedures block network packets with timestamps equal to or newer than the barrier timestamp. The barrier timestamp is moved once per cycle by the real-time container agent. Therefore, the nondeterministic propagation time of network packets within industrial control equipment and / or across physical networks is largely flattened, and network packets can be delivered deterministically in batches in each cycle.

[0012] However, the inventors have identified situations in which, despite these measures, network packets transmitted by one of the control applications are delivered to another control application in one cycle or another (such as in the next or the next cycle, or such as in the same cycle or the next cycle), depending on the nondeterministic environment.

[0013] More specifically, when a queuing procedure merely moves its barrier timestamp to the current time in response to a move command, different queuing procedures in the same or different industrial control devices may establish different barrier timestamps, and contention may occur. Furthermore, due to internal timing dependent on the network stack or similar mechanisms, incoming network packets may be released before or after the time a move command is received within a given period, further contention may arise.

[0014] Therefore, according to the proposed industrial control device, the movement command includes a timestamp indicating the start time of the corresponding cycle, which will be referred to below as the "logical execution time". This advantageously allows multiple queuing procedures to adjust their barrier timestamps to the same barrier timestamp. Furthermore, using the logical execution time, it is advantageous to adjust the barrier timestamp at the boundary of the cycle, i.e., at the beginning or end of the cycle, such that during any given cycle, any incoming network packet will always be received after or before the barrier timestamp.

[0015] Therefore, it is advantageous to improve the time deterministic behavior of industrial control equipment.

[0016] Specifically, industrial control equipment can be computing devices, such as industrial personal computers or embedded devices, configured to run an operating system with a network stack, such as a customized version of Linux. The network stack can be configured to forward network packets between the virtual and physical network interfaces of the industrial control equipment.

[0017] Network packets can be any suitable type of network packet. In particular, network packets can be UDP network packets. More specifically, network packets transmitted by control applications can be network packets addressed to pseudo-addresses and pseudo-ports and include identifiers (the control application does not need to know who consumes their output). Firewalls installed on industrial control equipment, or, in a preferred embodiment, rule-based network packet processing performed by queuing procedures during dequeueing, can match network packets based on identifiers and rewrite the headers of matching packets according to the communication mapping of the industrial control system, thereby enabling the orchestration of the industrial control system.

[0018] Specifically, the queuing procedure can be a unit configured to enqueue network groups, temporarily keep enqueued networks in the queue, and dequeue network groups upon request.

[0019] Specifically, when a network packet is about to pass through or has already passed through the associated network interface, the operating system or its components may request a queuing procedure to enqueue and / or dequeue the network packet.

[0020] In particular, queuing procedures can be operated according to the FIFO (First In First Out) principle, in which network packets are enqueued at one end of the queue and dequeued at the other end.

[0021] In particular, the operating system, network stack, queuing procedures, and the like can operate independently of the cycle, but can operate continuously in parallel with the processing of the cycled real-time container agents and control applications.

[0022] In this application, the term "multiple" refers to one or more quantities, or N≥1, or one of the singular and plural numbers.

[0023] Specifically, the corresponding container can containerize the corresponding control application by forming an abstraction layer that isolates the control application from other control applications hosted on the same industrial control equipment. Each containerized control application can be restricted to accessing only the corresponding portion of the industrial control equipment's resources, such as CPU, memory, I / O, and network bandwidth. Direct access by the containerized control application to the accessible portions of the industrial control equipment's resources may not be permitted. Access to the accessible portions of the industrial control system's resources may only be possible via the abstraction layer formed through the container.

[0024] In particular, multiple network interfaces of industrial control equipment can be interconnected.

[0025] Specifically, the timestamp for each network packet can be created when an egress network packet is created for transmission via one of the network interfaces or when an ingress network packet is received at one of the network interfaces. Alternatively, the timestamp for each network packet can be created by the appropriate queuing procedure when the corresponding network packet is enqueued.

[0026] In particular, a real-time container agent can be a unit that configures industrial control equipment to execute containerized control applications at constant intervals.

[0027] More specifically, the maintenance process can further include establishing the start time (logical execution time) and length of each cycle by a real-time container agent, such as based on the clock of an industrial control device, for example.

[0028] More specifically, maintenance processing may further include, in response to receiving updates to the communication mappings of the industrial control system, the real-time container agent performing reconfiguration on how network packets are handled on the industrial control device. For example, the real-time container agent may reconfigure the routing tables, firewalls, or similar features of the industrial control device's operating system.

[0029] Specifically, a real-time container agent can perform output and input processing on behalf of the corresponding control application by using an abstraction layer formed by containers, which containerize the control application. For example, the real-time container agent can read output from the abstraction layer's virtual Linux GPIO filesystem and transfer the read output to the technology device, and the real-time container agent can receive input from the technology device and write the input to the virtual GPIO filesystem. In this way, the control application can perform virtual output and input processing using standardized GPIO interfaces. However, the actual output and input processing for controlling the technology device is performed by the real-time container agent during maintenance processing.

[0030] Specifically, technical equipment can be any industrial equipment that is adapted to be controlled by industrial control devices or systems. Examples of technical equipment include actuators, sensors, switches, and larger systems with various components, such as robots, factories, gas turbines, wind turbine generators, chemical plants, processing plants, power plants, power grids, or the like.

[0031] Specifically, inducing the execution of each control application during the remainder of the cycle may include generating and / or resuming the execution of the corresponding control application when the maintenance process is complete; and may include terminating and / or pausing the execution of the control application before the end of the cycle. For example, POSIX-based SIGTERM, SIGSUSPEND, and SIGRESUME signals can be used to implement termination, pause, and resumption.

[0032] The manner in which movement commands are transmitted to the queuing procedure is not particularly limited. For example, movement commands can be transmitted by: using a socket connecting the real-time container agent to the appropriate queuing procedure; by invoking an application programming interface published by the appropriate queuing procedure via the real-time container agent; by raising a flag and writing a value to a shared memory area via the real-time container agent; or by any other suitable means preferably ensuring rapid communication between the real-time container agent and the appropriate queuing procedure. In this document, "rapid communication" will mean that movement commands take precedence over network packets and do not need to go through queuing or the like.

[0033] In particular, the length of the constant period can be appropriately selected. Advantageous period lengths achievable using the proposed industrial control system include 100 milliseconds, more preferably 50 milliseconds, and most preferably 10 milliseconds.

[0034] According to an embodiment, the maintenance process includes synchronizing the start time of each cycle with the real-time container agent of other industrial control devices in the industrial control system.

[0035] This allows for advantageous synchronization and timing across the entire industrial control equipment. In other words, the logic execution time can be the same across all industrial control equipment, thereby promoting deterministic timing behavior throughout the entire industrial control system.

[0036] According to a further embodiment, the corresponding queuing procedure is configured to associate a timestamp indicating the current time with the network packet when enqueuing a network packet that does not yet have a timestamp.

[0037] Therefore, queuing procedures can advantageously determine whether to block network packets based on the period during which network packets are enqueued.

[0038] According to a further embodiment, the corresponding queuing procedure is configured to set the logical execution time of the current cycle or the logical execution time of the next cycle as its barrier timestamp.

[0039] Therefore, the time-deterministic delivery of network packets can be advantageously improved. The barrier timestamp is set to the boundary (start time or end time) of the current period. Thus, all network packets transmitted or received within the current period are processed in the same manner as the corresponding queuing procedure.

[0040] According to a further embodiment, the multiple queuing procedures include: a common inbound queuing procedure associated with the physical network interface of the industrial control equipment and configured to manage network packets entering the queue via the physical network interface; and a corresponding outbound queuing procedure associated with each of the virtual network interfaces of each container and configured to manage network packets exiting the queue via the corresponding virtual network interface.

[0041] A common inbound queue associated with the physical network interface advantageously enables time-deterministic multicast in industrial control systems. Separate outbound queues for each containerized app advantageously enable time-deterministic multicast of local network packets transmitted between containerized control applications on the same industrial control equipment.

[0042] According to a further embodiment, the public entry queuing procedure is further associated with each of the virtual network interfaces, which includes a first queue and a second queue, and is configured to: queue local network packets incoming via one of the virtual network interfaces and outgoing via another of the virtual network interfaces in the first queue, queue remote network packets incoming via a physical network interface in the second queue, maintain a first barrier timestamp for blocking local network packets in the first queue and a second barrier timestamp for blocking remote network packets in the second queue, and, in response to receiving a move command, set the first barrier timestamp to the logical execution time of the next cycle and set the second barrier timestamp to the logical execution time of the current cycle.

[0043] In particular, using the logic execution time as a first barrier timestamp applied to remote network packets advantageously enables remote network packets transmitted between different industrial control devices to be delivered to the receiving control application in a time-deterministic manner, and more particularly within two cycles after the cycle in which they are sent by the transmission control application.

[0044] In particular, using the logical execution time of the next cycle as a second barrier timestamp applied to local network packets advantageously enables local network packets transmitted between control applications of the same industrial control equipment to be delivered to the receiving control application in a time-deterministic manner, and more particularly in the next cycle after the cycle in which they are sent by the transmitting control application.

[0045] In particular, in cases where different barrier timestamps are applied to different types of network packets, classifying different types of network packets into two separate queues can advantageously improve the performance of the common entry queuing procedure. The common entry queuing procedure can maintain each packet in the queue classified by timestamp. Therefore, in the case of a dequeue request, only the first network packet in each queue needs to be checked to determine whether the network packet should be dequeued.

[0046] In particular, the case where a public entry queuing procedure is “associated with multiple network interfaces (such as virtual network interfaces and physical network interfaces)” also includes the case where a public entry queuing procedure is directly associated with only one network interface and indirectly associated with other network interfaces.

[0047] For example, "direct association" can be achieved using the Linux qdisc API. "Indirect association" can be achieved using mirror redirection, such as the Linuxmirred API, to "clone" or "steal" network packets from other network interfaces and feed them to one network interface.

[0048] A network interface can be one of multiple virtual network interfaces, physical network interfaces, and other pseudo-virtual network interfaces.

[0049] According to a further embodiment, the maintenance process includes, in the following order: an output processing step, namely, using technical equipment to perform output processing and transmitting a movement command to each of the outbound queuing procedures; and an input processing step, namely, using technical equipment to perform input processing and transmitting a movement command to the public inbound queuing procedure.

[0050] Specifically, the output processing step can be executed after the synchronization step.

[0051] It is important to note that transmitting a move command to the outbound queuing procedure can cause outbound local and remote network packets to be released from the outbound queuing procedure, and transmitting a move command to the inbound queuing procedure can cause inbound local and remote network packets to be released from the common inbound queuing procedure. Therefore, a sequence is established where output processing and the release of outbound network packets by utilizing the technical device occur first, followed by input processing and the release of inbound network packets by utilizing the technical device. This structure further facilitates time-deterministic behavior and establishes variable but well-defined time points at which reconfiguration can occur, such as before output processing, between output and input processing, or after input processing.

[0052] According to a further embodiment, the corresponding queuing procedure is configured to process network packets according to a configurable set of rules when dequeuing network packets.

[0053] In other words, the corresponding queuing procedure can advantageously implement rule-based processing of network packets as they are dequeued from their respective queues. Rule-based processing can be configured and used by a real-time container agent to induce routing and multicasting of network packets according to the configuration plan of the industrial control system. Advantageously, in this paper, rule-based processing can therefore occur at defined time points within each cycle in a time-deterministic manner. Furthermore, the inventors have discovered that significant performance gains can be achieved by implementing only a subset of rule-based processing, which is directly orchestrated within the queuing procedure, compared to using components of the network stack such as iptables in Linux to perform rule-based processing. This performance gain is particularly beneficial during reconfiguration of the industrial control system, i.e., when one or more of the rules need to change rapidly and in a time-deterministic manner.

[0054] According to a further embodiment, the configurable rule set can cause one or more of the following types of processing of network packets depending on the configured content: performing network address translation on the network packet; cloning the network packet and shortlisting the cloned network packet; immediately releasing the network packet if it is shortlisted; capturing a copy of the network packet for later replay; and dropping the packet if its per-cycle quota is exhausted in the current cycle.

[0055] The proposed cloning and inclusion candidate functionality can be advantageously used to enable multicasting when dequeuing individual network packets, even if only one network packet can be dequeued at a time.

[0056] The proposed capture function can be advantageously used to patch temporarily damaged data paths, for example, when a transmission control application needs to be blocked due to a state transition operation as part of a reconfiguration, and a receive control application can simultaneously operate using a copy of the input from the transmission control application. In this way, the functionality of the industrial control system can maintain limited degradation during such state transitions.

[0057] According to a further embodiment, the real-time container agent is configured to, during maintenance processing, implement communication mapping of the industrial control system belonging to the industrial control equipment by configuring corresponding rules in a configurable rule set of the corresponding queuing procedure.

[0058] In other words, based on the configuration plan specifying the communication mapping of the industrial control system, the proposed network address translation function of the queuing procedure can be advantageously used for orchestration, that is, for rewriting the destination address, port, and the like of network packets in transit.

[0059] Advantageously, in this paper, rule-based rewriting of network packets in transit for orchestration purposes occurs when network packets are dequeued from the corresponding queuing procedure. Since the real-time container agent uses move commands to control barrier timestamps, which in turn controls which network packets can and cannot be dequeued, it is possible to precisely control which network packets will be affected by rules changed at configuration time, and which will not be affected by rules changed at configuration time. Therefore, time-deterministic behavior in the case of reconfiguration can be further facilitated.

[0060] In this document, “communication mapping of an industrial control system belonging to an industrial control device” may specifically refer to any part of the rule set of a queuing procedure for a communication mapping that requires reconfiguration of at least one of the industrial control device’s queuing procedures.

[0061] According to further elements, the matching clause that can be configured for each configurable rule in the configurable rule set includes at least one criterion to match the packet header of the network packet and at least one criterion to match the timestamp of the network packet.

[0062] Therefore, using a criterion that matches the timestamps of network packets may be advantageous for defining network processing rules that apply only to network packets. These rules not only match the criterion to be matched with the packet header, but also apply only to network packets enqueued at a certain period, or no earlier than a certain period, or no later than a certain period. In other words, for example, it's possible to configure rules that will expire after a certain period or rules that will only be enforced at a logical future point in time.

[0063] Therefore, it may be advantageous to perform reconfiguration so that each network packet is processed correctly according to the period in which it was enqueued. This can further facilitate time-deterministic behavior during reconfiguration.

[0064] According to a further embodiment, the real-time container agent is configured to implement communication mapping by configuring a configurable set of rules for appropriate queuing procedures at one or more of the following time points: before input processing; between output processing and input processing; and after input processing.

[0065] In other words, in the configuration plan of the communication mapping that will be transmitted to the live container for reconfiguration, it can be specified whether the reconfiguration (such as the reconfiguration of the configurable rule set of the queuing procedure) is performed before output processing, between output processing and input processing, or after input processing.

[0066] Therefore, it is possible to reconfigure the communication mapping while the control application is in a static state. However, it is still not necessary to reserve a fixed time slice for such reconfiguration. Instead, the precise timing of reconfiguration can vary across different time periods, and / or within the same period, but across different industrial control devices, because reconfiguration is not related to precise time but to logical events, such as before or after the start of output and / or input processing.

[0067] In this way, time-deterministic behavior during reconfiguration can be further promoted in a favorable and efficient manner.

[0068] Any embodiment of the first aspect may be combined with any embodiment of the first aspect to obtain another embodiment of the first aspect.

[0069] According to the second aspect, an industrial control system includes a plurality of industrial control devices comprising any embodiment of the first aspect or thereof, whose physical operating interfaces are interconnected via a physical network.

[0070] For example, the physical network can be Ethernet. In particular, the physical network can apply Quality of Service (QoS) rules that ensure that network packets transmitted between containerized control applications are prioritized over other network traffic and are preferably delivered within the longest period of one of the constant cycles.

[0071] According to a third aspect, a method for operating an industrial control system is proposed. In this document, the industrial control system includes multiple industrial control devices, each configured to execute multiple control applications at a constant cycle. Each industrial control device includes: multiple containers, each containerizing a corresponding one of the control applications; multiple network interfaces, including physical network interfaces of the industrial control devices and multiple virtual network interfaces of the corresponding containers; multiple queuing procedures, each queuing procedure associated with at least one of the network interfaces and configured to manage network packets entering or leaving the queue via the corresponding network interface, and to block network packets in the queue having timestamps older than the barrier timestamps maintained by the corresponding queuing procedure; and a real-time container agent. The method includes: real-time container agents of each in an industrial control device globally synchronizing the start time of each cycle with each other across the industrial control system; thereafter, in each cycle, each real-time container agent performs maintenance processing, and only after the maintenance processing has been completed does it cause execution of each in a control application for the remainder of the cycle, wherein the maintenance processing includes: performing output processing and input processing on behalf of the control application of the respective industrial control device using technical devices connected to the respective industrial control device; and transmitting a move command to each in a queuing procedure of the respective industrial control device, the move command instructing the respective queuing procedure to advance its barrier timestamp, wherein the move command includes a timestamp indicating a logical execution time, the logical execution time being the start time of the respective cycle, and the method further includes the respective queuing procedure adjusting its barrier timestamp based on the logical execution time indicated by the timestamp included in the move command in response to receiving the move command.

[0072] According to the fourth aspect, a computer program product is proposed, comprising instructions that, when executed on an industrial control device having a physical network interface for an industrial control system comprising multiple industrial control devices, cause the industrial device to: be configured to execute multiple control applications at a constant cycle; form multiple containers, each container containerizing a corresponding one of the control applications, and each container having a virtual network interface; form multiple queuing procedures, each queuing procedure associated with at least one of the network interfaces and configured to manage network packets entering or leaving the queue via the corresponding network interface, and to block network packets in the queue having timestamps no older than the barrier timestamps maintained by the corresponding queuing procedure; and form a real-time container agent. In this document, the instructions, when executed on an industrial control device, further: cause the real-time container agent to perform maintenance processing in each cycle, and to perform each of the control applications only during the remainder of the cycle after the maintenance processing has been completed, wherein the maintenance processing includes: performing output and input processing on behalf of the control application using technical equipment connected to the industrial control device; and transmitting a move command to each of the queuing procedures, the move command instructing the corresponding queuing procedure to advance its barrier timestamp, wherein the move command includes a timestamp indicating a logical execution time, which is the start time of the corresponding cycle. The instructions further cause the corresponding queuing procedure, in response to receiving the move command, to adjust its barrier timestamp based on the logical execution time indicated by the timestamp included in the move command.

[0073] Computer program products, such as computer program devices, can be implemented as memory cards, USB sticks, CD-ROMs, DVDs, or as files downloadable from a server on a network. For example, such files can be provided by transmitting a file including the computer program product from a wireless communication network.

[0074] More specifically, the computer program product may include multiple modules, wherein each module implements a corresponding unit of an industrial control device, such as a queuing procedure and a real-time container agent. By way of example only, the computer program product may include an installable Linux qdisc, a Linux executable implementing a real-time container agent, and the like.

[0075] The embodiments and features described with reference to the industrial control equipment of the first aspect, with necessary modifications, are applicable to the industrial control system of the second aspect, the method of the third aspect, and the computer program product of the fourth aspect.

[0076] Further possible implementations or alternative solutions of the invention also encompass combinations of features described above or below with respect to the embodiments—combinations not explicitly mentioned herein. Those skilled in the art can also add individual or isolated aspects and features to the most basic form of the invention. Attached Figure Description

[0077] Further embodiments, features, and advantages of the invention will become apparent from the following description and dependent claims, taken in conjunction with the accompanying drawings, in which:

[0078] Figure 1 A schematic diagram illustrating the structural configuration of an industrial control system according to a first exemplary embodiment is shown;

[0079] Figure 2 A schematic diagram illustrating the structural configuration of an industrial control device according to a first exemplary embodiment is shown;

[0080] Figure 3 A schematic diagram illustrating the functional configuration of an industrial control device according to a first exemplary embodiment is shown;

[0081] Figure 4 A schematic diagram illustrating the functional configuration of an outbound queuing procedure according to a first exemplary embodiment is shown;

[0082] Figure 5 A schematic diagram illustrating the functional configuration of an entry queuing procedure according to a first exemplary embodiment is shown;

[0083] Figure 6 A timing diagram illustrating the flow of local network packets according to a first exemplary embodiment is shown;

[0084] Figure 7 The method steps of a method for operating an industrial control device according to a first exemplary embodiment are shown;

[0085] Figure 8 A timing diagram is shown to illustrate the flow of local network packets according to a comparable example;

[0086] Figure 9 A schematic diagram illustrating an exemplary functional configuration of an industrial control system according to a first exemplary embodiment is shown;

[0087] Figure 10 A timing diagram illustrating the flow of remote network packets according to a first exemplary embodiment is shown;

[0088] Figure 11 The method steps of a method for operating an industrial control system according to a first exemplary embodiment are shown;

[0089] Figure 12A timing diagram is shown to illustrate the flow of remote network packets according to a comparable example;

[0090] Figure 13 A schematic diagram illustrating the functional configuration of an industrial control system according to a second exemplary embodiment is shown;

[0091] Figure 14 The functional configuration of network packets according to a second exemplary embodiment is shown;

[0092] Figure 15 The illustration shows a configuration plan 500 for an industrial control system according to a second exemplary embodiment;

[0093] Figure 16 A schematic diagram illustrating the functional configuration of a queuing procedure according to a second exemplary embodiment is shown;

[0094] Figure 17 The illustration shows a configurable rule set according to a second exemplary embodiment;

[0095] Figure 18 The method steps of a method for operating an industrial control device according to a second exemplary embodiment are shown;

[0096] Figure 19 A timing diagram illustrating an example of reconfiguration in an industrial configuration according to a second exemplary embodiment is shown; and

[0097] Figure 20 A schematic diagram illustrating the functional configuration of an industrial control device based on a modified example is shown.

[0098] In the accompanying drawings, the same reference numerals denote the same or functionally equivalent elements, unless otherwise indicated. Detailed Implementation

[0099] Typical configuration of industrial control systems

[0100] Figure 1A schematic diagram of an industrial control system 1 according to a first embodiment is shown. The industrial control system 1 includes industrial control devices 100 and 200. Each of the industrial control devices 100 and 200 has a corresponding physical network interface 101, 201. The physical network interfaces 101 and 201 of the industrial control devices 100 and 200 are interconnected via a physical network 2. Each industrial control device 100 and 200 further includes physical input / output (I / O) ports 102 and 202. Corresponding technical devices 3 and 4 are connected to the corresponding I / O ports 102 and 202. The industrial control devices perform control of the technical devices 3 and 4 by inputting data to and outputting data from the technical devices 3 and 4 via the I / O ports 102 and 202. The processing of the input data and the processing of the data to be output as output data are performed by a control application executed at a constant cycle on the industrial control devices 100 and 200 of the industrial control system. Figure 1 (Not shown in the diagram) is used for execution. Control applications executed on different industrial control devices 100, 200 communicate with each other by exchanging network packets 7 on the physical network 2. The industrial control system 1 is configured such that the physical network 2 delivers network packets 7 to their destination within a time period equal to one of the lengths of a constant period. Therefore, the industrial control system 1 has real-time capability.

[0101] As a fairly simple example, device 3 could be a switch, and device 4 could be a light bulb. When switch 3 is pressed, industrial control system 100 reads the value indicating that switch 3 is pressed as input data via I / O port 102. Industrial control system 100 transmits a network packet 7 containing a command to turn on light bulb 4. Industrial control system 200 receives network packet 7 and outputs a signal via I / O port 202 that causes light bulb 4 to turn on. In reality, it goes without saying that devices 3 and 4 would be much more complex devices, such as robots, machines, and the like, including multiple sensors, actuators, and the like.

[0102] Figure 2 A schematic diagram illustrating the structural configuration of an industrial control device 100 according to a first exemplary embodiment is shown. The industrial control device 100 is a computing device including a processor 5, a memory 6, and a storage device 8. The industrial control device 100 further includes an I / O port 102 and a physical network interface 101. The processor 5, memory 6, storage device 8, physical network interface 101, and I / O port 102 are interconnected via at least one bus 9.

[0103] The computer program product can be stored in the storage device 8. The computer program product is loaded from the storage device 8 into the memory 8 and executed by the processor 6. When the computer program product is executed in this manner, the industrial control device 100 forms a system with… Figure 3 The various units of the functional configuration shown.

[0104] For example, industrial control equipment 100 can be The IOT2040 controller or something similar.

[0105] Figure 3 A schematic diagram illustrating the functional configuration of an industrial control device 100 according to a first exemplary embodiment is shown. The industrial control device 100 executes two control applications 113 and 123. The corresponding control applications 113 and 123 are containerized within corresponding containers 110 and 120, which isolate the containerized control applications 113 and 123 from each other and from the hardware of the industrial control device 100.

[0106] Specifically, containers 110 and 120 each form an abstraction layer of the hardware of the industrial control device 100, which includes, in particular, corresponding virtual I / O ports 112 and 122 and corresponding virtual network interfaces 111 and 121. For example, containers 110 and 120 can be implemented using Linux containers, LXC, Docker, and the like.

[0107] Virtual network interfaces 111 and 121 are interconnected with physical network interface 101. That is, network packets 7 sent by control application 113 via virtual network interface 111 can reach the virtual network interface 121 of another control application 123, and / or reach physical network interface 101 for use via physical network 2 (…). Figure 1 Further transmission.

[0108] In this document, the forwarding of network packet 7 from one of network interfaces 111, 121, and 101 to another of network interfaces 111, 121, and 101, and the forwarding of network packet 7 along physical network 2 ( Figure 1 Forwarding can be performed using techniques such as the kernel of the operating system of the corresponding control device 100, TCP / IP stack, device drivers, redirection, virtual bridges, and similar technologies. Within the industrial control device 100 and in the physical network 2 ( Figure 1 The forwarding of network packets on the virtual network interface 111 is performed in parallel with and in the background of the further processing described below. The precise timing of when network packet 7, which is queued at the virtual network interface 111 for transmission, is dequeued, forwarded, and when it arrives at another virtual network interface 112 or physical network interface 101 is at least somewhat uncertain.

[0109] The corresponding queuing procedures 150, 151, and 152 are associated with each of network interfaces 101, 111, and 121. Each queuing procedure is configured to enqueue network packet 7 upon request, manage network packet 7 in the queue, and dequeue network packet 7 upon request. For example, queuing procedures 150, 151, and 152 can be implemented as Linuxqdiscs.

[0110] Queuing procedures 151 and 152 are outgoing queuing procedures that apply only to network packets 7 that are being transmitted or going out. When the corresponding control applications 13 and 123 transmit network packet 7, they call the corresponding API of the network stack of the operating system of the industrial control device 100, such as the TCP / IP API. Internally, the operating system kernel then requests outgoing queuing procedure 151 to temporarily enqueue network packet 7. Queuing procedures 111 and 152 enqueue network packet 7 at the end of their queues. At different points in time, when virtual network interfaces 111 and 121 are ready to actually transmit network packet 7, a dequeue request is made to queuing procedures 151 and 152, and the first network packet 7 is dequeued from queuing procedures 111 and 152 and further transmitted.

[0111] Similarly, queuing procedure 151 is an inbound queuing procedure. When inbound network packet 7 is received by physical network interface 101, the kernel requests that network packet 7 be enqueued at the end of its queue in inbound queuing procedure 150. Later, when the kernel is ready to receive incoming network traffic, it requests that the foremost network packet 7 be dequeued from inbound queuing procedure 150.

[0112] It should be noted that the network packet 7 entering the queue in queuing procedure 151 can be sent from physical network interface 101 through the physical network ( Figure 1 2) The received remote inbound network packet 7, or the local inbound network packet 7 routed from one of the virtual network adapters 111, 121 to the other of the virtual network adapters 111, 121 via the physical network interface 101.

[0113] The functional configuration of industrial device 100 further includes a real-time container agent 104. The real-time container agent 104 is configured to create containers 110 and 120 to manage the execution of containers 110 and 120 at constant intervals, representing containerized control applications 113 and 123 performing I / O operations via physical I / O port 102, managing orchestration, i.e., appropriately configuring the forwarding, routing, network address translation, and multicasting of network packets 7, and ensuring the time-deterministic processing of network packets 7. The latter aspect will now be described in detail.

[0114] Deterministic delivery of network packets in time

[0115] Figure 4 A schematic diagram illustrating the functional configuration of an outbound queuing procedure 151 according to a first exemplary embodiment is shown. (Refer to...) Figure 3 and Figure 4 .

[0116] Outbound queuing procedure 151 manages network packets 71-73 in queue 91. Reference symbols 71-73 are used to specifically refer to a particular network packet within network packet 7 when network packet 7 is enqueued in queue 91. When network packets 71-73 are enqueued, a timestamp is associated with the newly enqueued network packet 71-73, and this timestamp is set to the current time at the moment the corresponding network packet 71-73 was enqueued in queue 91. Figure 4 The diagram illustrates queue 91, where network packets 71-73 are arranged chronologically based on their timestamps, with timestamps arranged from left to right and from oldest to newest; however, this is not required. Furthermore, an outgoing queuing procedure 151 maintains a barrier timestamp 17. When a dequeue request is received, the outgoing queuing procedure 151 checks if any network packet 71 held in queue 91 has a timestamp older than its barrier timestamp 17. If so, the network packet 71 with the oldest timestamp is dequeued in response to the dequeue request. If not, no packet is dequeued; that is, network packets 72-73 with newer timestamps than barrier timestamp 17 are blocked in queue 91.

[0117] The barrier timestamp 17 of the outbound queuing procedure 151 is managed by the real-time container agent 104. Specifically, when the outbound queuing procedure 151 receives a move command 14 from the real-time container agent 104, the outbound queuing procedure 151 moves the barrier timestamp 17. According to this exemplary embodiment, the move command 14 includes an indication 90 of a logical execution time, which is the start time of the current execution cycle. Therefore, the outbound queuing procedure 151 moves the barrier timestamp 17 to the right to the logical execution time indicated by the indication 90. If any network packet 72, 73 that was previously blocked is now more than the moved barrier timestamp 17 (now at...), the outbound queuing procedure 151 will move the barrier timestamp 17 to the right. Figure 4 If the barrier timestamp (top left of 17) is older, then they will no longer be blocked and can be dequeued when the next dequeue request is received.

[0118] In this document, the functionality of outbound queuing procedure 151 has been described in detail as an example. It should be understood that outbound queuing procedure 152 includes similar functionality.

[0119] Figure 5 A schematic diagram illustrating the functional configuration of an entry queuing procedure 150 according to a first exemplary embodiment is shown. (Refer to...) Figure 3 and Figure 5 .

[0120] Entry queuing procedure 150 manages two queues 92 and 93 for network packets 71-76. Reference symbols 71-76 are used to specifically refer to a particular network packet within network packet 7 when network packet 7 is enqueued in one of queues 92 and 93. When enqueuing network packets 71-76, entry queuing procedure 150 associates a timestamp indicating the current time with network packets 71-76. Furthermore, when enqueuing each of network packets 71-76, entry queuing procedure 150 determines whether the corresponding network packet 71-76 is a local network packet 71-73 that has been transferred via physical network interface 101 from one of containers 110 and 120 on industrial control equipment 100 to another container 110 or 120 on the same industrial control equipment 100. If network packet 71-76 is a local network packet 71-73, then that packet is enqueued in the first queue 92. Otherwise, network packets 74-76 are remote network packets 73-76 that have already been received via physical network interface 101. Remote network packets 73-76 are enqueued in the second queue 93. Both the first queue 92 and the second queue 93 are maintained in an ordered state of increasing timestamps from left to right. In addition, the entry queuing procedure 150 maintains two barrier timestamps 18 and 19. Barrier timestamp 19 is used for local network packets 71-73 in the first queue 92, and barrier timestamp 18 is used for remote network packets 74-76 in the second queue 93. When a dequeue request is received, the entry queuing procedure 150 checks whether queue 92 has a network packet 71 that is older than barrier timestamp 18 of the first queue 92, and whether the second queue 92 has a network packet 74 that is older than barrier timestamp 19 of the second queue 93. If any such network packets 71, 74 exist, the network packets 71, 74, 75 with the oldest timestamp are dequeued in response to the dequeue request. If not, no packets are dequeued, meaning network packets 73, 75, and 76 with newer timestamps than their corresponding barrier timestamps 18 and 19 are blocked in their respective queues 92 and 93. By having two separate queues 92 and 93 that maintain classification by timestamp, the queuing procedure 150 can advantageously and efficiently handle the different barrier timestamps 19 and 18 of the local packet queue 92 and the remote packet queue 93: upon receiving a dequeue request, only the oldest (leftmost) network packet 71 and 74 in each of the queues 92 and 93 needs to be checked for their corresponding barrier timestamps 18 and 19.

[0121] Therefore, when the queuing procedure 150 receives the move command 15 from the real-time container agent 102, the queuing procedure 150 moves the barrier timestamps 18 and 19. The move command 15 includes an indication 90 of the logical execution time. According to this exemplary embodiment, the queuing procedure 150 moves the barrier timestamp 19 of the local packet queue 92 to a time equal to the logical execution time indicated by the indication 90 plus the length of one period of a constant period. In other words, the barrier timestamp 19 of the local packet queue 92 is moved to the end time of the current period. In addition, the queuing procedure 150 moves the barrier timestamp 18 of the remote packet queue 93 to the logical execution time indicated by the indication 90, i.e., the start time of the current period. Thereafter, if any network packets 73, 75, 76 that were previously blocked are now older than the corresponding moved barrier timestamps 18, 18 of their respective queues 92, 93, they will no longer be blocked and can be dequeued when the next dequeue request is received.

[0122] The advantages of the interactions between the real-time container agent 104 and the queuing procedures 150, 151, and 152 disclosed above will now be referenced. Figure 6 This becomes even clearer in the description of the flows in the given local network packets 71-73.

[0123] Figure 6 A timing diagram is shown to illustrate the flow of local network packets 71-73 during two periods 10 and 20 according to a first exemplary embodiment. Figure 7 The method steps of operating an industrial control device 100 according to a first exemplary embodiment are shown.

[0124] refer to Figures 3 to 7 .exist Figure 6 In the diagram, processing phases 13, 16, 23, and 26 of the real-time container agent 104 are schematically visualized using a band 95. In each of the constant cycles, the real-time container agent 104 performs the following steps / runs through the following processing phases:

[0125] In step S1, the real-time container agent 104 initializes and determines the start time T1 of the current cycle, i.e., the logical execution time. After initialization, the real-time container agent 104 continues with maintenance processing 13. In step S4, the real-time container agent 104 executes output processing 11. During output processing 11, the real-time container agent 104 reads output data written by control applications 110 and 120 during previous cycles from virtual I / O ports 112 and 122, and outputs the output data to the technical device via physical I / O port 102. Figure 1(3) It should be noted that virtual I / O ports 112 and 122 can be implemented, for example, as a virtual Linux GPIO file system. Therefore, the real-time container agent 104 performs output processing 11 on behalf of containerized control applications 113 and 123. Furthermore, at some time t1 during output processing 11, the real-time container agent 104 transmits a move command 14 to outgoing queuing procedures 151 and 152 in step S5. In response to receiving the move command 14, outgoing queuing procedures 151 and 152 adjust their barrier timestamp 17 to the logical execution time T1 in step S6, and will dequeue any network packets 71 that are now older than the move barrier timestamp 17 that were enqueued therein when a subsequent dequeue request is received.

[0126] When output processing 11 is completed, in step S8, real-time container agent 104 begins executing input processing 12 at time t2. During input processing 12, real-time container agent 104 accesses the technical device (via physical I / O port 102). Figure 1 3) Read the input data and write the read input data to the virtual I / O ports 112 and 122 of containers 110 and 120. Therefore, the real-time container agent 104 performs input processing 12 on behalf of containerized control applications 113 and 123. Furthermore, at some time t3 during input processing 12, the real-time container agent 104 transmits a move command 15 to the entry queuing procedure 150 in step S9. In response to receiving the move command 15, the entry queuing procedure 150 adjusts the barrier timestamp 19 of its local queue 92 to the end time T2 of the first period 10 in step S10, and adjusts the barrier timestamp 18 of its remote queue 93 to the start time of the first period 10 in step S10. Thus, any local network packets 71 and 72 that are now older than the adjusted local barrier timestamp 19 and any remote network packets 74 that are now older than the adjusted remote barrier timestamp 18 are now ready to be dequeued upon subsequent dequeue requests.

[0127] In other words, at time point t4, when maintenance process 13 is completed, the barrier timestamp 17 of outgoing queuing procedures 151 and 152, the barrier timestamp 18 of remote queue 93 entering queuing procedure 150, and the barrier timestamp 19 of local queue 92 entering queuing procedure 150 are now as follows: Figure 6 It is set for the first cycle 10 as shown in the diagram.

[0128] Then, in step S12, after maintenance process 13 ends at time t4, real-time container agent 103 causes execution 16 of each of the control applications 113, 213 during the remainder of the first cycle 10, i.e., from time t4 until time T2. Real-time container agent 103 terminates and / or suspends control applications 113, 213 just before time T2. Therefore, the first cycle 10 ends.

[0129] Then, the second cycle 20 begins. The processes described above for the first cycle 10 are repeated for the second cycle 20. Specifically, the maintenance process 23 of the second cycle 20 includes input processing 21 and output processing 22; input processing 21 includes transmitting move commands 24, and output processing 22 includes transmitting move commands 25. When maintenance process 23 ends at time t9, the barrier timestamps 27, 28, and 29 of the corresponding queuing procedures 150, 151, and 152 are as follows: Figure 6 It is set for the second cycle 20 as shown in the diagram.

[0130] Now, examine the flow of network packet 7 in detail. For example... Figure 6 The network packet 7 shown is a network packet created by the control application 113 during its execution at time t5 in the processing phase 16 of the first cycle 10. Furthermore, as... Figure 6 The network packet 7 shown is a local network packet that will be delivered to the same industrial control device 100. Figure 1 , 2 Another control application 123 is executed on ).

[0131] At time t5, network packet 7 is enqueued into outgoing queuing procedure 151 associated with the first virtual network interface 111 and associated with timestamp t5. At time t5, the barrier timestamp 17 of outgoing queuing procedure 151 is set at the logical execution time T1 of the first cycle 10. Therefore, the timestamp t5 of network packet 7 is newer than the barrier timestamp 17 of outgoing queuing procedure 151. Therefore, network packet 7 is blocked in queue 91 of outgoing queuing procedure 151 for at least the remainder of the first cycle 10.

[0132] Then, at time t6 during the second period 20, movement command 24 is issued to the first outgoing queuing procedure 151, and the first outgoing queuing procedure 151 adjusts its barrier timestamp, which is now the barrier timestamp 27 at time T2>t5. Therefore, some amount of time after movement command 24 is issued at time t6, network packet 7 will be dequeued from the first outgoing queuing procedure 151. After that, industrial control device 100 ( Figure 1 , 2) is processed and forwarded as network packet 7 by the network stack of the operating system. In this exemplary embodiment, local network packet 7 is routed via the inner side of physical network interface 101. Thus, local network packet 7 will be enqueued in the common ingress queuing discipline 150, and associated with a new timestamp tx>t6. Herein, it is to be noted that at time t6, the barrier timestamp 19 of the local queue 92 of the ingress queuing discipline 150 is still set to T2. Since T2<t6<tx, network packet 7 will likely be blocked. However, at time t8, a move command 25 is transmitted to the ingress queuing discipline 150, in response to which the local queue 92 of the ingress queuing discipline 150 adjusts its barrier timestamp to become barrier timestamp 29, which is set at the end of the second period 20 at time T3. Since it can be safely assumed that T3>tx, network packet 7 with timestamp tx<T3 will now be released for dequeuing, even if it was initially blocked.

[0133] In particular, it is to be noted that the exact time tx at which network packet 7 is enqueued in the local queue 92 of the ingress queuing discipline 150 is unknown; it may be that tx>t8; in this case, network packet 7 will not be blocked at all; or it may be that tx<t8, in which case network packet 7 will first be blocked, but will be released shortly after t8. In either case, local network packet 7 will be released for dequeuing well within the second period 20. After dequeuing, local network packet 7 will be sent to local network interface 121, and will arrive at the control application 123 during its execution in the processing phase 26 of the second period 20.

[0134] Figure 8 shows a timing diagram for illustrating the flow of local network packet 7 according to a comparable example. Reference Figure 8 and Figure 3-5 .

[0135] In the comparable example, the move commands 14, 15 do not include the indication 90 of the logical execution time T1. Furthermore, the ingress queuing discipline 150 comprises only a single queue 93 for both remote and local network packets 7, and maintains a single barrier timestamp 18. At the respective points in time when the move commands 14 and 15 are received, the respective queuing disciplines 150, 151 and 152 set their barrier timestamps 17, 18 to the current time at which the respective move commands 14, 15 are received.

[0136] Thus, during the execution phase 16 of the first period 10, the barrier timestamps 17 of the egress queuing disciplines 151, 152 are set to time t1, and the barrier timestamp 18 of the ingress queuing discipline 150 is set to time t2, as Figure 8as shown in. Therefore, when the network packet 7 is created at time t5 and enqueued in the egress queuing procedure 151, it is blocked in the egress queuing procedure 151, similar to Figure 6 the case shown for the first exemplary embodiment in. However, in the second period 20, the processing of the network packet 7 differs from that in the first exemplary embodiment. That is, in the second period 20, at time t6, the barrier timestamp 27 of the egress queuing procedure 151 is moved to time t6>t5, and the network packet 7 is released from the egress queue 151. At some non-deterministic time after t6, the network packet 7 is enqueued in the ingress queuing procedure 150 and associated with a time tx>t6.

[0137] In a comparable example, at time t8, the barrier timestamp 28 of the ingress queuing procedure 150 is moved to time t8. Generally, the condition tx<t8 holds, because the local forwarding of the local network packet 7 is usually very fast. In this case, the network packet 7 will be released shortly after time t8 for dequeuing from the ingress queuing procedure 150. However, in the event of high load on the industrial control device 100 or the like where a race condition exists, the local network packet 7 may only be enqueued in the ingress queuing procedure 150 at a time tx that is also greater than t8. Now, if the local network packet 7 is enqueued with a timestamp tx>t8, the local network packet 7 will be blocked in the ingress queuing procedure 150 for the entire remainder of the period 20, and will only be released for dequeuing at a future point in time during a third period not shown.

[0138] That is, in the comparable example, the network packet 7 can either be released and received by the processing performed in the processing phase 26 of the second period 20, or in the case of high load, can only be released and received by the control application 123 in the processing phase of the third period. Therefore, the comparable example cannot guarantee time-deterministic delivery of the local network packet 7.

[0139] In contrast, thanks to the proposed logical execution time paradigm, in which the move commands 14, 15 comprise indications 90 of the respective logical execution times T1, T2, and the queuing procedures 150, 151, 152 adjust their barrier timestamps 17, 18, 19 based on the logical execution times T1, T2, and wherein in particular the ingress queuing procedure 150 manages a separate queue 92 for local networks, whose barrier timestamp 19 is set to the end 29 of the period 20, it is advantageously possible to ensure that, without any race conditions and in a time-deterministic manner, local network packets 7 are always received exactly one period after the period in which they have been transmitted.

[0140] Next, the flow of a remote network packet according to the first exemplary embodiment will be illustrated.

[0141] Figure 9 A schematic diagram of an exemplary functional configuration of the illustrated industrial control system 1 is shown. Figure 10 A timing diagram illustrating the flow of remote network packets 7 according to a first exemplary embodiment is shown, and Figure 11 The method steps of operating an industrial control system according to a first exemplary embodiment are shown. (Reference) Figures 9 to 11 .

[0142] Industrial control system 1 includes industrial control devices 100 and 200. Industrial control device 100 is connected to... Figure 10 The industrial control equipment shown and discussed above is the same as the industrial control equipment 200; in Figure 9 Some of its components are omitted. That is, as described above, industrial control device 100 includes two containerized control applications 113 and 123, each with access to virtual network interfaces 111 and 121, a physical network interface 101 interconnected with the virtual network interfaces 111 and 121, two outgoing queuing procedures 151 and 152 associated with the respective virtual network interfaces 111 and 121, an incoming queuing procedure 150 associated with the physical network interface 101, and a real-time container agent 104. Similarly, industrial control device 200 includes two containerized control applications 213 and 223, each with access to virtual network interfaces 211 and 221, a physical network interface 201 interconnected with the virtual network interfaces 211 and 221, two outgoing queuing procedures 251 and 252 associated with the respective virtual network interfaces 211 and 221, an incoming queuing procedure 250 associated with the physical network interface 201, and a real-time container agent 204. It should be noted that when referencing all queuing procedures, or any one of queuing procedures 150, 151, 152, 250, 251, 252, the common reference symbol 5050 can be used.

[0143] Now refer to Figures 9 to 11 This describes the flow of remote network packets 7 exchanged between industrial control devices 100 and 200.

[0144] exist Figure 10 In the diagram, processing stages 13, 16, 23, 26, 33, and 36 of the real-time container agent 104 of the first industrial control device 100 are schematically visualized using band 96. Processing stages 42, 46, 53, and 56 of the real-time container agent 204 of the second industrial control device 200 are schematically visualized using band 97.

[0145] The maintenance processes 13, 23, and 33 of real-time container agent 104 and the control application executions 16, 26, and 36 performed on the first industrial control device 100 during the corresponding periods 10, 20, and 30; and the maintenance processes 43, 53, and 63 of real-time container agent 204 and the control application executions 46, 56, and 66 performed on the second industrial control device 200 during the same corresponding periods 10, 20, and 30 are similar to those described above regarding... Figure 6 The maintenance procedures 13 and 23 and the control program execution 16 and 26 are described, and will not be described in detail again.

[0146] Similarly, Figure 11 The method of operating the industrial control system illustrated in the figure is similar to Figure 7 The method of operating industrial control equipment illustrated herein, and including performing steps S1, S4, S5, S6, S8, S9, S10, S12 in each of cycles 10, 20, and 30, wherein the steps are described above in relation to Figure 7 It has been described and will not be described in detail again.

[0147] However, the following facts should be noted: Figure 11 The method illustrated also includes an additional step S2, which is jointly performed by real-time container agents 104 and 204 immediately following initialization step S1. Step S2 involves synchronizing the start times T1, T2, and T3 of each cycle 10, 20, and 30 among the real-time container agents 104 and 204 of the industrial control devices 100 and 200 in the industrial control system 1. In other words, cycles 10, 20, and 30 in the industrial control system 1 are synchronized, and the logical execution times T1, T2, and T3, i.e., the start time of each of cycles 10, 20, and 30, are the same in each of the industrial control devices 100 and 200.

[0148] However, no fixed-time slices are reserved for maintenance processes 13, 23, 33, 43, 53, 63 and control app executions 16, 26, 36, 46, 56, 66. For example, in the second cycle 20, such as... Figure 10As shown, the maintenance process 53 performed by the real-time container agent 204 of the second industrial control device 200 (hereinafter also referred to as "second real-time container agent 204") takes more time than the maintenance process 23 performed by the real-time container agent 104 of the first industrial control device 100 (hereinafter also referred to as "first real-time container agent 104") in the second cycle 20. Therefore, the first real-time container agent 104 and the second real-time container agent 204 can issue corresponding movement commands 24, 25, 44, 45 at different time points t6, t7; t10; t12. By not fixing the time slices used for maintenance processes 13, 23, 33, the available time for control program execution 16, 26, 36, 46, 56, 66 in each cycle 10, 20, 30 of each of the industrial control devices 100, 200 is maximized. However, certain issues regarding timing and race conditions may potentially arise, which will be discussed in detail now.

[0149] More specifically, the flow of remote network packet 7 will be examined in detail below. Figure 10 The network packet 7 shown is a network packet created by the first control application 113 during its execution on the first control device 100 at time t5 in the processing phase 16 of the first cycle 10. Furthermore, as... Figure 11 The network packet 7 shown is a remote network packet to be delivered to a third control application 213 executed on another industrial control device 200.

[0150] On the first industrial control equipment 100 ( Figure 10 In the sequence (96), at time t5, network packet 7 is enqueued into the outgoing queuing procedure 151 associated with the first virtual network interface 111, and receives t5 as its associated timestamp. At time t5, the barrier timestamp 17 of the outgoing queuing procedure 151 is at logical execution time T1, meaning that the timestamp t5 of network packet 7 is newer than the barrier timestamp 17. Therefore, packet 7 is blocked in the outgoing queuing procedure 151 for at least the remainder of the first cycle 10.

[0151] Then, still on the first industrial control device 100, at time t6 during the second cycle 20, movement command 24 is issued to the first outgoing queuing procedure 151, and the first outgoing queuing procedure 151 adjusts its barrier timestamp to become the barrier timestamp 27 set at the new logical execution time T2. Time T2 is newer than time t5. Therefore, some time after movement command 24 is issued at time t6, network packet 7 will be dequeued from the first outgoing queuing procedure 151.

[0152] After that, the kernel of the operating system, the network stack, and the like of the industrial control devices 100, 200 operating in the background will take charge of the network packet 7, and route it through the physical network interface 101 and through the physical network 2 until it reaches the physical network interface 201 of the second industrial control device 200. There, the remote network packet 7 will be enqueued at time tx in the ingress queuing discipline 250 associated with the physical network interface 201 of the second industrial control device 200. That is, the remote network packet 7 in the ingress queuing discipline 250 is associated with the timestamp tx. Due to the delay of the physical network 2, the routing of the network packet 7 takes some time, and the time tx will be much later than the time t5. However, it is impossible to tell exactly when the time tx will be. The only guarantee made to ensure real-time capability is that the time tx will be less than the end time T3 of the second cycle 20, that is, tx<T3. Therefore, it is known that T2<t5<tx<T3.

[0153] At time tx, when the remote network packet 7 is enqueued in the ingress and egress queuing discipline 250 on the second industrial control device 200 ( Figure 10 the band 97 therein), the barrier timestamp of the ingress queuing discipline 250 will most likely be the barrier timestamp 58 set at T2. If the physical network 2 operates extremely fast, the barrier timestamp of the ingress queuing discipline 250 may even still be T1 (barrier timestamp 48). This is the case if tx<t12, that is, tx is before the time t12 at which the move command 55 is transmitted to the ingress queuing discipline 250, so that the barrier timestamp of the ingress queuing discipline 250 is moved to become the barrier timestamp 58 set at T2.

[0154] However, in either case, tx>T2>T1. Therefore, the network packet 7 will be blocked in the ingress queuing discipline 250 at least for the remainder of the second cycle 20. Then, in the third cycle 30 in band 97, at time t16, the move command 65 is issued to the ingress queuing discipline 250. In response, the ingress queuing discipline 250 adjusts its barrier timestamp to become the barrier timestamp 68 set at time T3. Now, the remote network packet 7 is ready to be dequeued from the ingress queuing discipline 250, because tx<T3.

[0155] In particular, it has been shown that no matter which delay the physical network 2 currently experiences, and even in the case of an unusually long maintenance process 53 on the second industrial control device 200, as long as tx<T3 can be guaranteed, the local network packet 7 will be delivered in the third cycle 30 for processing in the processing phase 66 of the third control application 213, which third cycle 30 is the second cycle after the first cycle 10 in which it has been transmitted. Therefore, time-deterministic delivery of the remote network packet 7 is advantageously possible.

[0156] Figure 12 A timing diagram illustrating the flow of local network packet 7 according to a comparable example is shown. (Reference) Figure 12 and Figure 9 .

[0157] In comparable examples, move commands 14, 15, 24, 25, 34, 35, 44, 45, 54, 55, 64, and 65 do not include indication 90 for logical execution time T1. Figure 4 , 5 Furthermore, queuing procedures 150 and 250 each consist of a single queue 93 for both remote and local network packets. Figure 5 ), and maintain a single barrier timestamp 18 ( Figure 5 When receiving movement commands 14, 15, 24, 25, 34, 35, 44, 45, 54, 55, 64, and 65, the corresponding queuing procedure 5050 sets its barrier timestamps 17, 18, 27, 28, 37, 38, 47, 48, 57, 58, 67, and 68 to the corresponding current time when the movement commands 14, 15, 24, 25, 34, 35, 44, 45, 54, 55, 64, and 65 were received at its location, as shown below. Figure 12 As shown in the image.

[0158] Therefore, in the first industrial control device 100, during the execution phase 16 of the first cycle 10 ( Figure 12 During the period of 96), the barrier timestamp 17 of the queuing procedure 151 is set to time t1, such as Figure 12 As shown in the diagram. Therefore, when network packet 7 is created at time t5 and enqueued in outgoing queuing procedure 151, it is blocked in outgoing queuing procedure 151, similar to... Figure 12 In the case shown in the first exemplary embodiment, since t5 > t1. However, in the second cycle 20, the processing of network packets 7 received on the second industrial control device 200 differs from the first exemplary embodiment.

[0159] In other words, in the second cycle 20, it is still on the first industrial control device 100 ( Figure 12 In the sequence 96), at time t6, the outgoing queuing procedure 151 adjusts its barrier timestamp to become the barrier timestamp 27 set at time t6 > t5. After some time, network packet 7 is dequeued from outgoing queuing procedure 151 and forwarded through physical network 2 until it reaches the second industrial control device 200. Figure 12 The 97 in the middle) is enqueued in the queuing procedure 250 and associated with the timestamp tx>t6.

[0160] In a comparative example, at time t12, the ingress queuing discipline 250 adjusts its barrier timestamp to become the barrier timestamp 58 set at time t12. Generally, the condition tx>t12 holds, because the remote forwarding of the remote network packet 7 is rather slow. In this case, the network packet 7 will be blocked in the ingress queuing discipline 250 associated with the physical network interface 201 of the second industrial control device 200 at least for the remainder of the second cycle 20, similarly to the above combined with Figure 12 the situation described for the first exemplary embodiment. Therefore, after the move command 65 causes the ingress queuing discipline 250 to adjust its barrier timestamp to become the barrier timestamp 68 at time t16, the network packet 7 will be ready to be dequeued in the third control cycle 30. That is, the network packet 7 will be made available to the control application 213 during its execution phase 66 in the third cycle 30.

[0161] However, in a scenario where the maintenance processing 53 of the second real-time container agent 204 takes a relatively large amount of time, for example due to many I / O operations being performed, and / or in a scenario where the physical network 2 operates rather fast, it may occur that the remote network packet 7 is enqueued in the ingress queuing discipline 250 at time tx, which is before time t12 at which the move command 55 is transmitted to the ingress queuing discipline 250, i.e. tx<t12. In this case, when the network packet 7 is enqueued at time tx, it is initially blocked in the ingress queuing discipline 250, because tx>t3 (barrier timestamp 48). However, when the barrier timestamp of the ingress queuing discipline 250 becomes the barrier timestamp 58 due to the move command 55 at time t12, the timestamp tx of the remote network packet 7 is now older than the barrier timestamp 58, i.e. tx<t12. Therefore, the remote network packet 7 will already be ready for dequeuing, and will be dequeued during the second cycle 20. As a result, the network packet 7 will have arrived at the control application 213 in its execution phase 56 in the second cycle 20.

[0162] Therefore, the comparative example cannot guarantee temporal deterministic behavior, because as has already been shown, the remote network packets 7 will be released in one cycle 20 or two cycles 30 after the cycle 10 in which they have been created.

[0163] In contrast, according to the first exemplary embodiment, thanks to the proposed logical execution time paradigm, it is possible to advantageously ensure that, without any race conditions and in a temporally deterministic manner, the local network packets 7 are always received exactly two cycles after the cycle in which they have been transmitted.

[0164] General discussion of orchestration requirements

[0165] Figure 13A schematic diagram illustrating the functional configuration of an industrial control system 1 according to a second exemplary embodiment is shown. The second exemplary embodiment is based on and compatible with the first exemplary embodiment. The structural and functional configurations described with respect to the first exemplary embodiment are also applicable to the second exemplary embodiment, and detailed descriptions thereof are omitted for brevity.

[0166] Industrial control system 1 includes a large number of N industrial control devices. As an example, a first industrial control device 100, a second industrial control device 200, and an Nth industrial control device 300 are shown. Each industrial control device 100, 200, and 300 performs multiple containerized ( Figure 13 For containers not shown in the image, see [reference]. Figure 3 Industrial control applications 113, 123, 213, 223, 313, and 323 will be collectively referred to by reference numeral 1313. Each control application 1313 communicates using corresponding virtual network interfaces 111, 121, 211, 221, 311, and 321. Each industrial control device 100, 200, and 300 has corresponding physical network interfaces 101, 201, and 301, and these physical network interfaces 101, 201, and 301 are interconnected via a physical network 2 for transmitting network packets 7 between the industrial control devices 100, 200, and 300.

[0167] Containerized control applications 1313 form distributed embedded control applications for controlling multiple technology devices ( Figure 13 Not shown in the image, see [link / reference]. Figure 1 (3 and 4 in the middle).

[0168] In a comparable example of "hard-wired" communication, each of control applications 1313 knows which network packet 7 it should transmit to in other control applications 1313. Therefore, control application 1313 can directly address all network packets 7 to the IP address, port, or similar address of a specific control application within control application 1313. However, at any time, if a hardware or software reconfiguration is made, such as when a technical device is moved to a different location, or when functionality is redistributed within a distributed embedded control application, or similar, each affected control application 1313 will need to be updated or reconfigured to reflect the changed or added IP address or port of its communication partner 1313. Let us assume, for example, that control application 113 is a control application that performs I / O using a sensor (not shown) and generates network packets 7 containing raw or pre-processed sensor data. It is expected that the code and configuration of control application 113 should not need to be updated, regardless of which or how many other control applications in control application 1313 might want to use the sensor data generated by control application 113. Therefore, the second exemplary embodiment has different methods regarding how network packets 7 are generated and addressed.

[0169] Figure 14 The functional configuration of network packet 7 according to a second exemplary embodiment is shown. (Reference) Figure 14 and Figure 13 Network packet 7 includes a packet header 701 and payload data 702. Network packet 7 may be a UDP packet. Packet header 701 includes routing information 710 and a data identifier 711. Routing information 710 may include the IP address, port, and the like of the receiving control application 1313. According to a second exemplary embodiment, when a control application such as control application 113 initially creates network packet 7, routing information 710 is set to a pseudo-value that does not indicate any existing destination. However, the newly created network packet 7 includes a data identifier 711, which identifies the type of data transmitted by network packet 7. By way of example only, data identifier 711 may identify that payload data 702 is sensor data from a specific sensor. Thus, data identifier 711 may include a data type ID, a sensor ID, and one or more of the like. Sensor data identifier 711 may be implemented as a separate data field in packet header 701. Alternatively, the sensor data identifier 711 can be encoded by pseudo routing information 710, that is, a specific pseudo destination IP address and / or port can be associated with each data type ID, sensor ID, and the like.

[0170] In other words, control application 1313 creates network packets 7 that do not include valid routing information 710 but include valid data identifiers 711. The data identifier 711 can be a numeric ID or other suitable identifier that identifies the type of data contained in payload 702, such as, for example, "temperature data from sensor number x", "switch number x is pressed", and the like. The task of correctly addressing and routing network packets 7 to all other control applications 1313 is left to the orchestration facility 550 of the industrial control network 1. The orchestration facility 550 has a configuration plan 500, which can be edited by a human user or through another technical device, configured to derive the communication mapping of the industrial network 1 from the configuration plan 500, and enable individual industrial control devices 100, 200, 300 to implement the communication mapping.

[0171] Figure 15 An exemplary configuration plan 500 of an industrial control system 1 according to a second exemplary embodiment is illustrated. (Reference) Figure 13-15For each of the control applications 1313, the configuration plan 500 includes corresponding control application entries 511 and 512. For each control application entry 511 and 512, the configuration plan 500 specifies a control application identifier 501 that identifies the corresponding control application 1313, a control device identifier 502 that specifies the industrial control devices 100, 200, and 300 on which the control application 1313 is installed, and specifies the container on which the control application 1313 is installed. Figure 3 The container identifier 503 in 110 and 120, multiple transmission data identifiers 504 each specifying a specific type of data transmitted by the corresponding control application 1313, and multiple receive data identifiers 505 each specifying a specific type of data that the corresponding control application 1313 wishes to receive. The formats of the transmission data identifiers 504 and receive data identifiers 505 can correspond to the above combination. Figure 14 The format of the data identifier 711 included in the described network group 7. It will be understood that it is possible to automatically derive the communication mapping of the industrial control network 1 from the configuration plan 500, that is, to specify which control applications 1313 transmit which types of data to which other control applications 1313.

[0172] Further reference Figure 13-15 The orchestration facility 550 derives communication mappings from the configuration plan 500 and transmits at least a corresponding portion of the communication mappings to the real-time container agents 104, 204, and 304 of the respective industrial control devices 100, 200, and 300. According to a second exemplary embodiment, the respective real-time container agents 104, 204, and 304 further include orchestration functions configured to implement the received portion of the communication mappings on the respective industrial control devices 100, 200, and 300.

[0173] In this paper, the idea is that after a network packet 7 without valid routing information 710 is sent by the corresponding control application 1313, some components of the corresponding industrial control devices 100, 200, 300 determine, based on a communication mapping, which one or more of their respective industrial control devices 100, 200, 300 the network packet 7 should be transmitted to. If more than one industrial control device 100, 200, 300 includes a control application 1313 using the type of data specified by data identifier 711, multicast is performed; that is, a copy of the network packet 7 is generated for each additional industrial control device 100, 200, 300 to which the network packet 7 is to be sent. Then, network address translation is performed to set the routing information 710 in the packet header 701 to the correct address for the network packet 7 or for each of the multicast copies of the network packet 7 for the corresponding industrial control device 100, 200, 300.

[0174] Similarly, when some components on receiving control devices 100, 200, and 300 receive an incoming remote network packet 7, they again examine the data identifier 711 of the received network packet 7 and, based on the communication mapping specified by configuration plan 500, determine which of the containerized local control applications 1313 on the local industrial control devices 100, 200, and 300 should receive the network packet 7. If more than one of the local control applications 1313 should receive the network packet, multicast is performed, that is, a copy of the network packet 7 is generated for each additional local control application 1313. Then, network address translation is performed on each copy of the network packet 7 to set the routing information 710 in the corresponding packet header 701 to the correct local address of the virtual network interface 111, 121, 211, 221, 311, 321 of the corresponding local control application 1313.

[0175] In this way, flexibility is gained when designing and redesigning distributed embedded control applications. When the communication mapping changes, a new configuration plan 500 can be provided, and the orchestration facility 550, in conjunction with real-time container agents 104, 204, 304 and additional components of industrial control systems 100, 200, 300, automatically adapts the addressing, multicasting, and routing of network packets 7 without requiring changes to the code and configuration of the actual control application 1313.

[0176] Firewalls such as Linux iptables are routinely used as "some components" to perform orchestration operations such as network switching, multicast, and the like. However, problems arise when time-deterministic dynamic updates are required.

[0177] Still referencing Figures 13-15 Let's consider a scenario where some control applications 1313 are updated to use a newer data protocol format for the same type of data, i.e., data with the same data identifiers 711, 504, and 505. For example, consider a scenario where a containerized control application 113 on industrial control device 100 transmits network packets 77 with a specified data format to a containerized control application 213 on industrial control device 200. Figure 16 Furthermore, consider that control application 123 is an updated version of control application 113, and control application 223 is a corresponding updated version of control application 213. The updated control applications 213 and 223 perform the same functions, but use different payload data 702 formats for their network packets 78. Figure 16When control applications 113 and 213 are updated to control applications 213 and 223, it is important that the orchestration is also updated in such a way that all packets transmitted from control application 113 arrive at control application 213, and all packets transmitted from control application 123 arrive at control application 223, and that in any given period, only one pair of control applications 113 and 213 or control applications 123 and 223 communicates with each other. As described with respect to the first exemplary embodiment, the fact that it takes two periods for any remote network packet 7 to propagate from its origin to its destination exacerbates this task.

[0178] Figure 16 A timing diagram is shown illustrating an example of reconfiguration in an industrial control system 1 according to a second exemplary embodiment. Specifically, Figure 16 The first industrial control device 100 is shown during four cycles of 10, 20, 30, and 40. Figure 13 The first real-time container agent 104 ( Figure 13 The processing phase 96 and the second real-time container agent 204 ( Figure 13 The processing stage is 97. (See reference 97.) Figure 13-16 .

[0179] In each of cycles 10, 20, 30, and 40, the corresponding real-time container agents 104 and 204 execute maintenance processes 13, 23, 33; 43, 54, 63, and 83. Each maintenance process includes output processes 11, 21, 31, 41, 51, 61, and 81, which include transmitting move commands 14, 24, 34, 44, 54, 64, and 84 to outbound queuing procedures 151 and 152. Figure 3 ); and input processing 12, 22, 32, 42, 52, 62, 82, including entry queuing procedures 150 for transmitting movement commands 15, 25, 35, 45, 55, 65, 85 to the corresponding industrial control devices 100, 200, 300. Figure 3 After maintenance processes 13, 23, 33, 43, 53, 63, and 83 are completed, control program execution 16, 26, 36, 45, 56, 66, and 86 are performed in each cycle 10, 20, 30, and 40. For details, refer to the description of the first exemplary embodiment.

[0180] Figure 6The control program execution stages 16, 26, 36, 46, 56, 66, and 86 indicate which of the two versions of control applications 113, 123; 213, and 223 is in service during the corresponding periods 10, 20, 30, and 40. Being in service specifically means that network packets generated by control application 1313 in service are transmitted over and received from physical network 2. Being out of service means that no network packets are routed from or to control application 1313 out of service.

[0181] Specifically, such as Figure 16 As shown, control application 113 exits service after cycle 10, while control application 123 is put into service during cycle 20. Correspondingly, after two cycles, control application 213 exits service after cycle 30, while control application 223 is put into service during control cycle 40.

[0182] In cycle 10, control application 113 creates network packet 77 at time t5 using an older data format. Network packet 77 is blocked in outgoing queuing procedure 151 until movement command 24 is issued to outgoing queuing procedure 151 in cycle 20. In cycle 20, network packet 77 is transmitted to control device 200 via physical network 2 and enqueued in incoming queuing procedure 250. There, it is blocked until movement command 65 is issued to incoming queuing procedure 250. Then, in the execution phase 66 of the second industrial control device 200 in the third cycle 30, network packet 77 is received by control application 213, which understands the older data format.

[0183] Accordingly, in cycle 20, control application 213 creates network packet 78 at time t17 using the newer data format. Network packet 78 is blocked in outgoing queuing procedure 151 until movement command 34 is issued to outgoing queuing procedure 151 in the third cycle 30. In the third cycle 30, network packet 78 is transmitted to control device 200 via physical network 2, where it is queued in incoming queuing procedure 250. There, it is blocked until movement command 85 is issued to incoming queuing procedure 250 in the fourth cycle 40. Then, in the execution phase 86 of the second industrial control device 200 in the fourth cycle 40, network packet 78 is received by the updated control application 223, which understands the newer data format.

[0184] In this document, it is important to note that during cycle 20, both network packet 77 (enqueued and then released by move command 25) destined for control application 213 and network packet 78 (newly enqueued) destined for control application 223 pass through the network stack of the first industrial control device. Furthermore, if control application 113 continues to execute, another network packet 77 may become newly enqueued during cycle 20; however, this other network packet 77 will not be routed anywhere and should be discarded.

[0185] Now, suppose that after control application 213 goes out of service, i.e., during maintenance process 23 of the second cycle 20, the first real-time container agent 104 simply reconfigures a standard firewall, such as Linux iptables, to stop re-addressing and forwarding network packets 77 with older data formats. In this case, network packet 77 (which was created in the first cycle 10 but is still in the outgoing queue procedure 151) Figure 9 The network packet 77, if enqueued in the queue, may not be forwarded to the second industrial control device 200 until some time after the move command 24 is issued in the second cycle 20 and enters the queuing procedure 151. More specifically, if network packet 77 only passes through the firewall after its reconfiguration, the readdressing rule matching network packet 77 may have been removed.

[0186] Even if the first real-time container agent 104 reconfigures the firewall only near the end of maintenance process phase 23, there is no guarantee that network packet 77 has been dequeued and has passed through the standard firewall at that point in time. Therefore, when orchestrating with a standard firewall, there are still race conditions that could disrupt time-deterministic behavior when reconfiguring during the operation of applications 122, 123, 213, and 213, which could cause network packet 77 to be lost during reconfiguration.

[0187] Temporal deterministic reconfiguration of communication mapping

[0188] According to the second exemplary embodiment, in order to achieve time-deterministic reconfiguration, real-time container agents 104 and 204 are not orchestrated using standard firewalls, but instead are orchestrated using queuing procedure 5050.

[0189] Figure 17 A schematic diagram illustrating the functional configuration of a queuing procedure 5050 according to a second exemplary embodiment is shown. The queuing procedure 5050 of the second exemplary embodiment is similar to the queuing procedure 5050 of the first exemplary embodiment, and similar features will not be described again.

[0190] In addition to the functions of managing network packets 71, 72, 73, queue 91, maintaining barrier timestamp 17, and blocking network packets 72 and 73 that are newer than barrier timestamp 71, queuing procedure 5050 further includes a filtering unit 94. The filtering unit 94 has a configurable rule set 600 and processes network packet 71 when it is dequeued from queue 91 according to the configurable rule set 600. That is, when in... Figure 17 When a dequeue request is received in the scenario shown, network packets 71 older than the barrier timestamp 17 will first be processed by the filtering unit 94 according to the configurable rule set 600 before being released from the queue 91.

[0191] Figure 18 The illustration shows a configurable rule set 600 according to a second exemplary embodiment. (Reference) Figure 17 and Figure 18 The configurable rule set 600 includes multiple rules 601. Each rule 601 includes a matching clause 610 and multiple action clauses 613 and 614. The matching clause 610 includes a group header criterion 611 and a timestamp criterion 612.

[0192] When network packet 71 is dequeued from queue 91 of queuing procedure 5050, filtering unit 94 compares the header criterion 611 of matching clause 610 with the header 701 of network packet 71. If header criterion 611, which may include regular expressions, wildcard statements, and the like, matches the header 701 of network packet 71, filtering unit 94 further compares the timestamp criterion 711 of matching clause 610 with the timestamp of network packet 71 associated with queue 91. Timestamp criterion 711 may include statements such as "older than...", "not older than...", "newer than...", "not newer than...", and the like. If network packet 71 matches both header criterion 611 and timestamp criterion 612, filtering unit 94 continues processing network packet 71 according to each of the action clauses 613, 614 specified for the corresponding rule 601.

[0193] In this way, using a matching clause 610 that includes both header standard 611 and timestamp standard 612, it is advantageous to filter network packets 71 that both have a certain data identifier 711 and belong to one or more specific time periods 10, 20, 30, 40 (…). Figure 16 (It has been created or received in one or more specific time periods 10, 20, 30, 40).

[0194] refer to Figures 13-18Action clauses 613 and 614 can specify one or more of the following actions:

[0195] The first configurable action a) is to perform network address translation on network packets 7 and 71. Network address translation refers to rewriting the routing information 710 included in the packet header 701. Action a) can be advantageously used for orchestration purposes. That is, the matching clause 610 can be used to match network packets 7 and 71 with a specific data identifier, and the action clauses 613 and 614 specifying action a) can be used to route the rewritten routing information 710 in the packet header 701 of network packets 7 and 71 in order to route network packets 7 and 71 to their correct recipient.

[0196] The second configurable action b) is to clone network packets 7 and 71 and add the cloned network packets to the candidate list. The third configurable action c) is to immediately release network packets 7 and 71 (dequeue network packets 7 and 71) if the network packets are added to the candidate list. In this document, adding to the candidate list may include adding a candidate list flag to the cloned network packets. Cloning may include creating a copy of network packets 7 and 71 and appropriately adjusting the routing information 710 in the copy of network packets 7 and 71. Actions b) and c) can be used in combination to perform multicast. That is, only a single network packet 7 and 71 can be dequeued at each dequeue request. However, if action b) is used to create a clone of network packets 7 and 71 to implement multicast, then the clone will then be dequeued immediately at the next dequeue request without undergoing further filtering using action c).

[0197] The fourth configurable action is to capture copies of network packets 7, 71 for later replay (action d). The capture and replay functions of queuing procedure 151 can be advantageously used to repair damaged links during reconfiguration. For example, consider the following reconfiguration where control switches back and forth between remote communication between control applications 1313 executed on different industrial devices 100, 200, 300, delivered after two cycles, and local communication between control applications 1313 executed on the same industrial control devices 100, 200, 300, delivered after one cycle. During the switching, a gap will occur where no network packets 7, 77, 78 are received during one of the cycles 10, 20, 30, 40. In this case, network packets 7 transmitted in the previous cycles 10, 20, 30, 40 can be captured and then retransmitted after one cycle 10, 20, 30, 40 to close the gap. The retransmission can be requested by the corresponding real-time container agents 104, 204, 304 to the corresponding queuing procedure 5050.

[0198] The fifth configuration action is action e): if the periodic quota for network packets 7 and 71 is exhausted in the current period 10, 20, 30, or 40, then network packets 7 and 71 are discarded. Configuring periodic quotas for network packets 7 and 71 that match specific criteria 611 and 612 may help avoid network congestion and ensure compliance with real-time requirements. For example, certain batch services may be considered optional, and if quotas are exceeded, they can be discarded without negative consequences. Alternatively, if the quota for basic network services is exceeded, an error condition can be generated, or an interruption can be implemented.

[0199] The configuration described for queuing procedure 5050 in the second exemplary embodiment applies to both entry queuing procedures 150, 250 and exit queuing procedures 151, 152, 251, 252 in the second exemplary embodiment. If queuing procedure 5050 maintains two separate queues 92, 93 (… Figure 5 For example, if a network packet 71 or 74 enters a queuing procedure 150 or 250, then any network packet 71 or 74 to be dequeued can be processed by the same filtering unit 94 using the same filtering rule set 600, or there can be different filtering rule sets 600 for local packet 71 to be dequeued and for remote packet 74.

[0200] In other words, the queuing procedure 5050 of the second embodiment implements the core firewall function, which can be advantageously used to perform orchestration in a time deterministic manner, that is, precisely at the time when network packet 71 is dequeued from the corresponding queuing procedure 5050, and more specifically, only for network packets 7, 71 that meet specific timestamp criteria 612 (belonging to specific periods 10, 20, 30, 40).

[0201] Figure 19 The method steps of operating industrial control devices 100, 200, and 300 according to a second exemplary embodiment are shown. (See reference...) Figure 13-19 The following will now describe how the real-time configuration agents 104, 204, and 304 of each of the industrial control devices 100, 200, and 300 perform orchestration using the firewall functionality described above in the queuing procedure 5050.

[0202] In each cycle 10, 20, 30, 40, steps S1 to S12 are performed on each of the industrial control systems 100, 200, 3000.

[0203] Steps S1, S2, S4, S5, S6, S8, S9, S10, and S12 correspond to the combination. Figure 7 and 11The steps of the method described in the first exemplary embodiment are as follows: Specifically, in step S1, real-time container agents 104, 204, and 304 are initialized. In step S2, real-time container agents 104, 204, and 304 synchronize the start times T1, T2, T3, and T4 of periods 10, 20, 30, and 40 with each other. In step S4, output processing 11, 21, 31, 41, 51, 61, and 81 is performed, which includes transmitting movement commands 14, 24, 34, 44, 54, 64, and 84 to the corresponding outgoing queuing procedures 5050, and the outgoing queuing procedures 5050 adjusting their barrier timestamps 17 in step S6. Figure 3 In step S8, after step S4 is completed, input processing 12, 22, 32, 42, 52, 62, 82 is performed, which includes transmitting movement commands 15, 25, 35, 45, 55, 65, 85 to the corresponding entry queuing procedure 5050 in S9, and the entry queuing procedure 5050 adjusts their barrier timestamps 18, 19 in S10. Figure 4 In step S12, after maintenance processes 13, 23, 33, 43, 53, 63, and 83 are completed, real-time container agents 104, 204, and 304 trigger the execution of the corresponding control applications 1313 that are containerized on the corresponding industrial control systems 100, 200, and 300 during the remainder of the corresponding cycles 10, 20, 30, and 40.

[0204] Furthermore, the method further includes a reconfiguration step S3 performed before the output processing step S4, a reconfiguration step S7 performed between the output processing step S4 and the input processing step S8, and a reconfiguration step S11 performed after the input processing step S8.

[0205] During each of the reconfiguration steps S3, S7, and S11, if necessary, the corresponding real-time container agents 104, 204, and 304 can update their communication mappings by reconfiguring the configurable rule set 600 of the corresponding queuing procedure 550. Figure 16 In the timing diagram, the time points before, between, and after the output processing S4 and the input processing S8—at which reconfiguration steps S3, S7, and S11 can be performed—are indicated as corresponding reconfiguration hooks 901, 902, 903, 904, 905, and 906.

[0206] For reference above Figure 16 In the specific example discussed, a time-deterministic reconfiguration is implemented, performing the following processing:

[0207] In the second period 20, on the first industrial control device 100 (with 96), at the reconfiguration ring 901 before the output processing 21, the real-time container agent 104 reconfigures the outgoing queue 151. More specifically, the real-time container agent 104 locates the existing rule 601 of the configurable rule set 600 of the outgoing queue 151, which performs network address translation via one of its action clauses 613, 614, said network address translation causing the network packet 77 transmitted by the control application 113 to be routed towards the second industrial control device 200 via the physical network interface 101. The real-time container agent 104 then changes the timestamp 612 criterion of the located rule 601 to "timestamp <= T2". The real-time container agent 104 further marks the rule 601 for deletion after time T3. In this way, the rule 601 remains active throughout the second period 10, and will apply to the network packets 77 associated with timestamp t5 < T2. However, the rule 601 will advantageously not apply to network packets 77, 78 generated in the second period 20 and having a timestamp newer than T2. Furthermore, after the second period 20 ends, the rule 601 will be deleted. The deletion of the rule 601 marked for deletion may be performed, for example, by a garbage collection process that is executed every period, or at each reconfiguration, or at predetermined time intervals, or the like.

[0208] Furthermore, also at the reconfiguration ring 901, the real-time container agent 104 creates a new rule 602 in the configurable rule set 600 of the outgoing queue 152 of the first industrial control device 100, the new rule 602 having action clauses 613, 614 that perform network address translation to cause the network packet 78 transmitted by the control application 123 to be routed towards the second industrial control device 200 via the physical network interface 101. The timestamp criterion 612 of the new rule 602 is set to "timestamp > T2". In this way, the newly configured rule 602 will apply to the network packets 78 transmitted by the control application 123 in the second period 20, but will not apply to any similar network packets that may have already been enqueued in the outgoing queuing procedure 152 due to previous execution of the control application 123 when it was not yet in service, such as during the first period 10.

[0209] Similarly, in the fourth cycle 40, on the second industrial control device 200 (with 97), at the reconfiguration ring 905 between the output processing 84 and the input processing 85, the real-time container agent 204 reconfigures the ingress queuing procedure 250. More specifically, the real-time container agent 204 locates the existing rule 601 of the configurable rule set 600 of the ingress queue 150, which performs network address translation via one of its action clauses 613, 614, the network address translation causing the network packet 77 received via the network 2 to be routed via the physical network interface 201 to the virtual network interface 211 used by the control application 213. The real-time container agent 204 then changes the timestamp 612 criterion of the located rule 601 to "timestamp < T3" and / or deletes the located rule 601. In this way, network packets 77 that still arrive in cycle 30 will not be forwarded to the control application 213 starting from the fourth cycle 40.

[0210] Furthermore, still at the reconfiguration ring 905, the real-time container agent 204 creates a new rule 602 in the configurable rule set 600 of the egress queuing procedure 252 of the second industrial control device 200, the new rule 602 having action clauses 613, 614 that perform network address translation to cause the network packet 78 received via the network 2 to be routed via the physical network interface 201 to the virtual network interface 212 used by the control application 223. The timestamp criterion 612 of the new rule 602 is set to "timestamp > T3". In this way, when the network packet 78 is to be dequeued in the fourth cycle after the move command 85 has already been issued, the newly configured rule 602 will apply to the network packet 78 even if it carries a timestamp T3 < tx < T4, since it was already received in the previous cycle 30 and enqueued in the ingress queue 250.

[0211] Referring above to Figure 16 the reconfiguration processing described is merely an example. A person skilled in the art will appreciate that even for the cases described above, the reconfiguration rings 901 and 905 are not the only possible reconfiguration rings 901-906 that can be used to perform the reconfiguration. Furthermore, a large number of different reconfiguration scenarios are conceivable, such as moving a control application 1313 from one industrial control device to another and the like.

[0212] In each such case, the precise decision as to which rule 601 is to be configured with which timestamp criterion 611 at which of the reconfiguration rings 901-906 can be made by a human operator and can be appended to the configuration plan 500. Alternatively, these decisions can be made automatically by the orchestration engine 500 and / or the real-time configuration agents 104, 204, 304 based on automated analysis of the configuration plan 500 and / or the communication mapping derived therefrom.

[0213] In summary, according to the second exemplary embodiment, when the correct rules 601 and 602 are configured with the correct timestamp standard 612, then for each conceivable reconfiguration scenario, it is possible to formulate rules 601 and 602 such that reconfiguration is performed in a time deterministic manner, wherein only desired network packets 77 and 78 are forwarded and unwanted network packets are discarded, and wherein there are no gaps or contention situations.

[0214] The effectiveness of this technology is particularly facilitated by the following features: rule set 600 is executed at a defined time point, i.e., when the network dequeues from the corresponding queuing procedure 5050; reconfiguration can be executed at a defined time point, i.e., before output processing S4, between output processing S4 and input processing S8, or after input processing S8; and each configurable rule can have a timestamp standard 611 that specifies a time aspect that defines which network packets 7 the rule will be applied to based on the associated timestamps of network packets 7.

[0215] Although the invention has been described with reference to preferred embodiments, it will be apparent to those skilled in the art that modifications are possible in all embodiments.

[0216] In the first exemplary embodiment, inbound queuing procedure 150 is associated with physical network interface 101, and corresponding outbound queuing procedures 151, 152 are associated with corresponding virtual network interfaces 111, 121. However, other configurations are possible. For example, if multicast is not required for inbound network packet 7, no queuing procedure can be associated with physical network interface 101, and the corresponding inbound queuing procedure 150 can be associated with each of the virtual network interfaces 111, 121. Alternatively, if local communication between multiple control applications is not desired, or if time determinism is not required for such local communication, it is also conceivable not to have a barrier-enabled queuing procedure associated with virtual network interfaces 111, 121, and not to associate both a single outbound queuing procedure 151 and a single inbound queuing procedure 152 with physical network interface 101.

[0217] Figure 20A functional configuration of another possible modified example of an industrial control system 100 compatible with the first or second exemplary embodiment is shown. Also in the modified example, outgoing queuing procedures 151, 152 are associated with each of the virtual network interfaces. However, unlike the configuration shown for the exemplary embodiment, in the modified example, virtual network interfaces 111, 121 and physical network interface 101 are interconnected using a virtual bridge 103. In this way, outgoing network packets 7 dequeued from outgoing queues 151, 152 are placed on the virtual bridge 103 and directly reach their destination (which may be physical network interface 101 or another of virtual network interfaces 111, 121). An additional pseudo-virtual network interface 108 is provided, and a common ingoing queuing procedure 150 is associated with pseudo-virtual network interface 108. Local network packets transmitted from control applications 113, 123 to another of control applications 113, 123 are not placed on the virtual bridge 103, but are redirected from virtual network interfaces 11, 121 to virtual network interface 108 via redirections 106, 107. Remote inbound network packets arriving at physical network interface 101 are not placed on virtual bridge 103, but are instead redirected to virtual network interface 108 via redirection 105. Upon arrival at virtual network interface 108, local or remote inbound network packets are queued in common inbound queuing procedure 150. The pseudo-network interface is unidirectionally connected to virtual bridge 103, such that local or remote inbound network packets that later dequeue from common inbound queuing procedure 150 are placed on virtual bridge 103 and can then directly reach their final destination—either of virtual network interfaces 111 or 121.

[0218] It has a public entry queuing procedure that is directly associated with the pseudo-virtual network interface 108. Figure 20 The modified example configuration shown advantageously allows public access queuing procedure 150 to be indirectly associated with each of virtual network interfaces 111, 121 and physical network interface 101 using redirects 105, 106, 107. This configuration is suitable for operating systems that only allow queuing procedures to be directly associated with a single network interface. Furthermore, bridging and redirection may be more efficient than routing. The modified example advantageously avoids internal routing of local network packet 7 via physical network interface 101.

[0219] In the exemplary and modified examples, the inbound queuing procedure 150 manages two separate queues 92, 93, one for local network packets 7 and one for remote network packets 7. However, it is also possible to manage local and remote inbound network packets in a single queue 91 while still applying different barrier timestamps 18, 19 to the local versus remote network packets 7 in the single queue 91. Furthermore, if accelerated delivery in the next cycle of the local network packets 7 is not required, the inbound queuing procedure 150 may also use only a single barrier timestamp 17 with a single queue 91 and adjust the single barrier timestamp 17 to the local execution time, as indicated in move command 15 90. In this case, the local network packets 7 will also be delivered in the second cycle after in which they have been created, similar to the remote network packets 7. Such a configuration may even be advantageous because when a receiving control application 123 that has already received local network packets from a transmission control application 123 on the same industrial control device 100 is moved to a different industrial control device 200, there will be no gaps in communication, and capture and replay are not necessary.

[0220] In an exemplary embodiment, when network packet 7 is enqueued in the corresponding queuing procedures 150, 151, and 152, a timestamp is associated with network packet 7 through queuing procedures 150, 151, and 152. However, it is also possible that network packet 7 is associated with a timestamp when it is created locally (outgoing packet) and / or when it is received (incoming packet), for example, through the network driver, network stack, or similar mechanism of the operating system of industrial control device 100. In this case, these pre-existing timestamps can be used by the corresponding queuing procedures 150, 151, and 152. The described effect is achieved as long as the timestamp is not transmitted over the physical network and / or updated when received via the physical network 2; that is, as long as the timestamp allows for determining the period of creation of outgoing network packets and the period of reception of incoming network packets.

[0221] The Linux operating system has been mentioned in various places, and Linux terminology such as "qdisc" has been used. However, those skilled in the art will understand that any operating system with networking capabilities, including free operating systems, proprietary operating systems, POSIX and non-POSIX operating systems, can be used to implement the disclosed features. Furthermore, features disclosed as software can also be implemented in hardware, such as through custom ASICs, custom processing units, and the like.

[0222] exist Figure 3 , 9In 1 and 13, the corresponding industrial control devices 100, 200 include two control applications 113, 123 or 213, 223. However, it will be understood that any industrial control device in industrial control devices 100, 200, 300 may also include only a single control application 1313, or may include three or more control applications 1313.

[0223] Additional References:

[0224] [1]Telschig, Knapp: "Towards Safe Dynamic Updates of DistributedUpdates of Distributed Embedded Applications in Factory Automation", 2017,22 nd IEEE International Conference on Emerging Technologies and Factory Automation

[0225] [2] Telschig, Knapp:"A Real-Time Container Architecture for Dependable Distributed Embedded Applications",2018,14 th IEEE International Conference on Automation Science and Engineering (CASE)

[0226] [3]Telschig, Knapp: "Synchronous Reconfiguration of DistributedEmbedded Applications during Operation", 2019, IEEE International Conference onSoftware Architecture (ICSA)

[0227] [4]Telschig, Knapp: "Time-Critical State Transfer during Operation ofDistributed Embedded Applications", 2019,17 thIEEE International Conference on Industrial Informatics(INDIN)

[0228] Reference figures

[0229] 1 Industrial Control Systems

[0230] 2 Physical Network

[0231] 3. Technical Equipment

[0232] 4. Technical Equipment

[0233] 5 processors

[0234] 6 memory

[0235] 7 Network Packets

[0236] 8 storage devices

[0237] 9-bus

[0238] 10 First Cycle

[0239] 11 Output Processing

[0240] 12 Input Processing

[0241] 13 Maintenance and Handling

[0242] 14 Move Commands

[0243] 15. Move Command

[0244] 16 Control Application Execution

[0245] 17 Barrier Timestamp

[0246] 18 Barrier Timestamps

[0247] 19 Barrier Timestamps

[0248] 20 Second Cycle

[0249] 21 Output Processing

[0250] 22 Input Processing

[0251] 23 Maintenance and Handling

[0252] 24. Move command to the exit queue procedure

[0253] 25. Move to the queueing procedure.

[0254] 26 Control Application Execution

[0255] 27 Barrier Timestamp

[0256] 28 Barrier Timestamps

[0257] 29 Barrier Timestamps

[0258] 30 Third cycle

[0259] 31 Output Processing

[0260] 32 Input Processing

[0261] 33 Maintenance and Handling

[0262] 34 Move Commands

[0263] 35. Move command

[0264] 36 Control Application Execution

[0265] Barrier timestamps 37 and 38

[0266] 40 Fourth Period

[0267] 41 Output Processing

[0268] 42 Input Processing

[0269] 43 Maintenance and Handling

[0270] 44 Move Command

[0271] 45. Move command

[0272] 46 Control Application Execution

[0273] Barrier timestamps 47 and 48

[0274] 51 Output Processing

[0275] 52 Input Processing

[0276] 53 Maintenance and Handling

[0277] 54 Move Command

[0278] 55 Move Command

[0279] 56 Control Application Execution

[0280] Barrier timestamps 57 and 58

[0281] 61 Output Processing

[0282] 62 Input Processing

[0283] 63 Maintenance and Handling

[0284] 64 Move Commands

[0285] 65 Move Command

[0286] 66 Control Application Execution

[0287] 71-78 Network Packets

[0288] 81 Output Processing

[0289] 82 Input Processing

[0290] 83 Maintenance and Handling

[0291] 84 Move Command

[0292] 85 Move Command

[0293] 86 control application execution

[0294] 90 indicates the timestamp of the logic execution time.

[0295] 91 queue

[0296] 92 local queues

[0297] 93 Remote Queues

[0298] 94 filter units

[0299] 95. Processing phase of the real-time container agent

[0300] 96 Processing stage on the first industrial control equipment

[0301] 97 Processing stage on second industrial control equipment

[0302] 100 Industrial Control Equipment

[0303] 101 Physical Network Interface

[0304] 102 physical I / O ports

[0305] 103 Virtual Bridge

[0306] 104 Real-time Container Proxy

[0307] 105-107 Redirect

[0308] 108 pseudo-virtual network interfaces

[0309] 110 container

[0310] 111 Virtual Network Interface

[0311] 112 Virtual I / O Ports

[0312] 113 Control Application

[0313] 122 Virtual I / O Ports

[0314] 121 Virtual Network Interface

[0315] 123 Control Application

[0316] 150 enter the queuing procedure

[0317] 151 Queueing Procedures

[0318] 152 Queueing Procedures

[0319] 250 enter the queuing procedure

[0320] 251 Queueing Procedures

[0321] 252 Queueing Procedures

[0322] 200 Industrial Control Equipment

[0323] 201 physical network port

[0324] 202 physical I / O ports

[0325] 204 Real-time Container Proxy

[0326] 211 Virtual Network Interface

[0327] 213 Control Application

[0328] 221 Virtual Network Interface

[0329] 223 Control Application

[0330] 300 Industrial Control Equipment

[0331] 311 Virtual Network Interface

[0332] 313 Control Application

[0333] 321 Virtual Network Interface

[0334] 323 Control Application

[0335] 500 Configuration Plan

[0336] 501 Control Application Identifier

[0337] 502 Control Device Identifier

[0338] 503 Container Identifier

[0339] 504 Transmit Data Identifier

[0340] 505 Receive Data Identifier

[0341] 511 and 512 Control Application Entries

[0342] 550-unit scheduling facility

[0343] 600 configurable rule sets

[0344] Rule 601

[0345] 610 Matching Clause

[0346] 611 Head Standard

[0347] 612 timestamp standard

[0348] Action Clauses 613 and 614

[0349] 701 Group Head

[0350] 702 Payload Data

[0351] 710 Routing Information

[0352] 711 Data Identifier

[0353] Reconfiguration ring before 901 output processing

[0354] Reconfiguration loop between 902 I / O processes

[0355] Reconfiguration ring after 903 input processing

[0356] Reconfiguration ring before 904 output processing

[0357] Reconfiguration loop between 905I / O processes

[0358] Reconfiguration ring after 906 input processing

[0359] 1313 (one or more) Control applications (general / common reference symbols)

[0360] 5050 (one or more) queuing procedures (general / common reference symbols)

[0361] S1-S12 Method Steps

[0362] t time

[0363] T1-T4 cycle start time, logic execution time

[0364] t n Time point

Claims

1. An industrial control device (100) for an industrial control system (1) comprising a plurality of industrial control devices (100, 200), said industrial control device (100) being configured to execute a plurality of control applications (113, 123) at a constant cycle, and comprising: - Multiple containers (110, 120), each container containerizing a corresponding one in the control application (113, 123). - Multiple network interfaces (101, 111, 121), including the physical network interface (101) of the industrial control device (1000) and multiple virtual network interfaces (111, 121) of the corresponding containers (110, 120). - Multiple queuing procedures (150, 151, 152), each queuing procedure associated with at least one of network interfaces (101, 111, 121), and configured to manage network packets (7) entering or leaving via the corresponding network interface (101, 111, 121) in queues (91, 92, 93), and to block network packets in queues (91, 92, 93) having timestamps older than the barrier timestamps (17, 18, 19) maintained by the corresponding queuing procedure (150, 151, 152). - A real-time container agent (104) configured to perform maintenance processing (13, 23) in each cycle (10, 20) and to cause the execution (16, 26) of each of the control applications (113, 123) only after the maintenance processing (13, 23) has been completed, wherein the maintenance processing (13, 23) includes: - Representing control applications (113, 123), output processing (11, 21) and input processing (12, 22) are performed using technical devices (3) connected to the industrial control equipment (100), and - Transmit a movement command (14, 15) to each of the queuing procedures (150, 151, 152), the movement command instructing the corresponding queuing procedure (150, 151, 152) to advance its barrier timestamp (17, 18, 19), wherein The movement commands (14, 15) include a timestamp (90) indicating the logical execution time, which is the start time (T1, T2) of the corresponding period (10, 20), and The corresponding queuing procedures (150, 151, 152) are configured to adjust their barrier timestamps (17, 18, 19) in response to receiving a move command (14, 15) based on the logical execution time indicated by the timestamp (90) included in the move command (14, 15).

2. The industrial control equipment according to claim 1, The maintenance processes (13, 23) include: - Synchronize the start time (T1, T2) of each cycle (10, 20) with the real-time container agent (204) of the other industrial control devices (200) of the industrial control system (1).

3. The industrial control equipment according to claim 1 or 2, The corresponding queuing procedures (150, 151, 152) are configured to associate a timestamp indicating the current time with the network packet (7) when enqueuing a network packet (7) that does not yet have a timestamp.

4. The industrial control equipment according to any one of claims 1 to 3, The corresponding queuing procedures (150, 151, 152) are configured to set the logical execution time of the current cycle (10) or the logical execution time of the next cycle (20) as its barrier timestamp (17, 18, 19).

5. The industrial control equipment according to any one of claims 1 to 4, The plurality of queuing procedures (150, 151, 152) include: - A public entry queuing procedure (150), associated with the physical network interface (101) of the industrial control device (100), and configured to manage network packets (7) entering through the physical network interface (101) in queues (92, 93), and - The corresponding outgoing queuing procedure (151, 152) is associated with each of the virtual network interfaces (111, 121) in each of the containers (110, 120) and is configured to manage the network packets (7) that are transmitted in the queue (91) via the corresponding virtual network interface (111, 121).

6. The industrial control equipment according to claim 5, The public entry queuing procedure (150) is further associated with each of the virtual network interfaces (111, 121), which includes a first queue (92) and a second queue (93), and is configured to: - Enqueue the local network packets (7, 71-73) that are incoming via one of the virtual network interfaces (111, 121) and outgoing via another of the virtual network interfaces (111, 121) into the first queue (92). - Enqueue the remote network packets (7, 74-76) received via the physical network interface (101) into the second queue (93). - Maintain a first barrier timestamp (19) for blocking local network packets (71-73) in the first queue (92) and a second barrier timestamp (18) for blocking remote network packets (74-76) in the second queue (93), and - In response to receiving the move command (15), the first barrier timestamp (19) is set to the logical execution time of the next cycle (20), and the second barrier timestamp (18) is set to the logical execution time (T1) of the current cycle (10).

7. The industrial control equipment according to any one of claims 5 or 6, The maintenance process (13) includes the following steps in the following order: The output processing step involves using the technical device (3) to perform output processing (11) and transmitting the movement command (14) to each of the outbound queuing procedures (151, 152), and The input processing step involves using the technical device (3) to perform input processing (12) and transmitting the movement command (15) to the public entry queuing procedure (150).

8. The industrial control equipment according to any one of claims 1 to 7, The corresponding queuing procedures (150, 151, 152) are configured to process network packets (7, 71-78) according to a configurable rule set (600) when dequeuing network packets (71).

9. The industrial control equipment according to claim 8, The configurable rule set (600) can, depending on the configured content, cause one or more of the following types of processing for network packets (7, 71-78): - Perform network address translation on network packets (7, 71-78); -Clone network packets (7, 71-78) and add the cloned network packets (7, 71-78) to the list of candidates; - If network packet (7, 71-78) is listed as a candidate, release network packet (7, 71-78) immediately; - Capture copies of network packets (7, 71-78) for later replay; and - If the per-cycle quota for network packet (7, 71-78) is exhausted in the current cycle (10, 20, 30, 40), then discard network packet (7, 71-78).

10. The industrial control equipment according to claim 8 or 9, The real-time container agent (104) is configured to realize the communication mapping of the industrial control system (1) belonging to the industrial control device (100) by configuring the corresponding rule (601) in the configurable rule set (600) of the corresponding queuing procedure (150, 151, 152) during the maintenance process (13).

11. The industrial control device according to any one of claims 8 to 10, The matching clause (610) that can be configured for each configurable rule (601) of the configurable rule set (600) includes at least one criterion (611) to match the packet header (701) of the network packet (7, 71-78) and at least one criterion (612) to match the timestamp of the network packet (7, 71-78).

12. The industrial control equipment according to any one of claims 8 to 11, The maintenance process (13) includes the following steps in the following order: The output processing step involves using the technical device (3) to perform output processing (11) and transmitting the movement command (14) to each of the outgoing queuing procedures (151, 152); as well as The input processing step involves using the technical device (3) to perform input processing (12) and transmitting the move command (15) to the entry queuing procedure (151, 152). The real-time container agent (104) is configured to implement communication mapping by configuring a configurable set of rules (600) of corresponding queuing procedures (150, 151, 152) at one or more of the following time points: before output processing (11); between output processing (11) and input processing (12); and after input processing (12).

13. An industrial control system (1) comprising a plurality of industrial control devices (100, 200) according to any one of the preceding claims, wherein their physical network interfaces (101, 201) are interconnected via a physical network (2).

14. A method of operating an industrial control system (1), the industrial control system (1) comprising a plurality of industrial control devices (100, 200), each industrial control device being configured to execute a plurality of control applications (113, 123, 213, 223) at a constant cycle, each industrial control device (100, 200) comprising: - Multiple containers (110, 120), each container containerizing a corresponding one of the control applications (113, 123, 213, 223). - Multiple network interfaces, including physical network interfaces (101, 201) for the corresponding industrial control devices (100, 200) and multiple virtual network interfaces (111, 121, 211, 221) for the corresponding containers (110, 120). - Multiple queuing procedures (150, 151, 152, 250, 251, 252), each queuing procedure associated with at least one of network interfaces (101, 111, 121, 201, 211, 221), and configured to manage network packets (7) entering or exiting in queues (91, 92, 93) via the corresponding network interface (101, 111, 121, 201, 211, 221), and to block network packets (7) in queues (91, 92, 39) having timestamps older than the barrier timestamps (17, 18, 19) maintained by the corresponding queuing procedures (150, 151, 152, 250, 251, 252), and - Real-time container agent (104, 204), The method includes - Each of the real-time container agents (104, 204) in the industrial control devices (100, 200) globally synchronizes the start times (T1, T2, T3) of each cycle (10, 20, 30) with each other (S2) across the industrial control system (1). After this, in each cycle (10, 20, 30), each real-time container agent (104, 204) performs maintenance processes (S3-S11), and only after the maintenance processes (S2-S11) have been completed does it trigger the execution of each in the control application for the remainder of the cycle (10, 20, 30) (S12), wherein the maintenance processes (S2-S11) include: - Control applications (113, 123, 213, 223) representing the corresponding industrial control devices (100, 200) utilize technical devices (3, 4) connected to the corresponding industrial control devices (100, 200) to perform output processing (S4) and input processing (S8), and - Move commands (14, 15, 24, 25, 34, 35, 44, 45, 54, 55, 64, 65) are sent to each transmission (S5, S9) in the queuing procedures (150, 151, 152, 250, 251, 252) of the corresponding industrial control equipment (100, 200), the move commands instructing the corresponding queuing procedures (150, 151, 152, 250, 251, 252) to advance their barrier timestamps (17, 18, 19), wherein The movement command includes a timestamp (90) indicating the logical execution time, which is the start time (T1, T2, T3) of the corresponding period (10, 20, 30), and The method further includes: - The corresponding queuing procedure, in response to receiving the movement command (14, 15), adjusts (S6, S10) its barrier timestamp (17, 18, 19) based on the logical execution time indicated by the timestamp (90) included in the movement command (14, 15).

15. A computer program product comprising instructions which, when executed on an industrial control device (100) having a physical network interface (101) for an industrial control system (1) comprising a plurality of industrial control devices (100, 200), cause the industrial device (100) to: - Configured to execute multiple control applications (113, 123) at constant cycles (10, 20, 30); - Form multiple containers (110, 120), each container containerizing a corresponding one of the control applications (113, 123), and each container has a virtual network interface (111, 121). - Multiple queuing procedures (150, 151, 152) are formed, each queuing procedure being associated with at least one of the network interfaces (101, 111, 121) and configured to manage network packets (7) entering or leaving the corresponding network interface (101, 111, 121) in queues (91, 92, 93), and to block network packets (7) with timestamps older than the barrier timestamps (17, 18, 19) maintained by the corresponding queuing procedures (150, 151, 152) in the queues (91, 92, 93). - Form a real-time container agent (104), The instructions, when executed on the industrial control device (100), further include: - This causes the real-time container agent (104) to perform maintenance processes (13, 23) in each cycle (10, 20, 30), and to perform (16, 26) each of the control applications (113, 123) only after the maintenance processes (13, 23) have been completed, wherein the maintenance processes (13, 23) include: - Representing control applications (113, 123), output processing (11, 21) and input processing (12, 22) are performed using technical devices (3) connected to the industrial control equipment (1000), and - Transmit a move command (14, 15) to each of the queuing procedures (150, 151, 152), the move command instructing the corresponding queuing procedure (150, 151, 152) to advance its barrier timestamp (17, 18, 19), wherein the move command (14, 15) includes a timestamp (90) indicating a logical execution time, the logical execution time being the start time (T1, T2) of the corresponding period (10, 20), and - This causes the corresponding queuing procedures (150, 151, 152) to adjust their barrier timestamps (17, 18, 19) in response to receiving a move command (14, 15) based on the logical execution time indicated by the timestamp (90) included by the move command (14, 15).

Images