Highly available delivery of ordered IOT message streams

Abstract

The invention relates to highly available delivery of ordered IOT message streams. There is disclosed a method for communicating messages between a first software application in a first container and a second software application in a second container, comprising the steps of: receiving, through a first connectivity service associated with the first container, an ordered sequence of messages sent by the first application; transmitting, by the first connectivity service, at least two copies of each message to a second connectivity service associated with the second container, where the at least two copies are routed over at least part of the redundant path, and where each copy indicates a location of the original message in the original ordered sequence; and forwarding, by the second connectivity service, at least one copy of each message to the second application in an order given by the original ordered sequence.

Description

[0001] This application is a divisional application of the invention patent application with application date of January 30, 2022, application number 202210114160.3, and invention name “Highly Available Delivery of Ordered IOT Message Streams”. Technical Field

[0002] The present invention relates to the delivery of ordered message streams in Internet of Things (IoT) applications, such as controlling and / or monitoring industrial processes executed on industrial equipment. Background Art

[0003] Industrial applications are increasingly adopting message-oriented middleware specifically designed for the Internet of Things. This type of middleware, such as an MQTT broker, can be used to process data streams from industrial devices sent via Internet protocols, potentially providing production optimization during runtime. MQTT is used as an example of message-oriented middleware in this document, but other examples (such as AMQP, DDS, XMPP, and JMS) can also be used.

[0004] MQTT specifies different Quality of Service (QoS) levels for message delivery. However, the behavior of broker implementations varies greatly, especially in the event of broker instance failures. There are clustered multi-instance broker implementations that aim to minimize the impact of broker instance failures. Experiments show that for QoS1 (at least once delivery) messages, most experiments cannot achieve reliable and ordered message delivery (no message loss, no message reordering). For applications that rely on reliable and ordered message delivery, such as online stream processing applications, application developers have two options: (1) rely on a specific broker implementation and trust its mechanisms after intensive testing; (2) implement additional mechanisms as part of the application logic to compensate for message loss and reordering (for example, by introducing application-layer message sequence numbers and implementing retransmission and reordering mechanisms). For option (1), application developers also need to trust the broker's reliable message delivery, for example after a software update of the broker. Due to the lack of an end-to-end monitoring concept for such middleware-based delivery, this requires extensive and repeated testing or contractual agreements with the broker vendor. On the other hand, option (2) makes the application implementation more complex and quickly leads to an undesirable mixture of application and data delivery logic. Summary of the Invention

[0005] The present invention provides a method for transmitting messages between a first software application in a first container and a second software application in a second container. As used herein, the term "container" includes any suitable virtualized and / or compartmentalized execution environment in which an application can run. For example, containers and pods in the Kubernetes framework can be used.

[0006] In the course of the method, a first connectivity service associated with the first container receives an ordered sequence of messages sent by the first application. This first connectivity service transmits at least two copies of each message to a second connectivity service associated with the second container. The at least two copies are routed over at least partially redundant paths. This means that on at least one leg and / or at least one station of the journey from the first connectivity service to the second connectivity service, the at least two copies are handled and / or transmitted by different hardware and / or software entities, so that a failure of such an entity does not cause both copies to be lost.

[0007] The partially redundant paths need not be completely disjoint. For example, the paths can use the same network interface and network connectivity, but then use different proxy IP addresses and proxy instances to deliver the redundant messages. In a very strict setting, two different network interfaces and physical networks are used.

[0008] Each copy of a message indicates the position of the original message in the original ordered sequence. This means that even if the copies of different messages do not arrive at the second connectivity service in sequence, the second connectivity service still knows in what order the messages need to be forwarded to the second application.

[0009] The second connectivity service forwards at least one copy of each message to the second application in the order given by the original ordered sequence. Whether multiple copies of the same message can be forwarded to the second application depends on the desired QoS level of the specific use case.

[0010] In the MQTT nomenclature of QoS levels, QoS level 1 means that at least one copy of a message is forwarded. Thus, multiple copies of one message can be forwarded to the second application before a copy of the next new message later in the original ordered sequence is forwarded. But once this copy of the new message has been forwarded, no further copies of the previous message that can have arrived later at the second connectivity service are forwarded. One example of data that can be suitably delivered with QoS level 1 is a stream of measurement values from a sensor. It is important that each measurement value arrives at the second application, and that the measurement values arrive in the correct order, so that the second application can react to the measurement values containing a temporal trend. But if measurement values arrive repeatedly at the second application, this does not hinder the work of the second application.

[0011] QoS Level 2 in MQTT nomenclature means that only one copy of a message is forwarded. Thus, when a second connectivity service receives the first copy of a message, it can forward this copy of the message to the second application and discard any other copies of the message that arrive later. An example of data that might be suitable for delivery at QoS Level 2 is a command to actuate a toggle switch that toggles a valve between an open and closed state. Here, if the second copy of the message is forwarded to the second application, the effect of the first copy of the message is negated. Another example is a command to start an internal combustion engine using an electric starter. Attempting to start the engine again while it's already running could damage the motor and / or starter.

[0012] Multipath delivery significantly improves the reliability of message streaming. Because message order is crucial, if a single message is completely lost, delivery to the second application is completely halted. No further messages can be delivered to the second application, allowing it to continue processing the data and provide the lost message once a new copy has been obtained. This advantageously reduces the probability of message loss or delay.

[0013] At the same time, no changes are required to either the first or second application. From the perspective of these applications, messages sent from the first application simply arrive at the second application in the correct order and with high reliability, as if the two containers hosting the applications were connected by a high-quality "virtual cable." Therefore, the method can also be used for "black box" applications for which no source code is available. Examples include control applications for field devices or other equipment that have been migrated from industrial controller hardware to virtualized containers.

[0014] Furthermore, the first and second connectivity services can enable legacy applications to communicate over newer networks and protocols, and even specifically utilize the more advanced features of these new networks and protocols. For example, control software originally written to communicate over HART, Modbus, or another fieldbus protocol can now be enabled to communicate over the MQTT protocol, and different types of messages sent from the control software can be mapped to the different QoS levels that MQTT has to offer.

[0015] Typically, it is advantageous to perform the communication between the first connectivity service and the second connectivity service through a middleware proxy.For example, such a middleware proxy may implement and / or manage a publish / subscribe communication model.

[0016] In a particularly advantageous embodiment, at least two of the at least partially redundant paths may be via:

[0017] ● Different instances of a middleware proxy configured to manage publish / subscribe communications between software applications; and / or

[0018] • different routing within the network; and / or

[0019] • different hardware interfaces to the network;

[0020] • different communication protocols; and / or

[0021] • different networks.

[0022] All these components represent points of failure that can cause a copy of a message to be lost. In particular, the middleware broker (e.g. an MQTT broker) is a source of errors, as the middleware broker tends to be under a very high load of simultaneous connections. Even if the software of the broker itself is free of faults, the execution environment in which the broker instance is deployed can suffer from temporary exhaustion of resources, e.g. memory, maximum number of concurrent processes, or maximum number of simultaneously open network connections.

[0023] In order to increase resilience, different instances of the middleware broker can be obtained from different vendors, and / or be chosen so as to be programmed differently. In this way, if one copy of a particular message causes problems or crashes in one instance, it is less likely that another copy of this particular message causes the same problems or crashes in another instance. For example, there can be a programming error in one instance that causes a buffer overflow when processing certain messages.

[0024] Temporary delays in the delivery of messages over the network or temporary bottlenecks in the processing by the middleware broker can cause a copy of a first message that is next to be forwarded to the second application according to the original ordered sequence to be delayed, but a copy of a second message that is later in the original ordered sequence to arrive on time. If the copy of the first message has not yet been received by the second connectivity service, but the copy of the second message that is later in the original ordered sequence has been received, then preferably the second connectivity service buffers the second message until the first message has been received and forwarded to the second application. In this way, the order in which the second application receives messages is restored without having to communicate back to the first connectivity service or another entity.

[0025] Preferably, if a copy of a message that is next to be forwarded to the second application according to the original ordered sequence has not been received by the second connectivity service within a predetermined period of time after a preceding message has been forwarded to the second application, the second connectivity service requests a retransmission of this message. In this way, a new copy of a lost message can be actively obtained, so that the stall time in which the second application cannot process further messages is minimized.

[0026] In another advantageous embodiment, the second connectivity service can acknowledge receipt of each message from the first connectivity service. The first connectivity service can buffer each message from the first application until at least one copy of this message is acknowledged as received by the second connectivity service. In this way, a more stringent guarantee can be provided that each message from the first application reaches the second application. The first application does not need to monitor whether all its messages are well received. It can simply assume that it is connected to the second application through a perfect "virtual cable". In particular, this avoids the complexity of providing a guarantee by the first application that the message delivery is free from "pollution" of the content of these messages. The software that generates the content is not mixed with the software that ensures the delivery. It is good practice to clearly separate software that performs different functions, in particular during software development. For example, one can avoid trying to improve one functionality and inadvertently introducing errors in another functionality. One example of an application for which it is particularly important to deliver each message in the correct order is an electrically transmitted flight control system in an aircraft. For example, if a command to lower the landing airspeed is executed, but not the previous command to deploy the flaps, the aircraft can stall and lose lift.

[0027] The acknowledgements and / or requests for retransmission can be forwarded from the second connectivity service to the first connectivity service, for example, by a centralized management entity. For example, this management entity can keep track of all messages sent by the first connectivity service and their successful delivery. This function of the management entity is somewhat similar to the function of a post office that keeps track of registered mail or parcels.

[0028] Alternatively or in combination with this function, the centralized management entity can be used to instantiate and / or manage the first connectivity service and / or the second connectivity service, and / or to monitor one or more instances of the middleware agent to check whether these instances are working properly.

[0029] In another advantageous embodiment, the first connectivity service aggregates multiple messages from the first application for transmission to the second connectivity service. Herein, "aggregation" includes every kind of processing that derives a reduced number of messages from the content of a given set of messages. One example of aggregation is to wrap the content from a given set of messages into one message. For example, by combining the content of several small messages into one larger message, protocol and other overhead can be saved. Furthermore, the chance that this combined message is successfully transmitted increases, assuming that each transmitted message has the same non-zero chance of being lost. Another example of aggregation is to form an average value from a series of measured values, for example temperatures.

[0030] In another advantageous embodiment, the first connectivity service blocks incoming messages that are destined for the first application. In this way, the first connectivity service acts as a "data diode" providing additional protection for the first application when it is exposed to the network. The first application cannot be taken over by malicious commands from the network. For example, a control application that was originally developed to run on a hardware controller can never have been intended to be exposed to the network. Thus, the programming of the control application can not have taken into account security against malicious input and contain other possible vulnerabilities such as buffer overflows or takeovers of its control flow.

[0031] In another advantageous embodiment, the first connectivity service can be a sidecar associated with the first container in the first pod and / or the second connectivity service can be a sidecar associated with the second container in the second pod. For example, pods and sidecars from the Kubernetes framework can be used. In the context of Kubernetes, a sidecar is a special container that provides an interface to an application in another container in the pod. The sidecar manages communication between the application and any entity outside the pod. In a more general context, pods can be used to group multiple containers that depend on each other in some way. For example, a pod can include a first container for processing video data and a second container for ingesting video data from a network.

[0032] In this way, the connectivity services are still running independently from the respective applications, but since they are in the same pod, the connectivity services can share some common resources, such as a file system or a namespace. Furthermore, in a virtualized environment, everything in the same pod will run on the same node. Thus, when a message is passed from the first application to the first connectivity service, or from the second connectivity service to the second application, the messages will communicate within the respective node and not over the network. For communication within the same node, it can be safely assumed that messages in an in-order stream all pass through in the correct sequence.

[0033] Similarly, other deployment options can be used to achieve the same purpose. For example, according to a Kubernetes deployment, multiple pods can also be grouped. In this case, the connectivity services can be implemented as pods of the deployment, rather than as sidecar containers that provide similar services as the described sidecar deployment option. For the reasons mentioned in the last paragraph, these pods can be deployed on the same physical node.

[0034] Thus, in another advantageous embodiment, the first connectivity service is deployed in a separate pod on the same physical node as the first pod with the first container and / or the second connectivity service is deployed in a separate container on the same physical node as the second pod with the second container.

[0035] In another advantageous embodiment, the first container can be deployed on an Internet of Things, IoT, edge gateway of an industrial device. The first application can then be configured to receive sensor data captured during execution of an industrial process by the device. The highly available delivery of messages to a second application processing the sensor data then allows for a higher degree of freedom with respect to the placement of the second container and the second application. This container can reside on the same edge gateway. But depending on which hardware resources (e.g. GPUs, FPGAs or other hardware accelerators) are needed for the processing and how much of these resources are needed, it can be advantageous to offload part or all of the processing to a second edge device or to the cloud. Due to the high availability of the communication provided by the present method, such offloading does not come at the cost of interrupting the processing due to the loss of one or more messages from the ordered sequence.

[0036] Thus, in another advantageous embodiment, the second container and / or at least one instance of a middleware agent configured to manage publish / subscribe communication between software applications is deployed on a second edge device or a cloud computing platform remote from the industrial device.

[0037] The methods can be fully or partially computer-implemented. Thus, the present application also provides one or more computer programs having machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform the methods described above. In particular, the virtualization platform and the one or more hardware controllers can be regarded as computers.

[0038] The present application also provides one or more non-transitory storage media and / or download products having the one or more computer programs. A download product is a product which can be sold in an online store for immediate implementation by download. The present application also provides one or more computers having the one or more computer programs and / or having the one or more non-transitory machine-readable storage media and / or the download products.

[0039] The present invention enables a multi-path message delivery mechanism transparent to the application. It leverages design diversity, where the present invention aggregates multiple middleware instances (e.g. multiple message delivery brokers) and combines them into a single resilient and reliable transport mechanism. The instances used can be e.g. different software versions of the same broker or even brokers provided by different vendors. By redundantly transmitting messages via multiple delivery paths, the likelihood of losing a message can be greatly reduced. Furthermore, the fluctuation of delivery latency can be reduced by delivering the first message copy received to the application without waiting for the temporarily slower delivery path. To optimize latency, the solution can leverage multiple message delivery QoS classes in parallel. For example, the solution can establish a QoS 0 MQTT session to achieve low latency, while redundantly transmitting copies via QoS1 or QoS2 session(s) in parallel. This way, in the best case, the application experiences a reliable QoS1 / 2 session with QoS 0 latency. Additional delivery path diversity can be achieved by using different network paths. Redundant physical network paths (e.g. two Ethernet), different network technologies (wireless and wired), or multi-path transport mechanisms (e.g. multi-path TCP) can be used.

[0040] By implementing the delivery mechanism outside the application, the solution is reusable for any application using message-oriented middleware communication and does not complicate the application implementation. The solution ensures in-order message delivery and enables other advanced delivery mechanisms, e.g. message aggregation. For this, messages are buffered before and after delivery.

[0041] By combining the described mechanisms, the solution is able to cope with hardware and software failures and is able to implement resilient message delivery, as well as demanding industrial application use cases.

[0042] The main use case of the present invention is for containerized systems, where the delivery mechanism can be implemented as a sidecar container providing transparent connectivity services to the application container, for example. Alternatively, the delivery mechanism can be implemented as a hardware device providing connectivity services to applications running on different devices in the form of a gateway or a proxy functionality. In both deployment options, the connectivity service behaves like an MQTT broker of the application, for example.

[0043] The present invention provides at least the following advantages:

[0044] • Multi-path delivery for message-oriented middleware (e.g. MQTT) with in-order, reliable and low-latency message delivery

[0045] • Dynamic delivery mechanism that automatically compensates for delivery path or broker instance failures

[0046] • Application implementation transparency and no need to change Commercial-Off-The-Shelf (COTS), middleware implementations or brokers;

[0047] • Network path diversity usage is a component of the solution (e.g. messages can be delivered via redundant or heterogeneous networks);

[0048] • Transparent support for message aggregation, message batching and protocol translation mechanisms;

[0049] • Reliable message transport independent of single (broker) implementation (design diversity and multi-vendor strategy);

[0050] • Exploitation of the potential of the middleware and broker ecosystem without dependency on specific vendors or implementations;

[0051] • Avoidance of the complexity of implementing recovery mechanisms related to transport as part of the application logic;

[0052] • Exploitation of the management features and capabilities of cloud-native and container orchestration environments for reliable (MQTT-based) communication;

[0053] • The Namur Open Architecture (NOA) Data Diode concept can be implemented as a component of the method.

[0054] The core idea comprises:

[0055] • Transparent end-to-end redundancy mechanisms for data delivery services using message-oriented middleware (e.g. MQTT);

[0056] • Multi-QoS level redundancy concept to enable low-latency message delivery with strong delivery guarantees;

[0057] • Mechanisms to exploit path and delivery mechanism diversity to enable low-latency reliable communication channels (e.g. MQTT QoS1 / 2 delivery with QoS0 latency);

[0058] • Flexible delivery mechanisms and concepts that can include advanced message delivery mechanisms such as batching, aggregation, data flow strategies (e.g. NOA Data Diode concept), or translation between delivery mechanisms (MQTT-AMQP bridging).

[0059] In the following, exemplary embodiments are described, without intending to limit the scope of the present application.

[0060] Applications use a locally deployed connectivity service to deliver application messages. This service behaves e.g. like an MQTT broker. In this case, the service is a broker delegate that maintains the actual (MQTT) sessions with external brokers. For other middleware examples, other types of sessions can be maintained through the service. As mentioned before, the connectivity service can be deployed as a sidecar container in a container orchestration system, as a software service or as a dedicated hardware device in other settings.

[0061] For QoS1 MQTT sessions and QoS2 MQTT sessions established by applications, the service establishes a set of redundant MQTT sessions with a dynamically managed list of external brokers for each MQTT topic used by the applications. A central coordinator (sidecar manager) manages the allocation of topics to broker instances and ensures that connectivity sidecars handling the same topics are connected to the same external redundant broker instances for these topics. To achieve the desired resilience against broker instance failures, the central coordinator ensures that messages for each QoS1 or QoS2 topic are delivered via at least two external brokers. This means that the connectivity sidecar replicates incoming messages for these topics and delivers the incoming messages to the brokers defined by the sidecar manager. The receiving service de-duplicates messages, buffers out-of-order messages and delivers the messages to subscribing applications in an ordered way. The allocation of topics to broker instances is handled dynamically by the sidecar manager and can be changed during runtime e.g. to compensate for a failed external broker instance.

[0062] To achieve low-latency delivery while maintaining delivery guarantees, the connectivity broker can be configured by the broker manager to internally append the MQTT messages of the distributed QoS1 / 2 application sessions via a QoS 0 session. While this approach obviously implies additional overhead regarding network load and CPU utilization, there are also some significant advantages. One advantage is that the broker / proxy network does not need to maintain a message queue and wait for the acknowledgement of the message delivery, which negatively impacts the delivery latency. In case of a QoS1 session, this reduces the delivery latency by approximately one round-trip time for each forwarding hop in the broker network, and for QoS2 messages, even two round-trip times can be saved. The two- and four-way handshakes required for QoS1 and QoS2, respectively, in the broker network are necessary for each forwarding hop among the broker instances, which means a potential high-latency reduction. Delivery on the QoS 0 delivery path happens with a best-effort delivery guarantee with low latency. In case of lost and reordered packets, the receiving connectivity broker waits for the slower QoS1 / 2 delivery. This means that as long as the QoS 0 delivery proceeds smoothly and does not encounter any bottlenecks, and thus also does not lose any messages, the application experiences the QoS1 / 2 session with the QoS 0 delivery latency. Only in case of message loss, which can happen for QoS 0 in case of broker failures (at most once delivery), the fallback to the QoS1 / 2 delivery latency occurs.

[0063] It should be noted that for the broker implementation, the application container itself does not need to be modified, as all described functionality is transparently implemented as part of the broker. Depending on the application and its configuration possibilities, the broker can even act as a delegate for an external proxy. This, however, assumes that the security mechanisms are also implemented outside the application, as usually assumed for the broker setup. In case of using security mechanisms within the application, e.g. hardcoded certificates for external proxy authentication, the usage of the solution can be limited or require the application to trust the certificates of the redundancy mechanism and its broker. This seems feasible, but is not discussed here any further.

[0064] In addition to the redundancy delivery mechanism, the connectivity broker is also used to implement complementary features. One key functionality is the reordering of messages before delivering them to the receiving application. For this purpose, the connectivity broker buffers messages and delivers them to the application only after all previous messages have been delivered. As mentioned before, for this, application layer sequence numbering can be used. Depending on the expected reordering, the size of the buffer can imply a significant overhead and slow down the message delivery. Careful tuning of the acceptable buffer size configuration and application specific policies on delivering incomplete buffers can be required. For strict stream processing applications, a larger buffer size combined with a retransmission mechanism can be the best solution. For other applications, reordering of only a few messages occasionally can use a smaller buffer size and handle the single undelivered message at the application layer.

[0065] Another message related mechanism can be implemented at the broker handling the connection to the sending application. To avoid sending many potentially small messages, the connectivity broker can buffer and batch the messages. This means that the payloads of the messages are combined to form larger messages. Especially for structured and text based data formats, this also allows implementing a compression mechanism to reduce the transmission size of the batched messages. In case of payloads in XML or JSON format, this can enable efficient compression of the payload by removing redundant information that would otherwise be delivered as part of smaller messages that are not batched. Overall, this optimizes the transmission bandwidth and message processing overhead at the cost of an increased delivery latency introduced by the time required for buffering and data compression. To limit the maximum latency introduced by buffering when waiting for a fixed number of messages, an upper time limit can be specified after which the batch is sent independent of the batch size. This mechanism can also reduce the processing load on the broker infrastructure, where experiments with MQTT brokers show that the processing is typically limited by the number of messages processed rather than the message size.

[0066] In addition to batching, application specific aggregation mechanisms can also be implemented as part of this approach, e.g. aggregating multiple message payloads into a single message payload. However, this requires knowledge of the data format and data model used. Considering structured data formats, an aggregation policy can be defined that extracts relevant numerical data fields from the MQTT messages and aggregates the fields into a predefined number of buffered messages. One example is to compute the average, minimum and maximum values and possibly other statistical properties for hundreds of buffered messages and deliver only a single aggregated message. To configure such application specific aggregation, the approach suggests exposing an API to specify the application specific aggregation mechanism, e.g. in the form of a small program or script to be run by the broker to perform the aggregation. Runtime and SDK concepts for such scripts can be envisioned.

[0067] In an extended version of the method, end-to-end delivery guarantees can be provided by implementing a buffering of application messages at the sending connectivity broker, which is connected to the application publishing data. This instance keeps the QoS 1 / 2 messages available until all receiving instances indicate to the broker manager that the messages have been delivered. This can happen when implementing a known acknowledgement mechanism, where a large batch of messages is acknowledged by the receiver at once. Alternatively, a negative acknowledgement (NACK) for individual messages or batches that have not been delivered within a given time frame is used to trigger a re-submission of such messages after the sending instance has been notified that the message has been successfully delivered to the broker. The broker manager informs the sending service about the received messages and requires the necessary retransmission. The sending service can remove the messages from its local buffer once all receivers have acknowledged the messages. The broker manager triggers this deletion by aggregated ACK messages.

[0068] This end-to-end delivery mechanism implies a significant overhead for the message delivery and makes the broker manager a potential bottleneck for the whole system. Therefore, its usage has to be carefully considered and is expected to be only applicable in particularly critical delivery scenarios or in case of highly unreliable broker instances. This end-to-end delivery mechanism is an optional extension to the original method, which works without additional message acknowledgement mechanisms.

[0069] The NOA (Namur Open Architecture) data diode concept can be implemented as a component of the proposed solution. For this, policies can be defined in the type and content of the data exchange that can pass through the connectivity broker. As the connectivity broker acts as a message delivery delegate for the application, these policies can be enforced in the best case by rejecting locally incoming messages to the application. As an additional protection layer, the receiving connectivity broker can also do similar filtering. In a containerized environment, for example, this would mean that if the application in the pod is configured as a data provider, messages can only leave the pod and thus data can be sent. In case of a pure data provider, the connectivity broker does not deliver incoming MQTT data messages to the application, thus implementing the data diode concept as a component of the method. When compared to other solutions, no additional containers or hardware boxes are needed.

[0070] The described approach hides the details of the data transfer from the application. While the described solution assumes e.g. MQTT for the connectivity broker-to-broker transfer, the solution can also be used to implement protocol translation. In this case, the application would still use e.g. MQTT to transfer and receive messages. However, internally, the connectivity broker can use a different transfer mechanism and / or protocol (e.g. AMQP). In this way, the described approach can be used to implement a migration to a different transfer mechanism. In this case, the transfer semantics of e.g. MQTT would have to be translated to the used protocol or mechanism. For this, also the redundant transfer mechanisms can be utilized, e.g. the descriptions related to different network paths. The main advantage of this approach is that it enables migration to other transfer mechanisms while keeping the application unaffected.

[0071] The connectivity broker manager can be implemented in a centralized or distributed fashion. While the connectivity broker manager is described in this approach and illustration as a logically centralized entity, the connectivity broker manager can be implemented in a decentralized fashion. In the decentralized case, multiple connectivity broker manager instances can run at any given time, with each instance managing a subset of the connectivity brokers. The manager instances coordinate to manage consistent end-to-end transfer paths. Coordination among the manager instances can follow any form of communication paradigm known from distributed software systems. In the extreme case, a fully decentralized (peer-to-peer) mechanism can be used. In this way, the solution can be scaled under heavy load, e.g. when end-to-end transfer guarantees are required and highly unreliable transfer paths are used, which would be desirable. In this case, the acknowledgement mechanism described earlier can put a high load on the connectivity broker manager, so it can be scaled by adding additional instances and distributing the load among the instances. In smaller setups, a single connectivity broker manager instance can be used. However, it is expected that in most cases a redundant setup with two managers will be used.

[0072] The described mechanism enables reliable and low-latency transfer through redundant message transfer. This comes at the cost of increased network traffic and CPU overhead. Depending on the network setup and pattern, this load can accumulate on the same network, or be distributed over different networks. Furthermore, the coordination of the connectivity brokers with the connectivity broker manager results in additional network traffic and requires computational resources to execute the mechanisms for message replication etc. and the mechanisms of the connectivity broker manager. This overhead can be considered as an optimization criterion. For example, in case of varying network load, the number of parallel transfers can be dynamically adjusted to avoid overload situations. In many cases, the available network bandwidth is not an issue, and it can be desirable to trade reliable transfer based on unreliable transfer paths. BRIEF DESCRIPTION OF DRAWINGS

[0073] In the following, the application is described using the drawings, which are not intended to limit the scope of the application. The drawings show:

[0074] Figure 1 An exemplary embodiment of the method 100 for transmitting messages 4 from a first application 11a to a second application 21a is shown;

[0075] Figure 2 An exemplary implementation of the method 100 on an loT edge gateway 6 and a cloud computing platform 7 is shown;

[0076] Figure 3 An exemplary implementation of the method 100 with diversity between two networks 8a, 8b is shown;

[0077] Figure 4 An exemplary implementation of the method for acknowledging receipt of messages 4 is shown.

[0078] Figure 1 is a schematic flow chart of an embodiment of the method 100 for transmitting messages 4 from a first application 11a to a second application 21a.

[0079] In step 110, a first connectivity service 12 receives messages 4 from a first application 11a executing in a first container 11. According to block 111, this first connectivity service 12 can also be used to block incoming messages destined to the first application 11a, thereby implementing a "data diode" allowing data flow in only one direction.

[0080] In step 120, the first connectivity service 12 sends at least two copies 4a-4c of each message to a second connectivity service 22. This second connectivity service 22 is associated with a second container 21 in which a second application 21a is executed. According to block 121, messages 4 can be aggregated for transmission to the second connectivity service 22. According to block 122, each message received from the first application 11a can be buffered until at least one copy 4a-4c of the message is acknowledged as received by the second connectivity service 22.

[0081] In Figure 1 In the example shown, the copies 4a-4c of messages 4 are routed to the second connectivity service 22 through different instances 31-33 of a middleware broker in the broker arrangement 3. That is, the paths to route the copies 4a-4c are at least partially redundant, in that unavailability of one broker instance 31-33 or discarding of a copy 4a-4c by one broker instance 31-33 will not deprive the second connectivity 22 of all copies 4a-4c of any particular message 4.

[0082] In step 130, the second connectivity service 22 forwards at least one copy of each received message 4 to the second application 21a in the order given by the original ordered sequence output by the first application 11a. In Figure 1In the example shown, only the copy 4a-4c that has been received first by the second connectivity service 22 is forwarded to the second application 21a.

[0083] According to block 131, it can be checked whether a copy 4a-4c of the first message 4 to be forwarded next according to the original ordered sequence has been received by the second connectivity service 22. If this is not the case (truth value 0), it is checked in block 132 whether a copy 4a-4c of a second message 4 later in the original ordered sequence has been received. If this is the case (truth value 1), the second message 4 is buffered in block 133 until the first message 4 that has to be forwarded next has been received and forwarded to the second application 21a.

[0084] According to block 134, it can be checked whether a copy of a message 4 to be forwarded next according to the original ordered sequence has been received by the second connectivity service 22 within a predetermined time limit after the preceding message 4 has been forwarded to the second application 21a. If this is not the case, a retransmission of this message is requested in block 135.

[0085] According to block 136, it can be acknowledged by the second connectivity service 22 that each message 4 has been received from the first connectivity service 12.

[0086] Figure 2 An exemplary implementation of the method 100 on an IoT edge gateway 6 and a cloud computing platform 7 is shown. The IoT edge gateway 6 is located on a facility of industrial equipment that executes an industrial process. The cloud computing platform 7 is located remote from the industrial equipment.

[0087] The first application 11a ingests data captured by sensors during execution of the industrial process. The container 11 with the first application 11a and the first connectivity service 12 are deployed in one common pod 1 on the IoT edge gateway 6.

[0088] The first connectivity service routes a copy 4a-4c of each message 4 to three different broker instances 31-33 of a broker service 3 on the cloud computing platform 7. In Figure 2 In the example shown, the copy 4c is routed through the network to the broker instance 33 using multipath delivery for increased redundancy. That is, in addition to using redundant broker instances 31-33 implemented as IoT hub instances in this example, a combination of redundant network paths (e.g. wired and mobile networks) is used with redundant MTCP transport paths. This shows a complex deployment scenario in which a supplementary level of redundancy is implemented.

[0089] A copy 4a-4c of each message 4 is delivered from the broker instances 31-33 to the second connectivity service 22. Similar to Figure 1and a copy is forwarded to a second application 21a executing in a second container 21. The second container 21 and the second connectivity service 22 are deployed in one common pod 2 on the cloud computing platform 7.

[0090] The first connectivity service 12 and the second connectivity service 22 are managed by a centralized management entity 51, which is deployed in its own pod 5 on the cloud computing platform 7. This centralized management entity 51 also monitors the proxy arrangement 3.

[0091] Figure 3 Another exemplary embodiment of the method 100 with network layer redundancy is shown. In case redundant networks 8a, 8b are available, the connectivity services 12, 22 connected to these networks using different interfaces eth0, eth1 can use these networks 8a, 8b to further improve the reliability of the delivery. In this case, the redundant proxies or proxy networks 31, 32 can operate in different networks 8a, 8b. This has the advantage of improving the resilience to network failures, as different physical network paths or technologies can be utilized. In a containerized environment, the network interfaces eth0, eth1 need to be exposed to the application pods 1, 2 and used by the connectivity services 12, 22. These can be physical or virtual network interfaces and can be mapped to e.g. different Ethernet or even different network technologies, e.g. wired and wireless technologies. Alternatively, also additional redundancy mechanisms can be used, e.g. using Multi-Path-TCP (MTCP) to utilize different network interfaces and delivery paths while avoiding the redundant proxy setup. In this case, the MTCP extension ReMP TCP can be used to establish an MTCP session with the proxy instance using redundant delivery paths.

[0092] Figure 4 Another exemplary embodiment of the acknowledgement of the receipt of a message 4 is shown. In this example, the second connectivity service 22 acknowledges the receipt of each copy 4a-4c of a message 4 by an ACK to the centralized management entity and requests a retransmission of a new copy in case no copy 4a-4c of a particular message 4 arrives within a predetermined time limit from the last message 4 being forwarded to the second application 21a by a NAK.

[0093] The existence of each message 4 is informed to the centralized management entity 51 by the first connectivity service 12, e.g. by the message ID. The centralized management entity 51 sends the ACKs it receives from the second connectivity service 22 together with the independent NAKs received from the second connectivity service 22 to the first connectivity service 12 in aggregated form. The first connectivity service 12 keeps a copy of each message 4 in a buffer 13 in the pod 1 until at least one copy 4a-4c of this message 4 is acknowledged as received by the second connectivity service 22.

[0094] List of reference signs:

[0095] 1 first pod for first container 11 and first connectivity service 12

[0096] 11 first container in pod 1

[0097] 11a first software application in first container 11

[0098] 12 first connectivity service in pod 1

[0099] 13 buffer for messages 4 in pod 1

[0100] 2 second pod for second container 21 and second connectivity service 22

[0101] 21 second container in pod 2

[0102] 21a second software application in second container 21

[0103] 22 second connectivity service in pod 2

[0104] 3 arrangement of proxy instances

[0105] 31-33 proxy instances

[0106] 4 message

[0107] 4a-4c copies of message 4

[0108] 5 pod for centralized management entity 51

[0109] 51 centralized management entity

[0110] 6 IoT edge gateway

[0111] 7 cloud computing platform

[0112] 8a, 8b network

[0113] 100 method for transmitting messages

[0114] 110 ordered sequence of received messages 4

[0115] 111 blocking incoming messages destined to first application 11a

[0116] 120 transmitting copies 4a-4c of messages 4 to second connectivity service 22

[0117] 121 aggregating multiple messages 4

[0118] 122 buffering messages 4 until acknowledgement

[0119] 130 Forwarding a copy 4a-4c of message 4 to the second application 21a

[0120] 131 Checking if the correct next message 4 has arrived

[0121] 132 Checking if the next message 4 in the sequence has arrived

[0122] 133 Buffering the next message 4

[0123] 134 Checking if the next message 4 has arrived before the deadline

[0124] 135 Requesting retransmission

[0125] 136 Acknowledging receipt of message 4

Claims

1. A method (100) for transferring messages (4) between a first software application (11a) in a first container (11) and a second software application (21a) in a second container (21), comprising the steps of: • receiving (110), by a first connectivity service (12) associated with the first container (11), an ordered sequence of messages (4) sent by the first application (11a); • transmitting (120), by the first connectivity service (12), at least two copies (4a-4c) of each message (4) to a second connectivity service (22) associated with the second container (21), wherein the at least two copies (4a-4c) are routed over at least partially redundant paths, and wherein each copy (4a-4c) indicates the position of the original message (4) in the original ordered sequence; and • forwarding (130), by the second connectivity service (22), at least one copy (4a-4c) of each message (4) to the second application (21a) in the order given by the original ordered sequence.

2. The method (100) according to claim 1, wherein at least two of the at least partially redundant paths are over: • different instances (31-33) of a middleware broker configured to manage publish / subscribe communication between software applications (11a, 21a); and / or • different routes within a network; and / or • different hardware interfaces to a network; • different communication protocols; and / or • different networks (8a, 8b).

3. The method (100) according to claim 2, wherein the different instances (31-33) of a middleware broker have been programmed differently and / or obtained from different vendors.

4. The method (100) of any one of claims 1 or 3, further comprising: If a copy (4a-4c) of a first message (4) to be forwarded next to the second application according to the original ordered sequence has not been received (131) by the second connectivity service (22), but a copy (4a-4c) of a second message (4) later in the original ordered sequence has been received (132), the second message (4) is buffered (133) until the first message (4) has been received and forwarded to the second application (21a).

5. The method (100) of any one of claims 1 to 4, further comprising: If a copy of a message (4) to be forwarded next to the second application (21a) according to the original ordered sequence is not received (134) by the second connectivity service (22) within a predetermined period after a previous message (4) has been forwarded to the second application (21a), a retransmission of this message (4) is requested (135).

6. The method (100) according to any one of claims 1 to 5, further comprising: • acknowledging (136), by the second connectivity service (22), receipt of each message (4) from the first connectivity service (12); and • acknowledging (137), by the first connectivity service (12), receipt of each message (4) from the second connectivity service (22). • buffering (122), by the first connectivity service (12), each message (4) received from the first application (11a) until the reception of at least one copy (4a-4c) of this message (4) is acknowledged by the second connectivity service (22).

7. The method (100) of claim 6, wherein the acknowledgement and / or the request for retransmission are forwarded from the second connectivity service (22) to the first connectivity service (12) by a centralized management entity (51).

8. The method (100) of any one of claims 1-7, further comprising: • aggregating (121), by the first connectivity service (12), multiple messages (4) from the first application (11a) for transmission to the second connectivity service (22).

9. The method (100) of any one of claims 1-8, further comprising: • blocking (111), by the first connectivity service (12), incoming messages destined to the first application (11a).

10. The method (100) of any one of claims 1 to 9, wherein the first connectivity service (12) is a sidecar associated with the first container (11) in a first pod (1) and / or the second connectivity service (22) is a sidecar associated with the second container (21) in a second pod (2).

11. The method (100) of any one of claims 1 to 9, wherein the first connectivity service (12) is deployed in a separate pod on the same physical node as the first pod (1) with the first container (11) and / or the second connectivity service (22) is deployed in a separate pod on the same physical node as the second pod (2) with the second container (21).

12. The method (100) of any one of claims 1 to 11, wherein • the first container (11) is deployed on an Internet of Things, IoT, edge gateway (6) of an industrial device, and • the first application (11a) is configured to receive sensor data captured during the execution of an industrial process by the device.

13. The method (100) of claim 12, wherein the second container (21) and / or at least one instance (31-33) of a middleware broker configured to manage publish / subscribe communication between software applications (11a, 21a) are deployed on a second edge device remote from the industrial device or on a cloud computing platform (7).

14. One or more computer programs comprising machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100) of any one of claims 1 to 13.

15. One or more non-transitory storage media and / or download products having the one or more computer programs of claim 14.

16. One or more computers having the one or more computer programs of claim 14 and / or having the one or more non-transitory storage media and / or download products of claim 15.

Images