Network telemetry apparatus and methods

Abstract

This disclosure relates to methods and apparatus for side-band telemetry monitoring. There is provided an apparatus for monitoring telemetry of data flows within a communications network, wherein the apparatus is communicatively coupled with a node within the communications network. The apparatus is configured to receive a data packet comprising first network flow instructions identifying at least a source address and a destination address, and to determine that the first network flow instructions have not been previously received by the apparatus. The apparatus is configured to generate a telemetry packet by incorporating second network flow instructions, based on the first network flow instructions, within the telemetry packet to cause the telemetry packet to take the same path through the communications network as the received data packet. The apparatus is further configured to incorporated telemetry data associated with a data flow of the data packet into the telemetry packet.

Description

[0001] NETWORK TELEMETRY APPARATUS AND METHODS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to the field of telemetry monitoring within a network to determine the nature of a network, in particular to methods of sending and receiving telemetry data to determine a network topology.

[0004] BACKGROUND

[0005] In the field of networking, cloud-native applications and the containerization technologies that enable those applications have become increasingly ubiquitous. Users typically break down applications into smaller units, sometimes called “Microservice”. Users then pack the service’s application code into so-called “containers” and deploy the containers in a cluster of physical or virtual servers. One popular tool that is frequently used to manage and automate the lifecycle of containers is Kubemetes (K8s).

[0006] In K8s terminology, the smallest unit of managing containers is a pod. In this disclosure, and in general, a pod has one or more container. K8s can be deployed to manage a pod’s lifecycle, e.g., to carry out operations such as spawn and destroy, pause, restart, and the like. K8s also enables pods to be deployed with high availability, and enables good load balancing and storage management characteristics, and good management of network communication in general.

[0007] Figure 1 shows an example of an application’s communication topology 100 between a plurality of pods. Arrows within the topology diagram 100 indicate network flows between respective pairs of pods, and each pod has an assigned IP address that is also indicated in figure 1. Generally, K8s assigns each pod an IP address, and so pods may contact each other directly. However, direct communication between pods is not recommended since pods can be ephemeral resources whose IP addresses may change when they are spawned, restarted, or destroyed. Fortunately, pods can also contact each other through a virtual IP address called “Service IP”. By using Service IP, a source pod may send requests to a destination pod using the “Service IP” as the destination IP. Figure 1 also indicates a pod-to-pod communication pathway 102 between a source pod 104 and a destination pod 106. During the communication, the source pod 104 attempts to access the service provided by the destination pod 106. To do so, the source pod 104 generates a packet with the destination IP set as the service node 108 that performs network address translation (NAT). Incoming packets at the destination pod 106 thus have the correct destination IP address, but the source address is in fact the ‘translated address’ which is the Service IP address belonging to the service node 108, due to NAT. Translated addresses generally define the address of the node that performs the NAT.

[0008] Using Service IPs provides various advantages including: (i) a more stable way to access pods, and (ii) improved load distribution and / or load balancing between multiple pods within an application topology. However, when using Service IPs, NAT may cause several address fields to be re-written including the source IP, the destination IP, source Port, and destination Port number, to forward the request to the appropriate pod. The alteration of these address fields, which may occur during NAT, can cause problems for telemetry monitoring. For example, determining an application’s topology can be difficult for methods that rely on parsing the data within packets because the existence or identity of various nodes within the application may be hidden or obscured by the NAT. In other words, due to the presence of NAT, the actual communication endpoints can be difficult or impossible to determine directly from the IP address fields of the packets of a network flow. The problem of determining network topology and application loads in large networks in these scenarios is exacerbated for larger networks, and so scalable methods for accurately solving the aforementioned problems are especially needed.

[0009] It is therefore an object of the present disclosure to solve the aforementioned problems, amongst other things, caused by NAT within a communications network by providing improved telemetry monitoring apparatus and methods. The foregoing and other objects are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0010] Particular embodiments are outlined in the attached independent claims, with other embodiments in the dependent claims.

[0011] SUMMARY

[0012] In view of the scenarios above, improved telemetry monitoring apparatus and methods are needed that enable accurate and scalable determination of network topologies and of telemetry monitoring within networks that use Service IPs. This can be done by providing, inter alia, a method of telemetry monitoring that generates cloned packets to form specialist telemetry packets, which are configured to be processed in the same way as the other flow packets in the network while offering more telemetry data space. This is advantageous because, amongst other things, this technique makes the solution network policy agnostic, is scalable for large networks, is applicable to cloud and physical networks, and is symmetric in that telemetry information can be sent in both directions (from source to destination, and vice versa).

[0013] A first aspect of the present disclosure provides an apparatus for monitoring telemetry of data flows within a communications network, wherein the apparatus is communicatively coupled with a node within the communications network. The apparatus is configured to: receive a data packet comprising first network flow instructions identifying at least a source address and a destination address; in dependence on determining that the first network flow instructions have not been previously received by the apparatus, generate a telemetry packet based on the data packet, the generating comprising: in dependence on the first network flow instructions, incorporating second network flow instructions within the telemetry packet that are configured to cause the telemetry packet to take the same path through the communications network as the received data packet; obtaining telemetry data associated with a data flow of the data packet and incorporating the obtained telemetry data into the telemetry packet; and cause the data packet and the telemetry packet to be transmitted towards a destination node defined by the destination address. The communications network may be a virtual cloud service network, or a physical data centre network. The node may be a host, and the apparatus may be a module configured to generate a telemetry packet, for example an injector module as described in this disclosure.

[0014] Advantageously, the telemetry packets enable the true (i.e., untranslated) endpoint addresses to be identified (i.e., such that the destination is able to identify the true, untranslated, source address) with low latency and without additional infrastructure. Further in this regard, telemetry monitoring using the telemetry packets is policy agnostic and thus if data packets are allowed by the network, so will the telemetry packets. Furthermore, a characteristic of the telemetry packets is that they enable a wide range of protocol coverage. Furthermore, the use of telemetry packets for telemetry monitoring has the advantage that it is highly scalable and can be applied in virtual networks such as cloud-native deployments, and also in physical networks, for example, physical high-speed data centre networks (DCNs). Furthermore, telemetry packets offer a significant amount of space for storing telemetry data, e.g., in a payload of the telemetry packet, and therefore allow more telemetry data to be stored.

[0015] In an example of the first aspect, the apparatus is configured to include the source address, identified in the data packet, in the telemetry data incorporated into the telemetry packet.

[0016] In an example of the first aspect, the apparatus is configured to generate the telemetry packet by duplicating the data packet and overwriting payload data of the duplicated data packet with the obtained telemetry data to thereby obtain the telemetry packet. In an example of the first aspect, apparatus is configured to obtain the telemetry data associated with the data flow of the data packet by looking-up network connection information from a data store, accessible by the apparatus, in dependence on the first network flow instructions.

[0017] In an example of the first aspect, the apparatus is configured to copy a time to live, TTL, indicator from the data packet into the telemetry packet. The TTL indicator, when in the data packet, is configured to be updated in response to the data packet passing through a hop in the communication network. The copy of the TTL indicator copied into in the telemetry packet is not updated further, e.g., because the TTL indicator is copied into a payload of the telemetry packet. In some examples, the TTL header is contained in an IP header of the data packet.

[0018] In an example of the first aspect, the apparatus is configured to enable generation of a telemetry packet via a socket in a user space, wherein the socket is configured to enable a user to generate telemetry packet to generate the second network flow instructions for incorporation within the telemetry packet.

[0019] In an example of the first aspect, the source address and a destination address are IP addresses, and wherein the apparatus is configured to determine that the source IP address of the received data packet has undergone network address translation prior to transmission towards the destination node, and wherein the obtained telemetry data incorporated within the telemetry packet comprises a record of the source IP address and a translated source IP address.

[0020] In an example of the first aspect, the apparatus is configured, in response to determining that received data packet has undergone network address translation, to store telemetry mapping data indicating an association between the source IP address, the translated source IP address, and the destination IP address.

[0021] In an example of the first aspect, apparatus is configured to include a telemetry identifier in the telemetry packet configured to allow a destination host, defined by the destination address, to distinguish between the data packet and the telemetry packet.

[0022] In an example of the first aspect, the apparatus is configured to use extended Berkeley Packet Filter, eBPF, technology, wherein the generation of the telemetry packet uses eBPF technology and is processed at an operating system level of the apparatus. The operating system level may also be referred to as the kernel level.

[0023] A second aspect of the present disclosure provides an apparatus for monitoring telemetry of data flows within a communications network, wherein the apparatus is comprised within a destination node of the communications network, the apparatus configured to: receive a data packet comprising first network flow instructions identifying at least a source address and a destination address, wherein the destination address defines the destination node; receive one or more telemetry packets having been generated, based on the data packet, by preceding respective one or more nodes within the communications network, wherein the one or more telemetry packets have, from the point of generation, travelled the same path through the communications network as the received data packet; determine that the one or more telemetry packets comprise network flow instructions that match the first network flow instructions, and in response to the determining: extract telemetry data from each of the one or more telemetry packets.

[0024] In an example of the second aspect, the apparatus is further configured to determine, in dependence at least on the extracted telemetry, at least a portion of a topology of the communications network. A third aspect of the present disclosure provides a method for monitoring telemetry of data flows within a communications network, the method comprising: receiving, at a node within the communications network, a data packet comprising first network flow instructions identifying at least a source address and a destination address; in dependence on determining that the first network flow instructions have not been previously received at the node, generating a telemetry packet based on the data packet, the generating comprising: in dependence on the first network flow instructions, incorporating second network flow instructions within the telemetry packet that are configured to cause the telemetry packet to take the same path through the communications network as the received data packet; obtaining telemetry data associated with a data flow of the data packet and incorporating the obtained telemetry data into the telemetry packet; and transmitting the data packet and the telemetry packet towards a destination node defined by the destination address.

[0025] A fourth aspect of the present disclosure provides a method for monitoring telemetry of data flows within a communications network, the method comprising: receiving, at a destination node within the communications network, a data packet comprising first network flow instructions identifying at least a source address and a destination address, wherein the destination address defines the destination node; receiving one or more telemetry packets having been generated, based on the data packet, by preceding respective one or more nodes within the communications network, wherein the one or more telemetry packets have, from the point of generation, travelled the same path through the communications network as the received data packet; determining that the one or more telemetry packets comprise network flow instructions that match the first network flow instructions, and in response to the determining: extracting telemetry data from each of the one or more telemetry packets.

[0026] A fifth aspect of the present disclosure provides a computer program stored in non-transitory form and including code instructions which when executed on one or more processor, causes the one or more processor to execute the method according to the third or fourth aspects.

[0027] BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present disclosure is described by way of example, with reference to the accompanying drawings, in which:

[0029] Figure 1 shows an illustration of an application’s communication topology, and an illustration of the observable addresses between pairs of applications;

[0030] Figure 2 shows an illustration of a side-band telemetry packet according to embodiments of the present disclosure; Figure 3 shows an illustration of a deployment between a pair of network applications of the telemetry apparatus according to embodiments of the present disclosure;

[0031] Figure 4 shows an illustration of a deployment between three hosts of a network of the telemetry apparatus according to embodiments of the present disclosure;

[0032] Figure 5 shows a schematic illustrating the communications workflow of the telemetry method according the network of figure 4;

[0033] Figure 6 shows a Linux networking stack being configured to perform an embodiment according to the present disclosure;

[0034] Figure 7 illustrates a method of generating a side-band telemetry packet according to embodiments of the present disclosure;

[0035] Figure 8 illustrates a method of telemetry monitoring using time to live, TTL, indicators according to embodiments of the present disclosure;

[0036] Figure 9 is a flowchart illustrating a method for monitoring telemetry of data flows within a communications network, according to embodiments of the present disclosure;

[0037] Figure 10 is a flowchart illustrating a method for monitoring telemetry, at a destination host, of data flows within a communications network, according to embodiments of the present disclosure; and

[0038] Figure 11 illustrates an example of an apparatus configured to perform the methods described herein.

[0039] DETAILED DESCRIPTION

[0040] In the present disclosure, reference to ‘communications network’ encompasses networks in general, including virtual networks such as virtual cloud native deployments, and physical networks such as physical high-speed data centre networks (DCNs). Further, eBPF and BPF may be used interchangeable, and refer to the same thing (i.e., extended Berkeley Packet Filter).

[0041] In the present disclosure, the terms ‘host’, ‘hop’, and ‘node’ may be used with equivalent meaning. In some examples, a node may refer to a host that hosts an apparatus, where the apparatus may be an injector, telemetry module, or tracing module.

[0042] In the present disclosure, reference to ‘apparatus’ encompasses tracing modules such as tracing modules 302, 304 shown in figure 3, or the injectors shown in figure 4 and figure 8. An apparatus contained in a destination host may be referred to as a ‘parser’. More generally, an apparatus in the present disclosure and appended claims encompasses modules or logic configured to implement the side-band telemetry techniques disclosed herein.

[0043] In the present disclosure, reference to ‘data packet’ encompasses a flow packet or packet of data having at least a source address and a destination address (for example, contained in a header such as a 5-tuple header) and which is not a side-band ‘telemetry packet’ according to presently disclosed embodiments of the SBT techniques.

[0044] In the present disclosure, reference to ‘network flow instructions’ encompasses the portion of a packet (including both data backs and telemetry packets) that contains at least a source address and a destination address. In general terms, network flow instructions comprise the portion of a packet that is configured to cause a module within a network to transmit the packet to the desired destination point. Network flow instructions therefore encompass packet headers and 5-tuple values. In the present disclosure, reference to ‘telemetry packet’ encompasses packets generated, based on a suitable data packet, for the purposes of telemetry monitoring in accordance with SBT techniques disclosed herein. Telemetry packets may equivalently be called ‘side-band’ packets in this disclosure.

[0045] In the present disclosure, reference to ‘telemetry data’ encompasses relevant data, sometimes called ‘identifying information’, that is obtained from a data packet e.g., for the purposes of storing the telemetry data in a telemetry packet.

[0046] In large clusters, having an observability system, i.e., a telemetry monitoring system, that provides insights on various characteristics of the systems is important for troubleshooting, and for maintaining the availability and security of deployed applications. An observability system can be configured to extract various events, and application, systems, and networking metrics from deployed pods. These metrics include the number of transferred bytes, error rates, latencies, utilization, and the like. Accurate reconstruction of an application’s topology can be an important tool to debug connectivity and performance issues in network clusters for example in Kubemetes (K8s) clusters, such as the topology 100 shown in figure 1. Once application topology has been reconstructed, e.g., showing which pods communicate with which, metrics determined by telemetry monitoring can be correlated to the flows of the application.

[0047] To construct the topology, current systems typically record the observed IP addresses in the packets generated by the pods at various vantage points. A vantage point may correspond to a node or host within a network application or may correspond to a telemetry monitoring point within a node. Telemetry monitoring, in which identifying information used to determine network topology is embedded in packets, typically falls into one of two categories which are in-band telemetry, and out-of-band telemetry. In-band telemetry may comprise, at a vantage point: marking packets with telemetry data, e.g., vantage point ID as packets are processed. Out-of-band telemetry may comprise, at a vantage point: sending a separate packet to a central telemetry server with the information about the observed packets. In other words, in-band telemetry comprises intercepting a data packet and using that data packet to carry the desired telemetry data, which is inserted at specified places within the data packet. In principle, no additional network servers are needed to carry out in-band telemetry. On the other hand, out-of-band telemetry comprises generating a dedicated telemetry packet, separate from the data packets, that are used to carry the telemetry data, and which requires an additional (centralised) server to which the out-of-band packets are sent. The centralised server aggregates the telemetry data to reconstruct the application’s topology.

[0048] In-band and out-of-band telemetry methods have their respective disadvantages. In the case of in-band telemetry, for hops / hosts in a network subsequent to the host that generates the in-band packet, the in-band packet needs to be parsed, and telemetry data offset needs to be calculated, and relevant telemetry data written. This causes a problem for multiple hops since the in-band telemetry header space limits the amount of telemetry data that can be sent, and all telemetry updates must be written within the same limited header. Out-of-band telemetry, although it offers larger telemetry data space compared to in-band telemetry, generally requires additional infrastructure. This additional infrastructure is typically a central entity / server that collects all the telemetry data from the various points in the path of a network flow. The central entity is problematic since it represents additional infrastructural overhead. Furthermore, the generation of out-of-band packets is done in the user space, which causes significant performance delays, as opposed to the much more efficient kernel space.

[0049] Moreover, if the observability system does not record the telemetry information at the appropriate vantages points (or, e.g., at a sufficient number of vantage points), the telemetry system may fail or may provide misleading information, especially when pods are communicating with each other using Service IPs. As explained above, the problem with Service IPs is that, despite their helpful ability to enable high availability and load balancing, Service IPs add multiple levels of indirection by virtue of network address translation (NAT) that may occur multiple times within a network flow. Thus, a receiving pod may not see the ‘true’ (i.e., untranslated) IP address of the source pod since the packet has already gone through at least one stage of NAT. Therefore, solutions are needed to trace packets as they go through these NAT stages and accurately reconstruct the application topology, especially for large networks which notoriously suffer from scalability issues in network monitoring systems (particularly for out-of-band monitoring systems).

[0050] In detail, NAT transforms network packets by mapping the packets from one flow, defined by a packet header which may be a so-called “5 tuple” containing Source IP, Source Port, Destination IP, Destination Port, Protocol, to another flow Therefore, depending on the vantage point (e.g., before NAT, on NAT device, or after NAT device) different flows may be observed, e.g., the original flow, or the transformed one. In a practical scenario in which a network may have thousands of concurrent flows between different source-destination pairs undergoing NAT operations, a significant amount of uniquely identifying information is necessary in order to unambiguously map transformed flows to their original ones, and vice versa.

[0051] This problem is usually solved in one of two general ways, in which I) identifying information can be derived from the packets and other sources, and II) identifying information may be embedded in packets (i.e., the data packets themselves in an in-band method, or in separate dedicated packets in an out-of-band method) for later identification.

[0052] In method I), systems that derive the unique information from packets passively monitor the network traffic and its state, and then attempt to map the transformed flows to their original ones. However, this category of methods has several problems. Firstly, any change in the network state, e.g., addition / destruction / update of a NAT entry, must be recorded if the change in topology is to be accurately captured. However, for a large-scale network, to reduce the collection and processing load, it may only be possible to collect this information periodically. In other words, large networks may be forced to limit the amount of time that can be spent enquiring about changes in network state. For such systems, there is an undesirable trade-off between accuracy and scalability, and as such the size of a network can be a limiting factor. In other methods in category I), the assumption is that there is enough uniquely identifying information that is invariant to NAT. This assumption depends on the network environment and the type of applications using the network. Thus, such methods are sensitive to the change in the traffic characteristics, requiring operators to regularly check whether the underlying assumptions about the traffic still hold. If such assumptions are not regularly checked / verified, the systems may misclassify and produce inaccurate mappings. Network size is therefore again a limiting factor.

[0053] In method II), identifying information is embedded in packet headers that, in theory, won’t be modified by the NAT operation. In some methods, this requires devices capable of marking packets with an identification number in an unused space in the packet header. This further requires devices to generate an out-of-band telemetry packet upon seeing a marked packet. Yet further, the system requires a central node to aggregate the telemetry packets and perform mapping. This extra overhead, and extra post-processing step in a central location, presents as a bottleneck in terms of processing, especially for larger systems. Moreover, sending the marking and un-marking commands to the nodes adds additional complexity to the system design, and adds latency into the network processing. In in-band methods, the original flow information can be embedded in a suitable header (e.g., a TCP options header). Although this approach eliminates the need for central post-processing infrastructure (needed in out-of-band methods), relying on suitable headers (e.g., TCP options header) limits the tracing capabilities to certain e.g., TCP) flows only. Furthermore, due to the limited space in the header, this method suffers from the disadvantage that only the source IP address of the originating (first) node may be embedded in the header in some examples. Therefore, if there are multiple NATs between a source and a destination, the information about the intermediate nodes cannot be preserved.

[0054] As mentioned, due to the presence of a NAT, the actual communication endpoints cannot be determined directly from the IP address fields of the packets of a network flow. In a cluster of pods, such as the topology 100 illustrated in figure 1, a virtual service performs NAT on the flow packets before they are forwarded to the destination (server) application or pods. Thus, from the perspective of the destination pod, the actual source details are not known. Conversely, for the source pod, the destination remains as the virtual service. So, from the communication endpoint’s perspective, the problem lies in getting telemetry data (e.g., IP addresses / L4 port numbers) of the actual peers. In the presence of multiple NAT gateways (e.g., multiple intermediary hosts or hops may contain entities that perform NAT) in the path of a network flow, it is equally important to construct the end- to-end network topology for observability and troubleshooting purposes. As described above, known methods have limitations in terms of scalability (due to packet processing latencies at a large scale), infrastructural overhead, and protocol coverage.

[0055] The present disclosure aims to solve at least these problems by a new type of telemetry referred to as ‘side-band’ telemetry. The side-band method of telemetry monitoring works, in one example, by generating (e.g., by cloning) data packets to form specialist ‘telemetry packets’. The telemetry packets are configured to contain network flow instructions based on the network flow instructions contained in the original packet (e.g., they may be a direct copy), such that the telemetry packet is configured to take the same path through the communications network as the received data packet. Thus, telemetry packets may transmit and process telemetry data in the same way as ‘standard’ data packets, and advantageously using the same network systems. Furthermore, the side-band telemetry packets offer more telemetry data space than in-band packets.

[0056] In more detail, the side-band telemetry methods and apparatus of the present disclosure solve at least the following problems:

[0057] 1. In the presence of NAT(s) in the network path, the side-band telemetry (SBT) enables the ‘true’ endpoint addresses to be identified (i.e., such that the destination source is able to identify the true source, and vice versa) with low latency and without external dependencies like additional infrastructure. Further in this regard, the side-band technology has a wide range of protocol coverage.

[0058] 2. The SBT technique enables the transfer of information of the end-points and any middle hops, i.e., intermediary hosts, within the network.

[0059] 3. In one embodiment, the SBT technique generates telemetry packets on the fly by cloning data packets. This embodiment has the advantage that it is highly scalable. Moreover, SBT can be applied, and is scalable, in virtual networks such as cloud-native deployments, and also in physical networks, for example, physical high-speed data centre networks (DCNs).

[0060] Figure 2 illustrates a telemetry packet 203 generated according to the side-band telemetry technique of the present disclosure. Figure 2 illustrates a source host 204 communicating with a destination host 206. The source host sends data packets 202 to the destination host 206 using the destination host’s IP address. If Service IP addresses are used, the destination IP address will be translated by a NAT entity, and the data packet will be transformed from one flow to another. Specifically, the packet header, which defines at least a source address and a destination address, will become translated by the NAT such that at least the source address is altered to the address of the NAT entity. In one specific example, the header is a 5-tuple value comprising Source IP, Source Port, Destination IP, Destination Port, and Protocol. In general terms in this disclosure, header data that contains the source and destination addresses may be referred to as ‘network flow instructions’.

[0061] With the SBT technique, a suitable data packet 202 may be used as a basis to form a dedicated ‘telemetry packet’ 203 containing network flow instructions 208 of its own, where those instructions are configured to cause the telemetry packet to take the same path through the communications network as the received data packet. In one example, the network flow instructions 208 of the telemetry packet 203 are a direct copy of the network flow instructions of the data packet. Thus, in one example the telemetry packet 203 is formed by cloning a suitable data packet, whose header value / 5-tuple value 208 is preserved. The ‘payload’ of data packet may also be copied, at least initially, to become the pay load 210 of the telemetry packet. However, the telemetry packet’s pay load is used to store telemetry data of the original packet, which includes the original network flow instructions (e.g., the packet header, which may be a 5-tuple value) of the data packet. The content in the payload of the original packet can therefore be removed for the telemetry packet. Thus, when the telemetry packet undergoes NAT, its header value 208 will be transformed in the same way as the header value contained in the data packet. However, the telemetry packet has been generated such that its pay load 210 contains telemetry data associated with a data flow of the data packet. Therefore, the original header value of the data packet, which identified at least the original source address and the destination address, is preserved in the telemetry packet. In this way, the telemetry packet can be used as a vessel to transport the untransformed flow data, associated with the original data packet, to the destination server without allowing the untransformed flow data to be affected by NAT operations.

[0062] As mentioned, SBT techniques enable the telemetry packets to be processed in the same way as the other data packets in the network. This is by virtue of the fact that the network flow instructions incorporated within the telemetry packet are configured to cause the telemetry packet to take the same path through the communications network as the data packet from which the telemetry packet has been derived or cloned. As an advantage over in-band telemetry methods, the side-band telemetry packets offer more space for storing telemetry data. This is because, for in-band methods, a suitable header (e.g., TCP options header) is used for storing the identifying information, which significantly limits the tracing capabilities to certain e.g., TCP) flows, and significantly limits the amount of data that can be stored. By contrast, side-band telemetry packets store telemetry data in the ‘payload’ area 210 of the packet. Furthermore, SBT techniques are policy agnostic. In other words, if data packets are allowed in the network, side-band telemetry packets will also be allowed and do not require additional network policies to be able to flow in the network (unlike, for example, out-of-band packets which need additional infrastructure). Another general advantage is that the presently disclosed SBT techniques are symmetric - i.e., the telemetry information can be sent in both directions from client (source) to server (destination), and from server (destination) to client (source) as well. Information on the actual peer is thus available at both endpoints.

[0063] Figure 3 shows an applicable scenario for the implementation of the SBT techniques. This example illustrates a specific system implementation implemented using extended Berkeley Packet Filter, eBPF, technology to perform at least some of the SBT operations, and shows a K8s deployment in which a client application 104 requests data from a (destination 106) server application. The view in figure 3 is thus a logical view of a K8s deployment. Figure 3 shows a K8s deployment where the client application, in the source pod 104 at the source host 204, establishes a connection with the server application 312 in a destination pod 106 at the destination host 206. The server application 312 is exposed via the Service 108 which is of type clusterIP (a K8s service which is exposed on a cluster-internal IP, and reachable from within the cluster). The client at the source node 104 sends packets to the “server application” 312 contained within the destination pod 106. The “Service” 108 is the entity that implements Service IPs, which performs NAT on data packets sent from the source pod 104 and forwards the packets to the server application 312 within the destination pod 106. The labelled “points of intercepting network flow packets” (306, 308, 310) show the points at which telemetry information is recorded, and thus represent logical data-collection positions within the flow of the telemetry packets. The points of intersection do not (necessarily) indicate physical data-collection points or entities within the hosts.

[0064] The eBPF-based network tracing modules 302, 304 are present both at the source and destination hosts. The tracing modules intercept the flow packets at interception points 306, 308, 310 as shown in figure 3. At the source host 204, the interception points 306 and 308 help to gather telemetry data that is incorporated into the generated telemetry packets. At the destination host 206, point 310 monitors all the incoming packets. In more detail, at interception point 306 the tracing module 302 collects network flow instructions (e.g., the 5-tuple value or packet header) from a data packet. At interception point 308, the tracing module 302 does the same again, however, the data packet will have undergone NAT at the Service entity 108. The information collected at 308 is thus transformed information. The information collected at both points 306, 306 is preferably included in the payload 210 portion of the telemetry packing, thus forming the telemetry data that is incorporated within the telemetry packet.

[0065] In some examples, the telemetry module only causes a telemetry packet to be generated upon determining that a received packet represents a ‘new’ connection; in other words, if the packet header is unknown. This helps prevent copies of telemetry packets from being made. For example, a source host 204 may generate a telemetry packet based on a newly received data packet. Both the (original) data packet and the telemetry packet may then be received at an intermediate host. The intermediate host may generate another telemetry packet based on the data packet since the packet is new to the intermediate host. However, the intermediate host will preferably be configured not to generate another telemetry packet based on the telemetry packet generated by the source host, since the intermediary is configured to recognise that the telemetry packet does not represent a ‘new’ connection. This can be determined by determining that the packet header of the data packet and the telemetry packet are the same.

[0066] Generally, a host within a network flow (whether it is a source, intermediate, or destination) may have at least one telemetry collection point implemented by a telemetry module such as module 302 or 304. Thus, each telemetry module can be configured to issue a new telemetry packet for every new connection. In practice, in some examples, the destination node may have no telemetry collection point at all if it is known that it does not perform any address translation.

[0067] Generally, the tracing apparatus 302 at a source host should be configured to collect at least the pre-translated address (i.e., the original source IP address, and port, contained in the packet header of the data packet). In other words, the pre-translation collection point 306 should be implemented. However, in some cases, the post-translation point 308 is optional. The pretranslated address should be collected and stored in the telemetry data of a side-band telemetry packet since the pre-translated address would otherwise be lost. Although it would be convenient to also record the post-translated address at 308 and store this in telemetry data of a side-band telemetry packet, this is not strictly necessary since the post-translated address may be determined / read anyway from the translated packet header of the data packet. In particular, it is preferable to collect both the pre- translated and post- translated addresses for L4 load balancing.

[0068] The tracing module 304 in the destination host 206 monitors all packets, and does not distinguish (initially, at least) between data packets and telemetry packets. The tracing modules 302, 304 further communicate with each other using the SBT channel for exchanging telemetry data (indicated by the solid line). For additional middle / intermediary hosts (not shown in figure 3), the tracing module(s), similar to tracing modules 302 and 304, may also be present at each of the middle hosts with a similar workflow as mentioned in this example.

[0069] Example implementations

[0070] Specific ways of implementing the side-band telemetry (SBT) technique are now disclosed. However, the implementation details, functions, apparatus, and technologies described in the foregoing are not intended to be limiting.

[0071] In a specific example implementation of the SBT techniques, the extended Berkeley Packet Filter, eBPF, technology is used to perform at least some of the SBT operations. Specifically, network apparatuses configured to perform SBT may use eBPF technology to instrument the Linux networking stack to extract relevant network flow information from incoming data packets. In some embodiments that clone data packets to obtain telemetry packets, the cloning may be implemented using eBPF technology to clone a suitable flow packet to be used as a telemetry packet for the respective network flow. In some examples, the cloning operation is done in the traffic control (TC) subsystem.

[0072] In some examples, SBT techniques utilize Conntrack, which is a native Linux module used for connection tracking. Information from Conntrack can be used to track network flow packets (i.e., data packets). Conntrack may also be used to map the pre-NAT and post-NAT (header) values of a packet undergoing address translations in a host / node. This method supports all the L4 protocols that are supported by Conntrack for connection tracking - TCP, UDP, ICMP, DCCP, SCTP, GRE. Preferably, the SBT functions carried out by eBPF operate in the kernel space (i.e., at the operating system level) which improves performance and reduces latency attributable to the negligible time spent at the kernel level on packet processing and monitoring. Alternatively, in some examples, there is an optional user space mode called ‘raw socket’ mode, described in further detail in the examples below. Figure 4 shows an example implementation using eBPF and illustrates a system diagram 400 of the SBT technique. Known components are presented in solid-lined boxes, and the additional components used by SBT-specific functionality are present in dotted boxes. The eBPF ‘Bee’ logo beside a component further denotes that either it is instrumented with an eBPF probe, or it is a part of eBPF technology. Similarly to figure 3, figure 4 illustrates a source host 204 and destination host 206, in addition to an intermediate host. The injectors 404, 406 shown in the source host 204 and middle host 406, respectively, perform the functions of cloning the data packets to obtain telemetry packets. Thus, the injector corresponds in functionality, at least in part, to the tracing module 302 shown in figure 3. The parser 408, of the destination host 206, may correspond to the tracing module 304 of figure 3. The functionality of the parser is disclosed in more examples below. Generally, the parser is configured to identify telemetry packets (and distinguish them from data packets) and process the telemetry packets accordingly. As shown in figure 4, the parser 408 may identify telemetry packets, parse the relevant telemetry data contained within the telemetry packets, and send the extracted telemetry data to storage, e.g., BPF data storage, as indicated in figure 4.

[0073] Figure 5 shows a communication / data workflow schedule 500 that illustrates the flow of data packets and telemetry packets as they would be processed by the system diagram of figure 4. The source host 204, middle host 402, and destination host 406 thus correspond to the hosts of the same name in figure 4.

[0074] In figures 4 and 5, the source host 204 is a Linux-based server, and it hosts the client pod 104. The source host 204 has the usual Linux network stack with native components like the Netfilter Conntrack module.

[0075] In figures 4 and 5, as in previous examples of figure 2 3, the client pod 104 is a K8s pod that runs a client application which communicates with the server pod 106 (also a K8s pod) running a server application 312. The server pod 106 in this example is hosted on the destination host 206 which is also a Linux-based server similar to the source host. The client application 104 sends flow packets (i.e., data packets) to the virtual service that operates the Service IP. The virtual service, on receiving the data packets, may perform necessary SNAT and DNAT to forward the flow packets to one of the backend pods. Further NATs may also be performed by the host if necessary to route the packets to the appropriate destination.

[0076] Conntrack is a native Linux Kernel module in the Netfilter framework that tracks network connections at the L4 level, e.g. TCP or ITDP connection in a host. Additionally, it also identifies data packets belonging to a connection, thus making the module a suitable point of instrumentation to track packets having undergone NAT. In other words, the Conntrack may be used to determine whether a packet contains a header value that has already been received, thus allowing it to identify a telemetry packet (providing that the telemetry packet is received its corresponding data packet). Conntrack maintains the records of connections only when they are confirmed as denoted by the ‘Connection Confirm Handler step’ in the system diagram.

[0077] The eBPF technology enables the instrumentation of various components in both the kernel space and user space. In this example, the Linux kernel stack is instrumented by attaching eBPF probes at various points in the stack. The modules that are instrumented with eBPF probes in this example are: Conntrack, connection confirm handler, injector 404, 406, and parser 408. BPF Data Storage are BPF maps that are key- value storage systems of the eBPF technology. The relevant details of a network connection are stored in these BPF maps for lookup and are thus used to populate the telemetry packets with the appropriate telemetry data collected from suitable data packets.

[0078] The injector modules 404, 406 generate telemetry packets. In one example this is done by cloning a suitable flow packet using an eBPF program at the egress network interface (which operates entirely in the kernel space). Alternatively, telemetry packets are generated by establishing a ‘raw socket’ in the user space to generate new telemetry packets. In this example, the socket is configured to enable a user to generate a telemetry packet to generate the second network flow instructions (i. e. , a packet header having the appropriate source and destination addresses, based on the data packet) for incorporation within the telemetry packet. In the example of figure 4 and 5, the injector 404, 406 performs lookup in the BPF data storage, extracts the telemetry data for the relevant data packet network flow, and writes / incorporates the data in the telemetry packet. In some examples, a time-to- live (TTL) indicator of a data packet at the respective hop in the network is included as a part of the telemetry data within the telemetry packet. TTL data can be especially helpful in determining the topology of a network that has many additional hops which do not have SBT capability. Additionally, the injector may be configured to mark the telemetry packets with a unique signature header. The unique signature header may then be used as a part of the identification mark by the parser 408 module at the destination host in order to identify telemetry packets for onward processing.

[0079] As shown in figure 4, the middle host 402 is another host in the path of a network flow of a data packet. There can be zero, one or multiple middle hosts in the path of a network flow in general. The middle host 402 in this case is a Linux-based server with routing and NAT capabilities. Additionally, the middle host also generates its own telemetry packets for every new connection that is confirmed. Thus, a plurality of telemetry packets may be received by the parser 408 of the destination host where each of the plurality of telemetry packets has been generated based on the same data packet. For example, a first telemetry packet is generated by the injector 404 of the source host and a second telemetry packet is generated, based on the same data packet, by the injector of the middle host.

[0080] As shown in figure 4, the parser module 408 at the destination host is an eBPF TC program running in the ingress network interface of the destination host 206. The parser monitors all the incoming packets and filters the telemetry packets. The filtering may be performed by identifying which packets contain a unique signature header (contained only in telemetry packets) generated by the injector modules either from the source or from the middle hosts. In response to identifying a telemetry packet, the parser stores the telemetry information extracted from the telemetry packets in the BPF Data Storage. Additionally, the parser 408 may be configured to discard the telemetry packets (once the telemetry information has been extracted) to prevent unnecessary response packets in the network that could be generated by the upper layers of the network stack otherwise. For all the other non-telemetry incoming packets, the parser forwards them for usual processing in the network stack as per the network policies.

[0081] The trace aggregator shown in figure 4 is a point of aggregation at the destination node. It receives telemetry data from the Parser and individual incoming flow information from the Conntrack via the respective BPF Data Storage. The trace aggregator thus correlates the telemetry data extracted from telemetry packets to the network flows of their corresponding data packets. Furthermore, the trace aggregator interacts with a Logs / Monitoring module to store the aggregate flow information. The Logs / Monitoring Module is a component that may be used to store all the logs in the relevant format from the trace aggregator. This could be further exposed with its own Application Programming Interfaces (APIs) to be consumed by front-end applications.

[0082] Figure 6 shows a schematic flowchart 600 illustrating an overview of the Linux networking stack and how it is instrumented at various points with eBPF probes for use with SBT techniques disclosed herein. The parts that are utilized specifically for SBT techniques have been marked with dotted lines. The schematic flowchart 600 thus provides an overview of how SBT techniques make use of the host Linux Network Stack. eBPF probes are attached to Netfilter Conntrack submodule and egress network interface for sender hosts. Correspondingly, Conntrack and ingress network interfaces are instrumented for receiver hosts. As mentioned previously, Conntrack helps in mapping the pre-NAT and post-NAT values of a network data packet, and this information at the client (i.e., source host) helps to retrieve the actual server details (i.e., source / destination IP address and L4 port number) which constitutes the telemetry data. Additionally, Conntrack information may be utilized for garbage collection both at the client and the server side.

[0083] Instrumentation at the network interfaces is done with TC programs. The ‘egress’ instrumentation (which may include the injector shown in figure 4), is particularly advantageous for SBT purposes because it implements the cloning-based telemetry packet generation described above (and in more detail below, with respect to figure 7). The ingress instrumentation is a parser, deployed at destination hosts, that looks for the telemetry packets at the receiver host. Since both Conntrack and network interfaces are instrumented in the kernel space, there are negligible packet processing latency penalties. This speed advantage makes SBT techniques suitable not only for virtual cloud deployments but also, beneficially, for high-speed physical DCNs as well.

[0084] Figure 7 illustrates the functionality of an injector module 404, which corresponds to the injector 404 shown in figure 4. As mentioned above, the injector can be realized in two modes: a) cloning mode, which operates in the kernel space, and b) raw socket mode, which operates in the user space. The main difference between the two modes lies in the level at which the modes are implemented. It should be appreciated though that both modes are capable of producing a side-band telemetry packet with the same parameters and configuration, and as described above, i.e., a telemetry packet having network flow instructions that is configured to cause the telemetry packet to take the same path through the communications network as the data packet used as the basis to form the telemetry packet.

[0085] Figure 7 illustrates the injector operating in cloning mode. When operating in cloning mode, the injector determines a suitable data packet to use as a basis for telemetry monitoring, and makes a copy of the data packet. The injector then overwrites the (pay load) data of the copied packet with the telemetry data obtained about the data packet (the telemetry data may have been). The injector keeps the header value the same as the original packet, in order that the telemetry packet that is configured to take the same path through the communications network as the copied data packet.

[0086] Figure 7 shows two data packets 700, 702 being received at a host. The injector selects the first data packet 700 (labelled with a ‘1 ’) as a suitable packet from which to clone a telemetry packet 202. The choice of a suitable packet depends on the implementation. In the case of TCP, choosing the first data packet is probable because the telemetry information is sent only when the connection between the end-points is confirmed and proceeds forward to exchange data. The header of the data packet 700 is copied directly into the telemetry packet, and so the header value 208 in the telemetry packet is, in this case, identical to the header value in the data packet 700. The payload 704 (which in this case is L4 payload) of the data packet (marked “application data” in figure 7), once copied into the telemetry packet, may be overwritten by, or replaced with, the telemetry data 210 obtained in respect of the data packet. Once the telemetry packet 203 has been obtained by incorporating the telemetry data into the payload region, the two original data packets 700, 702, and the side-band telemetry packet 203 are output for transmission to the destination node defined by the destination address contained in the header value 208.

[0087] In a more detailed, specific, example of the cloning mode, the injector runs in the egress network interface of the host, preferably as a traffic control (TC) type of eBPF program which has the ability to clone packets. The header, being a 5-tuple value of the packet (i.e., containing the five values of i) source address, ii) source port number, iii) destination address, iv) destination port number, and v) protocol) that identifies a network flow for the data packet, remains the same. Thus, the telemetry packet is, in effect, another flow packet generated by the injector, and the telemetry data is processed in the same way by subsequent hosts within the network. After cloning, a look-up is done for the relevant flow (i.e., the relevant telemetry data associated with a data flow of the data packet on which the cloning is performed) in the BPF Data Storage and the corresponding telemetry data is extracted and written in the L4 payload 210, thus overwriting the original payload copied from the data packet 208. Additionally, though not shown in figure 7, the telemetry packet is marked with a unique signature that serves as an identification of telemetry packets. The unique signature allows the destination host (specifically, the parser 408 configured in the destination host) to distinguish between telemetry packets and data packets in spite of them potentially sharing the same header value. When the telemetry packet is formed and ready to be transmitted, the packet is processed further at the egress interface and sent out of the host. The cloning mode runs in the kernel space (i.e., at the operating system level). The alternative mode, the raw socket mode, does not run in the kernel space. In the raw socket mode, the injector initiates the establishment of an L4 client socket. The 5-tuple value for a new telemetry packet can, in principle, be set as desired by the user, e.g., to mimic the 5-tuple value contained in the original data packet. The telemetry data is written into this packet by performing similar look-ups in the BPF Data Storage. Next, the telemetry packet is sent further down in the network stack for usual processing, as described above. The raw socket mode traverses the user space partly during the socket establishment phase. In both the raw socket mode and the cloning mode, it is generally assumed that the network has a fixed routing or a perflow routing, and so packets generated using the raw socket mode are still intended, and configured, to follow the same path through the network as the data packets from which they have been cloned. This is because the routing decision is made based on 5-tuple (for TCP / UDP) value, or other fixed protocol attributes (e.g. ICMP type and code).

[0088] It should nevertheless be appreciated that, although the kernel space used by the cloning mode provides a reduction in latency compared to the user space raw socket mode, the raw socket mode still retains the fundamental advantages of the side-band telemetry techniques disclosed herein. For example, the raw-socket SBT methods are scalable since they do not require additional infrastructure, and they can still generate accurate topology of large networks. Further, raw-socket-based side-band telemetry packets are more convenient than typical in-band packets since more space is provided (i.e., in the L4 pay load) to store telemetry data in a side-band packet. As with the cloning mode, the raw-socket mode is also agnostic to policy (i.e., rawsocket telemetry packets do not require additional network policies to be able to flow in the network).

[0089] Figure 8 shows a schematic 800 demonstrating time to live, TTL, information may be used to help construct an end-to-end topology of an application within a network. Specifically, figure 8 shows how TTL information, contained in the IP header of a packet, is used to re-construct the end-to-end topology for a network flow. When an IP packet flows through a network, the TTL value is reduced by one at each consecutive hop in the network, whether or not the hop performs NAT and whether or not the hop is configured with the side-band technology functionality disclosed herein. If a source host creates a packet with a TTL value of t, the TTL at the first network hop would be t - 1 , at the second hop would be t - 2, and so on. This functionality may advantageously be used in conjunction with SBT techniques to calculate distances between the source and each network hop in the path of a network flow.

[0090] At amiddle / intermediary host having SBT functionality, when an injector (e.g., injector 406 shown in figure 4) clones a suitable data packet, the current TTL value contained within the data packet (i.e. - the TTL value when received at the respective middle host) is copied and stored as a part of the telemetry data incorporated within the telemetry packet. Once stored in the telemetry data of the telemetry packet, the TTL is preserved and does not change at further hops. The stored TTL value can therefore be used to derive of the distance of the respective middle host from the source. The derivation of this distance, and thus the end- to-end topology, may be performed when all telemetry packets from all the hops in the network, including the source host, are aggregated at the destination host.

[0091] In detail, figure 8 shows a communication schematic 800 between a client application running in a K8s source pod 104, and a server pod 106 running inside a self-hosted container at the destination host 206. The middle host 402 is a NAT gateway that performs NAT and thus translates the source (IP) address for traffic flowing from the source to the destination host. The points of telemetry data collection are shown by numbers 1, 2, 3, and 4. In the presence of NAT, the IP addresses are translated as shown in the table 802. At point 1, the Original IP Port, in this example 10.0.0.8:51700, gets translated to 192.168.3.14:37300 as shown in the first row of the table 800. However, NAT does not affect the TTL value in the IP header (which thus remains 64. Assuming all the flow packets traverse the same route, at points 1, 2, 3 and 4 the TTL values 64, 58, 53 and 52 are extracted from the data packets and sent as a part of telemetry data contained within the telemetry packets (which, though not shown, will be created at injectors 404 and 406). The TTL values in adjacent rows of the table each differ by more than 1 , which means that there are multiple intermediary hops between the visible hops, which are not accessible (that is, accessible in the sense that the intermediary hops do not have the SBT functionality described in this disclosure). The TTL information, when aggregated at the destination host, provides the distance between each of the hops. Thus, from this example, the distance between the client pod 104 (whose original IP is 10.0.0.8) and the source host 204 (whose IP address is 192.168.3.14) can be calculated to be 1. Similarly, the distance between the source host 204 (IP address 192.168.3.14) and the middle host 402 (IP address 100.85.1.50) is 5, and between the middle host 402 (IP address 100.85.1.50) and the destination host 206 (IP address 7.182.10.233) the distance is 5. Finally, the distance between the destination host 206 (IP address 7.182.10.233) and the server pod 106 (IP address 172.0.0.2) is 1. The distance information could be crucial to understand the flow path, for example, to correlate a high latency with one of the paths.

[0092] Figure 9 is a flowchart showing a method of generating a side-band telemetry packet according to embodiments of the present disclosure. The flowchart relates to SBT functionality that would be implemented at a source host or middle host, for example, within the injectors 404, 406 shown in figure 4.

[0093] Step S100 comprises receiving, at a node within the communications network, a data packet comprising first network flow instructions identifying at least a source address and a destination address. The network flow instructions may therefore be a header value, for example, a 5-tuple value.

[0094] Step SI 02 comprises, in dependence on determining that the first network flow instructions have not been previously received at the node, generating a telemetry packet based on the data packet. Steps SI 04 and SI 06 described the method of generating the telemetry packet. As described above, the generation of the telemetry packet may be performed at the kernel level, e.g., using the ‘cloning mode’, or by using a ‘raw-socket’ mode implemented at a user-level.

[0095] Step SI 04 comprises, in dependence on the first network flow instructions, incorporating second network flow instructions within the telemetry packet that are configured to cause the telemetry packet to take the same path through the communications network as the received data packet. In a cloning mode example, this comprises copying the network instructions of the data packet to form the same header in the telemetry packet. Because the header values are the same, the two packets should be treated identically by subsequent hops in the network and therefore take the same path through the communications network.

[0096] Step SI 06 comprises obtaining telemetry data associated with a data flow of the data packet and incorporating the obtained telemetry data into the telemetry packet. The telemetry data may be obtained, as mentioned above, by a look-up in the BPF data storage, from which the telemetry data for the relevant data packet is determined, and subsequently written into the ‘payload portion of the data in the telemetry packet. Thus, whatever data was contained in the payload of the data packet is overwritten, or replaced, in the copy of the data packet by the determined telemetry data for that data packet. The telemetry packet is thus formed following the incorporation of the telemetry data into the pay load area of the telemetry packet.

[0097] Step S 108 comprises transmitting the data packet and the telemetry packet towards a destination node defined by the destination address.

[0098] Figure 10 is a flowchart showing a method monitoring telemetry, specifically from the point of view of a destination host. In a specific example, the flowchart relates to SBT functionality that would be implemented at a parser of a destination host, for example, parser 408 as shown in figures 4 and 5.

[0099] Step S200 comprises receiving, at a destination node within the communications network, a data packet comprising first network flow instructions identifying at least a source address and a destination address, wherein the destination address defines the destination node. Step S202 comprises receiving one or more telemetry packets having been generated, based on the data packet, by preceding respective one or more nodes within the communications network, wherein the one or more telemetry packets have, from the point of generation, travelled the same path through the communications network as the received data packet.

[0100] Step S204 comprises determining that the one or more telemetry packets comprise network flow instructions that match the first network flow instructions. In some examples, an explicit check that the data is matching may not be necessary. For example, in some cases the telemetry packets may have been generated to include a unique signature which i) identifies the packet to the destination host as a telemetry packet, and / or ii) correlates the telemetry packet to the data packet from which it has been copied. Therefore, the destination may simply determine that a packet must have flow instructions that match flow instructions from a data packet by virtue of identifying the unique signature. In other words, determining the presence of the unique signature may be used as a proxy to determining that the one or more telemetry packets comprise network flow instructions that match the first network flow instructions.

[0101] Step S206 comprises, in response to the determining performed at step S204, extracting telemetry data from each of the one or more telemetry packets. A specific example of this is described above in more detail in respect of the parser shown in figures 4 and 5.

[0102] Figure 11 shows an apparatus 1000 comprising a processor 1002 configured to process the datasets in the manner described herein. For example, the processor 1002 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The apparatus 1000 comprises a memory 1004 which is arranged to communicate with the processor 1002. Memory 1004 may be a non-volatile memory (e.g., permanent storage). The processor 1002 may also comprise a cache (not shown in Figure 10), which may be used to temporarily store data from memory 1004. The apparatus 1000 may comprise more than one processor 1002 and more than one memory 1004. The memory 1004 may store data that is executable by the processor 1002. The processor 1002 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor 1002 to perform its methods in the manner described herein.

[0103] In this disclosure, when the subject of a phase is described as being "configured to" or “arranged to”, followed by a term defining a condition or function, this is used to indicate that the subject of the phrase is in a state in which it has that condition, or is able to perform that function, without the subject being modified or further configured.

[0104] Some implementations may be described using the expressions “one / an embodiment” or “one / an implementation” or “one / an example”, along with their derivatives. These terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in some implementations” in various places in the specification are not necessarily all referring to the same embodiment. Moreover, unless otherwise noted the features described above are recognized to be usable together in any combination. Thus, any features discussed separately may be employed in combination with each other unless it is noted that the features are incompatible with each other.

[0105] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present disclosure may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the appended claims. The foregoing description of example embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Many modifications and variations are possible in light of this disclosure. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto. Future filed applications claiming priority to this application may claim the disclosed subject matter in a different manner and may generally include any set of one or more limitations as variously disclosed or otherwise demonstrated herein.

[0106] Abbreviations and Definitions

Claims

CLAIMS1. An apparatus for monitoring telemetry of data flows within a communications network, wherein the apparatus is communicatively coupled with a node within the communications network, the apparatus configured to: receive a data packet comprising first network flow instructions identifying at least a source address and a destination address; in dependence on determining that the first network flow instructions have not been previously received by the apparatus, generate a telemetry packet based on the data packet, the generating comprising: in dependence on the first network flow instructions, incorporating second network flow instructions within the telemetry packet that are configured to cause the telemetry packet to take the same path through the communications network as the received data packet; obtaining telemetry data associated with a data flow of the data packet and incorporating the obtained telemetry data into the telemetry packet; and cause the data packet and the telemetry packet to be transmitted towards a destination node defined by the destination address.

2. The apparatus of claim 1, wherein the apparatus is configured to include the source address, identified in the data packet, in the telemetry data incorporated into the telemetry packet.

3. The apparatus according to claim 1 or 2, wherein the apparatus is configured to generate the telemetry packet by duplicating the data packet and overwriting payload data of the duplicated data packet with the obtained telemetry data to thereby obtain the telemetry packet.

4. The apparatus according to any preceding claim, wherein the apparatus is configured to obtain the telemetry data associated with the data flow of the data packet by looking-up network connection information from a data store, accessible by the apparatus, in dependence on the first network flow instructions.

5. The apparatus according to any preceding claim, wherein the apparatus is configured to copy a time to live, TTL, indicator from the data packet into the telemetry packet.

6. The apparatus according to claim 1, wherein the apparatus is configured to enable generation of a telemetry packet via a socket in a user space, wherein the socket is configured to enable a user to generate telemetry packet to generate the second network flow instructions for incorporation within the telemetry packet.

7. The apparatus according to any preceding claim, wherein the source address and a destination address are IP addresses, and wherein the apparatus is configured to determine that the source IP address of the received data packet has undergone network address translation prior to transmission towards the destination node, and wherein the obtained telemetry data incorporated within the telemetry packet comprises a record of the source IP address and a translated source IP address.

8. The apparatus of claim 6, wherein the apparatus is configured, in response to determining that received data packet has undergone network address translation, to store telemetry mapping data indicating an association between the source IP address, the translated source IP address, and the destination IP address.

9. The apparatus according to any preceding claim, wherein apparatus is configured to include a telemetry identifier in the telemetry packet configured to allow a destination host, defined by the destination address, to distinguish between the data packet and the telemetry packet.

10. The apparatus according to any preceding claim, wherein the apparatus is configured to use extended Berkeley Packet Filter, eBPF, technology, wherein the generation of the telemetry packet uses eBPF technology and is processed at an operating system level of the apparatus.

11. An apparatus for monitoring telemetry of data flows within a communications network, wherein the apparatus is comprised within a destination node of the communications network, the apparatus configured to: receive a data packet comprising first network flow instructions identifying at least a source address and a destination address, wherein the destination address defines the destination node; receive one or more telemetry packets having been generated, based on the data packet, by preceding respective one or more nodes within the communications network, wherein the one or more telemetry packets have, from the point of generation, travelled the same path through the communications network as the received data packet; determine that the one or more telemetry packets comprise network flow instructions that match the first network flow instructions, and in response to the determining: extract telemetry data from each of the one or more telemetry packets.

12. The apparatus as claimed in claim 11, wherein the apparatus is further configured to determine, in dependence at least on the extracted telemetry, at least a portion of a topology of the communications network.

13. A method for monitoring telemetry of data flows within a communications network, the method comprising : receiving, at a node within the communications network, a data packet comprising first network flow instructions identifying at least a source address and a destination address; in dependence on determining that the first network flow instructions have not been previously received at the node, generating a telemetry packet based on the data packet, the generating comprising: in dependence on the first network flow instructions, incorporating second network flow instructions within the telemetry packet that are configured to cause the telemetry packet to take the same path through the communications network as the received data packet; obtaining telemetry data associated with a data flow of the data packet and incorporating the obtained telemetry data into the telemetry packet; and transmitting the data packet and the telemetry packet towards a destination node defined by the destination address.

14. A method for monitoring telemetry of data flows within a communications network, the method comprising: receiving, at a destination node within the communications network, a data packet comprising first network flow instructions identifying at least a source address and a destination address, wherein the destination address defines the destination node; receiving one or more telemetry packets having been generated, based on the data packet, by preceding respective one or more nodes within the communications network, wherein the one or more telemetry packets have, from the point of generation, travelled the same path through the communications network as the received data packet; determining that the one or more telemetry packets comprise network flow instructions that match the first network flow instructions, and in response to the determining: extracting telemetry data from each of the one or more telemetry packets.

15. A computer program stored in non-transitory form and including code instructions which when executed on one or more processor, causes the one or more processor to execute the method according to claim 13 or 14.

Images