A low-overhead in-band full-network telemetry method and system based on compressed sensing

Abstract

The application discloses a low-overhead in-band full-network telemetry method and system based on compressed sensing, and belongs to the technical field of computer network monitoring and management. In the control plane, a measurement path is generated according to a network topology, and a sensing matrix is constructed; by using a multi-path aggregation value and the sensing matrix, high-precision original metadata of full-network links are reconstructed through sparse optimization solving with band constraint; in the data plane, a source routing is used to guide a probe packet to transmit along a preset path, and metadata are compressed and aggregated at each hop, so that the packet only carries an aggregation value with a fixed length. While realizing fine-grained full-network state visibility, the application fundamentally avoids the problem that the overhead increases with the path, significantly reduces bandwidth consumption, avoids the MTU limit, has high reconstruction accuracy and strong deployment feasibility, and solves the problems that the existing in-band network telemetry (INT) causes large overhead, is linearly related to the path length, is easy to exceed the MTU limit and the like due to hop-by-hop addition of metadata.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer network monitoring and management, and particularly relates to a low-overhead in-band full-network telemetry method and system based on compressed sensing. BACKGROUND

[0002] Fine-grained network visibility is an important basis for network management, which can provide key data support for multiple applications such as traffic engineering, anomaly detection, fault localization, and congestion control. In recent years, the application of in-band network telemetry (INT) technology has greatly improved network observability, and has been regarded as one of the important development directions of future network management.

[0003] INT realizes fine-grained monitoring of network state by recording key information of network devices in real time during data packet transmission. Specifically, the source node receives the telemetry command issued by the controller, and inserts its own metadata (such as port state, link utilization, etc.) into the data packet. The telemetry instruction is used to instruct the intermediate network devices on the data packet path to collect the required INT metadata, and insert it into the data packet hop by hop. Finally, the end node extracts all the INT metadata from the data packet and reports it to the controller, while continuing to forward the original data packet to the destination host. The entire telemetry process is transparent to the upper layer service and does not affect the normal transmission of data packets.

[0004] Since INT needs to attach metadata at each hop on the data packet forwarding path, the number of network devices on the path directly determines the length of the telemetry information of the INT data packet. With the growth of the path, the metadata field in the INT data packet grows linearly, thereby bringing significant network overhead: on the one hand, a large amount of network bandwidth is occupied, affecting the transmission of normal business traffic; on the other hand, network switches need additional computing resources to process telemetry data, increasing the processing burden of the device; in addition, the uploading of a large amount of telemetry data may also cause congestion of the southbound interface bandwidth, affecting the interaction efficiency of the control plane and the data plane. In the full-network telemetry scenario, under the condition that each hop of metadata has visibility, this problem is particularly prominent, which seriously limits the feasibility of INT in large-scale network environments.

[0005] To address the high overhead problem of INT, various optimization schemes have been proposed. For example, sINT and Sel-INT reduce packet overhead by reducing the spatio-temporal coverage of telemetry, but these methods rely on sampling mechanisms, which can result in incomplete data and loss of network status information at critical moments, making it difficult to meet the requirements of fine-grained network-wide measurement. The PINT scheme uses flow-level information aggregation to encode telemetry data into multiple packets, thereby reducing the overhead of individual packets, but it cannot provide detailed link-level telemetry data, making it difficult to meet the application requirements of congestion diagnosis, link-level traffic engineering, and other applications that rely on accurate link information. In addition, although Sketch-based methods can reduce the transmission burden of some telemetry data, they still rely on traditional INT mechanisms and do not fundamentally solve the inherent overhead problem of INT.

[0006] In recent years, the INT-MC scheme uses matrix completion technology to optimize telemetry data collection methods by planning measurement paths and measuring only part of the links, and using matrix filling algorithms to infer the metadata of unmeasured links, thereby reducing the overhead of in-band telemetry. However, this scheme still faces the following challenges in actual deployment: on the one hand, it still needs to carry telemetry metadata for the complete path in the data packet, which may result in a data packet length exceeding the maximum transmission unit (MTU) when the path is long or multiple telemetry information needs to be collected, affecting the effective transmission of data; on the other hand, INT-MC requires a large amount of data accumulation to perform matrix filling, otherwise the accuracy of the inference cannot be guaranteed, limiting its applicability in small-scale or dynamic network environments.

[0007] In summary, although existing INT schemes have made some progress in reducing telemetry overhead, they still face two challenges: first, some low-overhead measurement methods rely on sampling mechanisms, resulting in incomplete telemetry data and failing to meet the requirements of fine-grained network-wide monitoring; second, optimization schemes such as INT-MC are limited by MTU constraints in actual deployment and require a large amount of data accumulation to be effective, making the implementation threshold high. Therefore, how to minimize bandwidth overhead while ensuring per-hop metadata visibility and avoiding data packet length exceeding the limit remains a key technical problem that needs to be solved. SUMMARY

[0008] In view of the above-mentioned deficiencies of the prior art, the present application provides a low-overhead in-band full-network telemetry method and system based on compressed sensing. The present application innovates the traditional "hop-by-hop addition" metadata mode to "hop-by-hop compression aggregation". In the data plane, the probe packet is transmitted along the predetermined path, and each hop device performs linear aggregation on the local metadata and the fixed-length digest value in the packet, thereby greatly reducing the bandwidth and processing overhead. In the control plane, using a small amount of aggregated digest and a pre-built sensing matrix, a sparse reconstruction algorithm that fuses network physical constraints is used to restore the original metadata of all links in the network with high precision, thereby ensuring the visibility of the network link level while fundamentally avoiding the problem of packet exceeding MTU.

[0009] The specific technical solutions of the present application are as follows:

[0010] The present application discloses a low-overhead in-band network telemetry method based on compressed sensing, which comprises compressing and measuring network telemetry data in the data plane of a network switch, and restoring complete telemetry data based on a sparse reconstruction technology in the control plane, including the following steps:

[0011] S1, measurement path and sensing matrix construction: based on the network topology, a plurality of measurement paths are generated, and a sensing matrix is constructed according to the measurement paths, wherein the rows of the sensing matrix correspond to the total number of measurement paths m, and the columns correspond to the total number of unidirectional links in the network n;

[0012] S2, data plane compression collection: according to the measurement paths, the source routing mechanism is used to guide the transmission of probe packets in the network; in the data plane, when the probe packet passes through each hop network device on the measurement path, the network device performs aggregation operation on the metadata of the current hop or link and the digest value carried in the probe packet, generates an updated digest value, and transmits an aggregated digest value of a predetermined fixed length on the entire measurement path, wherein the fixed length is determined according to the total number of metadata types to be collected and the value range of each metadata, so as to prevent numerical overflow in the aggregation process;

[0013] S3, measurement digest collection: after the probe packet reaches the destination node or the designated collection point, the fixed-length aggregated digest value is extracted;

[0014] S4, control plane reconstruction: in the control plane, using the aggregated digest values collected from the plurality of measurement paths and the sensing matrix, a sparse reconstruction algorithm based on compressed sensing theory is executed to restore the predetermined metadata of the link level in the network.

[0015] Preferably, the specific method for generating multiple measurement paths in step S1 is as follows: a random walk-based strategy is adopted, in which the controller randomly selects m nodes as starting nodes in the network topology graph, and in each step, the next hop node is randomly selected from all the neighboring nodes of the current node with equal probability and moved, and this is repeated t times to generate a random path of length t; by performing m independent walks, m measurement paths with both path connectivity and matrix non-correlation are generated.

[0016] The value of m is the total number of measurement paths. The value of m is determined according to the compressed sensing theory: for the accurate reconstruction of K-sparse signals, m satisfies m ≥ C·K·log(n / K), where C is a constant, K is the sparsity, and n is the total number of network links.

[0017] The value of m is in the range of [5, 20], in order to balance measurement overhead and reconstruction accuracy;

[0018] The t is the number of hops for a single path, and the value of t ranges from [4, 10]. If the number of hops is too short, it cannot provide sufficient information gain, while if the number of hops is too long, it may introduce redundancy and reduce the sparsity of the sensing matrix.

[0019] The measurement path is implemented using a source routing mechanism.

[0020] Furthermore, the specific method for constructing the sensing matrix in step S1 is as follows: based on the m measurement paths generated in step S1, the controller constructs a matrix with dimension m in memory. Routing-aware sensor matrix .in, To measure the total number of paths, This represents the total number of unidirectional links in the network. This matrix is ​​a binary matrix, and its elements are... Defined as: if the first Measurement path After the first time in the network Link Then matrix elements ,otherwise The sensing matrix It accurately describes the linear relationship of the subsequent data plane measurement process and is the core basis for controlling the plane to reconstruct data;

[0021] Furthermore, the measurement path generated in step S1, which has the characteristics of path connectivity and matrix non-correlation, can enhance the accuracy of compressed sensing reconstruction.

[0022] Furthermore, in step S1, the elements in the sensing matrix are used to indicate whether a certain measurement path passes through a certain link, in order to maintain the linearity of compressed sensing measurements.

[0023] Preferably, in the step S2, the aggregation operation is specifically weighted linear accumulation: each network device performs weighted linear accumulation on the telemetry data of the node and the existing aggregated digest value in the probe packet as it passes; in actual deployment, considering the hardware limitations of programmable switches, the weight is preferably set to 1, that is, simple addition operation is adopted, so as to fully utilize the efficient support of switch hardware for basic arithmetic operations.

[0024] Preferably, the step S2 is implemented in a programmable switch environment, the telemetry packet format is defined based on P4 or similar protocols, and the hop-by-hop aggregation operation is performed in the switch pipeline.

[0025] Preferably, the structure design of the probe packet in the step S2 includes a source routing part and an INT metadata part, the source routing part carries pre-defined measurement path information to ensure that the probe packet can be strictly transmitted according to the routing-aware matrix; the INT metadata aggregation part includes a short INT instruction header and at least one fixed-length digest field, which are used for hop-by-hop aggregation in the data plane.

[0026] The INT instruction header is used to indicate what type of metadata needs to be collected by the network device on the path; each digest field corresponds to a type of metadata, and is used to roll up the aggregated value of the metadata in the entire transmission path; wherein the metadata type includes queue delay, link utilization, and packet loss.

[0027] Further, in the step S2, the source routing mechanism is used to accurately control the forwarding path of the probe packet, which aims to ensure that the actual measurement process in the data plane strictly follows the linear model defined by the sensing matrix .

[0028] Further, in the source routing mechanism in the step S2, the controller encodes the hop-by-hop information (preferably the sequence of egress port IDs of each switch along the way) of each path into the source routing field in the packet header when generating the probe packet of the path; when the network device receives the probe packet carrying the source routing information, it will ignore the regular routing table lookup process, and directly forward according to the next hop information specified in the packet header, thereby ensuring that the trajectory of the probe packet completely corresponds to the row vector of the sensing matrix.

[0029] Further, in the step S2, the format design of the probe packet mainly includes: a source routing part and an INT metadata part.

[0030] Further, in the step S2, the source routing part is usually placed between the Ethernet header and the IP header; the INT metadata aggregation part is located after the transport layer header (such as the UDP header), and each network device performs accumulation based on the existing aggregated value in the probe packet to support transparent transmission and reduce bandwidth overhead.

[0031] Further, the data plane compression collection process in step S2 is embodied as a hop-by-hop compression aggregation operation: when the probe packet carrying the source routing information is transmitted along the predetermined path in the data plane, each hop network device (preferably a programmable switch supporting P4 language programming) on the path will in turn perform the following atomic operations to realize efficient compression aggregation and forwarding of metadata:

[0032] (1) Recognize that this is a probe packet through the packet specific field (EtherType);

[0033] (2) Analyze the INT instruction header to determine the type of metadata to be collected;

[0034] (3) Obtain the corresponding metadata instantaneous value (denoted as ) of the current hop device or its designated outgoing link;

[0035] (4) Perform an efficient linear aggregation operation on the metadata value and the corresponding type of digest value (denoted as ) carried in the packet; the simplest and most effective way is linear accumulation, i.e. = + ; this addition operation is a nanosecond-level operation for the arithmetic logic unit (ALU) of modern switch chips, almost without introducing additional processing delay;

[0036] (5) Update the old digest field in the packet using the newly generated digest value ;

[0037] (6) Forward the packet to the next hop according to the source routing header information.

[0038] Further, in step S2, the metadata refers to various types of state information that can be collected by network devices in the data plane, including network performance indicators such as queueing delay, hop latency, link utilization, queue length, etc. These metadata reflect the real-time running state of network devices and links and are standard monitoring objects in network telemetry technology.

[0039] Further, in step S2, the aggregation operation mode can ensure that the telemetry data on the entire probe path is aggregated hop by hop during transmission, maintaining a linear accumulation relationship to reduce the overhead while improving the recoverability of telemetry data.

[0040] Further, the specific method of step S3 is: when the probe packet completes the traversal of the whole path, it reaches the endpoint node (INT Sink node) of the path or a designated centralized collection server; then, the node extracts the fixed-length aggregated digest value from the packet and uploads it to the central controller through the out-of-band network.

[0041] Preferably, the sparse reconstruction algorithm in step S4 solves the optimization problem based on the alternating direction multiplier method (ADMM), first restores the sparse vector, and then obtains the complete telemetry data through a predetermined transformation matrix; and in the solving process, the physical boundary constraint information of the predetermined metadata is fused.

[0042] Preferably, the physical boundary constraint information is the value interval of each link metadata derived based on the linear equation set established by the sensing process and the physical value range of the metadata itself.

[0043] Further, in step S4, the core idea of the sparse reconstruction algorithm is to transform the reconstruction problem from directly solving the dense to solving the sparse . The relationship between the two is , where is the corresponding inverse transformation matrix. The reconstruction model thus becomes .

[0044] Further, in step S4, when deriving the value interval of each link metadata, in order to significantly improve the reconstruction accuracy and accelerate the convergence of the algorithm, an innovative preprocessing step is introduced before formally performing sparse reconstruction: for each to-be-solved link metadata , a more compact effective value interval than physical common sense is derived, and this step fuses two kinds of information:

[0045] (1) Linear equation set constraint: the linear equation set established by the sensing process contains the internal correlation between variables itself;

[0046] (2) Physical priori constraint: any network metadata has its inherent physical value range, and preferably the value range of link utilization rate must be between [0, 1], and the queue length must be between [0, the maximum buffer depth of the switch].

[0047] Further, in step S4, the specific algorithm for deriving the value interval of each link metadata is:

[0048] (1) Gaussian elimination is performed on the augmented matrix to transform it into row-echelon form, thereby obtaining the linear expression between the leading variables and the free variables;

[0049] (2) Taking the physical value interval of the free variable as the starting point, the value range of the main variable is derived along the linear expression through interval arithmetic operation;

[0050] (3) Taking the intersection of the derived range and the physical range of the main variable itself, a more compact interval is obtained;

[0051] (4) Iterating the above process until the variable interval converges, and finally obtaining a globally consistent and greatly reduced solution space.

[0052] Further, in step S4, the sparse reconstruction problem can be represented as a norm minimization problem based on the solution of the ADMM constrained optimization problem, specifically including:

[0053] (1) Construct an optimization problem that integrates the compact boundary constraint:

[0054]

[0055] (2) Solve by using the alternating direction multiplier method ADMM, decompose the original problem into multiple more easily solvable subproblems by introducing auxiliary variables and augmented Lagrangian functions, effectively decoupling the norm in the objective function and the complex boundary constraint imposed on the variable ;

[0056] Each subproblem after decomposition usually has an efficient closed-form solution or a simple projection solution, in which the subproblem involving the norm can be solved by a soft threshold shrinkage operator; the subproblem involving the quadratic term and the boundary constraint is solved by projecting the solution of a linear equation set to the precomputed effective interval;

[0057] (3) The optimal sparse frequency domain vector is obtained by iterative solution of the ADMM algorithm , and then the original high-precision metadata vector of all n links in the entire network can be finally recovered through a one-time inverse transformation.

[0058] Further, in step S4, the sparse reconstruction algorithm integrates the physical boundary constraint information of the predetermined metadata to improve the reconstruction accuracy and accelerate the algorithm convergence.

[0059] Further, the method can realize high-precision network state monitoring under low overhead conditions through real-time monitoring and analysis in a large-scale data center network or wide area network environment.

[0060] The application also discloses an application based on the method, which is applied to a large-scale data center network, a wide area network environment or other complex topological environments, and can realize high-precision network state monitoring under low overhead conditions.

[0061] The application further discloses a low-overhead in-band network telemetry system based on compressed sensing, which is used for realizing the method, and the system comprises the following components.

[0062] A plurality of network devices are configured to perform the following operations: receiving a probe packet carrying source routing information; identifying predetermined metadata of a current device or a connection link thereof; performing an aggregation operation on the predetermined metadata and a digest value carried in the probe packet to generate an updated digest value, the digest value having a fixed length; replacing an old digest value in the probe packet with the updated digest value, and forwarding the probe packet according to the source routing information;

[0063] At least one control entity is configured to perform the following operations: generating a plurality of measurement paths based on a network topological structure; constructing a corresponding sensing matrix; initializing a probe packet, including source routing information based on the measurement paths and an initial digest value; receiving the fixed-length aggregated digest value from the probe packet arriving at a destination node or a designated collection point; and performing a compressed-sensing-based sparse reconstruction algorithm by using the collected aggregated digest value and the sensing matrix to recover the predetermined metadata at a link level in the network.

[0064] Preferably, the network device is a programmable switch, which is programmed based on a P4 protocol to realize the aggregation operation; and the control entity is an SDN controller or a network management server.

[0065] The application has the beneficial technical effects that:

[0066] 1. The application aggregates metadata on a path into a fixed-length digest for transmission in a data plane by using a compression aggregation technology, so that the telemetry bandwidth overhead is irrelevant to the length of the measurement path, and the bandwidth overhead can be reduced to 1 / 5 or even lower of the original bandwidth overhead of a traditional INT method, thereby greatly saving network resources. Meanwhile, since the fixed-length digest is transmitted, the problem of packet exceeding the MTU limit caused by too long path or too many types of metadata in the traditional INT method is fundamentally avoided, and the universality and robustness of the telemetry scheme are enhanced.

[0067] 2、Although only the compressed summary information is transmitted, with the help of the compressed sensing sparse reconstruction algorithm of the control plane, the application can restore the detailed, link-level metadata of all monitored links in the network, realize the global network state visibility, and overcome the granularity sacrifice defect of the existing low-overhead scheme. By adopting random walk to generate a deployable sensing matrix with good mathematical properties (low RIP), and combining the advanced reconstruction algorithm based on ADMM and incorporating physical boundary constraints, the application can realize high-precision metadata recovery (the preferred normalized mean square error NMSE can be less than 0.1), ensuring the accuracy and usability of the telemetry data.

[0068] 3、In addition, the sensing matrix construction (random walk) and path control (source routing) mechanisms proposed by the application both consider the deployability of the actual network environment, can be effectively implemented in programmable switches or SDN environments, and ensure the practicability and feasibility of the technical solution. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A CS-INT architecture diagram provided by the application;

[0070] Figure 2 A workflow diagram of the in-band network-wide telemetry method described by the application;

[0071] Figure 3 A CS-INT probe packet diagram. Among them, the ETH header is an Ethernet header; the SR header is a segment routing header; the IPv4 header is an IPv4 header; the UDP header is a user datagram protocol header; the INT header is an in-band network telemetry header; and the INT data is in-band network telemetry data. The SR header includes multiple segment routing nodes (SR_1 to SR_n), each node consisting of a port ID (PortID) and a flag bit (flag). The INT data includes multiple data summaries (digest_1 to digest_4). DETAILED DESCRIPTION

[0072] The application will be specifically described below in combination with the drawings and examples. Obviously, the described examples are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application.

[0073] The application provides a low-overhead in-band network telemetry (INT) method and system based on compressed sensing (CS), hereinafter referred to as CS-INT. The scheme aims to solve the core problems of the linear growth of measurement overhead with path length, large bandwidth occupation and possible exceeding of MTU limit caused by hop-by-hop embedding of metadata in the existing INT technology (especially the classic INT mechanism); at the same time, the application is also committed to overcoming the shortcomings of the existing low-overhead INT scheme in ensuring fine granularity (link level) and full-network visibility. The core of the application lies in utilizing the inherent sparsity of network metadata in the Fourier transform domain, and realizing comprehensive and accurate perception of network status under low overhead by innovatively combining the data plane compression aggregation measurement mechanism with the control plane high-precision sparse reconstruction mechanism.

[0074] Embodiment 1: Full-network telemetry system based on compressed sensing

[0075] This embodiment describes the system architecture for implementing the method of the application. Figure 1 A typical deployment architecture of the CS-INT system of the application is shown.

[0076] (1) System composition:

[0077] The CS-INT system mainly includes two parts of a controller and a data plane network device;

[0078] The controller, as the core control entity of the system, is deployed in the control plane and is responsible for global measurement planning, probe packet generation and metadata reconstruction. Specifically, the controller generates measurement paths and constructs a sensing matrix based on network topology information; creates a probe packet containing source routing information and an initial digest value and injects it into the network; collects the aggregated digest reported by the data plane, executes the sparse reconstruction algorithm based on ADMM, and restores the original metadata of the links in the whole network. The controller can be implemented as an SDN controller, a network management server or a dedicated telemetry analysis platform;

[0079] The data plane network device includes a source switch, an intermediate forwarding switch and a destination switch, all of which use devices supporting programmable data plane, preferably Barefoot Tofino switches based on P4 programming. The source switch is responsible for receiving the probe packet issued by the controller and starting forwarding; the intermediate switch identifies the probe packet, extracts the local metadata (such as Figure 1 different metadata types identified by different colors), performs aggregation operation with the digest value carried by the packet and then continues forwarding; the destination switch completes the aggregation operation of the last hop and reports the final compressed metadata to the controller through the out-of-band management network.

[0080] (2) System workflow:

[0081] During system operation, the controller first performs measurement path planning. Based on the known network topology, the controller generates multiple measurement paths using a random walk strategy and constructs the corresponding sensor matrix. This path information is encoded into probe messages to guide subsequent data plane measurements.

[0082] After generating a probe message, the controller injects it into the source switch. The probe message contains three key parts: source routing information to specify the forwarding path, an INT header to indicate the type of metadata to be collected, and a digest field initialized to zero to store the aggregation result;

[0083] After the probe message enters the data plane, it is forwarded hop-by-hop according to the path specified by the source route. For example... Figure 1 As shown, the packet originates from the source switch, passes through multiple intermediate switches, and finally reaches the destination switch. At each hop, the switch performs the same aggregation operation: identifies the probe packet, extracts local metadata (identified by different colors in the diagram, such as orange, blue, and green representing different types of metadata), adds it to the digest value carried in the packet, updates the digest field, and forwards it to the next hop. This process ensures that compressed metadata accumulates continuously throughout the path, but the packet length remains constant.

[0084] When the probe packet reaches the destination switch, a digest value containing compressed metadata is extracted and reported to the controller via the out-of-band management network. The controller collects digest values ​​from multiple measurement paths and, using the previously constructed sensor matrix, executes a sparse reconstruction algorithm based on compressed sensing theory. By solving an optimization problem, the controller can accurately recover the original metadata information of all links in the entire network from these compressed digests.

[0085] The entire process realizes a shift from "hop-by-hop appending" to "hop-by-hop aggregation". When probe messages are transmitted in the network, they only carry a fixed-length digest instead of an ever-growing sequence of metadata, which greatly reduces telemetry overhead. At the same time, the intelligent reconstruction of the control plane ensures the complete visibility of the entire network status.

[0086] Example 2: Basic Methods for Network-wide Telemetry Based on Compressed Sensing

[0087] The core idea of ​​this invention is to draw on compressed sensing theory and innovate the traditional INT's "hop-by-hop appending" data mode into a "hop-by-hop compressed aggregation" mode. This method performs efficient linear compression measurement of telemetry metadata in the data plane, ensuring that probe messages carry only a fixed-length aggregated value (digest) during transmission. Subsequently, in the control plane, using the collected aggregated values, a high-precision sparse reconstruction algorithm that incorporates network physical characteristics accurately recovers the original metadata of all links in the entire network.

[0088] The basic flow of the present application is described in this embodiment, as shown in the following steps: Figure 2

[0089] S1, measurement path and sensing matrix construction:

[0090] The goal of this step is to build a sensing matrix on the controller side that meets both the network physical topology constraints and has excellent mathematical properties (high non-correlation), which is the theoretical basis and practical premise for realizing high-precision compressed sensing reconstruction.

[0091] Based on the network topology structure, a plurality of measurement paths are generated, and a sensing matrix is constructed according to the measurement paths, wherein the rows of the sensing matrix correspond to the measurement paths, and the columns correspond to the network links.

[0092] S2, data plane compression collection:

[0093] The goal of this step is to use programmable network devices to guide the probe packet to travel according to the preset path in the data plane, and to complete the online compression and aggregation of metadata with extremely low overhead.

[0094] According to the measurement path, the source routing mechanism is used to guide the transmission of the probe packet in the network; in the data plane, when the probe packet passes through each hop network device on the measurement path, the network device performs aggregation operation on the predetermined metadata of the current hop or link and the digest value carried in the probe packet, generates an updated digest value, and transmits a fixed-length aggregated digest value on the entire measurement path.

[0095] S3, measurement summary collection:

[0096] After the probe packet reaches the destination node or the designated collection point, the fixed-length aggregated digest value is extracted.

[0097] S4, control plane reconstruction:

[0098] This step is one of the technical cores of the present application, and its goal is to use the collected digest values (constitute the observation vector y) and the pre-constructed sensing matrix , through advanced algorithms to accurately restore the original metadata of all links in the network;

[0099] In the control plane, using the aggregated digest values collected from the plurality of measurement paths and the sensing matrix, based on the compressed sensing theory, the sparse reconstruction algorithm is executed to restore the predetermined metadata of the network link level.

[0100] Embodiment 3: optimization construction method of measurement path and sensing matrix

[0101] ​This embodiment is a further refinement of step S1 in embodiment 2, focusing on the variant of the sensing path generation strategy and the maintenance mechanism of the sensing matrix, to further improve the reconstruction accuracy and system adaptability;

[0102] This phase is executed on the control entity (referred to as "controller" for short), and the ultimate goal is to build a sensing matrix which can mathematically guarantee high-precision reconstruction and physically fully comply with network topology constraints. This sensing matrix is a bridge connecting the physical measurement of the data plane and the mathematical reconstruction of the control plane, and its design directly determines the final performance of the technical solution of the present application.

[0103] (I) Network topology discovery and characterization

[0104] As a preparatory work of this phase, the primary task of the controller is to obtain the complete, real-time and accurate topology structure of the network to be measured. In a preferred embodiment, the network to be measured is a software-defined network (SDN), and the controller, as the core of the SDN architecture, naturally has a global network view through its southbound interface protocol (such as OpenFlow), and can accurately grasp the connection relationship (link) between all network devices (nodes) and their ports in the whole network in real time;

[0105] In other alternative embodiments, for traditional networks that do not use SDN architecture, the controller can obtain the topology through active probing or passive listening. Preferably, the controller can deploy one or more monitoring agents, which discover neighbor devices and their port information by running the Link Layer Discovery Protocol (LLDP), or listen to link state broadcasts by participating in the network's routing protocol (such as Open Shortest Path First, OSPF, Intermediate System to Intermediate System, IS-IS), thereby collecting the information needed to build the global network topology;

[0106] Regardless of the method used, the controller will ultimately abstract and characterize the collected topology information into a mathematical graph model . Among them, represents the set of all network devices (such as switches, routers) participating in telemetry in the network, i.e., the node set; represents the set of physical or logical communication links connecting these devices. For ease of accurate description, the present application regards each bidirectional physical link as two unidirectional links in opposite directions, so is a directed edge set. The total number of unidirectional links in the network is denoted as , which will be the dimension of the metadata signal vector to be reconstructed later. The controller needs to continuously maintain the accuracy of the graph model, and when the network topology changes (such as link interruption or addition of new devices), the graph model can be updated in a timely manner .

[0107] (ii) Sensing path generation that balances deployability and non-correlation

[0108] The selection of measurement paths is the core of compressed sensing telemetry. The ideal set of measurement paths corresponds to the sensing matrix. Two seemingly contradictory conditions must be met simultaneously: first, each path must be a real, walkable entity within the network, i.e., "deployability"; second, the paths should be as random and uncorrelated as possible, ensuring the sensor matrix... The column vectors are highly linearly independent, i.e., "highly uncorrelated". In compressed sensing theory, this is usually characterized by the Restricted Isometry Property (RIP), which is a mathematical prerequisite for ensuring high-precision reconstruction.

[0109] This invention employs an innovative strategy based on random walk to generate measurement paths. The specific algorithm flow of this strategy is as follows:

[0110] (1) Parameter initialization: The controller initializes parameters according to the network size. Two key parameters are set: desired reconstruction accuracy and acceptable measurement overhead.

[0111] The total number of measurement paths, m: This value directly relates to the measurement's "compression rate". According to compressed sensing theory, in order to accurately reconstruct a signal in a certain domain... Sparse signals, The value of should satisfy ,in It is a constant. In practice, The value of can be as small as possible while ensuring reconstruction accuracy, so as to reduce measurement overhead;

[0112] Fixed length (step size) for each path This parameter determines the detection range and complexity of a single measurement path. Longer paths can aggregate information from more links at once, but may also increase the risk of autocorrelation within the path. The choice needs to strike a balance between information gain and the introduction of noise;

[0113] (2) Path generation iterative process: The controller executes a loop. Each iteration generates an independent and unique measurement path. In the... Second-rate In progress:

[0114] Randomly select the starting node: from the set of all nodes in the network In particular embodiments, a node is randomly selected following a uniform distribution as the starting point of the path; as the starting point of the path;

[0115] Performing a fixed-step random walk: from the starting node Perform a random walk of length starting from the starting node At step , assume the current node is , the controller queries the topology graph to obtain the set of all outgoing neighbor nodes of . Then, a node is randomly selected from this neighbor set with equal probability as the destination node for the next step. This directed link from to is recorded;

[0116] Path sequence recording: repeat the above walk process times, and a random path sequence consisting of directed links is obtained. The controller stores this path sequence;

[0117] In an optional embodiment, to avoid generating self-loops or overly simple paths, the random walk strategy can be optimized. Preferably, a “non-backtracking random walk” can be used, i.e., when selecting the next hop, the node of the previous hop is prohibited from being selected, unless the current node is a dead end (has only one neighbor). This optimization still maintains the principle of equal probability selection from the optional neighbor nodes, only the candidate node set is narrowed down;

[0118] In other optional embodiments, although theoretically, a “weighted random walk” can be considered, i.e., the controller maintains a counter of the number of times a link is measured, and when randomly selecting the next hop, a link with a smaller number of measurements is given a higher probability of being selected. However, the core scheme of the present application uses equal probability random selection, which is simple to implement, guarantees good randomness, and experimental results have shown excellent reconstruction performance. The weighted method as an optional optimization means will increase the complexity of the system, and is not necessary in actual deployment.

[0119] (Three) Construction and maintenance of the sensing matrix

[0120] When measurement paths ​​After all the generation is completed, the controller constructs a binary sensing matrix with dimension of in memory according to the path information . The construction rule of the matrix is very intuitive: for the element of the matrix at the th row and the th column, the value is defined as:

[0121] ;

[0122] The constructed sensing matrix accurately describes how the aggregated summary value obtained by the th measurement in the subsequent data plane is linearly combined from the real metadata of all the links in the network, that is, . The matrix is one of the core inputs of the subsequent control plane reconstruction algorithm. The controller needs to store the matrix and re-execute the flow of this stage to generate and update the sensing matrix when the network topology changes significantly.

[0123] Embodiment 4: P4 programmable implementation of data plane compression collection

[0124] This embodiment is a further refinement of step S2 in embodiment 2, which describes in detail how to efficiently and accurately implement the compression aggregation operation on a programmable data plane (taking P4 language as an example);

[0125] In the specific implementation of the weighted linear accumulation, theoretically, each node can dynamically calculate the weight coefficient according to the link characteristics or traffic state. However, considering the feasibility of actual deployment, especially in the programmable switch environment based on P4, complex floating-point multiplication operations will consume a lot of computing resources or simply not be supported. Therefore, the preferred embodiment of the present application uniformly sets the weight to 1, that is:

[0126] = ;

[0127] This simple addition operation is a nanosecond-level atomic operation for the arithmetic logic unit (ALU) of modern switch chips (such as Barefoot Tofino), which is not only simple to implement, but also almost does not introduce additional processing delay. Experimental results show that even with unit weight, CS-INT can still achieve high-precision metadata reconstruction (NMSE<0.1), proving the effectiveness of this simplified design.

[0128] (I) Detailed design and packaging of CS-INT probe packet

[0129] The header structure of CS-INT probe packet needs to be carefully designed to carry necessary information. Referring to Figure 3 (CS-INT probe packet structure), a typical implementation can be based on standard network protocol stack (e.g. Ethernet / IPv4 / UDP). Key additional or modified parts include;

[0130] (1) Source Routing (SR) stack: used to precisely control the packet path. Can be placed in the network layer header (e.g. IPv4 option) or after the transport layer header. The stack contains a series of labels, each corresponding to a hop on the path. Each label (preferably designed as 8 bits) can contain:

[0131] ① Egress Port ID (e.g. 7 bits): indicates which port the switch should forward the packet from;

[0132] ② Flag (e.g. 1 bit): can be used to indicate whether it is the end of the path or other special processing;

[0133] When the switch processes the packet, it pops the top SR label, forwards according to the egress port ID, and points to the next label;

[0134] (2) INT metadata header: this part is in the payload of the transport layer protocol such as UDP, and its structure design needs to consider functionality and efficiency;

[0135] ① INT Header: usually one or more bytes of field, can use bitmap or type-length-value (TLV) format to specify the type of metadata to be collected. For example, an 8-bit header, the 0th bit is 1 representing collecting "queue delay", the 1st bit is 1 representing collecting "link utilization", this format can support simultaneously collecting 8 types of metadata;

[0136] ② INT data / digest: following the header is one or more fixed-length digest fields. Their number and order strictly correspond to the metadata types specified in the header. In order to prevent data overflow during long path aggregation and ensure sufficient precision, the length of each digest field should have a large enough length, such as 48-bit or 64-bit unsigned integer. When the packet is first injected into the network by the controller, all digest fields are initialized to zero.

[0137] (II) Source Routing-based path execution

[0138] The control plane according to the selected random walk path The sequence of egress port IDs corresponding to each hop is encoded into an SR label stack, which is put into the SR field of the first probe packet;

[0139] The data plane switch (assuming a programmable switch, such as Tofino programmed using P4) upon receiving a CS-INT probe packet (identifiable by Ethernet header type or other identifier) parses the top label of the SR stack in its ingress processing pipeline (such as ingress_parser of TNA architecture);

[0140] The egress port of the next hop is determined according to the egress port ID in the label;

[0141] Before sending the packet to the egress pipeline or in the egress pipeline (such as egress stage of TNA architecture), the following metadata aggregation operation is performed;

[0142] The switch sends out the packet from the specified egress port, at which time the SR stack pointer in the packet header has been updated (pointing to the next label, or marked as empty / ending);

[0143] In this way, the probe packet can be transmitted in the network strictly according to the random walk path corresponding to the sensing matrix of a row preset by the controller.

[0144] (III) P4 implementation of hop-by-hop compression and aggregation in the data plane

[0145] The method of the present application is particularly suitable for implementation on programmable switches supporting P4 language, and the data plane processing logic thereof can be accurately and efficiently defined. When a CS-INT probe packet arrives at a P4 switch, the processing procedure thereof is as follows:

[0146] (1) Packet parsing (Parser): The parser of the switch is programmed to be able to identify the special EtherType of the CS-INT probe packet. Once matched, it will further parse the SR header and the INT metadata header, and store the field values in these headers into the metadata bus of the parser output for use by the subsequent pipeline stages;

[0147] (2) Ingress pipeline processing (Ingress Pipeline):

[0148] ① Path control: The switch reads the SR label at the top of the stack from the parsed SR header metadata, which is the egress port ID of the current switch that needs to send out the packet. The ID is saved in the egress metadata, which is used to guide the final forwarding decision;

[0149] ​②Metadata collection: The switch acquires the corresponding state value from its internal register or metadata source according to the content of the INT instruction header;

[0150] ③Queue latency / residence latency: By reading the timestamp register (TS) at the ingress and egress pipeline respectively, ,the difference between the two is the residence latency (Hop_Latency) of the packet in the switch, which includes the queuing latency;

[0151] ④Queue length: Read the current or average length register (queue_depth) of the egress queue associated with the predetermined egress port;

[0152] ⑤Link utilization: Read the periodically updated port utilization register. Let the collected metadata value be ;

[0153] ⑥Core aggregation operation: This is the key step of the present application. The arithmetic logic unit (ALU) of the switch performs a simple addition operation, adding the just-collected metadata value to the current value of the corresponding type of digest field carried in the packet to obtain a new digest value: ;

[0154] ⑦Packet modification: The switch writes the calculated new digest value back to the INT data part of the packet, overwriting the original .

[0155] (3) Egress pipeline processing (Egress Pipeline):

[0156] ①SR label update: The switch performs a "pop" operation on the SR header of the packet, removing the top label, so that the forwarding instruction of the next-hop switch is exposed at the new top of the stack;

[0157] ②Packet forwarding: Finally, the packet is sent out from the corresponding physical port according to the egress port ID determined in the ingress pipeline;

[0158] Through the above process, the summary value carried by the probe packet is added by the metadata of each hop as it passes through each hop, while the total length of the packet remains unchanged.

[0159] Embodiment 5: Implementation method of measurement summary collection and observation vector construction

[0160] This embodiment is a further refinement of step S3 in embodiment 2, focusing on the specific implementation method of measurement summary collection, reporting and observation vector construction.

[0161] When a probe packet finishes its whole journey of aggregation defined by its SR header , it enters the collection phase:

[0162] (1) Probe termination and digest extraction: The end node of a path is called INT Sink. This role can be played by the last network device of the path, or a high-performance server specially configured to receive telemetry packets. When the INT Sink receives a CS-INT probe packet, it identifies that this is a completed measurement packet by checking whether the SR label stack is empty, etc. Then it extracts one or more fixed-length aggregation digest values of fixed length from the INT data part of the packet ;

[0163] (2) Reporting controller: The INT Sink node sends the extracted digest value , along with its corresponding path index (or other information that can uniquely identify this measurement), to the central controller through a reliable out-of-band management network channel. This channel can be a secure gRPC connection, a UDP stream, or any other suitable mechanism for communication between the controller and the data plane device;

[0164] (3) Observation vector construction: The controller continuously receives the digest values reported by each INT Sink in the network. Each time a is received, it is stored in the corresponding position of an observation vector . When the controller successfully collects the measurement results of all preset paths, a complete dimensional observation vector is constructed. This vector is the only data input for the next stage of global state reconstruction.

[0165] Embodiment 6: High-precision reconstruction algorithm integrating physical boundary constraints

[0166] This embodiment is a deepening of step S4 in embodiment 2, which details how to achieve high-precision metadata reconstruction in the control plane by introducing physical boundary constraints and advanced optimization algorithms;

[0167] This phase is the wisdom core of the entire technical solution. The controller uses its powerful computing power to restore the high-dimensional, fine global link metadata vector from the collected, highly compressed observation vector .

[0168] (I) Mathematical modeling of high-precision reconstruction:

[0169] (1) Basic linear model: As mentioned before, the whole measurement process can be mathematically represented as a linear equation system: This is a typical underdetermined system, because the number of measurements is usually designed to be much smaller than the number of unknown link metadata to achieve low overhead. Without other information, this equation system has infinite solutions;

[0170] (2) Exploiting sparsity prior: Compressive sensing exploits the sparsity of a signal as a prior knowledge to find the unique sparse solution of an underdetermined equation system. The whole-network link metadata vector (e.g. the vector consisting of the queue delay of all links in the network) itself might not be sparse, but it usually exhibits high sparsity in some fixed transform domain (e.g. Fourier domain, cosine transform domain, wavelet domain, etc.). This means that there exists an orthogonal transform matrix (preferably the inverse discrete Fourier transform matrix) such that can be represented as while the vector has only a few non-zero elements (i.e. K-sparse, ). Substituting this relation into the basic model, the reconstruction problem is transformed from solving the dense to solving the sparse :

[0171] ;

[0172] where is the equivalent sensing matrix of .

[0173] (II) Calculation of variable value interval incorporating physical constraints

[0174] In order to further improve the reconstruction accuracy and enhance the robustness of the algorithm in the real environment with measurement noise, the present invention introduces an innovative variable value interval tightening algorithm as a preprocessing step before formally solving the optimization problem. The algorithm aims to derive a tighter effective value interval for each to-be-solved link metadata ;

[0175] The algorithm ingeniously incorporates two kinds of information:

[0176] Algebraic constraints from the linear equation system: The equation system itself implies the linear relationship between all variables .

[0177] Prior constraints from the physical world: Any network metadata has its inherent physical value range. For example, the value range of link utilization must be [0, 1]; the value of delay must be non-negative; the queue length cannot exceed the maximum buffer depth configured by the switch.

[0178] The detailed flow of the algorithm is as follows:

[0179] (1) Variable relationship simplification: for the augmented matrix Perform Gaussian-Jordan elimination to convert it into row-reduced echelon form Through this form, all variables can be easily divided into principal variables and free variables , and their explicit linear relationship expressions are obtained: where the elements of matrix and vector are directly read out by ;

[0180] (2) Interval initialization: initialize a value interval for all variables , which is the known, widest physical boundary of the variable;

[0181] (3) Iterative tightening: perform an iterative process until all variable intervals converge. In each iteration:

[0182] ① Forward propagation: using the current value interval of all free variables , through the rules of interval arithmetic, calculate a new, possibly narrower value range of all principal variables along the linear relationship ;

[0183] ② Interval intersection: intersect the new range of the principal variable calculated in the last step with its own old interval to obtain the updated principal variable interval;

[0184] ③ Backward propagation: using the updated, more compact interval of the principal variable, substitute it back into the linear relationship to form new constraints on the value range of the free variable, and solve the new interval of the free variable under these constraints;

[0185] ④ Convergence judgment: compare the change in the width of all variable intervals before and after this iteration. Specifically, when the interval width change rate of all variables is less than 1%, or the number of iterations reaches the preset upper limit (such as 20 times), it is considered that the interval has converged, and the iteration ends. In actual implementation, for a network of n=42 links, it usually converges within 5-10 iterations;

[0186] The output of this process is a set of highly optimized, globally consistent variable value intervals which greatly reduces the feasible solution space of the subsequent optimization problem, and is one of the key techniques to achieve high-precision reconstruction.

[0187] (Three) ADMM-based constrained sparse optimization solution

[0188] Combined with the sparsity requirement and the boundary constraints calculated in the previous step, the final sparse reconstruction problem is accurately formalized as the following constrained basis pursuit problem:

[0189] ;

[0190] This is a structured convex optimization problem, but due to the non-smoothness of the objective function norm, and the complexity of the constraint conditions (including equality constraints and box constraints imposed on the transformed variables), an advanced optimization algorithm needs to be used for solution. The preferred embodiment of the present application uses the alternating direction method of multipliers.

[0191] The solution process is as follows:

[0192] (1) Problem equivalence transformation (variable splitting): In order to decouple the complex constraints, an auxiliary variable is introduced and set to . The original problem is equivalent to:

[0193] ;

[0194] Now the constraint conditions are clearly separated on different variables;

[0195] Construct the augmented Lagrangian function: For the coupled constraint , the augmented Lagrangian function is constructed as:

[0196] ;

[0197] Where is the dual variable, is the penalty parameter. is an indicator function, representing all constraints on , i.e. and , , otherwise it is infinite;

[0198] (2) ADMM iterative update steps: Solve the problem by alternately and iteratively minimizing . In the th iteration, the following three update steps are performed:

[0199] Update: fixed and solving a minimization problem with respect to . This subproblem is a standard Lasso problem, which has an efficient closed-form solution, given by the soft-thresholding operator acting on + , i.e. ;

[0200] Update: fixed and solving a minimization problem with respect to . This subproblem is a quadratic programming problem, which requires finding a point in the feasible set that satisfies the linear constraints and the box constraints such that its Euclidean distance to + is minimized. This is essentially a projection onto a convex set problem, which can be solved efficiently;

[0201] Update (dual variable update): ;

[0202] (3) Convergence check and result output: repeat the above iteration steps until the algorithm converges (e.g., the primal residual and the dual residual are both smaller than a set tiny threshold). After convergence, the algorithm outputs the optimal sparse frequency-domain vector .

[0203] (Four) Original metadata recovery and application

[0204] Finally, the controller performs a simple inverse transformation, which can recover the final full-network link metadata vector from the sparse vector :

[0205] ;

[0206] This recovered vector is the core technical product provided by the method of the present application. It is no longer scattered, local measurement data, but a structured, global, high-dimensional digital snapshot. This snapshot accurately contains the high-fidelity estimate value of the same performance indicator (such as queue delay, link utilization, packet loss count, etc.) of all one-way links in the network within a specific measurement period in the form of a vector.

[0207] This high-precision, full-network range metadata snapshot has a very high application value, and it constitutes a solid data foundation for realizing intelligent and automated network management and operation. The controller can perform subsequent processing and application on the periodically restored metadata vector , which can specifically include but is not limited to the following aspects:

[0208] (1) Real-time monitoring and visualization of network status: The controller can present the data in the restored vector in real time on the management interface of the network operation personnel. For example, by combining the full-network topology graph with the metadata, the load or latency status of all links can be dynamically displayed in the form of a heat map, so that the network bottlenecks and congestion hotspots are immediately apparent, greatly improving the observability of the network;

[0209] (2) Fine-grained and proactive traffic engineering: Based on the accurate and complete view of the load of all network links, traffic engineering applications can make better global routing decisions. For example, when it is detected that the utilization rate of some links is about to reach the threshold, part of the traffic can be actively and smoothly migrated to the path with lighter current load, achieving load balancing and avoiding congestion, thereby improving the resource utilization rate and user experience of the entire network;

[0210] (3) Rapid and accurate fault diagnosis and root cause location: When performance degradation (such as increased application access latency) occurs in the network, traditional troubleshooting methods often consume a lot of time and effort. Using the full-network link-level latency or packet loss data provided by the present application, a fault diagnosis system can quickly locate the problem to a specific link or several links, achieving rapid tracing from "alarm" to "root cause" and significantly shortening the fault recovery time;

[0211] (3) Network security situation awareness and anomaly detection: Network attacks (such as DDoS attacks, port scanning) or abnormal device behavior often leave traces on network metadata. By machine learning modeling on the continuous and full-quantity metadata time series provided by the present application, a high-precision anomaly detection model can be trained. This model can discover data points deviating from the normal behavior pattern in real time, thereby timely alerting potential security threats or device failures;

[0212] (4) Quantitative verification of service level agreement (SLA): For critical businesses that require strict SLA guarantees (such as financial transactions, remote medical care), the performance data of the full-network end-to-end path provided by the present application (which can be obtained by accumulating the performance values of each link on the path) can be used for continuous and quantitative verification and reporting of SLA indicators (such as latency, jitter, packet loss rate), providing a reliable basis for service quality management.

[0213] In summary, the application organically combines the innovative construction method of the measurement path and the sensing matrix, the efficient compression measurement mechanism of the data plane, and the high-precision sparse reconstruction algorithm of the control plane together, and completely proposes a novel end-to-end low-overhead in-band full-network telemetry technical solution. The application solves the contradiction between the deployability and high non-correlation of the sensing matrix through the path generation strategy based on random walk; through the hop-by-hop aggregation and fixed-length digest transmission mechanism of the data plane, the traditional INT telemetry overhead linearly increasing with the path and the MTU over-limit problem are fundamentally solved; through the reconstruction process of the control plane combining the physical boundary constraint and adopting the ADMM optimization algorithm, high-fidelity recovery of the full-network element data under extremely low measurement overhead is ensured.

[0214] The method and system disclosed by the application can realize all-around and fine-grained state monitoring of a large-scale complex network with much lower resource consumption than the prior art, and the high-quality telemetry data generated thereby can effectively enable intelligent traffic engineering, rapid fault location, network security and SLA guarantee and other advanced network applications, and has important technical value and broad application prospect for promoting the automation and intelligent development of the next-generation network.

[0215] Test example:

[0216] In order to verify the effectiveness and significant effect of the CS-INT method of the application, the following comparative experiments are carried out:

[0217] Experimental setup:

[0218] ① Test platform: NSFNet topology composed of 14 Barefoot Tofino programmable switches (42 links)

[0219] ② Comparative scheme: classic INT, INT-Path, INT-React, INT-MC (sampling rate 25%)

[0220] ③ Test index: bandwidth overhead

[0221] The experimental results are shown in Table 1:

[0222] Table 1: Bandwidth overhead comparison of different INT schemes

[0223]

[0224] The data in Table 1 shows that the bandwidth overhead of the CS-INT scheme of the present application is only about 1 / 6 of the traditional hop-by-hop additional INT scheme (such as INT, INT-Path), and is even about 30% lower than the INT-MC scheme with a sampling rate of 25%. This proves that the CS-INT scheme, by transmitting a compression summary of fixed length, fundamentally overturns the proportional relationship between telemetry overhead and path length, and realizes an order of magnitude reduction in bandwidth overhead without sampling and ensuring data integrity, thereby solving the core defect of the traditional INT packet that is easy to exceed the MTU.

[0225] As can be seen from the above examples and test examples, the present application generates a measurement path with good mathematical properties and physical reach based on network topology in the control plane through random walk and other strategies, and constructs a corresponding sensing matrix; in the data plane, the source routing mechanism is used to guide the transmission of the probe packet along the preset path, and a compression and aggregation operation is performed on the network element data at each hop, so that the packet carries only a fixed-length aggregation summary value throughout the path; finally, the control plane uses the aggregation summary values collected from multiple paths and the sensing matrix to solve a sparse optimization problem that integrates the physical boundary constraints of the element data, and reconstructs the original element data of all links in the network with high precision. Compared with the prior art, the present application realizes full-network and fine-grained link state visibility, fundamentally solves the problem of increasing telemetry overhead with path growth, significantly reduces bandwidth consumption, avoids MTU limitation, and has the advantages of high reconstruction accuracy, strong deployment feasibility, etc.

[0226] Although the embodiments of the present application have been disclosed as above, they are not limited to the applications listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application. For those skilled in the art, for those skilled in the art, various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and therefore the present application is not limited to specific details without departing from the general concept defined by the claims and equivalent ranges.

Claims

1. A low-overhead in-band network telemetry method based on compressed sensing, characterized in that, The method comprises the following steps of: S1, measurement path and sensing matrix construction: based on network topology, a plurality of measurement paths are generated, and a sensing matrix is constructed according to the measurement paths, wherein the rows of the sensing matrix correspond to the total number of measurement paths m, and the columns correspond to the total number of unidirectional links n in the network; S2, data plane compression collection: according to the measurement paths, the probe packet is transmitted in the network by using the source routing mechanism; in the data plane, when the probe packet passes through each hop network device on the measurement path, the network device aggregates the metadata of the current hop or link with the digest value carried in the probe packet to generate an updated digest value, and transmits an aggregated digest value of a preset fixed length on the entire measurement path, wherein the fixed length is determined according to the total number of metadata types to be collected and the value range of each metadata type, so as to prevent numerical overflow in the aggregation process; S3, measurement digest collection: after the probe packet reaches the destination node or the designated collection point, the aggregated digest value of the fixed length is extracted; S4, control plane reconstruction: in the control plane, the aggregated digest values from the plurality of measurement paths and the sensing matrix are used to execute a sparse reconstruction algorithm based on the compressed sensing theory to restore the predetermined metadata at the link level in the network; In the step S2, the aggregation operation is specifically weighted linear accumulation: each network device performs weighted linear accumulation on the telemetry data of the current node and the aggregated digest value in the probe packet when the probe packet passes through; in actual deployment, considering the hardware limitations of programmable switches, the weight is set to 1, that is, simple addition operation is used, so as to fully utilize the efficient support of switch hardware for basic arithmetic operations; The structure design of the probe packet in the step S2 includes a source routing part and an INT metadata part, the source routing part carries pre-defined measurement path information to ensure that the probe packet can be strictly transmitted according to the routing sensing matrix; the INT metadata aggregation part includes a short INT instruction header and at least one fixed-length digest field, which is used for hop-by-hop aggregation in the data plane; The INT instruction header is used to indicate which type of metadata needs to be collected by the network device on the path; each digest field corresponds to a type of metadata, and is used to roll up the aggregation value of the metadata in the entire transmission path; wherein the metadata types include queue delay, link utilization and packet loss number.

2. The method of claim 1, wherein, In the step S1, the specific method for generating a plurality of measurement paths is: a random walk based strategy is adopted, m nodes are randomly selected as starting nodes in the network topology graph by the controller, in each step, the next hop node is randomly selected from all neighbor nodes of the current node with equal probability and moved, and the process is repeated t times to generate a random path with a length of t; by performing m independent walks, m measurement paths with path connectivity and matrix irrelevance are generated; m is the total number of measurement paths, and the value of m is determined according to the compressed sensing theory: for accurate reconstruction of a K-sparse signal, m satisfies m ≥ C·K·log(n / K), wherein C is a constant, K is the sparsity, and n is the total number of network links; The value of m ranges from 5 to 20, so as to balance the measurement overhead and reconstruction accuracy. t is the number of hops of a single path, and the value of t ranges from 4 to 10; if the number of hops is too short, sufficient information gain cannot be provided, and if the number of hops is too long, redundancy may be introduced and the sparsity of the sensing matrix is reduced. The measurement paths are realized by a source routing mechanism.

3. The method of claim 1, wherein, The step S2 is realized in a programmable switch environment, the format of the telemetry message is defined based on a P4 protocol, and a hop-by-hop aggregation operation is performed in a switch pipeline.

4. The method of claim 1, wherein, In the step S4, the sparse reconstruction algorithm is based on an alternating direction multiplier method (ADMM) to solve an optimization problem, a sparse vector is first recovered, and then complete telemetry data is obtained through a preset transformation matrix; and in the solving process, physical boundary constraint information of the predetermined metadata is fused.

5. The method of claim 4, wherein, The physical boundary constraint information is a value interval of each link metadata derived based on a linear equation group established in a sensing process and a physical value range of the metadata itself.

6. The method according to any one of claims 1 to 5, characterized in that, The method can be applied to a large-scale data center network, a wide area network environment or other complex topology environments, and high-precision network state monitoring can be realized under low overhead conditions.

7. A low-overhead in-band network telemetry system based on compressed sensing for implementing the method of any of claims 1-5, characterized in that, The system comprises: A plurality of network devices configured to perform the following operations: receiving a probe message carrying source routing information; identifying predetermined metadata of the current device or a connected link; performing an aggregation operation on the predetermined metadata and a digest value carried in the probe message to generate an updated digest value, the digest value having a fixed length; replacing an old digest value in the probe message with the updated digest value, and forwarding the probe message according to the source routing information; At least one control entity configured to perform the following operations: generating a plurality of measurement paths based on a network topology; constructing a corresponding sensing matrix; initializing a probe message, including source routing information based on the measurement paths and an initial digest value; receiving the fixed-length aggregated digest value from the probe message arriving at a destination node or a designated collection point; using the collected aggregated digest value and the sensing matrix, performing a sparse reconstruction algorithm based on compressed sensing to recover the predetermined metadata at a link level in the network.

8. The system of claim 7, wherein, The network device is a programmable switch programmed based on a P4 protocol to realize the aggregation operation; and the control entity is an SDN controller or a network management server.

Images