Low-overhead in-band whole-network telemetering method and system based on compressed sensing

Abstract

The invention discloses a low-overhead in-band whole-network telemetering method and system based on compressed sensing, and belongs to the technical field of computer network monitoring and management. According to the method, a measurement path is generated on a control plane according to network topology, a sensing matrix is constructed, and original metadata of a whole network link is reconstructed with high precision through constrained sparse optimization solution by utilizing a multi-path aggregation value and the sensing matrix; a detection message is guided to be transmitted along a preset path in a data plane by means of a source route, and metadata is compressed and aggregated in each hop, so that the message only carries an aggregation value with a fixed length. According to the method, the problem that the overhead is increased along with the path is fundamentally avoided while the state of the fine-grained whole network is visible, the bandwidth consumption is remarkably reduced, the MTU limitation is avoided, and the method has high reconstruction precision and strong deployment feasibility; the method solves the problems of high overhead, linear correlation with path length, high possibility of exceeding MTU limitation and the like caused by adding metadata hop by hop in the existing in-band network telemetry (INT).

Description

Technical Field

[0001] This invention relates to the field of computer network monitoring and management technology, and in particular to a low-overhead in-band full-network telemetry method and system based on compressed sensing. Background Technology

[0002] Fine-grained network visibility is a crucial foundation for network management, providing critical data support for various applications such as traffic engineering, anomaly detection, fault location, and congestion control. In recent years, the application of in-band network telemetry (INT) technology has significantly improved network observability and is considered one of the important development directions for future network management.

[0003] INT (Instantaneous Telemetry) enables fine-grained monitoring of network status by recording key information about network devices in real time during packet transmission. Specifically, the source node receives telemetry commands from the controller and inserts its own metadata (such as port status and link utilization) into the data packet. This telemetry command instructs intermediate network devices along the data packet path to collect the necessary INT metadata and insert it hop-by-hop into the data packet. Finally, the end node extracts all INT metadata from the data packet and reports it to the controller, while simultaneously forwarding the original data packet to the destination host. The entire telemetry process is transparent to upper-layer services and does not affect the normal transmission of data packets.

[0004] Since INT requires the addition of metadata at each hop in the packet forwarding path, the number of network devices along the path directly determines the length of the telemetry information in the INT packet. As the path grows, the metadata fields in the INT packet grow linearly, resulting in significant network overhead: on the one hand, it consumes a large amount of network bandwidth, affecting the transmission of normal business traffic; on the other hand, network switches need additional computing resources to process telemetry data, increasing the processing burden on devices; in addition, the uploading of a large amount of telemetry data may also lead to southbound interface bandwidth congestion, affecting the interaction efficiency between the control plane and the data plane. In a network-wide telemetry scenario, where the metadata at each hop is visible, this problem is particularly prominent, severely limiting the feasibility of INT in large-scale network environments.

[0005] To address the high overhead of INT (Instantaneous Telemetry), several optimization schemes have been proposed. For example, sINT and Sel-INT reduce packet overhead by decreasing the spatiotemporal coverage of telemetry. However, these methods rely on sampling mechanisms, which may lead to incomplete data and loss of network status information at critical moments, making it difficult to meet the requirements of fine-grained network-wide measurement. The PINT scheme uses a stream-level information aggregation method to encode telemetry data into multiple packets, thereby reducing the additional overhead of individual packets. However, it cannot provide detailed link-level telemetry data, making it difficult to meet the application requirements of congestion diagnosis and link-level traffic engineering, which rely on accurate link information. In addition, although the Sketch-based method can reduce the transmission burden of some telemetry data, its core still relies on the traditional INT mechanism and does not fundamentally solve the inherent overhead problem of INT.

[0006] In recent years, the INT-MC scheme has optimized telemetry data acquisition by utilizing matrix completion technology. By planning the measurement path, it measures only a portion of the links and uses matrix completion algorithms to infer metadata for the unmeasured links, thereby reducing in-band telemetry overhead. However, this scheme still faces the following challenges in practical deployment: Firstly, it still needs to carry the complete path's telemetry metadata in the data packets. When the path is long or multiple types of telemetry information need to be collected, the data packet length may exceed the Maximum Transmission Unit (MTU), affecting effective data transmission. Secondly, INT-MC requires a sufficient amount of data accumulation to perform matrix completion; otherwise, the inference accuracy is difficult to guarantee, limiting its applicability in small-scale or dynamic network environments.

[0007] In summary, while existing INT schemes have made some progress in reducing telemetry overhead, they still face two challenges: First, some low-overhead measurement methods rely on sampling mechanisms, resulting in incomplete telemetry data and failing to meet the needs of fine-grained monitoring across the entire network. Second, optimization schemes such as INT-MC are constrained by MTU in practical deployments and require a sufficient amount of data accumulation to be effective, making implementation difficult. Therefore, how to minimize bandwidth overhead and avoid exceeding packet length limits while ensuring the visibility of metadata for each hop remains a critical technical challenge that urgently needs to be addressed. Summary of the Invention

[0008] To address the aforementioned shortcomings of existing technologies, this invention provides a low-overhead in-band network telemetry method and system based on compressed sensing. This invention innovates the traditional "hop-by-hop appending" metadata model into "hop-by-hop compressed aggregation." In the data plane, probe packets are transmitted along a predetermined path, and each hop device linearly aggregates its local metadata with a fixed-length digest value in the packet, significantly reducing bandwidth and processing overhead. In the control plane, using a small number of aggregated digests and pre-built sensing matrices, a sparse reconstruction algorithm incorporating network physical constraints is used to accurately recover the original metadata of all links in the entire network. This ensures network-wide link-level visibility while fundamentally avoiding the problem of packets exceeding the MTU.

[0009] The specific technical solution of the present invention is as follows: This invention discloses a low-overhead in-band network telemetry method based on compressed sensing. The method includes compressing and measuring network telemetry data in the network switch data plane, and recovering complete telemetry data in the control plane based on sparse reconstruction technology, comprising the following steps: S1. Measurement Path and Sensor Matrix Construction: Based on the network topology, multiple measurement paths are generated, and a sensor matrix is ​​constructed according to the measurement paths, wherein the rows of the sensor matrix correspond to the total number of measurement paths m, and the columns correspond to the total number of unidirectional links n in the network. S2. Data Plane Compressed Acquisition: Based on the measurement path, the probe packet is guided to be transmitted in the network using a source routing mechanism. In the data plane, when the probe packet passes through each hop network device on the measurement path, the network device aggregates the metadata of the current hop or link with the digest value carried in the probe packet to generate an updated digest value. An aggregated digest value of a preset fixed length is then transmitted over the entire measurement path. The fixed length is determined based on the total number of metadata types to be collected and the numerical range of each metadata type to prevent numerical overflow during the aggregation process. S3. Measurement Digest Collection: After the probe message arrives at the destination node or the designated collection point, the fixed-length aggregated digest value is extracted. S4. Control Plane Reconstruction: In the control plane, using the aggregated summary values ​​collected from multiple measurement paths and the sensing matrix, a sparse reconstruction algorithm is executed based on compressed sensing theory to recover the predetermined metadata at the link level in the network.

[0010] Preferably, the specific method for generating multiple measurement paths in step S1 is as follows: a random walk-based strategy is adopted, in which the controller randomly selects m nodes as starting nodes in the network topology graph, and in each step, the next hop node is randomly selected from all the neighboring nodes of the current node with equal probability and moved, and this is repeated t times to generate a random path of length t; by performing m independent walks, m measurement paths with both path connectivity and matrix non-correlation are generated. The value of m is the total number of measurement paths. The value of m is determined according to the compressed sensing theory: for the accurate reconstruction of K-sparse signals, m satisfies m ≥ C·K·log(n / K), where C is a constant, K is the sparsity, and n is the total number of network links. The value of m is in the range of [5, 20], in order to balance measurement overhead and reconstruction accuracy; The t is the number of hops for a single path, and the value of t ranges from [4, 10]. If the number of hops is too short, it cannot provide sufficient information gain, while if the number of hops is too long, it may introduce redundancy and reduce the sparsity of the sensing matrix. The measurement path is implemented using a source routing mechanism.

[0011] Furthermore, the specific method for constructing the sensing matrix in step S1 is as follows: based on the m measurement paths generated in step S1, the controller constructs a matrix with dimension m in memory. Routing-aware sensor matrix .in, To measure the total number of paths, This represents the total number of unidirectional links in the network. This matrix is ​​a binary matrix, and its elements are... Defined as: if the first Measurement path After the first time in the network Link Then matrix elements ,otherwise The sensing matrix It accurately describes the linear relationship of the subsequent data plane measurement process and is the core basis for controlling the plane to reconstruct data; Furthermore, the measurement path generated in step S1, which has the characteristics of path connectivity and matrix non-correlation, can enhance the accuracy of compressed sensing reconstruction. Furthermore, in step S1, the elements in the sensing matrix are used to indicate whether a certain measurement path passes through a certain link, in order to maintain the linearity of compressed sensing measurements.

[0012] Preferably, in step S2, the aggregation operation is specifically a weighted linear accumulation: when a probe packet passes through, each network device performs a weighted linear accumulation of the telemetry data of its own node and the existing aggregated digest value in the probe packet; in actual deployment, considering the hardware limitations of the programmable switch, the weight is preferably set to 1, that is, a simple addition operation is used, so as to make full use of the switch hardware's efficient support for basic arithmetic operations.

[0013] Preferably, step S2 is implemented in a programmable switch environment, defining the telemetry message format based on P4 or a similar protocol, and performing hop-by-hop aggregation operations in the switch pipeline.

[0014] Preferably, the structure of the probe packet in step S2 includes a source routing part and an INT metadata part. The source routing part carries predefined measurement path information to ensure that the probe packet can be transmitted strictly according to the routing awareness matrix. The INT metadata aggregation part includes a short INT instruction header and at least one fixed-length digest field for hop-by-hop aggregation in the data plane. The INT instruction header is used to indicate what type of metadata the network devices on the path need to collect; each of the digest fields corresponds to a metadata type, which is used to accumulate the aggregate value of that metadata throughout the entire transmission path; wherein the metadata types include: queue latency, link utilization, and packet loss.

[0015] Furthermore, in step S2, the use of a source routing mechanism to precisely control the forwarding path of probe packets aims to ensure that the actual measurement process in the data plane strictly follows the parameters set by the sensor matrix. The defined linear model; Furthermore, in step S2, when the controller generates a probe packet for each path, it encodes the hop-by-hop information of that path (preferably the outgoing port ID sequence of each switch along the path) into the source routing field in the packet header. When the network device receives a probe packet carrying source routing information, it will ignore the regular routing table lookup process and forward it directly according to the next-hop information specified in the packet header, thereby ensuring that the trajectory of the probe packet corresponds completely to the row vector of the sensing matrix. Furthermore, in step S2, the format design of the probe message mainly includes: the source routing part and the INT metadata part; Furthermore, in step S2, the source routing part is usually placed between the Ethernet header and the IP header; the INT metadata aggregation part is located after the transport layer header (such as the UDP header), and each network device accumulates the aggregation value based on the existing value in the probe packet to support transparent transmission and reduce bandwidth overhead. Furthermore, the data plane compression acquisition process described in step S2 is specifically manifested as a hop-by-hop compression aggregation operation: when a probe packet carrying source routing information is transmitted along a predetermined path in the data plane, each hop network device on the path (preferably a programmable switch supporting programming languages ​​such as P4) will sequentially perform the following atomic operations to achieve efficient compression aggregation and forwarding of metadata: (1) The message is identified as a probe message by a specific field (EtherType); (2) Parse the INT instruction header to determine the type of metadata to be collected; (3) Obtain the instantaneous value of the corresponding metadata of the local hop device or its designated outgoing link (denoted as ). ); (4) Match the metadata value with the corresponding digest value of the type carried in the message (denoted as...). To perform an efficient linear aggregation operation, the simplest and most efficient method is linear accumulation, i.e. = + This addition operation is a nanosecond-level operation for the arithmetic logic unit (ALU) of modern switch chips, introducing almost no additional processing latency. (5) Use the newly generated digest value Overwrite the old digest field in the update message; (6) Based on the source routing header information, forward the packet to the next hop.

[0016] Furthermore, in step S2, the metadata refers to various status information that network devices can collect in the data plane, including network performance indicators such as queuing delay, hop latency, link utilization, and queue length. This metadata reflects the real-time operating status of network devices and links and is a standard monitoring object in network telemetry technology. Furthermore, in step S2, the aggregation operation mode can ensure that the telemetry data on the entire detection path is aggregated hop by hop during transmission, maintaining a linear accumulation relationship, so as to improve the recoverability of telemetry data while reducing overhead.

[0017] Furthermore, the specific method of step S3 is as follows: after the probe message completes the traversal of the entire path, it reaches the end node of the path (INT Sink node) or a designated centralized collection server; subsequently, the node extracts the fixed-length, aggregated digest value from the message. It is then uploaded to the central controller via an out-of-band network.

[0018] Preferably, the sparse reconstruction algorithm in step S4 is based on the Alternating Direction Multiplier Method (ADMM) to solve the optimization problem. First, the sparse vector is recovered, and then the complete telemetry data is obtained through a preset transformation matrix. Furthermore, during the solution process, the physical boundary constraint information of the predetermined metadata is fused.

[0019] Preferably, the physical boundary constraint information is the value range of each link metadata derived from the linear equations established by the sensing process and the physical value range of the metadata itself.

[0020] Furthermore, in step S4, the core idea of ​​the sparse reconstruction algorithm lies in transforming the reconstruction problem from directly solving dense problems... Transformed into solving sparse The relationship between the two is as follows: ,in This is the corresponding inverse transformation matrix. The reconstructed model therefore becomes... .

[0021] Furthermore, in step S4, when deriving the value range of each link metadata, in order to significantly improve the reconstruction accuracy and accelerate algorithm convergence, an innovative preprocessing step is introduced before the formal sparse reconstruction: for each link metadata to be determined... Derive a more compact effective value range than common sense in physics. This step integrates two types of information: (1) Linear equation system constraints: The linear equation system established during the sensing process It inherently contains the intrinsic relationship between variables; (2) Physical prior constraints: Any network metadata has its inherent physical value range. The preferred link utilization value range must be between [0,1] and the queue length must be between [0, maximum buffer depth of the switch].

[0022] Furthermore, in step S4, the specific algorithm for deriving the value range of each link metadata is as follows: (1) For augmented matrices Gaussian elimination is performed to transform the expression into the simplest echelon form, thus obtaining a linear expression between the main variables and the free variables; (2) Starting from the physical range of the free variables, the range of the main variables is derived by performing interval arithmetic operations along the linear expression; (3) Take the intersection of the derived range and the physical range of the main variable itself to obtain a more compact interval; (4) Iterate the above process until the intervals of each variable converge, and finally obtain a globally consistent and significantly reduced solution space.

[0023] Furthermore, in step S4, the constraint optimization problem based on ADMM is solved, and the sparse reconstruction problem can be represented as a... Norm minimization problems specifically include: (1) Construct an optimization problem that incorporates tight boundary constraints: ; (2) The Alternating Direction Multiplier Method (ADMM) is used to solve the problem. By introducing auxiliary variables and augmented Lagrangian functions, the original problem is decomposed into several subproblems that are easier to solve, effectively decoupling the objective function. Norm and applied to variables Complex boundary constraints; Each subproblem after decomposition typically has an efficient closed-form solution or a simple projective solution, where... The norm subproblems can be solved using the soft threshold shrinkage operator; subproblems involving quadratic terms and boundary constraints can be solved by solving the system of linear equations and then projecting them onto the pre-calculated effective interval. (3) The optimal sparse frequency domain vector is obtained by iteratively solving the ADMM algorithm. Then, through an inverse transformation This allows for the eventual recovery of the original high-precision metadata vectors for all n links in the entire network. .

[0024] Furthermore, in step S4, the sparse reconstruction algorithm integrates the physical boundary constraint information of the predetermined metadata in order to improve the reconstruction accuracy and accelerate the convergence of the algorithm.

[0025] Furthermore, the method enables high-precision network status monitoring under low-overhead conditions by performing real-time monitoring and analysis in large-scale data center networks or wide area network environments.

[0026] The present invention also discloses an application based on the method, which applies the method to large-scale data center networks, wide area network environments or other complex topology environments, and can achieve high-precision network status monitoring under low overhead conditions.

[0027] This invention further discloses a low-overhead in-band network telemetry system based on compressed sensing for implementing the method, the system comprising: Multiple network devices are configured to perform the following operations: receive probe packets carrying source routing information; identify predetermined metadata of the current device or its connection link; aggregate the predetermined metadata with the digest value carried in the probe packet to generate an updated digest value, the digest value having a fixed length; replace the old digest value in the probe packet with the updated digest value, and forward the probe packet according to the source routing information; At least one control entity is configured to perform the following operations: generate multiple measurement paths based on the network topology; construct corresponding sensing matrices; initialize probe messages containing source routing information and initial digest values ​​based on the measurement paths; receive the fixed-length aggregated digest values ​​from probe messages arriving at the destination node or a designated collection point; and use the collected aggregated digest values ​​and the sensing matrices to execute a sparse reconstruction algorithm based on compressed sensing to recover the predetermined metadata at the link level in the network.

[0028] Preferably, the network device is a programmable switch, programmed based on the P4 protocol to implement the aggregation operation; the control entity is an SDN controller or a network management server.

[0029] The beneficial technical effects of this invention are as follows: 1. This invention employs compression aggregation technology to aggregate metadata along a path into a fixed-length digest for transmission in the data plane. This makes the telemetry bandwidth overhead independent of the measurement path length, reducing bandwidth overhead to 1 / 5 or even lower compared to the traditional INT method, thus significantly saving network resources. Furthermore, because a fixed-length digest is transmitted, it fundamentally avoids the problem of traditional INT methods where excessively long paths or numerous metadata types cause packets to exceed the MTU limit, enhancing the universality and robustness of the telemetry scheme. 2. Although only compressed summary information is transmitted, by leveraging the compressed sensing sparse reconstruction algorithm of the control plane, this invention can recover detailed, link-level metadata of all monitored links in the network, achieving global network state visibility and overcoming the disadvantage of sacrificing granularity in existing low-overhead schemes. By employing random walks to generate a deployable sensing matrix with good mathematical properties (low RIP), and combining it with an advanced reconstruction algorithm based on ADMM and incorporating physical boundary constraints, this invention can achieve high-precision metadata recovery (preferred normalized mean square error NMSE can be below 0.1), ensuring the accuracy and availability of telemetry data; 3. In addition, the sensor matrix construction (random walk) and path control (source routing) mechanisms proposed in this invention both take into account the deployability of actual network environments and can be effectively implemented in programmable switches or SDN environments, ensuring the practicality and feasibility of the technical solution. Attached Figure Description

[0030] Figure 1 A CS-INT architecture diagram provided for this invention; Figure 2 This is a schematic diagram illustrating the workflow of the in-band full-network telemetry method described in this invention; Figure 3 This is a schematic diagram of a CS-INT probe packet. The packet consists of: an ETH header (Ethernet header); an SR header (Segment Routing header); an IPv4 header (IPv4 header); a UDP header (User Datagram Protocol header); an INT header (In-band Network Telemetry header); and INTdata (In-band Network Telemetry data). The SR header contains multiple segment routing nodes (SR_1 to SR_n), each consisting of a port ID and a flag. INTdata contains multiple data digests (digest_1 to digest_4). Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0032] This invention provides a low-overhead in-band network telemetry (INT) method and system based on compressed sensing (CS), hereinafter referred to as CS-INT. This scheme aims to solve the core problems of existing INT technologies (especially the classic INT mechanism), such as the linear increase in measurement overhead with path length, large bandwidth consumption, and potential exceeding of MTU limits caused by hop-by-hop metadata embedding. Simultaneously, this invention also strives to overcome the shortcomings of existing low-overhead INT schemes in ensuring fine-grained (link-level) and network-wide visibility. The core of this invention lies in utilizing the inherent sparsity of network metadata in the Fourier transform domain and other domains, and innovatively combining a data plane compression aggregation measurement mechanism with a control plane high-precision sparse reconstruction mechanism to achieve comprehensive and accurate perception of network status with low overhead.

[0033] Example 1: A network-wide telemetry system based on compressed sensing This embodiment describes the system architecture for implementing the method of the present invention. Figure 1 The typical deployment architecture of the CS-INT system of this invention is shown.

[0034] (1) System composition: The CS-INT system mainly consists of two parts: the controller and the data plane network equipment. The controller, as the core control entity of the system, is deployed in the control plane and is responsible for global measurement planning, probe packet generation, and metadata reconstruction. Specifically, the controller generates measurement paths and constructs a sensor matrix based on network topology information; creates probe packets containing source routing information and initial digest values ​​and injects them into the network; collects aggregated digests reported from the data plane, executes an ADMM-based sparse reconstruction algorithm, and recovers the original metadata of the entire network link. The controller can be implemented as an SDN controller, a network management server, or a dedicated telemetry analysis platform. The data plane network equipment includes source switches, intermediate forwarding switches, and destination switches, all of which use devices that support a programmable data plane, preferably Barefoot Tofino switches based on P4 programming. The source switches are responsible for receiving probe packets from the controller and initiating forwarding; intermediate switches identify probe packets and extract local metadata (such as...). Figure 1 The metadata types (identified by different colors in the packet) are aggregated with the digest value carried in the packet and then forwarded. The destination switch completes the final hop aggregation operation and reports the final compressed metadata to the controller through the out-of-band management network.

[0035] (2) System workflow: During system operation, the controller first performs measurement path planning. Based on the known network topology, the controller generates multiple measurement paths using a random walk strategy and constructs the corresponding sensor matrix. This path information is encoded into probe messages to guide subsequent data plane measurements. After generating a probe message, the controller injects it into the source switch. The probe message contains three key parts: source routing information to specify the forwarding path, an INT header to indicate the type of metadata to be collected, and a digest field initialized to zero to store the aggregation result; After the probe message enters the data plane, it is forwarded hop-by-hop according to the path specified by the source route. For example... Figure 1 As shown, the packet originates from the source switch, passes through multiple intermediate switches, and finally reaches the destination switch. At each hop, the switch performs the same aggregation operation: identifies the probe packet, extracts local metadata (identified by different colors in the diagram, such as orange, blue, and green representing different types of metadata), adds it to the digest value carried in the packet, updates the digest field, and forwards it to the next hop. This process ensures that compressed metadata accumulates continuously throughout the path, but the packet length remains constant. When the probe packet reaches the destination switch, a digest value containing compressed metadata is extracted and reported to the controller via the out-of-band management network. The controller collects digest values ​​from multiple measurement paths and, using the previously constructed sensor matrix, executes a sparse reconstruction algorithm based on compressed sensing theory. By solving an optimization problem, the controller can accurately recover the original metadata information of all links in the entire network from these compressed digests. The entire process realizes a shift from "hop-by-hop appending" to "hop-by-hop aggregation". When probe messages are transmitted in the network, they only carry a fixed-length digest instead of an ever-growing sequence of metadata, which greatly reduces telemetry overhead. At the same time, the intelligent reconstruction of the control plane ensures the complete visibility of the entire network status.

[0036] Example 2: Basic Methods for Network-wide Telemetry Based on Compressed Sensing The core idea of ​​this invention is to draw on compressed sensing theory and innovate the traditional INT's "hop-by-hop appending" data mode into a "hop-by-hop compressed aggregation" mode. This method performs efficient linear compression measurement of telemetry metadata in the data plane, ensuring that probe messages carry only a fixed-length aggregated value (digest) during transmission. Subsequently, in the control plane, using the collected aggregated values, a high-precision sparse reconstruction algorithm that incorporates network physical characteristics accurately recovers the original metadata of all links in the entire network.

[0037] This embodiment describes the basic process of the present invention, such as... Figure 2 As shown, it includes the following steps: S1. Measurement path and sensor matrix construction: The goal of this step is to construct a sensing matrix on the controller side that satisfies both network physical topology constraints and has excellent mathematical properties (high non-correlation). This is the theoretical basis and practical prerequisite for achieving high-precision compressed sensing reconstruction. Based on the network topology, multiple measurement paths are generated, and a sensing matrix is ​​constructed according to the measurement paths, where the rows of the sensing matrix correspond to the measurement paths and the columns correspond to the network links.

[0038] S2, Data Plane Compression Acquisition: The goal of this step is to use programmable network devices to guide probe packets along a preset path in the data plane and to complete online compression and aggregation of metadata with extremely low overhead. According to the measurement path, the probe message is guided to be transmitted in the network using a source routing mechanism. In the data plane, when the probe message passes through each network device on the measurement path, the network device aggregates the predetermined metadata of the current hop or link with the digest value carried in the probe message to generate an updated digest value, and transmits a fixed-length aggregated digest value over the entire measurement path.

[0039] S3. Measurement summary collection: After the probe message arrives at the destination node or the designated collection point, the fixed-length aggregated digest value is extracted.

[0040] S4. Control Plane Reconstruction: This step is one of the core technologies of this invention, and its objective is to utilize the collected data... A summary value (constituting the observation vector y) and a pre-constructed sensing matrix It accurately recovers all data in the network using advanced algorithms. The original metadata of the link; In the control plane, using the aggregated summary values ​​collected from multiple measurement paths and the sensing matrix, a sparse reconstruction algorithm is executed based on compressed sensing theory to recover the predetermined metadata at the link level in the network.

[0041] Example 3: Optimization Method for Constructing Measurement Path and Sensing Matrix This embodiment is a further refinement of step S1 in embodiment 2, focusing on the variations of the sensing path generation strategy and the maintenance mechanism of the sensing matrix, so as to further improve the reconstruction accuracy and system adaptability. This phase is executed on the control entity (referred to as the "controller"), and its ultimate goal is to construct a sensing matrix. This matrix mathematically guarantees high-precision reconstruction and physically fully conforms to network topology constraints. This sensing matrix serves as a bridge connecting the physical measurement of the data plane with the mathematical reconstruction of the control plane; its design directly determines the final performance of the technical solution of this invention.

[0042] (I) Network topology discovery and characterization As part of the preparatory work for this stage, the controller's primary task is to acquire a complete, real-time, and accurate topology of the network under test. In a preferred embodiment, the network under test is a software-defined network (SDN). As the core of the SDN architecture, the controller naturally possesses a global network view through its southbound interface protocol (such as OpenFlow), enabling it to grasp the real-time and accurate connections (links) between all network devices (nodes) and their ports. In other alternative implementations, for traditional networks that do not employ an SDN architecture, the controller can acquire the topology through active probing or passive listening. Preferably, the controller can deploy one or more monitoring agents that discover neighboring devices and their port information by running the Link Layer Discovery Protocol (LLDP), or by listening to link-state broadcasts through network routing protocols (such as Open Shortest Path First (OSPF) or Intermediate System to Intermediate System (IS-IS), thereby collecting the information needed to construct the entire network topology. Regardless of the method used, the controller will ultimately abstract and represent the collected topological information as a mathematical graphical model. .in, It represents the set of all network devices (such as switches and routers) participating in telemetry in the network, i.e., the set of nodes; This represents a set of physical or logical communication links connecting these devices. For ease of precise description, this invention treats each bidirectional physical link as two unidirectional links in opposite directions. It is a set of directed edges. The total number of unidirectional links in the network is denoted as . This will be the dimension of the metadata signal vector to be reconstructed subsequently. The controller needs to continuously maintain the accuracy of this graph model and be able to update the graph model in a timely manner when the network topology changes (such as link interruption or the addition of new devices). .

[0043] (ii) Sensing path generation that balances deployability and non-correlation The selection of measurement paths is the core of compressed sensing telemetry. The ideal set of measurement paths corresponds to the sensing matrix. Two seemingly contradictory conditions must be met simultaneously: first, each path must be a real, walkable entity within the network, i.e., "deployability"; second, the paths should be as random and uncorrelated as possible, ensuring the sensor matrix... The column vectors are highly linearly independent, i.e., "highly uncorrelated". In compressed sensing theory, this is usually characterized by the Restricted Isometry Property (RIP), which is a mathematical prerequisite for ensuring high-precision reconstruction.

[0044] This invention employs an innovative strategy based on random walk to generate measurement paths. The specific algorithm flow of this strategy is as follows: (1) Parameter initialization: The controller initializes parameters according to the network size. Two key parameters are set: desired reconstruction accuracy and acceptable measurement overhead. The total number of measurement paths, m: This value directly relates to the measurement's "compression rate". According to compressed sensing theory, in order to accurately reconstruct a signal in a certain domain... Sparse signals, The value of should satisfy ,in It is a constant. In practice, The value of can be as small as possible while ensuring reconstruction accuracy, so as to reduce measurement overhead; Fixed length (step size) for each path This parameter determines the detection range and complexity of a single measurement path. Longer paths can aggregate information from more links at once, but may also increase the risk of autocorrelation within the path. The choice needs to strike a balance between information gain and the introduction of noise; (2) Path generation iterative process: The controller executes a loop. Each iteration generates an independent and unique measurement path. In the... Second-rate In progress: Randomly select the starting node: from the set of all nodes in the network In the process, a node is randomly selected following a uniform distribution. As the first The starting point of the path; Perform a random walk with a fixed step size: starting from the starting node To begin, execute a sequence of lengths... The random walk. In the first... step Assuming the current node is Controller query topology graph get set of all outgoing neighbor nodes Then, a node is randomly selected from that set of neighbors with equal probability. As the next target node. From arrive This directed link , It was recorded; Path sequence record: Repeat the above walking process This will give you a result from A random path sequence consisting of links connected end to end. The controller stores this path sequence; In an alternative embodiment, the random walk strategy can be optimized to avoid generating self-loops or overly simple paths. Preferably, a "non-backtracking random walk" can be used, where choosing the next hop prohibits selecting the node from the previous hop unless the current node is a dead end (having only one neighbor). This optimization still maintains the principle of selecting from available neighboring nodes with equal probability, only narrowing down the set of candidate nodes. In other alternative implementations, while a "weighted random walk" could theoretically be considered—where the controller maintains a counter for the number of times a link is measured, assigning a higher probability of selection to links with fewer measurements when randomly choosing the next hop—the core solution of this invention employs equal-probability random selection. This approach is simple to implement, guarantees good randomness, and has proven to achieve excellent reconfiguration performance experimentally. The weighted method, as an optional optimization technique, increases system complexity and is not essential in practical deployments.

[0045] (III) Construction and maintenance of the sensing matrix when Measurement path Once all generation is complete, the controller uses this path information to construct a dimension-1 in memory. binary sensing matrix The rules for constructing this matrix are very intuitive: for the first... Line number Column elements Its value is defined as: ; This completed sensor matrix It precisely describes the subsequent data plane, the first... The aggregated summary value obtained from this measurement How did it come from the entire network? Real metadata of the link Linear combination, i.e. This matrix is ​​one of the core inputs to the subsequent control plane reconstruction algorithm. The controller needs to store this matrix and re-execute the process of this stage to generate and update the sensing matrix when the network topology changes significantly.

[0046] Example 4: P4 Programmable Implementation of Data Plane Compression Acquisition This embodiment is a further refinement of step S2 in embodiment 2, and describes in detail how to efficiently and accurately implement the compression and aggregation operation on the programmable data plane (taking P4 language as an example); In the specific implementation of the weighted linear accumulation, theoretically, each node can dynamically calculate the weight coefficient based on link characteristics or traffic status. However, considering the feasibility of practical deployment, especially in a P4-based programmable switch environment, complex floating-point multiplication operations would consume a large amount of computing resources or be completely unsupported. Therefore, the preferred embodiment of the present invention sets the weight uniformly to 1, that is: = ; This simple addition operation is a nanosecond-level atomic operation for the arithmetic logic unit (ALU) of modern switch chips (such as Barefoot Tofino), which is not only simple to implement but also introduces almost no additional processing latency. Experimental results show that even with unit weights, CS-INT can still achieve high-precision metadata reconstruction (NMSE < 0.1), demonstrating the effectiveness of this simplified design.

[0047] (I) Detailed design and encapsulation of CS-INT probe messages The header structure of CS-INT probe messages needs to be carefully designed to carry the necessary information. (Refer to...) Figure 3 As shown in the (CS-INT probe packet structure), a typical implementation can be based on a standard network protocol stack (such as Ethernet / IPv4 / UDP). Key additions or modifications include; (1) Source Routing (SR) Stack: Used for precise control of packet paths. It can be placed after the network layer header (such as the IPv4 option) or the transport layer header. This stack contains a series of labels, each label corresponding to a hop on the path. Each label (preferably designed to be 8 bits) can contain: ① Egress Port ID (e.g., 7 bits): Indicates from which the switch should forward the packet; ② Flag bit (e.g., 1 bit): can be used to indicate whether it is the end of a path or other special processing; When processing a packet, the switch consumes (pops) the SR tag at the top of the stack, forwards it according to the outgoing port ID, and moves the pointer to the next tag; (2) INT metadata header: This part is located in the payload of transport layer protocols such as UDP, and its structural design needs to take into account both functionality and efficiency; ①INT Header: This is typically a field of one or more bytes, which can use bitmap or type-length-value (TLV) formats to specify the types of metadata to be collected. For example, an 8-bit header, where bit 0 is 1, represents collecting "queue latency," and bit 1 is 1, represents collecting "link utilization." This format can support telemetry of eight types of metadata simultaneously. ②INT data / digest field: Following the command header are one or more fixed-length digest fields. Their number and order strictly correspond to the metadata type specified in the command header. To prevent data overflow during long-path aggregation and to ensure sufficient precision, each digest field should be of a sufficiently large length, such as a 48-bit or 64-bit unsigned integer. All digest fields are initialized to zero when the message is first injected into the network by the controller.

[0048] (ii) Path execution based on source routing The control plane is based on the selected random walk path. Encode the outgoing port ID sequence corresponding to each hop into an SR tag stack and put it into the first hop. In the SR field of each probe message; When a data plane switch (assuming it is a programmable switch, such as a Tofino programmed with P4) receives a CS-INT probe message (which can be identified by the Ethernet header type or other identifiers), it parses the tag at the top of the SR stack in its ingress processing pipeline (such as the ingress_parser in a TNA architecture). The next-hop forwarding port is determined based on the outgoing port ID in the label; Before sending the message to the egress pipeline or during the egress pipeline (such as the egress phase of the TNA architecture), perform the following metadata aggregation operation. The switch sends the packet from the designated output port. At this time, the SR stack pointer in the packet header has been updated (pointing to the next tag, or marked as empty / end). In this way, the detection messages can strictly follow the controller's preset corresponding sensor matrix. a line random walk path Transmitted over the network.

[0049] (III) P4 Implementation of Data Plane Hop-by-Hop Compression Aggregation The method of this invention is particularly suitable for implementation on programmable switches that support the P4 language, and its data plane processing logic can be defined precisely and efficiently. When a CS-INT probe packet arrives at a P4 switch, its processing flow is as follows: (1) Message parser: The switch's parser is programmed to recognize a special EtherType in CS-INT probe packets. Once a match is found, it will further parse out the SR header and INT metadata header, and store the field values ​​in these headers into the metadata bus output by the parser for use in subsequent pipeline stages; (2) Ingress Pipeline Processing: ① Path Control: The switch reads the SR tag at the top of the stack from the parsed SR header metadata. This tag is the outgoing port ID for which the switch needs to forward the packet. This ID is stored in the outgoing metadata to guide the final forwarding decision; ② Metadata Acquisition: The switch obtains the corresponding status value from its internal registers or metadata source based on the content of the INT instruction header; ③ Queue delay / dwell time delay: By reading the timestamp registers at the inlet and outlet pipelines respectively ( , The difference between the two is the dwell time of the packet on this switch (Hop_Latency), which includes queuing delay; ④ Queue length: Reads the current or average length register (queue_depth) of the exit queue associated with the predetermined exit port. ⑤ Link Utilization: Read the periodically updated port utilization register. Assume the collected metadata value is... ; ⑥ Core Aggregation Operation: This is a key step in this invention. The switch's arithmetic logic unit (ALU) performs a simple addition operation, converting the newly acquired metadata value... The current value of the digest field of the corresponding type carried in the message. Add them together to get a new digest value: ; ⑦ Message modification: The switch will calculate the new digest value. Write back to the INT data section of the message, overwriting the original data. .

[0050] (3) Egress Pipeline Processing: ①SR tag update: The switch performs a "pop" operation on the SR header of the packet, removing the tag at the top of the stack, so that the forwarding instructions of the next-hop switch are exposed at the new top of the stack; ② Message forwarding: Finally, the message is sent out from the corresponding physical port according to the output port ID determined in the ingress pipeline; Through the above process, the digest value carried by the probe message is added to the metadata of that hop as it passes through each hop, while the total length of the message remains unchanged.

[0051] Example 5: Implementation Method for Measurement Summary Collection and Observation Vector Construction This embodiment is a further refinement of step S3 in embodiment 2, focusing on the specific implementation methods of measurement summary collection, reporting, and observation vector construction.

[0052] When a probe message completes all defined by its SR header After the initial aggregation phase, the collection phase begins: (1) Probe Endpoint and Digest Extraction: The endpoint node of the path is called the INT Sink. This role can be played by the last network device in the path, or by a high-performance server specifically configured to receive telemetry messages. When the INTSink receives a CS-INT probe message, it identifies that this is a message that has completed a measurement by checking whether the SR tag stack is empty, etc. Then, it extracts the final fixed-length aggregate digest value (one or more fixed-length aggregate digests) from the INT data portion of the message. ; (2) Reporting to the controller: The INT Sink node will extract the summary value Along with its corresponding path index (or other information that can uniquely identify this measurement) is sent to the central controller via a reliable out-of-band management network channel. This channel can be a secure gRPC connection, a UDP stream, or any other mechanism suitable for communication between the controller and data plane devices; (3) Observation Vector Construction: The controller continuously receives summary values ​​reported by each INT Sink in the network. Each time a summary value is received... Then store it in an observation vector. The corresponding location. When the controller successfully collects all After measuring the results of the preset paths, a complete... dimensional observation vector The construction is now complete. This vector will be the sole data input for the next stage of reconstructing the entire network state.

[0053] Example 6: High-precision reconstruction algorithm incorporating physical boundary constraints This embodiment is a further development of step S4 in embodiment 2, and elaborates in detail how to achieve high-precision metadata reconstruction in the control plane by introducing physical boundary constraints and advanced optimization algorithms; This stage is the intelligent core of the entire technical solution. The controller utilizes its powerful computing capabilities to process the collected, highly compressed observation vectors. In the process, high-dimensional, detailed metadata vectors of the entire network link are recovered. .

[0054] (I) Mathematical Modeling for High-Precision Reconstruction: (1) Basic linear model: As mentioned above, the entire measurement process can be mathematically represented precisely as a system of linear equations: This is a typical underdetermined system because of the number of measurements. It is typically designed to be much smaller than the amount of unknown link metadata. This is done to achieve low overhead. Without further information, this system of equations has infinitely many solutions; (2) Utilizing sparsity priors: Compressed sensing uses the sparsity of signals as prior knowledge to find the unique sparse solution to an underdetermined system of equations. (Full network link metadata vector) (For example, the vector composed of queue delays across the entire network) may not be sparse in itself, but it typically exhibits high sparsity in a fixed transform domain (such as the Fourier domain, cosine transform domain, wavelet domain, etc.). This implies the existence of an orthogonal transform matrix. (Preferred inverse discrete Fourier transform matrix), such that It can be represented as And vector Only a few non-zero elements are present (i.e., K-sparse). Substituting this relationship into the basic model, the problem is reconstructed from solving dense... This was transformed into solving sparse problems. : ; in It is The equivalent sensing matrix.

[0055] (ii) Calculation of the range of variable values ​​with integrated physical constraints To further improve reconstruction accuracy and enhance algorithm robustness in real-world environments with measurement noise, this invention introduces an innovative variable value range tightening algorithm as a preprocessing step before formally solving the optimization problem. This algorithm aims to tighten the variable value range for each piece of link metadata to be calculated. A more compact efficient range of values ​​than its original physical boundary is derived. ; The algorithm cleverly combines two types of information: Algebraic constraints from a system of linear equations: system of equations It inherently contains all variables. The linear relationship between them.

[0056] Prior constraints from the physical world: any network metadata has its inherent physical range of values. For example, link utilization must be in the range of [0,1]; latency must be non-negative; and queue length cannot exceed the maximum buffer depth configured on the switch.

[0057] The detailed process of the algorithm is as follows: (1) Simplification of variable relationships: for augmented matrices By performing Gauss-Jordan elimination, the problem is reduced to its simplest echelon form. In this way, all of them can be easily... Variables Divided into One main variable and free variables And obtain the explicit linear relationship expression between them: , where the matrix sum vector All elements are made of Read it directly; (2) Range initialization: for all variables Initialize a range of values, which is the widest known physical boundary. (3) Iterative Tightening: Execute an iterative process until the intervals of all variables converge. In each iteration: ① Forward propagation: utilizing all free variables The current value range, according to the rules of interval arithmetic, follows a linear relationship. Calculate all main variables A new, and possibly narrower, range of values; ②Interval Intersection: Find the intersection of the new range of the main variable calculated in the previous step with its old range to obtain the updated range of the main variable; ③ Backpropagation: Using the updated, more compact intervals of the main variables, substitute them back into the linear relationship to create new constraints on the range of values ​​of the free variables, and solve for the new intervals of the free variables under these constraints. ④ Convergence Judgment: Compare the changes in the interval widths of all variables before and after this iteration. Specifically, when the rate of change of the interval width of all variables is less than 1%, or the number of iterations reaches a preset upper limit (e.g., 20 times), the interval is considered to have converged, and the iteration ends. In practice, for a network with n=42 links, convergence is usually achieved within 5-10 iterations. The output of this process is a set of highly optimized, globally consistent variable value ranges. It greatly reduces the feasible solution space of subsequent optimization problems and is one of the key technologies for achieving high-precision reconstruction.

[0058] (III) Constrained Sparse Optimization Solution Based on ADMM Combining the sparsity requirement and the boundary constraints calculated in the previous step, the final sparse reconstruction problem is precisely formalized into the following constraint basis tracing problem: ; This is a structured convex optimization problem, but due to the objective function The non-smoothness of the norm and the complexity of the constraints (including equality constraints and box constraints imposed on the transformation variables) require advanced optimization algorithms for solving. This embodiment preferably uses the alternating direction multiplier method.

[0059] The solution process is as follows: (1) Problem equivalent transformation (variable splitting): In order to decouple complex constraints, an auxiliary variable is introduced. and make it equal to The original problem is equivalent to: ; The constraints are now clearly separated into different variables; Constructing the augmented Lagrangian function: for coupling constraints Construct the augmented Lagrange function: ; in, It is a dual variable. It is a penalty parameter. It is an indicator function that represents the... All constraints, i.e., when and hour, Otherwise, it is infinity; (2) ADMM iterative update steps: by alternately and iteratively minimizing To solve the problem. In the... In this iteration, the following three updates are performed: Update: Fixed and Seeking answers regarding The minimization problem. This subproblem is a standard Lasso problem with an efficient closed-form solution, achieved by applying a soft-threshold shrinkage operator. + It can be obtained ; Update: Fixed and Seeking answers regarding The problem is to minimize the condition. This subproblem is a quadratic programming problem, requiring the condition to satisfy linear constraints. and box constraints feasible set Go up and find a point Make it and + The Euclidean distance is minimized. This is essentially a projection problem onto a convex set, which can be solved efficiently; Update (dual variable update): ; (3) Convergence judgment and result output: Repeat the above iterative steps until the algorithm converges (e.g., the original residuals). and dual residual (All values ​​are less than a set small threshold). After convergence, the algorithm outputs the optimal sparse frequency domain vector. .

[0060] (iv) Recovery and application of original metadata Finally, the controller performs a simple inverse transform from the sparse vector. The final network-wide link metadata vector is recovered from the process. : ; This recovered vector This is the core technology product provided by the method of this invention. It is no longer fragmented, localized measurement data, but a structured, global, high-dimensional digital snapshot. This snapshot precisely contains all data from the entire network in vector form. A high-fidelity estimate of the same performance metric (such as queue delay, link utilization, packet loss count, etc.) for a single unidirectional link within a specific measurement period.

[0061] This high-precision, network-wide metadata snapshot has extremely high application value, forming a solid data foundation for intelligent and automated network management and maintenance. The controller can periodically recover the metadata vectors. Subsequent processing and application may include, but are not limited to, the following aspects: (1) Real-time monitoring and visualization of network status: The controller can recover the vector The data is displayed in real time on the management interface of network operations and maintenance personnel. For example, by combining the network topology map with metadata, the load or latency status of all links is dynamically displayed in the form of a heat map, making network bottlenecks and congestion hotspots clear at a glance, greatly improving network observability; (2) Refined and proactive traffic engineering: Based on an accurate and complete view of the load on all network links, traffic engineering applications can make better global routing decisions. For example, when it is detected that the utilization of some links is about to reach the threshold, some traffic can be proactively and smoothly migrated to the currently less loaded path to achieve load balancing, avoid congestion, and thus improve the utilization of network resources and user experience; (3) Fast and accurate fault diagnosis and root cause location: When network performance deteriorates (such as increased application access latency), traditional fault diagnosis methods are often time-consuming and labor-intensive. By using the network-wide link-level latency or packet loss data provided by this invention, the fault diagnosis system can quickly locate the problem to a specific link or several links, realizing rapid tracing from "alarm" to "root cause", and significantly shortening the fault recovery time; (3) Network security situation awareness and anomaly detection: Network attacks (such as DDoS attacks and port scanning) or abnormal device behavior often leave traces in network metadata. By performing machine learning modeling on the continuous and full metadata time series provided by this invention, a high-precision anomaly detection model can be trained. This model can detect data points that deviate from normal behavior patterns in real time, thereby promptly alerting potential security threats or device failures; (4) Quantitative verification of Service Level Agreement (SLA): For critical businesses that require strict SLA guarantees (such as financial transactions and telemedicine), the performance data of the entire network end-to-end path provided by this invention (which can be obtained by accumulating the performance values ​​of each link on the path) can be used to continuously and quantitatively verify and report SLA indicators (such as latency, jitter, and packet loss rate), providing a reliable basis for service quality management.

[0062] In summary, this invention organically combines an innovative method for constructing measurement paths and sensor matrices, an efficient compressed measurement mechanism in the data plane, and a high-precision sparse reconstruction algorithm in the control plane, thus proposing a novel, end-to-end, low-overhead in-band full-network telemetry solution. This invention resolves the contradiction between the deployability and high non-correlation of the sensor matrix through a path generation strategy based on random walks; it fundamentally solves the problems of linear growth in overhead and MTU exceeding limits in traditional INT telemetry by using hop-by-hop aggregation and fixed-length summary transmission mechanisms in the data plane; and it ensures high-fidelity recovery of full-network metadata with extremely low measurement overhead through a reconstruction process that integrates physical boundary constraints in the control plane and employs the ADMM optimization algorithm.

[0063] The method and system disclosed in this invention can achieve comprehensive and fine-grained status monitoring of large-scale complex networks with far lower resource consumption than existing technologies. The high-quality telemetry data generated can effectively empower various advanced network applications such as intelligent traffic engineering, rapid fault location, network security, and SLA assurance. It has important technical value and broad application prospects for promoting the automation and intelligent development of next-generation networks.

[0064] Test example: To verify the effectiveness and significant results of the CS-INT method of this invention, the following comparative experiments were conducted: Experimental setup: ① Test platform: NSFNet topology consisting of 14 Barefoot Tofino programmable switches (42 links) ② Comparison schemes: Classic INT, INT-Path, INT-React, INT-MC (sampling rate 25%) ③ Test metric: Bandwidth overhead The experimental results are shown in Table 1: Table 1. Comparison of bandwidth overhead for different INT schemes

[0065] The data in Table 1 shows that the bandwidth overhead of the CS-INT scheme of this invention is only about 1 / 6 of that of traditional hop-by-hop appending INT schemes (such as INT and INT-Path), and even about 30% lower than the INT-MC scheme with a sampling rate of 25%. This proves that CS-INT, through fixed-length compressed digest transmission, fundamentally overturns the direct proportional relationship between telemetry overhead and path length, achieving a reduction in bandwidth overhead by orders of magnitude without sampling and while ensuring data integrity, thus solving the core defect of traditional INT messages easily exceeding the MTU.

[0066] As can be seen from the above embodiments and test examples, this invention generates measurement paths with good mathematical properties and physical reachability based on network topology in the control plane through strategies such as random walks, and constructs corresponding sensing matrices. In the data plane, it uses source routing mechanisms to guide probe packets along preset paths, and performs compression and aggregation operations on network metadata at each hop, so that the packets carry only a fixed-length aggregated digest value throughout the entire process. Finally, the control plane uses the aggregated digest values ​​and sensing matrices collected from multiple paths to solve a sparse optimization problem that incorporates physical boundary constraints of metadata, and reconstructs the original metadata of all links in the entire network with high accuracy. Compared with the prior art, this invention achieves full-network, fine-grained link status visibility while fundamentally solving the problem of telemetry overhead increasing with path, significantly reducing bandwidth consumption, avoiding MTU limitations, and has advantages such as high reconstruction accuracy and strong deployment feasibility.

[0067] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, and for those of ordinary skill in the art, various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. Therefore, the present invention is not limited to the specific details without departing from the general concept defined by the claims and their equivalents.

Claims

1. A low-overhead in-band network telemetry method based on compressed sensing, characterized in that, The method includes compressing and measuring network telemetry data in the network switch data plane, and recovering complete telemetry data in the control plane based on sparse reconstruction technology, including the following steps: S1. Measurement Path and Sensor Matrix Construction: Based on the network topology, multiple measurement paths are generated, and a sensor matrix is ​​constructed according to the measurement paths, wherein the rows of the sensor matrix correspond to the total number of measurement paths m, and the columns correspond to the total number of unidirectional links n in the network. S2. Data Plane Compressed Acquisition: Based on the measurement path, a source routing mechanism is used to guide the transmission of probe packets in the network. The probe packets carry instruction information indicating the predetermined metadata types to be collected. In the data plane, when the probe packets pass through each hop network device on the measurement path, the network device aggregates the predetermined metadata of the current hop or link with the digest value carried in the probe packets to generate an updated digest value. An aggregated digest value of a preset fixed length is then transmitted over the entire measurement path. The fixed length is determined based on the total number of metadata types to be collected and the numerical range of each metadata type to prevent numerical overflow during the aggregation process. S3. Measurement Digest Collection: After the probe message arrives at the destination node or the designated collection point, the fixed-length aggregated digest value is extracted. S4. Control Plane Reconstruction: In the control plane, using the aggregated summary values ​​collected from multiple measurement paths and the sensing matrix, a sparse reconstruction algorithm is executed based on compressed sensing theory to recover the predetermined metadata at the link level in the network.

2. The method according to claim 1, characterized in that, The specific method for generating multiple measurement paths in step S1 is as follows: a random walk-based strategy is adopted, in which the controller randomly selects m nodes as starting nodes in the network topology graph. In each step, the next hop node is randomly selected from all the neighboring nodes of the current node with equal probability and moved. This is repeated t times to generate a random path of length t. By performing m independent walks, m measurement paths with both path connectivity and matrix non-correlation are generated. The value of m is the total number of measurement paths. The value of m is determined according to the compressed sensing theory: for the accurate reconstruction of K-sparse signals, m satisfies m ≥ C·K·log(n / K), where C is a constant, K is the sparsity, and n is the total number of network links. The value of m is in the range of [5, 20], in order to balance measurement overhead and reconstruction accuracy; The t is the number of hops for a single path, and the value of t ranges from [4, 10]. If the number of hops is too short, it cannot provide sufficient information gain, while if the number of hops is too long, it may introduce redundancy and reduce the sparsity of the sensing matrix. The measurement path is implemented using a source routing mechanism.

3. The method according to claim 1, characterized in that, In step S2, the aggregation operation is specifically a weighted linear accumulation: when a probe packet passes through, each network device performs a weighted linear accumulation of the telemetry data of its node and the existing aggregated digest value in the probe packet; in actual deployment, considering the hardware limitations of the programmable switch, the weight is preferably set to 1, that is, a simple addition operation is used, so as to make full use of the switch hardware's efficient support for basic arithmetic operations.

4. The method according to claim 1, characterized in that, Step S2 is implemented in a programmable switch environment, defining the telemetry message format based on P4 or a similar protocol, and performing hop-by-hop aggregation operations in the switch pipeline.

5. The method according to claim 3, characterized in that, The probe packet structure design in step S2 includes a source routing part and an INT metadata part. The source routing part carries predefined measurement path information to ensure that the probe packet can be transmitted strictly according to the routing awareness matrix. The INT metadata aggregation part contains a short INT instruction header and at least one fixed-length digest field for hop-by-hop aggregation in the data plane. The INT instruction header is used to indicate what type of metadata the network devices on the path need to collect; Each of the aforementioned summary fields corresponds to a metadata type, which is used to accumulate the aggregated value of that metadata throughout the entire transmission path; The metadata types mentioned include: queue latency, link utilization, and packet loss.

6. The method according to claim 1, characterized in that, The sparse reconstruction algorithm described in step S4 is based on the Alternating Direction Multiplier Method (ADMM) to solve the optimization problem. First, it recovers the sparse vectors, and then obtains complete telemetry data through a preset transformation matrix. Furthermore, during the solution process, it integrates the physical boundary constraint information of the predetermined metadata.

7. The method according to claim 6, characterized in that, The physical boundary constraint information is the value range of each link metadata derived from the linear equations established based on the sensing process and the physical value range of the metadata itself.

8. The application of the method according to any one of claims 1-7, characterized in that, Applying the method to large-scale data center networks, wide area network environments, or other complex topology environments enables high-precision network status monitoring with low overhead.

9. A low-overhead in-band network telemetry system based on compressed sensing, used to implement the method described in any one of claims 1-7, characterized in that, The system includes: Multiple network devices are configured to perform the following operations: receive probe packets carrying source routing information; identify predetermined metadata of the current device or its connection link; aggregate the predetermined metadata with the digest value carried in the probe packet to generate an updated digest value, the digest value having a fixed length; replace the old digest value in the probe packet with the updated digest value, and forward the probe packet according to the source routing information; At least one control entity is configured to perform the following operations: generate multiple measurement paths based on the network topology; construct corresponding sensing matrices; initialize probe messages containing source routing information and initial digest values ​​based on the measurement paths; receive the fixed-length aggregated digest values ​​from probe messages arriving at the destination node or a designated collection point; and use the collected aggregated digest values ​​and the sensing matrices to execute a sparse reconstruction algorithm based on compressed sensing to recover the predetermined metadata at the link level in the network.

10. The system according to claim 9, characterized in that, The network device is a programmable switch, programmed based on the P4 protocol to implement the aggregation operation; the control entity is an SDN controller or a network management server.

Images