Telemetering data structure and telemetering method

Abstract

The invention discloses a telemetering data structure and a telemetering method, and relates to the technical field of network communication. The problems of high switch resource overhead, low analysis speed, low calculation efficiency and the like in the prior art are solved. The method comprises the following steps: analyzing header information of a data packet, and obtaining the header information to obtain a data packet type; for the user data packet, required network data is collected according to the analyzed header information; storing the collected various network data in the form of a telemetry data graph; updating the synchronous control block, matching the user data packet with the flow table and forwarding; for a telemetering data packet, analyzing a telemetering head, obtaining a telemetering data graph field and a data type, and updating a synchronous control block; the kernel mode program determines whether fragmentation is carried out according to the to-be-written telemetry data and the MTU; embedding data into a telemetry data packet; updating header information of the telemetry data packet; performing re-verification on the telemetry data packet; and matching the flow table and forwarding. The method is applied to traffic engineering.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a telemetry data structure and a telemetry method. Background Art

[0002] With the rapid development of computer networks and information technology, the construction and management of networks for traditional data centers, enterprises, and Internet service providers are becoming increasingly challenging. At the same time, traditional network measurements are increasingly unable to meet the market demand for network operation and maintenance. The introduction of Software-Defined Networking (SDN) provides a new network architecture and network measurement idea - network telemetry.

[0003] Compared with traditional network measurement technologies such as sFlow, IPFIX, Netflow, and PSAMP, network telemetry can collect fine-grained network status information on demand. The existing network telemetry mechanisms are mainly used in congestion control, traffic engineering, and other fields. As a new network measurement technology, current research on network telemetry focuses on network telemetry mechanisms, encoding and transmission of telemetry data, and telemetry task scheduling. It is mainly used in traffic engineering, fault location, and other fields. Related research results are in the initial stage of "just being able to measure is fine".

[0004] In-band network telemetry (INT) is a commonly used passive telemetry technology, but it also has many limitations, including: (1) The network telemetry range is limited by the layout of telemetry points. Inappropriate telemetry point layout makes it difficult to obtain a global network view. Before conducting network telemetry, the network administrator needs to specify the detection path and telemetry data indicators in advance, which cannot be dynamically adjusted during the telemetry process. (2) INT encapsulates telemetry instructions and network data into user data packets in implementation. On the one hand, this will reduce the payload ratio of user data packets and reduce user experience, while limiting the maximum data volume of INT telemetry. On the other hand, INT needs to embed telemetry instructions and extract telemetry data in user data packets at the sending and receiving ends of the telemetry, which will further affect the transmission of user data packets.

[0005] Based on the problems of limited network detection range and insufficient scalability of INT, it is necessary to propose a network telemetry method that improves the efficiency of network telemetry while providing multiple types of network data and ensures the robustness and reliability of network telemetry. Summary of the invention

[0006] The present invention aims to solve the problems existing in the prior art such as limited detection range, insufficient scalability, limited or excessively large amount of telemetry data, which results in large switch resource overhead, slow parsing speed, and low computing efficiency. A telemetry data structure and a telemetry method are provided. The scheme specifically includes:

[0007] A telemetry data structure, comprising:

[0008] Telemetry data graph, synchronization control block and telemetry data packet structure;

[0009] The telemetry data graph is used for ring storage with the device ID as the identifier, and the telemetry data graph is stored internally through a multi-element cross-linked list;

[0010] The synchronization control block is used to synchronize and manage the ring storage structure and the multi-element cross-linked list;

[0011] The telemetry data packet structure is used to organize and transmit the format of telemetry data, and the telemetry data packet structure includes a telemetry header field and a metadata field.

[0012] Furthermore, a preferred embodiment is provided, wherein the synchronization control block includes:

[0013] The synchronization control block read-write lock is used for read-write synchronization of the entire synchronization control block;

[0014] Telemetry data graph head pointer, used for the telemetry data graph to record the starting address of the ring storage structure data;

[0015] Telemetry data graph tail pointer, used for recording the last address of the ring storage structure data in the telemetry data graph;

[0016] A telemetry data graph expansion control module, used to store physical storage information of the telemetry data graph structure;

[0017] Telemetry data graph read and write synchronization module, used for telemetry data graph write operations, telemetry data graph storage updates, telemetry data graph expansion space physical storage information modification operations, and telemetry data graph read and copy operations;

[0018] A data extraction read pointer, used to store read pointer data of telemetry data to be written;

[0019] Data extraction storage free space is used to store the remaining space size to be written into the telemetry data storage;

[0020] A data extraction and expansion control module is used to store physical storage information of the expansion space of the telemetry data to be written;

[0021] Data extraction read-write synchronization module, used for writing data to be written, modifying physical storage information in the expansion space, and reading and copying written data;

[0022] The operation synchronization lock is used to synchronize operations such as embedding data into telemetry packets, counting network data and writing it into telemetry data graphs, and updating telemetry data type request information in kernel mode with operations such as writing statistics of network device information into telemetry data graphs and extracting requested telemetry data type information in user mode.

[0023] Based on the same inventive concept, the present invention also provides a telemetry data structure telemetry method, the method comprising:

[0024] S1: parse the header information of the data packet, obtain the header information and obtain the data packet type;

[0025] S2: For user data packets, collect the required network data based on the parsed header information;

[0026] S3: storing the collected various network data in the form of a telemetry data graph;

[0027] S4: Update the synchronization control block based on the collected network data and the kernel's write request to the telemetry data graph, and the user data packet matches the flow table and forwards it;

[0028] S5: For the telemetry data packet, parse the telemetry header, obtain the telemetry data field and data type, and update the synchronization control block;

[0029] S6: The kernel state program determines whether to fragment according to the telemetry data to be written and the MTU; and embeds the data into the telemetry data packet;

[0030] S7: Update the telemetry packet header information; perform recheck on the telemetry packet; match the flow table and forward.

[0031] Furthermore, a preferred embodiment is provided, wherein step S3 comprises:

[0032] S31: If the network data stored in the telemetry data map is initial data, the telemetry data map is initialized, using the device ID as an identifier;

[0033] S32: storing the source IP address, the destination IP address, and the current data flow information into the IP subgraph of the telemetry data graph, and constructing the IP subgraph based on the current data flow direction and the multi-element cross-link list;

[0034] S33: Update the network-wide statistical information graph of the telemetry data graph;

[0035] S34: Update the flow table statistics graph of the telemetry data graph.

[0036] Furthermore, a preferred embodiment is provided, wherein step S4 comprises:

[0037] S41: Try to obtain the read-write lock of the synchronization control block;

[0038] S42: Update various control information of the data graph;

[0039] S43: Release the read-write lock of the synchronization control block;

[0040] S44: if the timer reaches the update interval, the ring structure of the telemetry data graph and the synchronous control block data are updated at the end of the telemetry data statistics period;

[0041] S45: At the end of the telemetry data statistics period, the update timer attempts to obtain the synchronization control block read-write lock;

[0042] S46: After acquiring the read-write lock, the synchronization information of the data block to be written, the synchronization information of the telemetry data graph, and the storage information of the data to be written are reset;

[0043] S47: Record the position of the telemetry data graph header pointer of this cycle as the telemetry data graph header pointer of the previous cycle;

[0044] S48: Point the telemetry data graph head pointer to the telemetry data graph end pointer;

[0045] S49: setting the space between the last pointer and the last storage address of the physical storage space of the telemetry data graph to zero;

[0046] S410: Set the space between the first address of the physical storage space of the telemetry data graph and the head pointer of the telemetry data graph of the previous cycle to zero; release the read-write lock of the synchronization control block.

[0047] Furthermore, a preferred embodiment is provided, wherein step S5 comprises:

[0048] S51: After the kernel state updates the operation synchronization lock of the synchronization control block, the read-write lock of the synchronization control block is acquired;

[0049] S52: Read the metadata bitmap extracted from the telemetry data packet to determine the telemetry data type;

[0050] S53: traverse the multi-data linked list corresponding to the telemetry data type in the telemetry data graph;

[0051] S54: Calculate the required information entropy of various network data according to the information entropy calculation formula;

[0052] S55: writing the network data information entropy into the data storage area to be written;

[0053] S56: Update the operation synchronization lock of the synchronization control block and release the read-write lock of the synchronization control block.

[0054] Based on the same inventive concept, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a telemetry method of a telemetry data structure as described in any one of the above items.

[0055] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program executes a telemetry method for a telemetry data structure as described in any one of the above.

[0056] The present invention is beneficial in that:

[0057] The present invention solves the problems existing in the prior art, such as limited detection range, insufficient scalability, limited telemetry data volume or excessively large data volume, which results in large switch resource overhead, slow parsing speed, low computing efficiency, etc.

[0058] The present invention provides a telemetry data structure and a telemetry method. The method designs multi-data type telemetry and uses a metadata bitmap to multi-dimensionally identify the required telemetry data; designs a telemetry data graph structure for data storage, so that the network data stored on the SDN switch is more logical, and the statistical, integration and extraction efficiency of the data can be improved when storing more data. At the same time, it can even be persistently stored without considering the long-term storage occupancy of the switch; the telemetry data graph is implemented in a shared memory manner in physical storage, and is logically a ring storage, which can effectively improve the efficiency of kernel state and user state synchronization operations, and at the same time reduce the network device resource occupancy as much as possible; the synchronization control block uses an atomic variable collaboration method in design, which is a lock-free synchronization mechanism for the read and write synchronization of the telemetry data graph, and is also applied to the collaborative operation of kernel state and user state programs. The synchronization logic is clear and clear, and then the interruption problem caused by locked synchronization and the problem of frequent recovery of the interrupt program context are avoided, which can effectively improve the telemetry efficiency. In addition, the present invention designs a telemetry header for multi-type telemetry data, and also takes precautions against possible problems such as packet loss, saturation attack, and excessive telemetry data volume, so as to ensure the applicability and robustness of the network telemetry method.

[0059] The invention is applied to the field of flow engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of a telemetry data structure according to implementation mode 1;

[0061] Figure 2 This is a system architecture diagram of the telemetry method described in Implementation Method 3. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0063] Implementation method 1, see Figure 1 The telemetry data structure described in this embodiment includes:

[0064] Telemetry data graph, synchronization control block and telemetry data packet structure;

[0065] The telemetry data graph is used for ring storage with the device ID as the identifier, and the telemetry data graph is stored internally through a multi-element cross-linked list;

[0066] The synchronization control block is used to synchronize and manage the ring storage structure and the multi-element cross-linked list;

[0067] The telemetry data packet structure is used to organize and transmit the format of telemetry data, and the telemetry data packet structure includes a telemetry header field and a metadata field.

[0068] The telemetry data graph described in this embodiment includes:

[0069] Device status information: CPU usage, memory usage, etc.

[0070] Device port information: forwarding queue length, receiving queue length, stack overflow packet count, receiving frame alignment error count, receiving CRC check error count, receiving packet quantity change rate, receiving data byte quantity change rate, etc.

[0071] IP information: data packet count stored with source IP as identifier, destination IP address identifier, flow information storage address, destination IP address count, etc.

[0072] Flow information: Data is stored according to the direction of the flow, including source IP storage address, destination IP storage address, protocol type entropy, destination IP information entropy, destination port information entropy, etc.

[0073] Telemetry data packet information: receiving timestamp, forwarding timestamp, receiving port, forwarding port, etc.;

[0074] Network statistics: ICMP message count, message type entropy, source IP information entropy, source IP growth rate, destination IP information entropy, destination port information entropy, flow count with arrival interval less than the specified threshold, etc.

[0075] The data structure also stores flow table information and flow table item information, and stores telemetry data graphs through a HASH structure to achieve fast retrieval.

[0076] The telemetry data packet structure of this implementation mode is structurally divided into a telemetry header field and a metadata field, mainly including:

[0077] The session ID is a field designed for possible packet loss and saturation attacks, and consists of a detection cycle number and a sequence number within the cycle. In specific implementation, when telemetry starts, the telemetry initial node sends multiple network telemetry messages with the same detection cycle number and different sequence numbers to the SDN network;

[0078] The message length is used to indicate the total length of the data set of the current network telemetry data packet, and is also convenient for determining the integrity of the telemetry data;

[0079] The fragment ID consists of a message index and a subsequent fragment identifier, and is mainly used for fragmentation when the network telemetry data packet exceeds the MTU. In specific implementations, the message index is mainly used to identify the sequence number of the fragmented packet. The subsequent fragment identifier is mainly used to determine whether there are fragmented network telemetry packets in the future;

[0080] The metadata bitmap is used to identify the data type; the number of source data is used to identify the number of metadata fields; the main fields include:

[0081] Let metadata bitmap be α1 α2 … α n , a total of n bits, the metadata bitmap is represented as:

[0082]

[0083] in, is a custom symbol, indicating bit string concatenation, β x Identify multiple types of data, μ xy Identify various fine-grained data under the corresponding data type;

[0084] The Datapath-ID of a network device in an SDN environment can uniquely identify an SDN switch. The device ID is expressed as:

[0085] Switchi id ={datapathi id}

[0086] Device status information includes the growth rate of flow entries, flow entry space occupancy, device memory occupancy, CPU occupancy, etc. The device status information is expressed as follows:

[0087] Switchi info ={rate1,rate2,…,rate n}

[0088] The optional network status information is divided into multiple categories such as device level, flow level, and full network level. Each category is further divided into multiple features. The optional network status information is represented as follows:

[0089]

[0090] The telemetry data graph of this embodiment adopts a ring storage structure, which can effectively recycle the storage space. When the end of the telemetry data graph reaches the upper limit of the storage capacity, the new telemetry data will overwrite the old data from the beginning of the graph. This ensures the continuous recording of telemetry data and prevents storage space overflow. The telemetry data graph uses a multi-element cross-linked list for storage, which can flexibly organize and manage telemetry data. The multi-element cross-linked list can achieve fast access and retrieval of telemetry data through the association between the nodes of the linked list. Through this data structure, the storage and management requirements of various types of telemetry data can be efficiently handled. The synchronization control block is used to synchronize and manage the storage and access process of the telemetry data graph. It can ensure data consistency and integrity during multi-threaded or concurrent operations. The synchronization control block records the read and write status, pointer position and other information of the graph, so that multiple operations can be performed mutually exclusive to avoid data conflicts and errors. The telemetry data packet structure organizes and transmits telemetry data in a certain format. The telemetry header field and metadata field provide necessary information, including data type, timestamp, device identification, etc., so that the receiver can accurately parse and process telemetry data. By defining a unified data packet structure, telemetry data can be easily transmitted, stored and analyzed.

[0091] The designs of the circular storage, multi-element cross-linked list, synchronization control block and data packet structure of the telemetry data structure described in this embodiment can effectively organize, store, manage and transmit telemetry data, improve data utilization and processing efficiency, and ensure data integrity and consistency.

[0092] Implementation method 2: This implementation method further limits the telemetry data structure described in implementation method 1. The synchronization control block includes:

[0093] The synchronization control block read-write lock is used for read-write synchronization of the entire synchronization control block;

[0094] Telemetry data graph head pointer, used for the telemetry data graph to record the starting address of the ring storage structure data;

[0095] Telemetry data graph tail pointer, used for recording the last address of the ring storage structure data in the telemetry data graph;

[0096] A telemetry data graph expansion control module, used to store physical storage information of the telemetry data graph structure;

[0097] Telemetry data graph read and write synchronization module, used for telemetry data graph write operations, telemetry data graph storage updates, telemetry data graph expansion space physical storage information modification operations, and telemetry data graph read and copy operations;

[0098] A data extraction read pointer, used to store read pointer data of telemetry data to be written;

[0099] Data extraction storage free space is used to store the remaining space size to be written into the telemetry data storage;

[0100] A data extraction and expansion control module is used to store physical storage information of the expansion space of the telemetry data to be written;

[0101] Data extraction read-write synchronization module, used for writing data to be written, modifying physical storage information in the expansion space, and reading and copying written data;

[0102] The operation synchronization lock is used to synchronize operations such as embedding data into telemetry packets, counting network data and writing it into telemetry data graphs, and updating telemetry data type request information in kernel mode with operations such as writing statistics of network device information into telemetry data graphs and extracting requested telemetry data type information in user mode.

[0103] The synchronization control block read-write lock described in this embodiment is a lock-free synchronization mechanism realized by atomic operations; the telemetry data graph head pointer stores the read pointer data of the telemetry data graph structure, which is used for the telemetry data graph to record the starting address of the circular storage structure data; the telemetry data graph tail pointer stores the write pointer data of the telemetry data graph structure, which is used for the telemetry data graph to record the last address of the circular storage structure data, that is, the starting address of data addition; the telemetry data graph extension control module is used for the physical storage information of the telemetry data graph structure such as the remaining space size, the first address, and the extension space; the telemetry data graph read-write module realizes a lock-free synchronization mechanism through atomic operations, which is mainly divided into mutual exclusion locks (used for telemetry data graph write operations, telemetry data graph storage updates, telemetry data graph extension space physical storage information modification operations, etc.), RCU locks (telemetry data graph read, copy operations, etc.); data The extracted read pointer is used to store the read pointer data of the telemetry data to be written, that is, the first address of the telemetry data to be written; the data extraction storage free space is used to store the remaining space size of the telemetry data storage to be written; the data extraction extension control information is used to store the physical storage information of the extended space of the telemetry data to be written; the data extraction read and write synchronization module implements a lock-free synchronization mechanism through atomic operations, which is mainly divided into mutual exclusion locks (used for write operations of data to be written, modification operations of physical storage information of extended space, etc.) and RCU locks (reading and copying operations of data to be written, etc.); the operation synchronization module is used for kernel-mode embedding of data into telemetry data packets, counting network data and writing into telemetry data graphs, telemetry data type request information updates and other operations and user-mode counting of network device information and writing into telemetry data graphs, extracting requested telemetry data type information, etc., which is called the operation synchronization lock.

[0104] The synchronization control block read-write lock in this embodiment realizes a lock-free synchronization mechanism through atomic operations, improves concurrent performance and efficiency, realizes data synchronization and management, and ensures data consistency and integrity during multi-threaded or concurrent operations; the telemetry data graph head pointer and tail pointer accurately record the read and write positions of the telemetry data graph, facilitate data addition and extraction operations, determine the start and end addresses of the ring storage structure, and ensure continuous recording of recycled storage space; the telemetry data graph expansion control module stores the physical storage information of the telemetry data graph, such as the remaining space size, the first address, the expansion space, etc., provides more flexible storage management, records and manages physical storage information, supports dynamic expansion and optimizes the use of storage space; the telemetry data graph read-write module adopts a lock-free synchronization mechanism realized by atomic operations, improves the concurrent performance and efficiency of read and write operations, avoids data conflicts, supports multi-threaded read and write operations on the telemetry data graph, and ensures data integrity and consistency. The data extraction read pointer, storage free space and extended control information store the relevant information of the telemetry data to be written, which facilitates the writing and management of data, improves the utilization rate of data storage space, records the location of the data to be written, the size of the free space and the extended space information, and supports the effective addition and management of telemetry data. The data extraction read and write synchronization module adopts a lock-free synchronization mechanism implemented by atomic operations, supports high-concurrency read and write operations, improves the access and management efficiency of data, ensures the synchronization and consistency of the read and write operations of the data to be written, and avoids data conflicts and errors; the operation synchronization module ensures the synchronization of operations between kernel mode and user mode, avoids data loss or errors, realizes synchronization and coordination between different operations, and ensures the correct processing and transmission of data.

[0105] The synchronization control block of this embodiment is mainly used to achieve efficient storage of telemetry data, concurrent performance of read and write operations, consistency and integrity of data, effective use of storage space, and synchronization and coordination of operations.

[0106] Implementation method three, see Figure 2 The present embodiment describes a telemetry data structure telemetry method, the method comprising:

[0107] S1: parse the header information of the data packet, obtain the header information and obtain the data packet type;

[0108] S2: For user data packets, collect the required network data based on the parsed header information;

[0109] S3: storing the collected various network data in the form of a telemetry data graph;

[0110] S4: Update the synchronization control block based on the collected network data and the kernel's write request to the telemetry data graph, and the user data packet matches the flow table and forwards it;

[0111] S5: For the telemetry data packet, parse the telemetry header, obtain the telemetry data field and data type, and update the synchronization control block;

[0112] S6: The kernel state program determines whether to fragment according to the telemetry data to be written and the MTU; and embeds the data into the telemetry data packet;

[0113] S7: Update the telemetry packet header information; perform recheck on the telemetry packet; match the flow table and forward.

[0114] This implementation method can accurately obtain the type of data packet, distinguish between user data packets and telemetry data packets, and determine the type of data packet by parsing the data packet header information; based on the header information, it can determine the network data that needs to be collected, avoid the collection and processing of irrelevant data, and improve efficiency; the collected network data is stored in the form of a telemetry data graph, which can conveniently manage and operate multiple data points. It has the characteristics of a ring storage structure, stores network data in the telemetry data graph, retains the order and relevance of the data, and provides efficient data storage and processing; by updating the synchronization control block, it can ensure data synchronization and consistency, avoid data conflicts and errors, update the synchronization control block according to the collected network data, perform flow table matching and forwarding of user data packets, and ensure the correctness of data Accurate processing and transmission; by parsing the telemetry header, the telemetry data graph fields and data types can be obtained, the telemetry data can be accurately identified, the synchronization control block can be updated, the relevant information of the telemetry data graph can be recorded, and the subsequent storage and management of telemetry data can be supported; according to the telemetry data to be written and the maximum transmission unit (MTU), it is determined whether data fragmentation is required to adapt to network transmission, the telemetry data to be written is embedded into the telemetry data packet, and necessary fragmentation processing is performed to ensure the integrity and transmissibility of the data; the header information of the telemetry data packet is updated according to the operation to ensure the accuracy and consistency of the header information; data rechecking is performed to detect errors in data transmission, the flow table of the telemetry data packet is matched, and forwarding operations are performed according to the header information to ensure the correct transmission and processing of the data.

[0115] The present implementer performs appropriate processing and collection of network data according to the data packet type, stores the data in a telemetry data map, updates the synchronization control block, and processes, verifies, and forwards the telemetry data packets to ensure the correctness, consistency, and efficient transmission of the data.

[0116] Embodiment 4: This embodiment further limits the telemetry method of telemetry data structure described in embodiment 3, and step S3 includes:

[0117] S31: If the network data stored in the telemetry data map is initial data, the telemetry data map is initialized, using the device ID as an identifier;

[0118] S32: storing the source IP address, the destination IP address, and the current data flow information into the IP subgraph of the telemetry data graph, and constructing the IP subgraph based on the current data flow direction and the multi-element cross-link list;

[0119] S33: Update the network-wide statistical information graph of the telemetry data graph;

[0120] S34: Update the flow table statistics graph of the telemetry data graph.

[0121] In this implementation, the network-wide statistical information graph of the telemetry data graph is updated, including document type entropy, source IP address information entropy, source IP address growth rate, destination IP address information entropy, destination port information entropy, flow count with arrival interval less than a specified threshold, etc.; the flow table statistical information graph of the telemetry data graph is updated, including the flow table item growth rate, the number of flow table messages, etc.

[0122] Advantages of this implementation method: By initializing the telemetry data graph, a cleared and prepared data structure can be provided for the network data collected subsequently, ensuring the neatness and consistency of the data; by storing the network data in the IP subgraph according to the source IP address, destination IP address and data flow information, the network data can be easily classified and organized, and higher readability and operability can be provided. The IP subgraph is established in the form of the flow direction of the data flow and the construction of a multi-element cross-linked list to better represent and analyze the data information related to the IP address; by updating the whole network statistical information graph, the summary statistical data of the entire network can be recorded and maintained in real time, including traffic, number of connections, delay and other information, to help monitor the operation status of the entire network, provide comprehensive performance analysis and evaluation of the entire network, provide data support for network optimization, troubleshooting and resource planning, and help optimize the performance and reliability of the network; by updating the flow table statistical information graph, the usage of each flow table can be recorded and counted, including the number of matches, forwarding times, hit rate and other information, to help monitor the load and efficiency of the flow table, provide analysis and optimization of the flow table, and help adjust and optimize the rules of the flow table to improve the forwarding performance and resource utilization of the network.

[0123] This implementation method manages, analyzes and optimizes network data and performance by initializing a telemetry data graph, storing network data in an IP sub-graph according to IP addresses and data flow information, and updating a network-wide statistics graph and a flow table statistics graph. These steps can provide a clear data structure and statistics, help improve the observability and manageability of the network, and provide basic data support for network operation and maintenance and decision-making.

[0124] Embodiment 5: This embodiment further limits the telemetry data structure telemetry method described in Embodiment 3, and step S4 includes:

[0125] S41: Try to obtain the read-write lock of the synchronization control block;

[0126] S42: Update various control information of the data graph;

[0127] S43: Release the read-write lock of the synchronization control block;

[0128] S44: if the timer reaches the update interval, the ring structure of the telemetry data graph and the synchronous control block data are updated at the end of the telemetry data statistics period;

[0129] S45: At the end of the telemetry data statistics period, the update timer attempts to obtain the synchronization control block read-write lock;

[0130] S46: After acquiring the read-write lock, the synchronization information of the data block to be written, the synchronization information of the telemetry data graph, and the storage information of the data to be written are reset;

[0131] S47: Record the position of the telemetry data graph header pointer of this cycle as the telemetry data graph header pointer of the previous cycle;

[0132] S48: Point the telemetry data graph head pointer to the telemetry data graph end pointer;

[0133] S49: setting the space between the last pointer and the last storage address of the physical storage space of the telemetry data graph to zero;

[0134] S410: Set the space between the first address of the physical storage space of the telemetry data graph and the head pointer of the telemetry data graph of the previous cycle to zero; release the read-write lock of the synchronization control block.

[0135] This embodiment can expand the detection range by updating the control information of the data map, the synchronization information of the telemetry data map, and resetting the synchronization information and storage information of the data block to be written. This means that network data can be collected and counted more comprehensively, providing more extensive telemetry information.

[0136] Concurrent access and modification of the telemetry data graph can be achieved by using synchronization control block read-write locks. This provides a scalable way to handle large-scale telemetry data, ensuring that data can be updated and processed efficiently in a high-concurrency environment.

[0137] By updating the ring structure of the telemetry data graph and the synchronous control block data at the end of the telemetry data statistical cycle, the problem of limited or excessive telemetry data can be solved. The ring structure can recycle the stored space, and the update of the synchronous control block can ensure that the data is correctly recorded and managed.

[0138] By resetting the storage space and head pointer of the telemetry data graph and setting part of the storage space to zero, the switch resource overhead can be effectively controlled and the parsing speed and computing efficiency can be improved. This can optimize the overall system performance and reduce resource usage and processing time.

[0139] This step obtains and releases the read-write lock of the synchronization control block, updates the control information and synchronization information of the data graph, and resets the storage space and pointers, thereby solving the problems existing in the prior art, such as limited detection range, insufficient scalability, limited or large amount of telemetry data, large switch resource overhead, slow parsing speed, and low computing efficiency, thereby providing a better solution and optimization effect.

[0140] Embodiment 6: This embodiment further limits the telemetry method of telemetry data structure described in embodiment 3, and step S5 includes:

[0141] S51: After the kernel state updates the operation synchronization lock of the synchronization control block, the read-write lock of the synchronization control block is acquired;

[0142] S52: Read the metadata bitmap extracted from the telemetry data packet to determine the telemetry data type;

[0143] S53: traverse the multi-data linked list corresponding to the telemetry data type in the telemetry data graph;

[0144] S54: Calculate the required information entropy of various network data according to the information entropy calculation formula;

[0145] S55: writing the network data information entropy into the data storage area to be written;

[0146] S56: Update the operation synchronization lock of the synchronization control block and release the read-write lock of the synchronization control block.

[0147] This implementation method obtains the read-write lock of the synchronization control block after updating the operation synchronization lock of the synchronization control block in the kernel state, which can ensure that the operation of updating the synchronization control block is performed in a synchronous environment. This helps to avoid concurrency conflicts and data inconsistency problems, and improves the stability and reliability of the entire system.

[0148] This embodiment can determine the type of telemetry data by reading the metadata bitmap extracted from the telemetry data packet, so that subsequent processing can be performed on specific data types to improve the accuracy and efficiency of data processing.

[0149] This implementation method can process and calculate the data in a targeted manner by traversing the multi-data linked list corresponding to the telemetry data type in the telemetry data graph and calculating the required network data information entropy according to the information entropy calculation formula. This helps to extract useful information, reduce the processing overhead of redundant data, and improve the efficiency of data processing.

[0150] This implementation achieves accurate data processing and efficient storage by acquiring the read-write lock of the synchronization control block, determining the data type, traversing the data linked list, calculating the information entropy and writing it into the storage area, and updating and releasing the operation synchronization lock of the synchronization control block, thereby ensuring the synchronization and consistency of data operations.

[0151] Embodiment 7: A computer device described in this embodiment includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a telemetry method of a telemetry data structure as described in any one of embodiments 3 to 6.

[0152] Embodiment 8: A computer-readable storage medium described in this embodiment, wherein the computer-readable storage medium is used to store a computer program, and the computer program executes a telemetry data structure telemetry method described in any one of embodiments 3 to 6.

[0153] Implementation mode 9: This implementation mode provides a specific example of the telemetry data structure telemetry method described in implementation mode 3, and is also used to explain implementation modes 4 to 6. Specifically:

[0154] S1, parse the packet header information; Figure 1 As shown in the network telemetry data packet structure diagram of the network telemetry method based on the telemetry data diagram and the synchronization control block in this embodiment, it is assumed that the following telemetry data packets and user data packets are received:

[0155] ETH header (MAX1, MAC2, ...); IP header (IP1, IP2, 254, ...); telemetry header (0, 216, 0, (00011111) 2, 1)

[0156] That is, for this telemetry data packet, the source network device is: MAC1, IP1; the destination network device is: MAC2, IP2, and the protocol type is 254; the telemetry packet: the detection cycle number is 0, the sequence number is 0, the message length is 216, the message index is 0, the telemetry packet is not fragmented, and the telemetry metadata bitmap is the collection data type is (0001)2, the data feature is (1111)2, and the number of metadata is 1.

[0157] ETH header (MAC3, MAC4, ...); IP header (IP3, IP4, 17, ...); UDP

[0158] That is, for this user's data packet, the source network device is: MAC3, IP3; the destination network device is: MAC4, IP4, encapsulating UDP datagram.

[0159] S2, for user data packets, the network data extraction module collects the required network data through the network data collection method. Figure 2 As shown, the telemetry data graph ring storage structure and the synchronization control block structure diagram of the network telemetry method based on the telemetry data graph and the synchronization control block in this embodiment adopt a ring queue, an information control block and a multi-element cross data linked list for storage and control in implementation, mainly for storing the data to be stored in the telemetry data graph, such as the above-mentioned IP3, IP4, message length, protocol type, etc.

[0160] S3: The data storage module stores the collected network data in the form of telemetry data graphs; updates the synchronization control block according to the collected network data and user write requests;

[0161] The synchronization mechanism of the synchronization control block is a lock-free synchronization method implemented by atomic operations, mainly including mutex locks and RCU locks. The mutex lock is maintained by an atomic variable to ensure strong consistency; the RCU lock is implemented by a read queue, a write queue, and a queue-maintained atomic variable to ensure strong mutual exclusion of reading and writing, and concurrent operations of read-copy-update.

[0162] S4, user data packets match the flow table; execute the corresponding action set according to the flow table; determine the forwarding port and forward it to the outbound port queue for forwarding by the scheduling module;

[0163] S5, for telemetry data packets, parse the telemetry header; extract the metadata bitmap field of the telemetry header to determine the required network data type; update the synchronization control block through the metadata bitmap, user read request, and kernel write request; match the telemetry data packet with the flow table; wait for the user-mode program to extract data; the parsed data is as follows:

[0164]

[0165] S6, the user-mode program transfers the telemetry data packet and the data to be written to the kernel-mode data packet fragmentation module; the fragmentation module determines whether to perform fragmentation according to the length of the telemetry data packet and the length of the data to be written; and embeds the data into the telemetry data packet;

[0166] If the amount of telemetry data is large, assuming 1300 bytes, it needs to be fragmented. The fragmented telemetry data packet is divided into: ETH header (MAX1, MAC2, ...); IP header (IP1, IP2, 254, ...); telemetry header (0, 216, 0, (00011111) 2, 1, ...)

[0167] S7, updates the telemetry data packet network protocol stack and telemetry header information; performs recheck on the telemetry data packet; executes the flow table action set; and the scheduling module forwards the telemetry data packet.

[0168] The original telemetry data packet before forwarding needs to update the fragment ID, the structure is: ETH header (MAC1, MAC2, ...); IP header (IP1, IP2, 254, ...); telemetry header (0, 216, 1, (00011111) 2, 1)

[0169] The fragmented telemetry data packet needs to update the telemetry header: ETH header (MAC1, MAC2, ...); IP header (IP1, IP2, 254, ...); telemetry header (0, 1316, 2, (00011111) 2, 1, ...)

[0170] The above further describes in detail the technical solution provided by the present invention in conjunction with the accompanying drawings in order to highlight the advantages and benefits, and is not intended to be a limitation of the present invention. Any modification, combination of implementation modes, improvement and equivalent substitution of the present invention within the spirit and scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. A telemetry data structure, characterized in that: The structure comprises: Telemetry data graph, synchronization control block and telemetry data packet structure; The telemetry data graph is used for ring storage with the device ID as the identifier, and the telemetry data graph is stored internally through a multi-element cross-linked list; The synchronization control block is used to synchronize and manage the ring storage structure and the multi-element cross-linked list; The telemetry data packet structure is used to organize and transmit the format of telemetry data, and the telemetry data packet structure includes a telemetry header field and a metadata field.

2. A telemetry data structure according to claim 1, characterized in that: The synchronization control block comprises: The synchronization control block read-write lock is used for read-write synchronization of the entire synchronization control block; Telemetry data graph head pointer, used for the telemetry data graph to record the starting address of the ring storage structure data; Telemetry data graph tail pointer, used for recording the last address of the ring storage structure data in the telemetry data graph; A telemetry data graph expansion control module, used to store physical storage information of the telemetry data graph structure; Telemetry data graph read and write synchronization module, used for telemetry data graph write operations, telemetry data graph storage updates, telemetry data graph expansion space physical storage information modification operations, and telemetry data graph read and copy operations; A data extraction read pointer, used to store read pointer data of telemetry data to be written; Data extraction storage free space is used to store the remaining space size to be written into the telemetry data storage; A data extraction and expansion control module is used to store physical storage information of the expansion space of the telemetry data to be written; Data extraction read-write synchronization module, used for writing data to be written, modifying physical storage information in the expansion space, and reading and copying written data; The operation synchronization lock is used to synchronize operations such as embedding data into telemetry packets, counting network data and writing it into telemetry data graphs, and updating telemetry data type request information in kernel mode with operations such as counting network device information and writing it into telemetry data graphs and extracting requested telemetry data type information in user mode.

3. A telemetry data structure telemetry method, characterized in that: The method comprises: S1: parse the header information of the data packet, obtain the header information and obtain the data packet type; S2: For user data packets, collect the required network data based on the parsed header information; S3: storing the collected various network data in the form of a telemetry data graph; S4: Update the synchronization control block based on the collected network data and the kernel's write request to the telemetry data graph, and the user data packet matches the flow table and forwards it; S5: For the telemetry data packet, parse the telemetry header, obtain the telemetry data field and data type, and update the synchronization control block; S6: The kernel state program determines whether to fragment according to the telemetry data to be written and the MTU; and embeds the data into the telemetry data packet; S7: Update the telemetry packet header information; perform recheck on the telemetry packet; match the flow table and forward.

4. A telemetry data structure telemetry method according to claim 3, characterized in that: The step S3 comprises: S31: If the network data stored in the telemetry data map is initial data, the telemetry data map is initialized, using the device ID as an identifier; S32: storing the source IP address, the destination IP address, and the current data flow information into the IP subgraph of the telemetry data graph, and constructing the IP subgraph based on the current data flow direction and the multi-element cross-link list; S33: Update the network-wide statistical information graph of the telemetry data graph; S34: Update the flow table statistics graph of the telemetry data graph.

5. A telemetry data structure telemetry method according to claim 3, characterized in that: The step S4 comprises: S41: Try to obtain the read-write lock of the synchronization control block; S42: Update various control information of the data graph; S43: Release the read-write lock of the synchronization control block; S44: if the timer reaches the update interval, the ring structure of the telemetry data graph and the synchronous control block data are updated at the end of the telemetry data statistics period; S45: At the end of the telemetry data statistics period, the update timer attempts to obtain the synchronization control block read-write lock; S46: After acquiring the read-write lock, the synchronization information of the data block to be written, the synchronization information of the telemetry data graph, and the storage information of the data to be written are reset; S47: Record the position of the telemetry data graph header pointer of this cycle as the telemetry data graph header pointer of the previous cycle; S48: Point the telemetry data graph head pointer to the telemetry data graph end pointer; S49: setting the space between the last pointer and the last storage address of the physical storage space of the telemetry data graph to zero; S410: Set the space between the first address of the physical storage space of the telemetry data graph and the head pointer of the telemetry data graph of the previous cycle to zero; release the read-write lock of the synchronization control block.

6. A telemetry data structure telemetry method according to claim 3, characterized in that: The step S5 comprises: S51: After the kernel state updates the operation synchronization lock of the synchronization control block, the read-write lock of the synchronization control block is acquired; S52: Read the metadata bitmap extracted from the telemetry data packet to determine the telemetry data type; S53: traverse the multi-data linked list corresponding to the telemetry data type in the telemetry data graph; S54: Calculate the required information entropy of various network data according to the information entropy calculation formula; S55: writing the network data information entropy into the data storage area to be written; S56: Update the operation synchronization lock of the synchronization control block and release the read-write lock of the synchronization control block.

7. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a telemetry method for a telemetry data structure as described in any one of claims 3 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program executes a telemetry data structure telemetry method as described in any one of claims 3-6.

Images