A data collection system for operation and maintenance platform based on big data

Abstract

The present invention discloses a data acquisition system for an operation and maintenance platform based on big data, and relates to the technical field of big data, including: capturing data packets from a network flow of an operation and maintenance platform and storing them in a buffer; preliminarily screening the data packets in the buffer, and assigning priorities to the data packets based on a dynamic weight scheduling mechanism to complete preliminary preprocessing of the data packets; assigning multiple decomposed parsing tasks to multiple threads for parallel processing based on a cyclic sorting and block optimization technology; formatting the parsing results, and performing batch parallel processing according to the data scale of the parsing results; writing the parsed data into a time series database in chronological order after formatting, and generating an index based on a tag field; and dynamically adjusting the cache size according to the throughput and maximum delay of the network traffic through a dynamic cache management mechanism, thereby avoiding the problems of data loss and processing delay.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a big data-based operation and maintenance platform data acquisition system. Background Art

[0002] With the rapid development of big data and network technology, the amount and complexity of data on the operation and maintenance platform have increased dramatically. In order to meet real-time performance requirements, traditional data acquisition systems have gradually evolved towards parallelization and distributed processing. The existing operation and maintenance platform data acquisition system mainly relies on single-threaded or simple multi-threaded models to capture and process data packets, lacking effective data priority scheduling mechanisms and dynamic cache management. This leads to resource bottlenecks in the system when facing high-concurrency, high-throughput data streams, and it is unable to process and store large amounts of data in a timely manner, affecting the timeliness and accuracy of the data.

[0003] The main deficiencies of the existing technology are reflected in two aspects: first, in the process of screening and processing data packets, there is a lack of an effective priority scheduling mechanism, and the priority processing order of data packets cannot be dynamically adjusted according to real-time network load and business needs; second, the existing system usually adopts a fixed-size cache or a simple FIFO mechanism, which cannot dynamically adjust the cache size according to the throughput and delay requirements of the data flow, resulting in data loss or processing delays. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an operation and maintenance platform data acquisition system based on big data to solve the problems of low efficiency of data packet priority scheduling, dynamic cache management and parallel data processing in the existing operation and maintenance platform data acquisition system under high concurrency environment.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a data collection system for an operation and maintenance platform based on big data, which comprises:

[0008] The packet capture module is used to capture data packets from the network flow of the operation and maintenance platform and store them in the buffer;

[0009] The screening preprocessing module is used to preliminarily screen the data packets in the buffer and assign priorities to the data packets based on the dynamic weight scheduling mechanism to complete the preliminary preprocessing of the data packets;

[0010] The parallel parsing module is used to decompose the high-priority data packets after preliminary preprocessing into multiple parsing tasks, and based on the circular sorting and block optimization technology, the decomposed multiple parsing tasks are assigned to multiple threads for parallel processing;

[0011] Batch processing module, used to format the analysis results and perform batch parallel processing according to the data size of the analysis results;

[0012] The storage module is used to format the parsed data, write it into the time series database in chronological order, and generate an index based on the tag field;

[0013] The retrieval and analysis module is used to quickly retrieve and analyze data stored in the time series database through API query statements.

[0014] As a preferred solution of the data collection system of the operation and maintenance platform based on big data described in the present invention, wherein: the capturing of data packets from the network flow of the operation and maintenance platform includes the following steps:

[0015] Initialize the network interface and set filtering rules, capture data packets in real time, extract timestamps from the packet headers, and append them to the metadata of the data packets;

[0016] Based on the timestamps of the data packets, the data packets are sorted in time sequence.

[0017] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data described in the present invention, wherein: storing the captured data packets into the buffer includes the following steps:

[0018] The size of the ring buffer is dynamically calculated based on the throughput and maximum latency requirements. The expression is:

[0019] ;

[0020] in, Indicates the size of the ring buffer, Indicates the maximum processing delay that can be tolerated. Indicates the maximum throughput of network traffic, Indicates the average size of a single data packet;

[0021] The sorted data packets are copied to the circular buffer and arranged in the order of timestamps. When the buffer reaches the maximum capacity, the earliest data packet is automatically discarded according to the first-in-first-out principle and the latest data packet is stored in the buffer.

[0022] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data of the present invention, the preliminary screening of the data packets in the buffer zone includes the following steps:

[0023] Extracting basic metadata from the data packet; the metadata includes source IP address and destination IP address, source port number and destination port number, protocol type and timestamp;

[0024] According to the protocol type of the data packet, high priority protocol and low priority protocol are screened;

[0025] Perform whitelist and blacklist screening based on source IP address and target IP address;

[0026] Filter high-priority ports and low-priority ports based on source port number and destination port number;

[0027] Perform timeliness screening based on the timestamp of the data packet.

[0028] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data described in the present invention, wherein: assigning priorities to the filtered data packets based on the dynamic weight scheduling mechanism, completing the preliminary preprocessing of the data packets includes the following steps:

[0029] After the screening is completed, a weight is assigned to each data packet, and the weight is dynamically adjusted according to the actual network status, business needs and current load;

[0030] By comparing the weight of each data packet, a heap sorting algorithm is used to sort the data packets from high to low according to the weight to form a priority;

[0031] Parse the IP header of the sorted high-priority data packets, extract necessary metadata and perform a preliminary check on the legitimacy of the data packets;

[0032] After the data packet passes the legitimacy check, the data packet is marked and classified to complete the initial preprocessing of the data packet.

[0033] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data of the present invention, wherein: the decomposition of the high-priority data packets after preliminary preprocessing into multiple parsing tasks includes the following steps:

[0034] The information in each data packet is divided into link layer, network layer, transport layer and application layer according to the order of protocol stack;

[0035] According to the layers of the protocol stack, multiple layers in the data packet are decomposed into multiple tasks;

[0036] For the link layer, extract the source MAC address and the destination MAC address, check the frame type field, determine whether it is an IPv4 or IPv6 packet, and determine the next parsing solution;

[0037] For the network layer, extract the source IP address, destination IP address, protocol type, TTL and fragment offset of the IP header, check the protocol type field, determine the transport layer protocol used by the data packet, parse the TTL, flag, fragment information, total length, checksum content of the IP header, and perform a legality check;

[0038] For the transport layer, extract the port number, sequence number, confirmation number, TCP flag bit and window size to check the legitimacy of the transport layer;

[0039] For the application layer, the HTTP request method, URL, request header, DNS query type and response record of the application layer data are extracted and parsed.

[0040] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data described in the present invention, the following steps are included: based on the circular sorting and block optimization technology, the decomposed multiple parsing tasks are assigned to multiple threads for parallel processing:

[0041] Determine the block size of the parsing task processed by each thread based on the capacity of L1 Cache and L2 Cache;

[0042] Divide each parsing task of the data packet into blocks and task units;

[0043] Select the loop that can be expanded, analyze the loop operations in the task, and determine the loop expansion factor based on the number of registers and cache size;

[0044] Put all the block-by-block and expanded parsing tasks into the task queue, and each thread takes tasks from the task queue for processing;

[0045] The task queue assigns parsing tasks to different threads according to the complexity and workload of the tasks;

[0046] Each thread executes the task unit independently, parses each part of the data packet, and dynamically adjusts the allocation of tasks for each thread according to the current network load and resource usage.

[0047] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data described in the present invention, the formatting of the analysis results and the batch parallel processing according to the data scale of the analysis results include the following steps:

[0048] Represent the parsing result as a three-dimensional tensor of batch size, number of header fields, and sequence length;

[0049] Divide the data into multiple blocks according to the batch size, and automatically adjust the number of parallel threads according to the number of header fields and sequence length;

[0050] Number the tasks of each block, use the modulus operation to take the modulus of the number of threads in the task number, and evenly distribute the tasks to different threads;

[0051] After receiving the task, each thread executes the processing task of the parsing results in parallel, generates an accurate timestamp based on the time information in the parsing results, adds necessary label information to each parsed data, and performs batch formatting on the assigned task partitions to generate standardized records.

[0052] As a preferred solution of the data acquisition system of the operation and maintenance platform based on big data described in the present invention, the following steps are included: after the parsed data is formatted, it is written into the time series database in chronological order:

[0053] Convert the batch formatted data into a format suitable for the time series database, and insert it into the time series database in batches through the API interface;

[0054] The inserted data is stored in the cache layer of the database. The cache layer is dynamically refreshed according to the write frequency and memory usage, the cache strategy is adjusted based on the access frequency of different data, and the cache size is dynamically adjusted according to the current memory usage and database load.

[0055] As a preferred solution of the data acquisition system for the operation and maintenance platform based on big data described in the present invention, the data written into the time series database generates an index based on the tag field, including the following steps:

[0056] Divide the data into multiple time partitions according to the timestamp of the data, generate a unique time series index for each written data, and partition and index the data according to the time range;

[0057] An inverted index structure is used to create indexes for commonly used tag fields, and differential encoding technology is used to compress fields that appear more frequently.

[0058] The beneficial effects of the present invention are as follows: by introducing a dynamic weight scheduling mechanism, the priority of data packets is dynamically adjusted according to the real-time network status, business needs and current load, ensuring that key data packets are processed first, thereby improving the response speed of the system and the accuracy of data processing. Secondly, based on parallel data parsing and batch processing, high-priority data packets are decomposed into multiple parsing tasks and processed in parallel through multi-threading, which greatly improves the parsing efficiency and makes full use of the computing resources of the multi-core system. In addition, the present invention uses a dynamic cache management mechanism to dynamically adjust the cache size according to the throughput and maximum delay of network traffic, thereby avoiding the problems of data loss and processing delays. Finally, the use of time series database and inverted index technology greatly optimizes the storage and retrieval efficiency of data, ensuring fast query and analysis in a big data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0060] Figure 1 This is a schematic diagram of the data collection system of the operation and maintenance platform based on big data in Example 1.

[0061] Figure 2 This is a schematic diagram of task decomposition in Example 1. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0063] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0064] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0065] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a data acquisition system for an operation and maintenance platform based on big data, including a packet capture module, a screening and preprocessing module, a parallel parsing module, a batch processing module, a storage module and a retrieval and analysis module.

[0066] The packet capture module is used to capture data packets from the network flow of the operation and maintenance platform and store them in the buffer; the screening and preprocessing module is used to perform preliminary screening of the data packets in the buffer, and assign priorities to the data packets based on the dynamic weight scheduling mechanism to complete the preliminary preprocessing of the data packets; the parallel parsing module is used to decompose the high-priority data packets after preliminary preprocessing into multiple parsing tasks, and based on the circular sorting and block optimization technology, the decomposed multiple parsing tasks are assigned to multiple threads for parallel processing; the batch processing module is used to format the parsing results and perform batch parallel processing according to the data scale of the parsing results; the storage module is used to write the parsed data into the time series database in chronological order after formatting, and generate indexes based on the label field; the retrieval and analysis module is used to quickly retrieve and analyze the data stored in the time series database through API query statements.

[0067] Specifically, capturing data packets from the network flow of the operation and maintenance platform and storing them in the buffer includes the following steps:

[0068] Initialize the network interface and set filtering rules, capture data packets in real time and extract timestamps from the header of the captured data packets, and attach them to the metadata of the data packets; sort the data packets in time series based on the timestamps of the data packets; dynamically calculate the size of the ring buffer according to the throughput and maximum delay requirements, the expression is:

[0069] ;

[0070] in, Indicates the size of the ring buffer, that is, the maximum number of packets that can be stored in the buffer. Indicates the maximum tolerable processing delay, in seconds (e.g. 1 second). This value indicates the longest expected processing time after a data packet arrives at the system. Indicates the maximum throughput of network traffic in bytes per second (Bytes / s). This value indicates the maximum traffic that the system needs to handle during peak hours. For high-speed network interfaces, such as 10 Gbps networks, The value can reach 1250 MB / s. Indicates the average size of a single data packet in bytes. This value can be obtained through experience or historical data. For example, in a typical Ethernet environment, the average size of a data packet is 1500 bytes;

[0071] The sorted data packets are copied to the circular buffer and arranged in the order of timestamps. When the buffer reaches the maximum capacity, the earliest data packet is automatically discarded according to the first-in-first-out principle and the latest data packet is stored in the buffer.

[0072] Specifically, once the system delay is detected to be excessive, the packet capture strategy is automatically adjusted, including:

[0073] Adjust the number of packet capture threads: The system can improve parallel processing capabilities by increasing the number of packet capture threads.

[0074] Optimize filtering rules: Under high load, the system can dynamically adjust filtering rules to reduce unnecessary traffic capture and ensure that key data packets can be processed quickly.

[0075] Buffer expansion: If frequent buffer overflows are detected, the system can increase the buffer size to increase its capacity, thereby reducing packet loss.

[0076] The data packets in the buffer are preliminarily screened and priorities are assigned to the data packets based on the dynamic weight scheduling mechanism. The preliminary preprocessing of the data packets includes the following steps:

[0077] Extract basic metadata from the data packet; the metadata includes the source IP address (Internet Protocol address) and the destination IP address, the source port number and the destination port number, the protocol type and the timestamp; filter high-priority protocols and low-priority protocols according to the protocol type of the data packet; filter whitelists and blacklists according to the source IP address and the destination IP address; filter high-priority ports and low-priority ports according to the source port number and the destination port number; filter timeliness according to the timestamp of the data packet; assign a weight to each data packet based on the extracted metadata information, and dynamically adjust the weight according to the actual network status, business needs and current load; sort the data packets from high to low according to the weight by comparing the weight of each data packet, using a heap sorting-based algorithm to form a priority; parse the IP header of the sorted high-priority data packet, extract the necessary metadata and perform a preliminary check on the legitimacy of the data packet; after the legitimacy check of the data packet passes, mark and classify the data packet to complete the preliminary preprocessing of the data packet.

[0078] Specifically, the sorting operations are as follows:

[0079] High-priority data packets: Set a priority threshold. Data packets with weights higher than the priority threshold will be immediately passed to the parallel parsing module to ensure that these critical data are processed first.

[0080] Low-priority packets: For packets with weights lower than the priority threshold, the current load situation determines whether to discard or delay processing. Specific strategies include: Delay processing: If the load is light, low-priority packets can be temporarily stored and parsed after processing high-priority data. Discard: If the load is heavy, low-priority packets may be directly discarded to free up resources to process more important packets.

[0081] Furthermore, the sorting algorithm is selected:

[0082] Heap Sort or Quick Sort are two commonly used sorting algorithms. The system selects a suitable sorting algorithm based on the number of data packets and real-time requirements.

[0083] Heap sort: Heap sort is suitable for scenarios with a large number of data packets but relatively low real-time requirements, and can maintain relatively stable performance in the worst case. Quick sort: Quick sort has better performance in average conditions and is suitable for scenarios with a small number of data packets or relatively uniform data distribution.

[0084] Specifically, the necessary metadata includes source and destination IP addresses, source and destination port numbers, protocol type (such as TCP, UDP, ICMP, etc.), packet length, and timestamp;

[0085] Going further, TCP refers to the Transmission Control Protocol, which is a connection-oriented, reliable transport layer protocol commonly used to transmit data packets; UDP refers to the User Datagram Protocol, which is a connectionless transport layer protocol commonly used to send small data packets; ICMP refers to the Internet Control Message Protocol, which is mainly used to transmit control information (such as the ping command) between network devices.

[0086] Specifically, the validity check of the data packet includes checking whether the checksum of the IP header is correct, checking whether the checksum of the TCP / UDP header is correct, and checking whether the length of the data packet is valid.

[0087] Decomposing the high-priority data packets after preliminary preprocessing into multiple parsing tasks includes the following steps:

[0088] The information in each data packet is divided into link layer, network layer, transport layer and application layer in the order of the protocol stack; according to the layers of the protocol stack, the multiple layers in the data packet are decomposed into multiple tasks; for the link layer, the source MAC address and the destination MAC address are extracted, the frame type field is checked, whether it is an IPv4 and IPv6 data packet, and the next parsing plan is determined; for the network layer, the source IP address, destination IP address, protocol type, TTL (lifetime) and fragment offset of the IP header are extracted, the protocol type field is checked, the transport layer protocol used by the data packet is determined, the TTL, flag, fragment information, total length, checksum content of the IP header are parsed, and a legitimacy check is performed; for the transport layer, the port number, sequence number, acknowledgment number, TCP flag and window size are extracted, and the legitimacy of the transport layer is checked; for the application layer, the HTTP request method, URL, request header, DNS query type and response record of the application layer data are extracted and parsed.

[0089] Going further, IPv4 and IPv6 are two versions of the Internet Protocol (IP), which is responsible for assigning addresses to devices in the network (such as computers, mobile phones, routers, etc.) and ensuring that data packets can be transmitted correctly in the network; HTTP refers to the Hypertext Transfer Protocol, which is used for data transmission in the network, usually for web page transmission; URL refers to Uniform Resource Locator, which is a standard address format for specifying the location of resources on the Internet; DNS is a system that converts domain names into IP addresses (such as 192.0.2.1). It is the phone book of the Internet, helping users access servers through easy-to-remember domain names without having to remember complex numerical IP addresses.

[0090] It should be noted that the way of parsing the data packet layer by layer according to the layers of the protocol stack (link layer, network layer, transport layer, application layer) makes the entire data packet processing flow clearer and more modular. The information of each layer can be parsed and processed independently, reducing the complexity of the analysis. Through this hierarchical parsing, the system can eliminate problems layer by layer and quickly locate abnormal information in the data packet, improving the parsing efficiency of the data packet; through the hierarchical parsing method, the system can carry out targeted processing of data at different levels according to business needs; by parsing these detailed data, the system can manage and analyze network traffic in a more refined manner.

[0091] Based on the loop sorting and block optimization technology, the decomposed multiple parsing tasks are assigned to multiple threads for parallel processing, including the following steps:

[0092] According to the capacity of L1 Cache (first-level cache) and L2 Cache (second-level cache), the size of the parsing task block processed by each thread is determined; each parsing task of the data packet is divided into blocks and divided into task units; all the block-based and expanded parsing tasks are placed in the task queue, and each thread takes out the task from the task queue for processing; the task queue assigns parsing tasks to different threads according to the complexity and workload of the tasks; each thread independently executes the task unit and parses various parts of the data packet, and dynamically adjusts the distribution method of each thread task according to the current network load and resource usage.

[0093] Specifically, the task allocation strategy is as follows:

[0094] Link layer and network layer parsing tasks can be assigned to thread A, which is responsible for extracting MAC address, source IP address, destination IP address, etc. Transport layer parsing tasks can be assigned to thread B, which is responsible for extracting port number, sequence number, TCP flag, etc. Application layer parsing tasks can be assigned to thread C, which is responsible for parsing HTTP requests or DNS query records.

[0095] It should be noted that task blocks of appropriate size can ensure that threads minimize cache misses when processing data, thereby increasing data processing speed; by dividing tasks into blocks, the granularity of tasks becomes finer, allowing threads to process smaller and more specific task units, reducing unnecessary waiting time; task queues are dynamically allocated according to task complexity and workload, ensuring that each thread obtains roughly the same amount of workload, avoiding overload or idleness of some threads, and effectively improving the parallel processing efficiency of the system; multi-threaded parallel processing enables multiple parsing tasks to be performed simultaneously, shortening the overall data processing time and improving the system's throughput; the task queue mechanism ensures that threads obtain tasks independently, reducing competition between threads, avoiding frequent locking and unlocking operations, and improving the efficiency of parallel processing.

[0096] The analysis results are formatted and batch processed in parallel according to the data size of the analysis results, including the following steps:

[0097] The parsing results are represented as a three-dimensional tensor of batch size, number of header information fields and sequence length; the data is divided into multiple blocks according to the batch size, and the number of parallel threads is automatically adjusted according to the number of header information fields and sequence length; the tasks of each block are numbered, and the modulus operation is used to take the modulus of the number of threads in the task number to evenly distribute the tasks to different threads; after each thread receives the task, it executes the processing task of the parsing results in parallel, generates an accurate timestamp based on the time information in the parsing results, adds necessary label information to each parsed data, and performs batch formatting on the assigned task partitions to generate standardized records.

[0098] Specifically, the operation of automatically adjusting the number of parallel threads is as follows:

[0099] When the batch size increases, the number of task partitions is automatically increased to ensure that the load of each thread remains balanced; when the number of header information fields increases, the tasks are further divided in dimension and assigned to multiple threads for processing.

[0100] It should be noted that by representing the parsing results as a three-dimensional tensor of batch size, number of header information fields and sequence length, the system can batch and parallelize a large number of parsing results. The data is divided into multiple blocks, and the number of parallel threads is dynamically adjusted according to the number of header information fields and sequence length, so that the data processing work can be reasonably allocated to multiple threads for execution. As the batch size increases, the task partition can be automatically expanded to ensure that the task load assigned to each thread is balanced, effectively prevent some threads from being overloaded, and improve the stability of the system. The remainder operation ensures that tasks can be evenly distributed to multiple threads, avoiding the problem of uneven task volume between threads, so that each thread can give full play to its processing capacity. By taking the remainder of the task number and the number of threads, the task allocation between threads is more reasonable, reducing the competition and synchronization overhead between threads, and improving the efficiency of parallel processing.

[0101] After the parsed data is formatted, it is written into the time series database in chronological order, and the index is generated based on the tag field. The following steps are included:

[0102] Convert the batch formatted data into a format suitable for the time series database, and insert it into the time series database in batches through the API interface; store the inserted data in the cache layer of the database, and dynamically refresh the cache layer according to the write frequency and memory usage, adjust the cache strategy based on the access frequency of different data, and dynamically adjust the cache size according to the current memory usage and database load; divide the data into multiple time partitions according to the timestamp of the data, generate a unique time series index for each written data, and partition and index the data according to the time range; use the inverted index structure to establish indexes for commonly used label fields, and use differential encoding technology to compress fields that appear more frequently.

[0103] In summary, the present invention introduces a dynamic weight scheduling mechanism to dynamically adjust the priority of data packets according to the real-time network status, business needs and current load, ensuring that key data packets are processed first, thereby improving the response speed of the system and the accuracy of data processing. Secondly, based on parallel data parsing and batch processing, high-priority data packets are decomposed into multiple parsing tasks and processed in parallel through multi-threading, which greatly improves the parsing efficiency and makes full use of the computing resources of the multi-core system. In addition, the present invention uses a dynamic cache management mechanism to dynamically adjust the cache size according to the throughput and maximum delay of network traffic, thereby avoiding the problems of data loss and processing delays. Finally, the use of time series database and inverted index technology greatly optimizes the storage and retrieval efficiency of data, ensuring fast query and analysis in a big data environment.

[0104] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of an operation and maintenance platform data acquisition system based on big data is provided.

[0105] The whole process of data packet capture, screening, parsing and processing in a high-concurrency network environment is simulated. The performance of the system in terms of data throughput, latency, data packet priority scheduling, parallel processing capability, etc. is evaluated and compared with the existing technology. Among them, traditional systems usually adopt single-threaded or simple multi-threaded models in data packet capture, processing and storage, while the present invention adopts a dynamic weight scheduling mechanism and parallel processing framework.

[0106] During the experiment, we first captured data packets from the network traffic of the operation and maintenance platform by initializing the network interface and setting filtering rules. Based on the timestamps extracted from the header of the captured data packets, we sorted the data packets in time series and stored them in a dynamically calculated ring buffer.

[0107] In the packet screening and preprocessing stage, metadata is extracted from the packet, including source IP address, destination IP address, protocol type, port number, etc. Based on the dynamic weight scheduling mechanism, the priority of each packet is dynamically adjusted according to the network status, business needs and current load. High-priority packets enter the parsing module first, and the packets are sorted from high to low according to the weight based on the heap sorting algorithm.

[0108] In the parsing phase, high-priority data packets are decomposed into multiple parsing tasks. Each data packet is decomposed into multiple layers of information such as link layer, network layer, transport layer and application layer, and parsed in the order of the protocol stack. Based on the L1 and L2 cache capacity, the task block size processed by each thread is determined, and loop unrolling and block optimization techniques are used to assign parsing tasks to different threads for parallel processing. Each thread independently executes the task unit, parses the data of the link layer, network layer, transport layer and application layer, and generates timestamp and label information based on the parsing results.

[0109] After parsing is completed, the data is processed in batches and parallelized, and the parsing results are represented as three-dimensional tensors (batch size, number of header information fields, and sequence length), and the number of threads is dynamically adjusted according to the data scale. Each thread performs data formatting tasks in parallel, writes the generated standardized records into the time series database in chronological order, and generates indexes based on the label fields to improve the efficiency of subsequent data retrieval and analysis.

[0110] It should be noted that the traditional system used in this implementation refers to the existing operation and maintenance platform data collection system based on static scheduling mechanism and simple multi-thread model. In these traditional systems, the capture, processing and storage of data packets are usually processed by fixed thread pool or single thread model, lacking the ability to dynamically adjust the priority of data packets.

[0111] The details are shown in Table 1 below:

[0112] Table 1 Experimental data record table

[0113]

[0114] It can be seen from the experimental data that the number of data packets processed by the system of the present invention in each experiment is significantly higher than that of the traditional system. For example, in Experiment 1, the traditional system processed 485,000 packets of data, while the system of the present invention processed 723,000 packets, and the processing efficiency was improved by nearly 50%. This is because the present invention introduces dynamic scheduling and multi-threaded parsing technology, which enables the system to maintain a high processing efficiency under high load conditions. However, the traditional system uses a fixed number of threads and a static scheduling mechanism, which leads to bottlenecks under high load conditions and cannot process more data packets.

[0115] The system of the present invention is also significantly better than the traditional system in terms of data processing delay. The average delay of the traditional system was above 125 ms in all experiments (up to 132 ms), while the delay of the system of the present invention was controlled between 85-93 ms. Such a gap proves the advantages of the present invention in dynamic buffer management and parallel processing, which can effectively reduce the time for data packets to wait for processing. Due to the lack of flexible buffer management in traditional systems, when network traffic fluctuates greatly, data packets need to queue for a long time to wait for processing, and the delay increases significantly.

[0116] Packet loss rate is an important indicator to measure the stability of the system under high load scenarios. The packet loss rate of the traditional system is higher than 8% in all experiments, and some experiments even exceed 10% (such as 10.1% in Experiment 4). In contrast, the packet loss rate of the system of the present invention is controlled between 2.5% and 3.2%, which greatly reduces the data loss. This is because the present invention adopts advanced cache management and priority scheduling strategies to ensure that high-priority data packets can be processed first when system resources are tight, reducing the occurrence of packet loss.

[0117] The processing throughput of the system of the present invention is close to the maximum throughput of network traffic (9.5-9.8 Gbps), while the throughput of the traditional system is between 8.0-8.4 Gbps. In a high-load network environment, the traditional system cannot fully utilize the bandwidth resources of the network due to large processing delays and high packet loss rates, resulting in limited processing throughput. The present invention can maintain a high throughput capacity in high-load scenarios through parallel processing and batch data processing technology.

[0118] It can be seen from the experimental data that the system of the present invention is significantly superior to the traditional system in terms of the number of data packets processed, average delay, packet loss rate and processing throughput. This advantage is mainly due to the dynamic weight scheduling mechanism, parallel processing framework and cache management technology of the present invention. In a high-concurrency, high-traffic operation and maintenance environment, the system of the present invention can better adapt to the fluctuations of network traffic, give priority to processing key data packets, and improve the overall processing capacity of the system. Due to the fixed buffer and scheduling mechanism, the processing efficiency and stability of the traditional system are obviously insufficient, especially when facing high throughput, it is prone to problems such as increased delay and increased packet loss rate.

[0119] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data collection system for an operation and maintenance platform based on big data, characterized by: include, The packet capture module is used to capture data packets from the network flow of the operation and maintenance platform and store them in the buffer; The screening preprocessing module is used to preliminarily screen the data packets in the buffer and assign priorities to the data packets based on the dynamic weight scheduling mechanism to complete the preliminary preprocessing of the data packets; The parallel parsing module is used to decompose the high-priority data packets after preliminary preprocessing into multiple parsing tasks, and based on the circular sorting and block optimization technology, the decomposed multiple parsing tasks are assigned to multiple threads for parallel processing; Batch processing module, used to format the analysis results and perform batch parallel processing according to the data size of the analysis results; The storage module is used to format the parsed data, write it into the time series database in chronological order, and generate an index based on the tag field; The retrieval and analysis module is used to quickly retrieve and analyze the data stored in the time series database through API query statements; Based on the loop sorting and block optimization technology, the decomposed multiple parsing tasks are assigned to multiple threads for parallel processing, including the following steps: Determine the block size of the parsing task processed by each thread based on the capacity of L1 Cache and L2 Cache; Divide each parsing task of the data packet into blocks and task units; Select the loop that can be expanded, analyze the loop operations in the task, and determine the loop expansion factor based on the number of registers and cache size; Put all the block-by-block and expanded parsing tasks into the task queue, and each thread takes tasks from the task queue for processing; The task queue assigns parsing tasks to different threads according to the complexity and workload of the tasks; Each thread independently executes the task unit, parses each part of the data packet, and dynamically adjusts the allocation of tasks for each thread according to the current network load and resource usage; The analysis results are formatted and batch processed in parallel according to the data size of the analysis results, including the following steps: Represent the parsing result as a three-dimensional tensor of batch size, number of header fields, and sequence length; Divide the data into multiple blocks according to the batch size, and automatically adjust the number of parallel threads according to the number of header fields and sequence length; Number the tasks of each block, use the modulus operation to take the modulus of the number of threads in the task number, and evenly distribute the tasks to different threads; After receiving the task, each thread executes the processing task of the parsing results in parallel, generates an accurate timestamp based on the time information in the parsing results, adds necessary label information to each parsed data, and performs batch formatting on the assigned task partitions to generate standardized records.

2. The data collection system for the operation and maintenance platform based on big data according to claim 1, characterized in that: The method of capturing data packets from the network flow of the operation and maintenance platform includes the following steps: Initialize the network interface and set filtering rules, capture data packets in real time, extract timestamps from the packet headers, and append them to the metadata of the data packets; Based on the timestamps of the data packets, the data packets are sorted in time sequence.

3. The data collection system for the operation and maintenance platform based on big data according to claim 2, characterized in that: Storing captured data packets in the buffer includes the following steps: The size of the ring buffer is dynamically calculated based on the throughput and maximum latency requirements. The expression is: ； in, Indicates the size of the ring buffer, Indicates the maximum processing delay that can be tolerated. Indicates the maximum throughput of network traffic, Indicates the average size of a single data packet; The sorted data packets are copied to the circular buffer and arranged in the order of timestamps. When the buffer reaches the maximum capacity, the earliest data packet is automatically discarded according to the first-in-first-out principle and the latest data packet is stored in the buffer.

4. The data collection system for the operation and maintenance platform based on big data according to claim 3, characterized in that: The preliminary screening of the data packets in the buffer includes the following steps: Extracting basic metadata from the data packet; the metadata includes source IP address and destination IP address, source port number and destination port number, protocol type and timestamp; According to the protocol type of the data packet, high priority protocol and low priority protocol are screened; Perform whitelist and blacklist screening based on source IP address and target IP address; Filter high-priority ports and low-priority ports based on source port number and destination port number; Perform timeliness screening based on the timestamp of the data packet.

5. The data collection system for the operation and maintenance platform based on big data according to claim 4, characterized in that: Based on the dynamic weight scheduling mechanism, the priority of the filtered data packets is assigned, and the preliminary preprocessing of the data packets is completed, including the following steps: After the screening is completed, a weight is assigned to each data packet, and the weight is dynamically adjusted according to the actual network status, business needs and current load; By comparing the weight of each data packet, a heap sorting algorithm is used to sort the data packets from high to low according to the weight to form a priority; Parse the IP header of the sorted high-priority data packets, extract necessary metadata and perform a preliminary check on the legitimacy of the data packets; After the data packet passes the legitimacy check, the data packet is marked and classified to complete the initial preprocessing of the data packet.

6. The data collection system for the operation and maintenance platform based on big data according to claim 5, characterized in that: Decomposing the high-priority data packets after preliminary preprocessing into multiple parsing tasks includes the following steps: The information in each data packet is divided into link layer, network layer, transport layer and application layer according to the order of protocol stack; According to the layers of the protocol stack, multiple layers in the data packet are decomposed into multiple tasks; For the link layer, extract the source MAC address and the destination MAC address, check the frame type field, determine whether it is an IPv4 or IPv6 packet, and determine the next parsing solution; For the network layer, extract the source IP address, destination IP address, protocol type, TTL and fragment offset of the IP header, check the protocol type field, determine the transport layer protocol used by the data packet, parse the TTL, flag, fragment information, total length, checksum content of the IP header, and perform a legality check; For the transport layer, extract the port number, sequence number, confirmation number, TCP flag bit and window size to check the legitimacy of the transport layer; For the application layer, the HTTP request method, URL, request header, DNS query type and response record of the application layer data are extracted and parsed.

7. The data collection system for the operation and maintenance platform based on big data according to claim 6, characterized in that: After the parsed data is formatted, it is written into the time series database in chronological order, including the following steps: Convert the batch formatted data into a format suitable for the time series database, and insert it into the time series database in batches through the API interface; The inserted data is stored in the cache layer of the database. The cache layer is dynamically refreshed according to the write frequency and memory usage, the cache strategy is adjusted based on the access frequency of different data, and the cache size is dynamically adjusted according to the current memory usage and database load.

8. The data collection system for the operation and maintenance platform based on big data according to claim 7, characterized in that: The data written into the time series database is indexed based on the tag field, including the following steps: Divide the data into multiple time partitions according to the timestamp of the data, generate a unique time series index for each written data, and partition and index the data according to the time range; An inverted index structure is used to create indexes for commonly used tag fields, and differential encoding technology is used to compress fields that appear more frequently.

Images