Processing system and method for accurate time of cloud environment data packet acquisition

Abstract

The invention provides a processing system and method for accurate time of cloud environment data packet acquisition, and the system comprises an acquisition module which carries out the timestamp standardization processing of an acquired data packet, adds a standardized timestamp into the data packet, and transmits the data packet; the forwarding module receives the data packet sent by the acquisition module, and determines forwarding or discarding of the data packet according to a preset condition; and the processing module is used for receiving the data packets of the acquisition module or the forwarding module, automatically identifying timestamp information in the data packets, determining whether to process the data packets or not according to a preset condition, and sorting the data packets. According to the invention, through intelligent timestamp identification and an adaptive time processing architecture, the problems of inaccurate data packet acquisition timestamp, low processing efficiency and the like in a cloud environment are solved, and accurate processing of data packet time is realized.

Description

Technical Field

[0001] The present invention relates to the field of data collection, monitoring and analysis. Specifically, it relates to a processing system and method for accurate time of cloud environment data packet collection. Background Art

[0002] Data packet collection and processing in the cloud environment refers to using packet capture technologies such as libpcap, eBPF or DPDK in the business network of the cloud, and then forwarding them to the processing end for processing. Due to the complex network environment (multi-layer tunnel protocol encapsulation) and numerous devices (routers, firewalls, switches, etc.), the simplest method is for the collection end and the processing center to directly capture data packets at both ends and process them respectively. However, the timestamp information of the data packets captured again at the processing end is inaccurate and requires timestamp standardization processing.

[0003] In the prior art, the following problems mainly exist in data packet timestamp processing:

[0004] 1. Inaccurate timestamp: In the cloud environment, due to network latency and clock synchronization problems, there is a deviation between the timestamp of the data packet at the processing end and the actual collection time. The prior art mainly relies on the capture timestamp and cannot guarantee the accuracy of time information.

[0005] 2. Inconsistent timestamp formats: The timestamp formats used at different collection points are inconsistent, including but not limited to accurate timestamps provided by manufacturers such as Asterfusion, IEEE 1588PTP timestamps, ordinary capture timestamps, etc., making it difficult to uniformly process the time information of data packets.

[0006] 3. Difficult to identify timestamps: It is difficult for the prior art to automatically identify whether a data packet contains an accurate timestamp. Especially after the data packet is forwarded through multiple layers, the timestamp information may be lost or distorted.

[0007] 4. Inflexible processing process: The processing process of the existing solutions is relatively fixed, making it difficult to dynamically adjust the processing strategy according to actual needs and lacking adaptability.

[0008] Patent document CN113347258A provides a method for data collection, monitoring and analysis under cloud traffic, including mirroring, collecting, encapsulating and identifying, and sending the traffic in the cloud environment, as well as collecting and network protocol analysis of the traffic in the cloud external environment. This method mainly focuses on data packet collection and transmission, but has the following limitations in timestamp processing:

[0009] 1. Simple timestamp processing: This solution mainly relies on the capture timestamp, lacks support and identification mechanisms for accurate timestamps, and cannot guarantee the accuracy of time information.

[0010] 2. Fixed processing architecture: This solution adopts a fixed processing architecture, lacking modular design and flexible configuration capabilities, and it is difficult to adapt to different network environments and processing requirements.

[0011] 3. Lack of timestamp standardization: This solution lacks a unified timestamp processing standard and it is difficult to handle timestamp formats from different sources.

[0012] In summary, the present invention aims to solve problems such as inaccurate timestamp of packet collection and low processing efficiency in the cloud environment through intelligent timestamp recognition and adaptive time processing architecture, and achieve precise processing of packet time. Summary of the Invention

[0013] Aiming at the defects in the prior art, the purpose of the present invention is to provide a processing system and method for accurate time of packet collection in the cloud environment.

[0014] According to a processing system for accurate time of packet collection in the cloud environment provided by the present invention, it includes:

[0015] A collection module that performs timestamp standardization processing on the collected packets, adds the standardized timestamp to the packets, and sends the packets.

[0016] A forwarding module that receives the packets sent by the collection module and decides whether to forward or discard the packets according to preset conditions.

[0017] A processing module that receives the packets from the collection module or the forwarding module, automatically identifies the timestamp information in the packets, decides whether to process the packets according to preset conditions, and sorts the packets.

[0018] Preferably, the collection module is used as the data entry of the system, encapsulates the packets with timestamps, adds metadata information, and sends the encapsulated packets to the forwarding module or the processing module.

[0019] Preferably, the forwarding module obtains the packets by packet capture. The packets include the packets with added timestamps and ordinary packets without added timestamps; and directly forwards the packets to be forwarded to the processing module.

[0020] Preferably, the processing module sorts the packets based on the timestamp and performs general processing on them; the general processing includes packet analysis and packet statistics.

[0021] According to a processing method for accurate time of packet collection in the cloud environment provided by the present invention, it includes:

[0022] Step S1: Collect network data packets and obtain real-time time information. Add the standardized timestamp to the data packet, then encapsulate the data packet, and add accurate timestamp information at the end of the data packet, and send the data packet;

[0023] Step S2: Receive the data packet, automatically identify whether the data packet contains accurate timestamp information. If the identification is successful, use this accurate timestamp information; otherwise, use the capture timestamp;

[0024] Step S3: Filter the data packet according to the configured filtering policy to decide whether to forward the data packet; Forward the data packet that meets the filtering conditions, and use the forward_info structure to record the forwarding information;

[0025] Step S4: Receive the data packet in Step S1 or Step S3, sort and perform general processing on the data packet according to the applied timestamp; After completing the processing of the current data packet, start the processing of the next data packet, and output the processed data packet.

[0026] Preferably, the step S1 includes the following sub-steps:

[0027] Step S1.1: Use the packet capture technologies such as libpcap, eBPF or DPDK to collect network data packets and obtain the time information of the data packets in real time; Store the collected data packet information in the packet_time structure;

[0028] Step S1.2: Convert the collected time information into a standard format to generate a standardized timestamp;

[0029] Step S1.3: Add the standardized timestamp to the data packet, and add accurate timestamp information at the end of the data packet. The timestamp adopts the IEEE 1588PTP format;

[0030] Step S1.4: Encapsulate the data packet with the timestamp, and use the packet_encap structure to encapsulate the data packet information;

[0031] Step S1.5: Send the encapsulated data packet, and use the send_info structure to record the sending information.

[0032] Preferably, the step S1.5 further includes:

[0033] After the processing of the current data packet is completed, return to Step S1.1 to start the processing of the next data packet, forming a continuous data packet processing loop.

[0034] Preferably, the step S2 includes the following sub-steps:

[0035] Step S2.1: Receive data packets, where the data packets include data packets with added timestamps and ordinary data packets without added timestamps; verify the integrity of the data packets and extract the timestamp information in the data packets.

[0036] Step S2.2: Automatically identify whether the data packet contains an accurate timestamp. If the identification is successful, use this accurate timestamp; otherwise, use the capture timestamp.

[0037] Preferably, step S2.2 includes the following sub-steps:

[0038] Step S2.2.1: Parse the IP header of the data packet to obtain the value of the IP Total Length field. Use the parse_ip_header function to parse the IP header in the data packet. This function takes the original data packet data as input and returns the parsed IP header information.

[0039] Step S2.2.2: Obtain the actual length of the data packet, including all header and payload data, for comparison with the IP TotalLength.

[0040] Step S2.2.3: Compare the actual length of the data packet with the IP Total Length to determine whether there are additional bytes. If the lengths are equal, it is an original data packet and does not contain an accurate timestamp. If the length is greater than the IP Total Length, there is an additional accurate timestamp.

[0041] Step S2.2.4: When the data packet length is equal to the IP Total Length, use the capture timestamp as the time information of the data packet.

[0042] Step S2.2.5: When the data packet length is greater than the IP Total Length, extract the timestamp data from the end of the data packet for verification.

[0043] Step S2.2.6: Check whether the length of the extracted timestamp data meets the preset requirements.

[0044] Step S2.2.7: Extract the seconds part from the timestamp data. Use the extract_timestamp function to extract the timestamp information from the data packet. This function takes the data packet information as input and returns the extracted timestamp.

[0045] Step S2.2.8: Calculate the difference between the extracted timestamp and the current system time.

[0046] Step S2.2.9: Determine whether the time difference is within the configured acceptable range.

[0047] Step S2.2.10: When the time difference is within the acceptable range, use the extracted accurate timestamp as the time information of the data packet;

[0048] Step S2.2.11: When the time difference exceeds the acceptable range, use the packet capture timestamp as the time information of the data packet.

[0049] Preferably, the sorting of data packets is completed based on a sorting algorithm or a circular buffer structure; the sorting algorithms include quicksort, heapsort, mergesort, and insertion sort.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. By parsing the IP header and checking the data packet length, the present invention can automatically identify whether the data packet contains an accurate timestamp; the system supports multiple timestamp formats (such as the accurate timestamps provided by manufacturers such as Asterfusion), and realizes the validity verification and automatic selection of timestamps. This mechanism can accurately identify and extract the time information in the data packet, providing a reliable time basis for subsequent processing.

[0052] 2. The present invention adopts a modular design, including three main components: a collection module, a forwarding module (optional), and a processing module; the system dynamically adjusts the processing strategy through a time quality evaluation mechanism, improving the flexibility and scalability of the system.

[0053] 3. The present invention unifies the processing standards for timestamps from different sources, realizing the standardization of time format conversion and time zone processing; the system provides a complete timestamp quality evaluation system, ensuring the consistency and reliability of time information.

[0054] 4. The present invention filters data packets according to the timestamp type and configuration strategy, supporting flexible configuration of processing rules; the system optimizes the use of system resources through an intelligent filtering mechanism, improving the processing efficiency and reducing the processing of invalid data.

[0055] Other beneficial effects of the present invention will be described in the specific implementation manner through the introduction of specific technical features and technical solutions. Those skilled in the art should be able to understand the beneficial technical effects brought by the technical features and technical solutions through these introductions. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects, and advantages of the present invention will become more apparent:

[0057] Figure 1 It is the system architecture diagram of the present invention.

[0058] Figure 2 It is the processing flow chart of the acquisition module in the present invention.

[0059] Figure 3 It is the processing flow chart of timestamp standardization in the present invention.

[0060] Figure 4 It is the processing flow chart of the forwarding module in the present invention.

[0061] Figure 5 It is the processing flow chart of timestamp recognition in the present invention.

[0062] Figure 6 It is the processing flow chart of the processing end module in the present invention.

[0063] Figure 7 It is the basic data packet format diagram of the present invention.

[0064] Figure 8 It is the data packet format diagram with additional precise timestamps of the present invention.

[0065] Figure 9 It is the IEEE 1588 PTP timestamp format diagram of the present invention. Specific embodiments

[0066] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all fall within the protection scope of the present invention.

[0067] As Figure 1 shown, a processing system for accurate time of cloud environment data packet acquisition includes:

[0068] An acquisition end module, a forwarding end module (optional), and a processing end module.

[0069] Module M1: Acquisition end module. As the data entry of the system, the acquisition end module is responsible for the acquisition of data packets and the addition of timestamps. This module acquires network data packets through packet capture technologies such as libpcap, eBPF, or DPDK, performs timestamp standardization processing on the acquired data packets, and adds the standardized timestamps to the data packets. Subsequently, the system encapsulates the data packets with timestamps, adds necessary metadata information, and finally sends the encapsulated data packets to the forwarding end or the processing end. The design of this module ensures the real-time nature of data packet acquisition and the accuracy of time information.

[0070] Optional module M2: Forwarding module. The forwarding module, as the middle layer of the system, is responsible for forwarding data packets. This module receives data packets from the collection end. Since the forwarding process also obtains data packets by packet capture, the received data packets may contain the accurate timestamps of the present invention or may be ordinary data packets without accurate timestamps. The system first identifies whether the data packets contain accurate timestamps and decides whether to forward or discard them according to the configuration. For the data packets that need to be forwarded, the system directly forwards them to the processing end, optimizes the forwarding performance, and ensures the reliable transmission of data packets. The introduction of this module provides a flexible data packet filtering and forwarding mechanism and enhances the configurability of the system.

[0071] Module M3: Processing module. The processing module is the core processing unit of the system and is responsible for processing data packets and applying timestamps. This module receives data packets from the collection end or the forwarding end, automatically identifies the timestamp information in the data packets, decides whether to process the data packets according to the configuration, applies accurate timestamps or ordinary timestamps, and sorts the data packets. Finally, the system performs general processing on the sorted data packets, including operations such as data packet analysis and statistics. The design of this module ensures the accuracy and efficiency of data packet processing and provides flexible processing strategy configuration at the same time.

[0072] A method for processing accurate time of data packet collection in a cloud environment, comprising:

[0073] Step 1.1: Data packet collection. As Figure 2 shown, the system uses packet capture technologies such as libpcap, eBPF, or DPDK to collect network data packets and obtain the time information of the data packets in real time. The collected data packet information is stored in the packet_time structure, which includes fields such as capture_time (collection time), timezone (timezone information, default is UTC), and precision (time precision, set to nanosecond level). As Figure Seven shown, the basic data packet format includes an Ethernet header, a network layer header, a transport layer header, and payload data.

[0074]

[0075] Step 1.2: Timestamp Standardization. The system converts the collected time information into a standard format to ensure the accuracy and consistency of timestamps. The timestamp standardization process includes operations such as time format conversion and time zone processing to generate standardized timestamps. This invention provides an implementation method based on Python, but is not limited to this, and other computer languages and methods can also be used to implement it. The standardization process uses the timestamp_standard structure to store processing information, including fields such as original_time (original timestamp), standardized_time (standardized timestamp), timezone_offset (time zone offset), and precision (time precision).

[0076] As Figure 3 shown, the specific processing steps are described as follows:

[0077] Step 1.2.1: Create a time zone object. The system creates a time zone object according to the configured time zone information for subsequent time conversion. This invention uses the pytz library in Python to implement it, but other time zone processing libraries or methods can also be used.

[0078] # Create a time zone object

[0079] source_tz = pytz.timezone(source_timezone) # Source time zone

[0080] utc_tz = pytz.UTC # UTC time zone

[0081] Step 1.2.2: Timestamp conversion. The system converts the original timestamp into a datetime object for easy time zone conversion. This invention uses the datetime library in Python to implement it, but other time processing libraries or methods can also be used.

[0082]

[0083] Step 1.2.3: Time zone conversion. The system converts the time to the UTC time zone to ensure the consistency of time information. This invention uses the astimezone method of datetime to implement it, but other time zone conversion methods can also be used.

[0084] # Convert the time zone to UTC

[0085] utc_dt = original_dt.astimezone(utc_tz)

[0086] Step 1.2.4: Calculate the time zone offset. The system calculates the offset between the source time zone and the UTC time zone for subsequent time standardization. This invention uses the utcoffset method of timezone to implement it, but other offset calculation methods can also be used.

[0087] # Calculate the time zone offset

[0088] tz_offset = source_tz.utcoffset(original_dt).total_seconds()

[0089] Step 1.2.5: Generate the standardized timestamp. The system converts the UTC time back to the timestamp format to generate a standardized timestamp. This invention uses the timestamp method of datetime to implement it, but other timestamp generation methods can also be used.

[0090] # Generate the standardized timestamp

[0091] standardized_ts = utc_dt.timestamp()

[0092] # Update the standardized structure

[0093] timestamp_standard.update({

[0094] 'original_time': original_timestamp,

[0095] 'standardized_time': standardized_ts,

[0096] 'timezone_offset': tz_offset,

[0097] 'precision': 'nanosecond'

[0098] })

[0099] To more comprehensively demonstrate the complete process of timestamp standardization, a complete example of a timestamp standardization function is provided below. This function integrates all the above steps and adds an error handling mechanism to ensure that even if the standardization process fails, the system can return a valid original timestamp. This implementation method has high reliability and robustness in practical applications.

[0100]

[0101]

[0102]

[0103] Step 1.3: Timestamp addition. The system adds the standardized timestamp to the data packet to ensure the integrity and readability of the timestamp information. The timestamp addition process includes operations such as the insertion of the timestamp field and format optimization. As Figure 8 shown, the system adds accurate timestamp information at the end of the data packet. The timestamp is in the IEEE 1588PTP format. As Figure 9 shown, it contains the following fields: timestamp (timestamp value), precision (precision information, in nanoseconds), timezone (timezone information, UTC), and format (time format, IEEE1588). The system uses the packet_encap structure to encapsulate the time information, which includes fields such as timestamp (timestamp information), capture_time (acquisition time), precise_timestamp (accurate timestamp), and time_delta (time deviation).

[0104] Step 1.4: Data packet encapsulation. The system encapsulates the data packet with the timestamp, and adds accurate timestamp information (such as the accurate timestamp provided by manufacturers like Asterfusion) at the end of the data packet. The encapsulation process includes operations such as adding the timestamp field and format optimization. The system uses the packet_encap structure to encapsulate the data packet information, which includes fields such as timestamp (timestamp information), capture_time (acquisition time), precise_timestamp (accurate timestamp), and time_delta (time deviation). Among them, the timestamp field stores the standardized timestamp information for subsequent time processing; the capture_time field stores the time information when the data packet is acquired for time synchronization and calibration; the precise_timestamp field stores the accurate timestamp information, such as the accurate timestamp provided by manufacturers like Asterfusion; the time_delta field stores the time deviation information for time synchronization and calibration.

[0105]

[0106] Step 1.5: Packet Sending. The system sends the encapsulated packets to the forwarding end or the processing end, ensuring reliable transmission of the packets and optimizing the transmission performance. The sending process includes packet fragmentation, retransmission mechanism, etc. The system uses the send_info structure to record the sending information, including fields such as packet_time (packet time information), send_time (sending time), and time_delta (time deviation). Among them, the packet_time field stores the complete packet time information, including timestamp, acquisition time, and precise timestamp, etc.; the send_time field records the time when the packet is sent, which is used for transmission delay calculation; the time_delta field stores the time deviation information, which is used for time synchronization and calibration. After the processing is completed, the system returns to Step 1.1 to start processing the next packet, forming a continuous packet processing loop.

[0107]

[0108] Step 2.1: Packet Receiving. As Figure 4 shown, the system receives the packets from the acquisition end by packet capture. Since the forwarding process also obtains the packets by packet capture, the received packets may contain the precise timestamp of the present invention or may be ordinary packets without the precise timestamp. The system verifies the integrity of the packets and extracts the timestamp information in the packets. The receiving process includes packet recombination, integrity verification, etc. The system uses the receive_info structure to record the receiving information, including fields such as packet_time (packet time information), receive_time (receiving time), time_delta (time deviation), and has_precise_timestamp (whether it contains a precise timestamp).

[0109]

[0110] Step 2.2: Timestamp Identification. As Figure 5As shown, the system automatically identifies whether the data packet contains a precise timestamp (such as the precise timestamp provided by manufacturers like Asterfusion). If it exists, the precise timestamp is used; otherwise, the capture timestamp is used. The timestamp identification process includes steps such as data packet length check, timestamp extraction and verification, and timestamp selection. The system uses the timestamp_detect structure to record the identification results, which includes fields such as detected_time (the detected timestamp), time_source (the timestamp source), time_valid (the time validity), time_delta (the time deviation), packet_length (the total length of the data packet), ip_length (the length in the IP header), has_extra_bytes (whether there are extra bytes), and validation_result (the verification result). The identification process is configured using the timestamp_config structure, which includes parameters such as acceptable_time_delta (the acceptable time deviation), timestamp_size (the size of the precise timestamp), min_packet_size (the minimum data packet size), and max_packet_size (the maximum data packet size).

[0111]

[0112] Step 2.2.1: Parse the IP header to obtain the total length. The system parses the IP header of the data packet to obtain the value of the IP TotalLength field, which is used to subsequently determine whether the data packet contains additional timestamp information. The system uses the parse_ip_header function to parse the IP header in the data packet. This function takes the raw data packet data as input and returns the parsed IP header information. The parsing process is implemented using the standard network library or general network parsing library provided by the system to ensure the accuracy and compatibility of IP header parsing.

[0113] # Parse the IP header to obtain the total length

[0114] ip_header = parse_ip_header(packet_data)

[0115] ip_length = ip_header['total_length']

[0116] Step 2.2.2: Check the actual length of the data packet. The system obtains the actual length of the data packet, including all headers and payload data, for comparison with the IP Total Length.

[0117] # Check the actual length of the data packet

[0118] packet_length = len(packet_data)

[0119] Step 2.2.3: Whether the length is greater than the IP length. The system compares the actual length of the data packet with the IP Total Length to determine whether there are extra bytes. If the lengths are equal, it means it is the original data packet without a precise timestamp; if the length is greater than the IP Total Length, there may be an additional precise timestamp.

[0120] # Check whether there are extra bytes

[0121] has_extra_bytes = packet_length > ip_length

[0122] Step 2.2.4: Use the capture timestamp. When the length of the data packet is equal to the IP Total Length, the system uses the capture timestamp as the time information of the data packet.

[0123]

[0124] Step 2.2.5: Extract the timestamp data at the end. When the length of the data packet is greater than the IP Total Length, the system extracts the timestamp data from the end of the data packet for verification.

[0125] # Extract the timestamp data at the end

[0126] timestamp_data = packet_data[ip_length:]

[0127] Step 2.2.6: Whether the timestamp data is sufficient. The system checks whether the length of the extracted timestamp data meets the requirements (default 16 bytes) to ensure the integrity of the timestamp data.

[0128]

[0129] Step 2.2.7: Extract the seconds of the timestamp. The system extracts the seconds part (Seconds) from the timestamp data for subsequent time difference calculation. The system uses the extract_timestamp function to extract the timestamp information from the data packet. This function receives the data packet information as input and returns the extracted timestamp. The extraction process is implemented using the standard time processing library or general time parsing library provided by the system to ensure the accuracy and compatibility of timestamp extraction.

[0130] # Extract the seconds of the timestamp

[0131] timestamp = extract_timestamp(timestamp_data)

[0132] Step 2.2.8: Calculate the time difference. The system calculates the difference between the extracted timestamp and the current system time to determine the validity of the timestamp.

[0133] # Calculate the time difference

[0134] current_time = time.time()

[0135] time_delta = abs(timestamp - current_time)

[0136] Step 2.2.9: Whether the time difference is within the acceptable range. The system determines whether the time difference is within the configured acceptable range (default 10 minutes) to verify the validity of the timestamp.

[0137] # Verify the time difference

[0138] is_acceptable = time_delta <= timestamp_config['acceptable_time_delta']

[0139] Step 2.2.10: Use the precise timestamp. When the time difference is within the acceptable range, the system uses the extracted precise timestamp as the time information of the data packet.

[0140]

[0141]

[0142] Step 2.2.11: Use the packet capture timestamp. When the time difference exceeds the acceptable range, the system uses the packet capture timestamp as the time information of the data packet.

[0143]

[0144] Step 2.3: Data packet filtering. The system filters the data packets according to the configured filtering policy to decide whether to forward the data packet.

[0145] 1. Filtering based on timestamp type: The system filters according to whether the data packet contains a precise timestamp. The filtering configuration includes:

[0146] · Whether it contains a precise timestamp (has_precise_timestamp);

[0147] · Whether to accept packets containing precise timestamps (accept_precise, default is true);

[0148] · Whether to accept normal packets (accept_normal, default is false).

[0149] 2. Filtering based on timestamp validity: The system verifies the validity of timestamps, including:

[0150] · Whether the length of the timestamp data meets the requirements (default is 8 bytes, where 4 bytes are for seconds and 4 bytes are for sub - seconds);

[0151] · Whether the time difference is within the acceptable range (default is 10 minutes);

[0152] · Whether the timestamp format is correct.

[0153] 3. Filter result recording: The filtering process uses the packet_filter structure to record the filtering results, including fields such as has_precise_timestamp (whether it contains a precise timestamp), accept_precise (whether to forward packets with precise timestamps), accept_normal (whether to forward normal packets), and filter_result (filtering result).

[0154]

[0155] Step 2.4: Packet forwarding. The system forwards packets that meet the filtering conditions to the processing end. The forwarding process uses the forward_info structure to record the forwarding information, including fields such as packet_time (packet time information), forward_time (forwarding time), and time_delta (time deviation).

[0156]

[0157] Step 3.1: Packet reception. As Figure 6As shown, the system receives data packets from the acquisition end or the forwarding end through packet capture. Since the receiving process also obtains data packets through packet capture, the received data packets may contain the precise timestamps of the present invention or may be ordinary data packets without precise timestamps. The system obtains a new packet capture timestamp, verifies the integrity of the data packets, and prepares for subsequent processing. The receiving process includes data packet recombination, integrity verification, etc. The system uses the process_info structure to record processing information, including fields such as packet_time (data packet time information), receive_time (receiving time), processing_delay (processing delay), and has_precise_timestamp (whether it contains a precise timestamp).

[0158]

[0159] Step 3.2: Timestamp identification. It is the same as Step 2.2: Timestamp identification.

[0160] Step 3.3: Data packet filtering. It is the same as Step 2.3: Data packet filtering.

[0161] Step 3.4: Apply timestamp. The system selects an appropriate timestamp according to the identification result and applies it to the data packet. The application process uses the timestamp_apply structure to record the application result, including fields such as packet_time (data packet time information), applied_timestamp (applied timestamp), time_source (timestamp source), and time_delta (time deviation).

[0162]

[0163] Step 3.5: Data packet sorting. The system sorts the data packets according to the applied timestamp to ensure that the data packets are processed in chronological order. The sorting process can adopt but is not limited to the following methods: common sorting algorithms such as quicksort, heapsort, mergesort, insertion sort, etc., or a circular buffer structure can also be adopted to optimize memory usage and improve sorting efficiency. The present invention provides an implementation method based on a circular buffer, but is not limited to this, and other sorting algorithms and data structures can also be used to achieve the same function.

[0164] Step 3.6: Data packet processing. The system performs general processing on the sorted data packets, including but not limited to operations such as data packet parsing, protocol analysis, traffic statistics, feature extraction, etc. The processing process adopts a modular design to support flexible expansion. The system sends the processing result to the downstream processing module to complete the processing flow of the current data packet. After processing is completed, the system returns to Step 3.1 to start processing the next data packet, forming a continuous data packet processing loop.

[0165] Those skilled in the art know that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a kind of hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structures within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or the structures within the hardware component.

[0166] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A processing system for collecting accurate time of cloud environment data packets, characterized in that: include: The acquisition module performs time stamp standardization on the collected data packets, adds the standardized time stamp to the data packets, and sends the data packets; The forwarding module receives the data packets sent by the acquisition module and decides whether to forward or discard the data packets according to preset conditions; The processing module receives the data packets from the acquisition module or the forwarding module, automatically identifies the timestamp information in the data packets, decides whether to process the data packets according to preset conditions, and sorts the data packets.

2. The system for processing cloud environment data packets for accurate time collection according to claim 1, characterized in that: The acquisition module is used for the data entry of the system, encapsulates the data packets with timestamps, adds metadata information, and sends the encapsulated data packets to the forwarding module or the processing module.

3. The processing system for collecting accurate time of cloud environment data packets according to claim 1, characterized in that: The forwarding module obtains data packets by capturing packets, wherein the data packets include data packets with timestamps added and common data packets without timestamps added; and directly forwards the data packets to be forwarded to the processing module.

4. The processing system for collecting accurate time of cloud environment data packets according to claim 1, characterized in that: The processing module sorts the data packets based on the timestamps and performs general processing on them; the general processing includes analysis of the data packets and statistics of the data packets.

5. A method for processing precise time of cloud environment data packet collection, based on the system for processing precise time of cloud environment data packet collection according to any one of claims 1 to 4, characterized in that: include: Step S1: Collect network data packets and obtain real-time time information, add standardized timestamps to the data packets, encapsulate the data packets, add accurate timestamp information at the end of the data packets, and send the data packets; Step S2: receiving a data packet, automatically identifying whether the data packet contains accurate timestamp information, if the identification is successful, using the accurate timestamp information, otherwise using the packet capture timestamp; Step S3: Filter the data packet according to the configured filtering policy and decide whether to forward the data packet; forward the data packet that meets the filtering conditions and use the forward_info structure to record the forwarding information; Step S4: receiving the data packets in step S1 or step S3, sorting and generally processing the data packets according to the application timestamp; after completing the processing of the current data packet, starting the processing of the next data packet, and outputting the processed data packet.

6. The method for processing accurate time of cloud environment data packet collection according to claim 5, characterized in that: The step S1 comprises the following sub-steps: Step S1.1: Use libpcap, eBPF or DPDK packet capture technology to collect network data packets and obtain the time information of the data packets in real time; store the collected data packet information in the packet_time structure; Step S1.2: converting the collected time information into a standard format to generate a standardized timestamp; Step S1.3: Add the standardized timestamp to the data packet, and add the accurate timestamp information at the end of the data packet. The timestamp adopts the IEEE 1588PTP format. Step S1.4: Encapsulate the data packet with the timestamp and use the packet_encap structure to encapsulate the data packet information; Step S1.5: Send the encapsulated data packet and use the send_info structure to record the sending information.

7. The method for processing the precise time of cloud environment data packet collection according to claim 6, characterized in that: The step S1.5 further includes: After the current data packet is processed, the process returns to step S1.1 to start processing the next data packet, thus forming a continuous data packet processing loop.

8. The method for processing the precise time of cloud environment data packet collection according to claim 5, characterized in that: The step S2 comprises the following sub-steps: Step S2.1: receiving data packets, including data packets with timestamps added and ordinary data packets without timestamps added; verifying the integrity of the data packets, and extracting the timestamp information in the data packets; Step S2.2: Automatically identify whether the data packet contains an accurate timestamp. If the identification is successful, use the accurate timestamp; otherwise, use the captured timestamp.

9. The method for processing accurate time of cloud environment data packet collection according to claim 8, characterized in that: The step S2.2 includes the following sub-steps: Step S2.2.1: Parse the IP header of the data packet, obtain the value of the IP Total Length field, and use the parse_ip_header function to parse the IP header in the data packet. The function receives the original data packet data as input and returns the parsed IP header information; Step S2.2.2: Get the actual length of the data packet, including all header and payload data, for comparison with IP TotalLength; Step S2.2.3: Compare the actual length of the data packet with the IP Total Length to determine whether there are extra bytes; If the lengths are equal, it is the original data packet and does not contain an accurate timestamp; if the length is greater than the IP Total Length, an additional accurate timestamp exists; Step S2.2.4: When the length of the data packet is equal to the IP Total Length, the packet capture timestamp is used as the time information of the data packet; Step S2.2.5: When the packet length is greater than the IP Total Length, extract the timestamp data from the end of the packet and prepare for verification; Step S2.2.6: Check whether the length of the extracted timestamp data meets the preset requirements; Step S2.2.7: Extract the seconds portion from the timestamp data, using the extract_timestamp function to extract the timestamp information from the data packet, which receives the data packet information as input and returns the extracted timestamp; Step S2.2.8: Calculate the difference between the extracted timestamp and the current system time; Step S2.2.9: Determine whether the time difference is within the configured acceptable range; Step S2.2.10: When the time difference is within the acceptable range, using the extracted accurate timestamp as the time information of the data packet; Step S2.2.11: When the time difference exceeds the acceptable range, the packet capture timestamp is used as the time information of the data packet.

10. The method for processing accurate time of cloud environment data packet collection according to claim 5, characterized in that: The data packets are sorted based on a sorting algorithm or a ring buffer structure; the sorting algorithm includes quick sort, heap sort, merge sort and insertion sort.

Images