Industrial network full-flow data storage and analysis method and system

Abstract

The application relates to the technical field of data management, in particular to a storage and analysis method and system for full-flow data of an industrial network, which comprises the following steps: acquiring switch mirror images and network interface data frames and numbering them according to timestamps, correcting abnormal frames to form an ordered queue, extracting address ports and protocols and storing them, establishing a position index according to time sections, and comparing and analyzing fields across files. In the application, continuous comparison of data frame timestamps is carried out and progressive numbering is given, abnormal time offset correction and sequence verification are combined, field consistency discrimination is introduced in the writing process to complete communication feature classification, and an association index is established through time section division and storage position mapping to form a corresponding relationship between the time dimension and the storage structure. In the analysis process, cross-position field comparison and protocol field splitting are combined to complete structured reorganization, the order and relevance of data organization are improved, and the information restoration ability and processing continuity in a differentiated communication environment are enhanced.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a method and system for storing and parsing full traffic data of industrial networks. Background Technology

[0002] Data management technology refers to a system of methods centered around core aspects such as data acquisition, storage, organization, indexing, retrieval, and parsing. It primarily involves collecting and recording multi-source data, establishing correspondences based on time sequences or identifier fields, segmenting and writing data to storage media, constructing an index structure, and then reading, parsing, and reassembling the data according to predetermined field rules. This achieves unified management and processing of data from different sources, in different formats, and across different time dimensions. Traditional industrial network full-traffic data storage and parsing methods target all message data generated during industrial network communication. This involves capturing data frames packet by packet at switching nodes or interfaces, writing each frame to a continuous file in capture order, and classifying and recording it according to the source address, destination address, port number, and protocol type in the message header. During the parsing phase, the storage file is read sequentially, and Ethernet header fields, IP header fields, and transport layer port information are extracted according to fixed byte offsets. The payload data is then segmented and reassembled line by line based on protocol identifiers to reconstruct the communication content.

[0003] Existing technologies perform linear writing based on capture time in actual operation. When faced with network fluctuations or multi-path transmission, time deviations are prone to accumulate. There is a lack of means to identify and correct abnormal time intervals, which causes local disorder in data arrangement. At the same time, the storage structure does not form a correspondence between time segments and locations. The retrieval process relies on segment-by-segment traversal, which limits efficiency. For example, when there is a sudden congestion or link switching in the industrial control network, the message arrival time is unevenly distributed. The parsing stage is prone to association mismatch or fragmentation, which in turn affects the continuity and accuracy of communication behavior analysis. Summary of the Invention

[0004] To address the technical problems existing in the prior art, embodiments of the present invention provide a method for storing and parsing full traffic data of industrial networks; To achieve the above objectives, the present invention adopts the following technical solution: a method for storing and parsing full traffic data of an industrial network, comprising the following steps: S1: Obtain the data frames output from the mirror port of the industrial switch and the industrial control network interface, extract the Ethernet frame timestamps, compare adjacent timestamps and assign them sequential numbers to obtain the message time sequence record. S2: Based on the message time sequence record, compare adjacent timestamps and adjust the position of abnormal data frames, reassign sequence numbers and compare the sequence again to obtain an ordered queue of industrial communication messages. S3: Based on the ordered queue of industrial communication messages, extract the source address, destination address, port number and protocol identifier, analyze the consistency of the fields, write the data frame to the storage file, and obtain the data corresponding to the storage location of the industrial message. S4: Based on the data corresponding to the storage location of the industrial message, extract the timestamp and divide the time segment, centrally divide the locations within the same time segment and map the corresponding relationship, associate the time segment with the location, and obtain the industrial message time location index set; S5: Based on the industrial message time location index set, locate the corresponding position of the time segment, read the header field and payload content of the data frame, compare the cross-file data frame fields, split according to the protocol field position, and obtain the industrial network message structured parsing dataset.

[0005] As a further embodiment of the present invention, the message time sequence record includes a timestamp set, a sequence number set, a capture order relationship, and a frame identifier mapping; the industrial communication message ordered queue includes a correction sequence, an anomaly marker, a rearrangement index, and a progressive relationship identifier; the industrial message storage location corresponding data includes address field classification, port category label, protocol type grouping, and file offset interval; the industrial message time location index set includes time segment division units, location interval mapping, index key-value pairs, and segment identifiers; and the industrial network message structured parsing dataset includes frame header structure fields, payload data blocks, cross-file splicing units, protocol layer structure models, and location association records.

[0006] As a further aspect of the present invention, the Ethernet frame timestamp refers to the time stamp when each data frame is collected; The location of the abnormal data frame refers to the position of the data frame that does not conform to the time progression order during the continuous comparison of timestamps.

[0007] As a further aspect of the present invention, the field consistency refers to the identical characteristics of differentiated data frames in the source address, destination address, port number, and protocol identifier fields; The cross-file data frame field refers to the corresponding field between data frames distributed in the differentiated storage file.

[0008] As a further aspect of the present invention, the specific steps of S1 are as follows: S101: Obtain the output data frames of the industrial switch mirror port and the industrial control network interface, extract the Ethernet frame timestamps and sort them according to the order of capture, compare adjacent timestamps and identify the time sequence to obtain the timestamp sequence. S102: Based on the timestamp sequence, continuously compare adjacent timestamps and identify positions with abnormal order, swap the data frame positions corresponding to the abnormal positions and readjust the order to obtain a time sequence. S103: Based on the time sequence, assign a sequential number to the data frame corresponding to each timestamp, map the timestamps to the sequential numbers, and obtain the message time sequence record.

[0009] As a further aspect of the present invention, the specific steps of S2 are as follows: S201: Based on the message time sequence record, compare the relationship between adjacent timestamps, identify the position of the data frame with abnormal sequence, extract the data frame number corresponding to the abnormal position, and obtain the abnormal position identification sequence. S202: Based on the abnormal location identifier sequence, swap the corresponding data frame positions and reorder the timestamp order, continuously compare the swapped data frames, analyze the time progression relationship, and obtain the time progression sequence; S203: Based on the time progression sequence, reassign the sequence number of each data frame, map the timestamp to the sequence number, and obtain an ordered queue of industrial communication messages.

[0010] As a further aspect of the present invention, the specific steps of S3 are as follows: S301: Based on the ordered queue of industrial communication messages, extract the source address, destination address, port number and protocol identifier of the data frame, compare the fields one by one and identify the field consistency distribution, classify the data frames according to field consistency, and obtain the message field classification set; S302: Based on the message field classification set, write each type of data frame into the storage file in sequence, extract the start byte position and end byte position of each data frame during the writing process, associate the start position and end position of the same data frame, and obtain the data frame position interval set. S303: Based on the set of data frame position intervals, map each data frame identifier to its corresponding start and end positions, associate the data frames with their corresponding positions, and obtain the data corresponding to the industrial message storage location.

[0011] As a further aspect of the present invention, the specific steps of S4 are as follows: S401: Based on the data corresponding to the storage location of the industrial message, extract the timestamp of the corresponding data frame and sort it according to the order of capture. Continuously compare the timestamps and identify the time segment boundaries. Divide the timestamps according to the segments to obtain a set of time segments. S402: Based on the set of time segments, extract the corresponding positions of data frames within the time segments, group and divide the positions while maintaining a consistent distribution of positions within the same time segment, map and associate the grouping results, and obtain the time segment position mapping relationship; S403: Based on the time segment location mapping relationship, associate and connect the time segments with their corresponding locations, and perform a unified index description on the corresponding location of each time segment to obtain the industrial message time location index set.

[0012] As a further aspect of the present invention, the specific steps of S5 are as follows: S501: Based on the industrial message time location index set, retrieve the corresponding location of the target time segment, extract the Ethernet header field, IP header field and transport layer field of the data frame in the corresponding file, extract the data frame order within the same time segment, and obtain the data frame field sequence; S502: Based on the data frame field sequence, compare the source address, destination address, port number and protocol identifier of the cross-file data frame item by item, connect the data frames with consistent fields and adjust the splicing position, and arrange the spliced ​​data frames continuously to obtain the cross-file message sequence. S503: Based on the cross-file message sequence, split the data frame according to the protocol field position, match the Ethernet header field, IP header field and transport layer field, associate the field order, and obtain the industrial network message structured parsing dataset.

[0013] A storage and parsing system for full traffic data of an industrial network includes: The message acquisition and sorting module acquires the data frames output from the mirror port of the industrial switch and the industrial control network interface, extracts the Ethernet frame timestamps and sorts them according to the order of capture, compares adjacent timestamps and assigns them sequential numbers, and matches the timestamps with the sequential numbers to obtain the message time sequence record. The sequence correction module, based on the message time sequence record, compares adjacent timestamps and adjusts the positions of data frames with abnormal order, reassigns sequence numbers to the adjusted data frames and compares the order again to obtain an ordered queue of industrial communication messages. The segmented writing module extracts the source address, destination address, port number and protocol identifier based on the ordered queue of industrial communication messages, performs consistency comparison and classification on the fields, writes the data frame to the storage file and extracts the start position and end position, and maps the data frame to the position to obtain the data corresponding to the storage position of the industrial message. The index generation module extracts timestamps and divides time segments based on the data corresponding to the storage location of the industrial message. It performs centralized division and mapping of the corresponding relationships within the same time segment, associates the time segment with the location, and obtains the industrial message time location index set. The splicing and parsing module, based on the industrial message time location index set, locates the corresponding position of the time segment, reads the header fields and payload content of the data frame, compares the fields of cross-file data frames and adjusts the splicing order, and splits them according to the protocol field position to obtain the industrial network message structured parsing dataset.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, by continuously comparing the timestamps of data frames and assigning progressive numbers, combined with abnormal time offset correction and sequence verification, and introducing field consistency judgment during the writing process to complete the classification of communication features, and establishing an associated index through time segment division and storage location mapping, the time dimension and storage structure are made to correspond. During the parsing process, cross-location field comparison and protocol field splitting are combined to complete the structured reorganization, thereby improving the orderliness and correlation of data organization, and enhancing the information restoration capability and processing continuity in differentiated communication environments. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the steps of the present invention; Figure 2 This is a detailed schematic diagram of S1 of the present invention; Figure 3 This is a detailed schematic diagram of S2 of the present invention; Figure 4 This is a detailed schematic diagram of S3 of the present invention; Figure 5 This is a detailed schematic diagram of S4 of the present invention; Figure 6 This is a detailed schematic diagram of S5 of the present invention; Figure 7 This is a system module diagram of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0019] Please see Figure 1 This invention provides a method for storing and parsing full traffic data of an industrial network, comprising the following steps: S1: Obtain the output data frames of the industrial switch mirror port and the industrial control network interface, extract the Ethernet frame timestamps and sort them according to the order of capture, determine the relationship between adjacent timestamps and assign them sequential numbers, match the timestamps with the sequential numbers, and obtain the message time sequence record sequence. S2: Based on the message time sequence, compare the relationship between adjacent timestamps and adjust the position of data frames with abnormal order. Reassign the order number to the adjusted data frames and compare the order again. Arrange the data frames in a time-progressive relationship to obtain an ordered queue of industrial communication messages. S3: Based on the ordered queue of industrial communication messages, extract the source address, destination address, port number and protocol identifier, analyze the consistency of field content and classify them, write the classified data frames into storage files in sequence and extract the start and end positions of writing, match the data frames with the positions to obtain the data corresponding to the storage positions of industrial messages. S4: Based on the data corresponding to the storage location of industrial messages, extract the timestamp of the corresponding data frame and divide the time segments in order. Then, divide the locations within the same time segment and perform corresponding relationship mapping. Associate the time segments with the locations to obtain the set of industrial message time location indexes. S5: Based on the industrial message time location index set, locate the corresponding position of the target time segment, read the header fields and payload content of the data frame in the corresponding file, compare the consistency of the data frame fields across files and adjust the splicing order, split and reassemble the spliced ​​data frame according to the protocol field position, and perform structured processing on the Ethernet header field, IP header field and transport layer field and match them with the position to obtain the industrial network message structured parsing dataset.

[0020] The message time sequence record includes a timestamp set, a sequence number set, a capture order relationship, and a frame identifier mapping. The ordered queue of industrial communication messages includes a correction sequence, anomaly markers, a rearrangement index, and a progressive relationship identifier. The data corresponding to the storage location of industrial messages includes address field classification, port category labels, protocol type grouping, and file offset intervals. The industrial message time location index set includes time segment division units, location interval mapping, index key-value pairs, and segment identifiers. The industrial network message structured parsing dataset includes frame header structure fields, payload data blocks, cross-file splicing units, protocol layer structure models, and location association records.

[0021] Please see Figure 2 The specific steps of S1 are as follows: S101: Obtain the output data frames of the industrial switch mirror port and the industrial control network interface, extract the Ethernet frame timestamps and sort them according to the order of capture, compare adjacent timestamps and identify the time sequence to obtain the timestamp sequence. First, raw traffic data frames from the industrial control network are captured online via the mirror port of an industrial switch. This operation directly connects to the underlying real-time communication link on the production line, obtaining raw Ethernet frames containing interactions between programmable logic controllers and distributed input / output modules. At the moment of capture, a high-precision clock generator at the hardware layer is invoked to add a nanosecond-level global timestamp to each incoming data frame, extracting the raw timestamp information of the Ethernet frame at the mirror moment. Because the mirror port in the industrial network may experience slight queuing delay jitter during backplane forwarding when handling instantaneous concurrent traffic, the order of the initially acquired timestamps may deviate slightly from their physical order on the bus. To meet the big data management needs of large-scale industrial sites, a metadata pre-annotation mechanism is simultaneously initiated at this stage to build underlying index support for subsequent massive message retrieval. Then, all captured timestamps are initially arranged according to the order in which they entered the mirror port. During execution, the values ​​of two adjacent timestamps are continuously extracted and logically compared to determine whether the latter timestamp is greater than the former. The specific determination logic is to calculate the time difference, i.e., subtract the former timestamp from the latter. If the difference is greater than 0, a positive temporal sequence is identified. If the difference is less than or equal to 0, a logical jump in the capture order or timestamp backtracking is identified. For example, in the control network of a digital factory, three data frames were captured consecutively, with extracted original timestamps of 10:30:00:105.002, 10:30:00:104.998, and 10:30:00:115.010 milliseconds, respectively. During processing, the difference between the first and second timestamps was calculated to be -0.004 milliseconds, indicating an abnormal capture order. The difference between the third and second timestamps was then calculated to be 10.012 milliseconds, indicating a positive evolution. By performing the above item-by-item comparison and identification on the entire capture queue, these timestamps with clear temporal relationships are finally combined sequentially to obtain a timestamp sequence.

[0022] S102: Based on the timestamp sequence, continuously compare adjacent timestamps and identify positions with abnormal order, swap the data frame positions corresponding to the abnormal positions and readjust the order to obtain the time sequence. First, deep error correction and sequence reordering operations are initiated based on the timestamp sequence to eliminate message timing reversals caused by network multipath transmission or buffer congestion. During execution, adjacent items in the sequence are retrieved for continuous numerical comparison to identify any sequence anomalies. The timestamp value of the current position and the immediately following timestamp value are read, and the incremental difference between them is calculated by subtracting the previous timestamp value from the subsequent timestamp value. If the calculated incremental difference is less than 0, the position is immediately defined as an out-of-sequence point. Once an out-of-sequence point is identified, the process triggers a memory swapping algorithm to exchange the physical pointers of the two original data frames corresponding to the out-of-sequence point in the storage buffer. This physical pointer tuning algorithm incorporates a distributed cache consistency protocol to ensure that the memory swapping operation maintains extremely high throughput when processing tens of millions of data packets, thereby achieving efficient big data management and real-time sequence correction. In the actual calculation, for the aforementioned out-of-sequence points of 105.002 milliseconds and 104.998 milliseconds, the incremental difference is calculated to be -0.004 milliseconds. A swap operation is performed, exchanging the positions of the 105.002 ms message, which was originally in the first position, with the 104.998 ms message, which was in the second position. After the swap, the sequence is adjusted to 104.998 ms, 105.002 ms, and 115.010 ms. To ensure the scientific accuracy of the adjustment, a timing correction reliability coefficient of 0.98 is introduced. This coefficient is set with reference to the jitter deviation range of industrial field clock synchronization protocols. It is calculated by logically associating the maximum offset of 5 microseconds from 1000 sets of normal communication cycles with a clock crystal frequency of 50 MHz. Experimental verification shows that after setting this coefficient for logical constraints, the timing correction accuracy reaches over 99.9%. This process involves iteratively comparing and swapping the entire sequence until the difference between all adjacent timestamps is non-negative, thus ensuring the consistency between the message sequence and the physical time flow, resulting in a time-ordered sequence.

[0023] S103: Based on the time sequence, assign a sequential number to the data frame corresponding to each timestamp, map the timestamp to the sequential number, and obtain the message time sequence record. First, a logical identifier is established for each calibrated data frame based on its chronological sequence. Starting from the beginning of the sequence, a linearly increasing counter is started, initially set to 1. This counter assigns a unique sequence number to each data frame sequentially. This process establishes a core mapping model that maps each microsecond-level timestamp to this increasing sequence number. In constructing this mapping model, a large-scale data management sharding logic for unstructured messages is introduced, distributing the sequence numbers to different logical buckets using a hash modulo method to improve the load balancing capability for large-scale concurrent queries on the backend. In actual execution, the first data frame at 104.998 milliseconds is assigned number 1, the second at 105.002 milliseconds is assigned number 2, and the third at 115.010 milliseconds is assigned number 3. This mapping relationship transforms the originally discrete time points into a numerical sequence with strong logical chronological significance. The process further retrieves message characteristic information, such as the message protocol type and its corresponding payload length, and aggregates and stores them along with their corresponding numbers. By performing the above assignment and association mapping operations on all data frames within the sequence, it is ensured that every communication moment is accurately recorded in the logical coordinate system. This process ultimately integrates all mapping entries to obtain the message time sequence record.

[0024] Please see Figure 3 The specific steps of S2 are as follows: S201: Based on the message time sequence, compare the relationship between adjacent timestamps, identify the position of the data frame with abnormal order, extract the data frame number corresponding to the abnormal position, and obtain the abnormal position identification sequence. First, a secondary check for microsecond-level logical jumps is initiated based on the message time sequence. The timestamps corresponding to two adjacent sequence numbers are extracted, and their time interval is calculated. To identify hidden sequence anomalies, this process compares the calculated time interval with a preset industrial communication logical step size threshold. This threshold is set to 15 milliseconds, referencing the minimum interval of periodic polling by PLCs in the control network. Statistical modeling of 48 consecutive hours of communication data from an industrial site yields an average message interval of 3 milliseconds and a maximum network jitter of 8 milliseconds. The sum of these two values ​​is multiplied by a redundancy safety factor of 1.3, resulting in a final setting of 14.3 milliseconds, rounded up to 15 milliseconds. This step, through precise removal and marking of abnormal locations, provides a clear data lineage for upper-level big data management components, preventing the accumulation of dirty data in storage. If the time interval between adjacent numbers exceeds 15 milliseconds, or if a stagnation occurs where the number increases but the timestamp remains unchanged, the location is determined to be a sequence anomaly data frame. In a practical example, if the timestamp of data frame number 100 is 500 milliseconds and the timestamp of data frame number 101 is 520 milliseconds, the difference is 20 milliseconds, exceeding the preset threshold of 15 milliseconds. The two data frame numbers, 100 and 101, corresponding to this abnormal location are then extracted. This process, through rapid scanning of records, identifies all nodes that do not conform to the smooth progression pattern of industrial communication. Finally, these marked abnormal numbers are collected to obtain an abnormal location identification sequence.

[0025] S202: Based on the abnormal position identifier sequence, swap the corresponding data frame positions and reorder the timestamp order, continuously compare the swapped data frames, analyze the time progression relationship, and obtain the time progression sequence; First, a precise reordering process is performed in memory for the abnormal location identifier sequences. The original data frames corresponding to the identifier sequences and their associated timestamps are extracted, and a position swapping algorithm is used to spatially permutate these data frames at abnormal locations. After the swap, the affected surrounding sequences are reordered, and the temporal progression relationship is analyzed. The core of the analysis logic lies in calculating the offset of the timestamp of each data frame relative to the start point of the sequence after the swap. To verify the rationality of the swap, a temporal evolution stability evaluation coefficient with a set value of 0.92 is introduced. This coefficient is obtained by calculating the ratio of the average deviation of historically successfully parsed packets to the real-time deviation. By pre-setting such a stability evaluation index in the big data management architecture, the reordering process of hot data can be automatically triggered, ensuring that the time slices accessed frequently always maintain the optimal logical order. In the actual calculation, if the time difference between two frames changes from a negative value to 5 milliseconds after the swap, the incremental value of 5 is multiplied by the coefficient 0.92, resulting in a score of 4.6. If this score is within the reasonable range of 3.5 to 5.5, the progression relationship is considered normal. This process ensures the monotonically increasing characteristic of the message stream on the time axis by performing logical consistency checks on the exchanged data frames, thus obtaining a time-progressive sequence.

[0026] S203: Based on the time-progressive sequence, reassign the sequential number of each data frame, map the timestamp to the sequential number, and obtain an ordered queue of industrial communication messages; First, the final reconstruction and encapsulation of the global serial number is performed based on the time-progressive sequence, discarding the previously generated intermediate serial numbers and starting a brand-new global serial number generation algorithm. Starting from the first position of the time-progressive sequence, the initial value of the sequence number is reassigned to 50001, and the sequence number of each subsequent data frame is incremented by 1 based on the previous value. A mapping relationship is established between the corrected precise timestamp and this new set of sequence numbers. To enhance data correlation, the source MAC address and destination MAC address of the data frame are introduced as auxiliary correlation items during the mapping process. This multi-dimensional correlation mapping is a key link supporting the traceability of "people, machines, materials, methods, and environment" in industrial big data management. By assigning a unique temporal fingerprint to each data packet, deterministic positioning is achieved at the scale of trillions of data points. In actual operation, for the first frame in the sequence, its timestamp is 104.998 milliseconds, and the established mapping relationship is: sequence number 50001 corresponds to timestamp 104.998 milliseconds, the associated source address is controller A, and the destination address is actuator B. This operational logic eliminates potential gaps in numbering during the correction process by refreshing the entire numbering system, ensuring that industrial communication messages have logically continuous and unique indices with absolute temporal significance. By performing the above reassignment and mapping association on thousands of rearranged data frames, an ordered queue of industrial communication messages is obtained.

[0027] Please see Figure 4The specific steps of S3 are as follows: S301: Based on the ordered queue of industrial communication messages, extract the source address, destination address, port number and protocol identifier of the data frame, compare the fields one by one and identify the field consistency distribution, classify the data frame according to the field consistency, and obtain the message field classification set; First, based on the ordered queue of industrial communication messages, the source address, destination address, port number, and protocol identifier of the data frame are extracted. The 48-bit source MAC address and destination MAC address are extracted by parsing the Ethernet frame header. Then, the network layer and transport layer fields are further parsed, extracting the source IP address and destination IP address from the IP header, and the 16-bit port number and protocol identifier from the transport layer. This process initiates a field consistency distribution identification mechanism, grouping data frames with the same communication characteristics (including source IP address, destination IP address, port number, and protocol identifier) ​​into the same business logic cluster. In the context of big data management, this step is essentially a multi-dimensional dimensionality reduction and clustering analysis of unordered data, reducing the processing overhead of subsequent online analysis through a pre-clustering strategy. The computational overhead is reduced; in the actual example, 500 packets were retrieved from the queue, and by comparison, it was found that 350 of them had the same source IP address of 192.168.1.100, destination IP address of 192.168.1.200, protocol identifier of TCP, and port number of 502. Packets with these consistent characteristics were assigned a unified classification label; in order to evaluate the classification aggregation degree, the classification consistency weight was set to 0.95. This weight was obtained by calculating the proportion of the number of packets with the same session to the total number of packets and adjusting it in combination with the protocol handshake frequency. In this way, the messy and intertwined mixed traffic was decomposed into several logical sets with clear communication directions, and finally the packet field classification set was obtained.

[0028] S302: Based on the message field classification set, write each type of data frame to the storage file in sequence, extract the start byte position and end byte position of each data frame during the writing process, associate the start position and end position of the same data frame, and obtain the data frame position interval set. First, based on the message field classification set, each type of data frame is sequentially written to the storage file. During the writing process, the start and end byte positions of each data frame in the storage file are extracted in real time. When writing the first frame of the first type of message, if the current file offset is 0, the start byte position is 0. If the frame length is 128 bytes, the end byte position is 127. When writing the second frame, the start byte position is automatically continuated to 128. This process establishes a unique association between the data frame's unique identifier and its position range. Retaining this physical layout information is a technical prerequisite for the hierarchical storage of hot and cold data in industrial big data management. Through precise position range mapping, historical redundant data can be smoothly migrated to the storage medium. By calculating the end byte position minus the start byte position plus 1, and performing length verification with the original frame length, it is ensured that no bytes are lost during the writing process. If the calculated result is 128, matching the original frame length, the record is valid. By performing this recording process on all classification sets, a data distribution map is constructed at the physical level, resulting in a set of data frame position ranges.

[0029] S303: Based on the set of data frame location intervals, map each data frame identifier to its corresponding start and end positions, associate the data frame with the location correspondence, and obtain the data corresponding to the industrial message storage location; First, each data frame identifier is mapped to its corresponding start and end positions based on a set of data frame position intervals. A unique sequential number is retrieved as the identifier and encapsulated with the physical location value to form an index entry. In practice, data frame number 50001 is mapped to start position 0 and end position 127. This process optimizes performance by calculating an index redundancy coefficient of 0.02, meaning the index table size is approximately 2% of the original message size. In storage engines for big data management, this compact index design effectively improves random access efficiency, stabilizing the addressing response time of large-scale messages within milliseconds or sub-milliseconds. Experimental results show that this mapping method eliminates the need for a full file scan in large-scale data retrieval scenarios; instead, the target data frame is quickly located via the index, significantly reducing retrieval overhead and improving read efficiency. Finally, this process integrates all index entries to obtain the data corresponding to the industrial message storage location.

[0030] Please see Figure 5 The specific steps of S4 are as follows:

[0031] S401: Based on the data corresponding to the storage location of industrial messages, extract the timestamps of the corresponding data frames and sort them according to the order of capture. Continuously compare the timestamps and identify the time segment boundaries. Divide the timestamps according to the segments to obtain a set of time segments. First, based on the storage location of industrial messages, the timestamps of the corresponding data frames are extracted and sorted according to the order of capture. Time segment boundaries are identified, and a fixed time window length of 1000 milliseconds is set as the dividing benchmark. This process continuously calculates the difference between the current timestamp and the starting timestamp of the segment. When the difference exceeds 1000 milliseconds, the segment is cut off at that point, and a new boundary is defined. This time-slicing technology is a core method for processing streaming data in big data management, allowing for incremental indexing and compressed storage of continuously generated industrial traffic in a micro-batch processing manner. In the actual example, the first timestamp is 0 milliseconds. When a timestamp of 1001 milliseconds is scanned, all messages within the previous 1000 milliseconds are included in the first segment. To handle long messages crossing boundaries, a boundary overlap buffer coefficient of 0.02 is introduced, set with reference to the average time taken for reassembling the largest fragmented industrial Ethernet packets. By scanning the timeline of the full data, a set of time segments is obtained.

[0032] S402: Based on the set of time segments, extract the corresponding positions of data frames within the time segments, group and divide the positions while maintaining a consistent distribution of positions within the same time segment, map and associate the grouping results, and obtain the time segment position mapping relationship; First, based on the time segment set, the corresponding positions of data frames within each time segment are extracted and grouped. All data frames falling within each time segment are retrieved, and their storage location information is grouped and encapsulated. For the 0 to 1000 millisecond interval, packets with numbers 50001 to 58000 are identified, and their corresponding storage coordinates are retrieved, such as those distributed in bytes 0 to 1024000 of file 1. Consistency verification logic is executed to calculate the length of consecutive position segments. By maintaining this alignment between time and physical address at the big data management level, the efficiency of range queries for specific time spans is improved. This mapping method transforms the abstract time range into a concrete disk addressing space. By grouping and mapping positions for all time segments, a consistent distribution of positions within the same time segment is maintained, resulting in a time segment position mapping relationship.

[0033] S403: Based on the time segment location mapping relationship, associate and connect time segments with corresponding locations, and perform a unified index description on the corresponding location of each time segment to obtain the industrial message time location index set; First, based on the time segment location mapping relationship, time segments are associated with their corresponding locations to construct an industrial message time location index set. The index descriptor contains an array of segment number, time span, file path, and starting offset address. For segment 2, the index record is: time 1001 milliseconds to 2000 milliseconds, and the physical location starts at byte 1024001 of file 1. The index density coefficient is set to 0.001, meaning one segment index is established for every 1000 messages. This multi-level index system improves the architectural layout of fast indexing and slow storage in big data management, enabling the location of physical blocks through secondary indexes in memory when facing sudden high-frequency traceability requests. Through this index construction, the data retrieval speed in large-scale industrial traceability scenarios is significantly improved, ultimately resulting in the industrial message time location index set.

[0034] Please see Figure 6 The specific steps of S5 are as follows: S501: Based on the industrial message time location index set, retrieve the corresponding location of the target time segment, extract the Ethernet header field, IP header field and transport layer field of the data frame in the corresponding file, extract the data frame order within the same time segment, and obtain the data frame field sequence; First, based on the industrial message time and location index set, the corresponding location of the target time segment is retrieved and the data frame fields are extracted. The physical location is retrieved according to the instruction, and the original binary data frame is extracted. The parsing module strips the Ethernet header, extracting the 20-byte IP header field and transport layer fields. The completeness of field extraction is monitored in real time, and the order of data frames within the same segment is extracted. In this deep parsing stage, big data management automatically initiates field-level semantic association, transforming the unstructured message payload into structured wide-table data that can be used for machine learning prediction. In the actual example, 100 consecutive data frames were extracted, their network layer protocols were identified, and the time-to-live and checksum of each frame were extracted. A protocol field matching rate benchmark of 1.0 was set to ensure that all standard fields were successfully identified, ultimately yielding the data frame field sequence.

[0035] S502: Based on the data frame field sequence, the source address, destination address, port number and protocol identifier of cross-file data frames are compared item by item. The data frames with consistent fields are ordered and the splicing position is adjusted. The spliced ​​data frames are then arranged continuously to obtain the cross-file message sequence. First, the source address, destination address, port number, and protocol identifier of cross-file data frames are compared item by item based on the data frame field sequence. When a segment with the same communication quadruple characteristics is identified in different files, the concatenation logic is initiated. The concatenation gap on the time axis is calculated and compared with a maximum allowable communication concatenation coefficient of 0.05. This coefficient is calculated based on an average communication cycle of 50 milliseconds, meaning the allowable threshold is 2.5 milliseconds. This logic effectively solves the session breakage problem caused by file slicing, which is common in big data management, and achieves complete restoration of the global session chain. If the time difference between the last frame of file A and the first frame of file B is 2 milliseconds, which is less than 2.5 milliseconds, it is determined to be a continuation of the same session. Concatenation is then performed, and the position offset is adjusted to obtain a cross-file message sequence.

[0036] S503: Based on cross-file message sequences, data frames are split according to the position of protocol fields, Ethernet header fields, IP header fields and transport layer fields are matched, and the field order is associated to obtain a structured parsing dataset of industrial network messages. First, the final industrial message structure parsing and dataset aggregation were completed. Following the layered definitions of standard industrial protocols, precise bit-field splitting was performed on cross-file message sequences, matching Ethernet header, IP header, and transport layer fields. For a Modbus read request, the MAC address, IP address, and application layer function code and register address were parsed out. This final dataset is the ultimate asset for industrial big data management; its highly structured characteristics can directly interface with production process optimization models, providing a data foundation for digital twin spatiotemporal backtracking. A field mapping weight model was established, setting the weight value to 0.98 to ensure the parsing depth reaches within the load. Finally, by associating all field sequences and logical relationships, the industrial network message structure parsing dataset was obtained.

[0037] Please see Figure 7 A storage and parsing system for full traffic data of an industrial network, comprising: The message acquisition and sorting module acquires the data frames output from the mirror port of the industrial switch and the industrial control network interface, extracts the Ethernet frame timestamps and sorts them according to the order of capture, compares adjacent timestamps and assigns them sequential numbers, and matches the timestamps with the sequential numbers to obtain the message time sequence record. The sequence correction module records the sequence of messages in time order, compares the adjacent timestamps and adjusts the position of data frames with abnormal order, reassigns the sequence number to the adjusted data frames and compares the order again to obtain an ordered queue of industrial communication messages. The segmented writing module extracts the source address, destination address, port number and protocol identifier based on the ordered queue of industrial communication messages. It performs consistency comparison and classification on the fields, writes the data frame to the storage file and extracts the start and end positions, and matches the data frame with the position to obtain the data corresponding to the storage position of the industrial message. The index generation module extracts timestamps and divides time segments based on the data corresponding to the storage location of industrial messages. It performs centralized division and mapping of locations within the same time segment, associates time segments with locations, and obtains a set of industrial message time and location indexes. The splicing and parsing module, based on the industrial message time and location index set, locates the corresponding position of the time segment, reads the header fields and payload content of the data frame, compares the fields of cross-file data frames and adjusts the splicing order, and splits them according to the position of the protocol fields to obtain the industrial network message structured parsing dataset.

[0038] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for storing and parsing full traffic data in an industrial network, characterized in that, Includes the following steps: S1: Obtain the data frames output from the mirror port of the industrial switch and the industrial control network interface, extract the Ethernet frame timestamps, compare adjacent timestamps and assign them sequential numbers to obtain the message time sequence record. S2: Based on the message time sequence record, compare adjacent timestamps and adjust the position of abnormal data frames, reassign sequence numbers and compare the sequence again to obtain an ordered queue of industrial communication messages. S3: Based on the ordered queue of industrial communication messages, extract the source address, destination address, port number and protocol identifier, analyze the consistency of the fields, write the data frame to the storage file, and obtain the data corresponding to the storage location of the industrial message. S4: Based on the data corresponding to the storage location of the industrial message, extract the timestamp and divide the time segment, centrally divide the locations within the same time segment and map the corresponding relationship, associate the time segment with the location, and obtain the industrial message time location index set; S5: Based on the industrial message time location index set, locate the corresponding position of the time segment, read the data frame header field and payload content, compare the cross-file data frame fields, split according to the protocol field position, and obtain the industrial network message structured parsing dataset. The specific steps for S3 are as follows: S301: Based on the ordered queue of industrial communication messages, extract the source address, destination address, port number and protocol identifier of the data frame, compare the fields one by one and identify the field consistency distribution, classify the data frames according to field consistency, and obtain the message field classification set; S302: Based on the message field classification set, write each type of data frame into the storage file in sequence, extract the start byte position and end byte position of each data frame during the writing process, associate the start position and end position of the same data frame, and obtain the data frame position interval set. S303: Based on the set of data frame position intervals, map each data frame identifier to its corresponding start and end positions, associate the data frame with the position correspondence, and obtain the data corresponding to the industrial message storage location; The specific steps of S4 are as follows: S401: Based on the data corresponding to the storage location of the industrial message, extract the timestamp of the corresponding data frame and sort it according to the order of capture. Continuously compare the timestamps and identify the time segment boundaries. Divide the timestamps according to the segments to obtain a set of time segments. S402: Based on the set of time segments, extract the corresponding positions of data frames within the time segments, group and divide the positions while maintaining a consistent distribution of positions within the same time segment, map and associate the grouping results, and obtain the time segment position mapping relationship; S403: Based on the time segment location mapping relationship, associate and connect the time segments with their corresponding locations, and perform a unified index description on the corresponding location of each time segment to obtain the industrial message time location index set; The specific steps of S5 are as follows: S501: Based on the industrial message time location index set, retrieve the corresponding location of the target time segment, extract the Ethernet header field, IP header field and transport layer field of the data frame in the corresponding file, extract the data frame order within the same time segment, and obtain the data frame field sequence; S502: Based on the data frame field sequence, compare the source address, destination address, port number and protocol identifier of the cross-file data frame item by item, connect the data frames with consistent fields and adjust the splicing position, and arrange the spliced ​​data frames continuously to obtain the cross-file message sequence. S503: Based on the cross-file message sequence, split the data frame according to the protocol field position, match the Ethernet header field, IP header field and transport layer field, associate the field order, and obtain the industrial network message structured parsing dataset.

2. The method for storing and parsing full-traffic data of an industrial network according to claim 1, characterized in that, The message time sequence record includes a timestamp set, a sequence number set, a capture order relationship, and a frame identifier mapping. The industrial communication message ordered queue includes a correction sequence, anomaly markers, a rearrangement index, and a progressive relationship identifier. The industrial message storage location corresponding data includes address field classification, port category labels, protocol type grouping, and file offset intervals. The industrial message time location index set includes time segment division units, location interval mappings, index key-value pairs, and segment identifiers. The industrial network message structured parsing dataset includes frame header structure fields, payload data blocks, cross-file splicing units, protocol layer structure models, and location association records.

3. The method for storing and parsing full traffic data of an industrial network according to claim 1, characterized in that: The Ethernet frame timestamp refers to the time stamp when each data frame is acquired; The location of the abnormal data frame refers to the position of the data frame that does not conform to the time progression order during the continuous comparison of timestamps.

4. The method for storing and parsing full traffic data of an industrial network according to claim 1, characterized in that: The field consistency refers to the identical characteristics of differentiated data frames in the source address, destination address, port number, and protocol identifier fields. The cross-file data frame field refers to the corresponding field between data frames distributed in the differentiated storage file.

5. The method for storing and parsing full traffic data of an industrial network according to claim 1, characterized in that, The specific steps of S1 are as follows: S101: Obtain the output data frames of the industrial switch mirror port and the industrial control network interface, extract the Ethernet frame timestamps and sort them according to the order of capture, compare adjacent timestamps and identify the time sequence to obtain the timestamp sequence. S102: Based on the timestamp sequence, continuously compare adjacent timestamps and identify positions with abnormal order, swap the data frame positions corresponding to the abnormal positions and readjust the order to obtain a time sequence. S103: Based on the time sequence, assign a sequential number to the data frame corresponding to each timestamp, map the timestamps to the sequential numbers, and obtain the message time sequence record.

6. The method for storing and parsing full-traffic data of an industrial network according to claim 1, characterized in that, The specific steps of S2 are as follows: S201: Based on the message time sequence record, compare the relationship between adjacent timestamps, identify the position of the data frame with abnormal sequence, extract the data frame number corresponding to the abnormal position, and obtain the abnormal position identification sequence. S202: Based on the abnormal location identifier sequence, swap the corresponding data frame positions and reorder the timestamp order, continuously compare the swapped data frames, analyze the time progression relationship, and obtain the time progression sequence; S203: Based on the time progression sequence, reassign the sequence number of each data frame, map the timestamp to the sequence number, and obtain an ordered queue of industrial communication messages.

7. A storage and parsing system for full traffic data of an industrial network, characterized in that, The system is used to implement the method for storing and parsing full-traffic data of an industrial network as described in any one of claims 1-6, and the system includes: The message acquisition and sorting module acquires the data frames output from the mirror port of the industrial switch and the industrial control network interface, extracts the Ethernet frame timestamps and sorts them according to the order of capture, compares adjacent timestamps and assigns them sequential numbers, and matches the timestamps with the sequential numbers to obtain the message time sequence record. The sequence correction module, based on the message time sequence record, compares adjacent timestamps and adjusts the positions of data frames with abnormal order, reassigns sequence numbers to the adjusted data frames and compares the order again to obtain an ordered queue of industrial communication messages. The segmented writing module extracts the source address, destination address, port number and protocol identifier based on the ordered queue of industrial communication messages, performs consistency comparison and classification on the fields, writes the data frame to the storage file and extracts the start position and end position, and maps the data frame to the position to obtain the data corresponding to the storage position of the industrial message. The index generation module extracts timestamps and divides time segments based on the data corresponding to the storage location of the industrial message. It performs centralized division and mapping of the corresponding relationships within the same time segment, associates the time segment with the location, and obtains the industrial message time location index set. The splicing and parsing module, based on the industrial message time location index set, locates the corresponding position of the time segment, reads the header fields and payload content of the data frame, compares the fields of cross-file data frames and adjusts the splicing order, and splits them according to the protocol field position to obtain the industrial network message structured parsing dataset.

Images