Dpu fine-grained packet reassembly method and system based on semantic perception

Abstract

The application discloses a DPU fine-grained packet reorganization method and system based on semantic perception and belongs to the technical field of communication protocol data processing. The method comprises the following steps: in a DPU environment, a shard set is generated based on original data packets, protocol length, sequence confirmation and load digest elements are extracted and fused for coding as semantic fingerprints, a shard connection graph is constructed, a reorganization path is generated through conflict suppression and boundary closure determination, and an aggregated load is de-duplicated and split to assemble a reorganized packet stream. Through hierarchical analysis of original data packets, protocol length elements, sequence confirmation elements and load digest elements are directly extracted, and normalization coding and semantic fingerprint mapping are completed in the DPU, so that real-time boundary identification and non-redundant splicing of application layer messages in concurrent TCP streams can be realized without relying on preset protocol templates.

Description

Technical Field

[0001] This invention relates to the field of communication protocol data processing technology, and in particular to a semantically aware DPU fine-grained message reassembly method and system. Background Technology

[0002] A Data Processing Unit (DPU) is a programmable hardware acceleration component deployed between a server and the network. It can directly parse network packets on a kernel-side data path, performing tasks such as protocol offloading, traffic shaping, and security policy execution. In industrial internet and data center scenarios, numerous TCP-based sessions carry application layer protocols such as Modbus TCP, MQTT, and OPC UA. The message boundaries of these protocols are not always aligned with TCP segment boundaries, often resulting in cross-segment messages or multiple messages concatenating within a single segment. A semantically aware DPU fine-grained packet reassembly method and system are proposed to address these needs, aiming to perform accurate segmentation and reassembly of application layer messages in concurrent TCP streams in real time on the DPU hardware.

[0003] Existing packet reassembly techniques on general-purpose processors typically employ packet capture frameworks based on libpcap or DPDK. These frameworks batch-copy raw data packets received from the network interface to a user-space buffer, then divide the packets into streams according to a five-tuple (source IP address, destination IP address, source port number, destination port number, and protocol type). Within each stream, the packets are sorted by TCP sequence number and output as a continuous stream of payload bytes. Some hardware offloading solutions embed the finite state machine for TCP retransmission and sorting into the network interface card (NIC) chip or FPGA. However, the detection of application-layer message boundaries still relies on fixed length fields or delimiters, requiring pre-configured protocol templates. Other methods attempt to locate message boundaries by scanning specific pattern strings within the payload using deep packet inspection (DPI). However, such scanning operations consume significant computational resources and are ineffective for encrypted traffic.

[0004] The aforementioned existing technical solutions exhibit several shortcomings when applied to DPU execution environments. Copying the entire payload to host memory for reanalysis undermines the DPU's design intent as a data path accelerator, making memory bandwidth and PCIe bus bandwidth bottlenecks and failing to meet the line-speed processing requirements of high-throughput scenarios. Boundary detection methods based on fixed protocol templates require maintenance personnel to manually write parsing rules for each industrial protocol. When device firmware upgrades lead to minor adjustments in message formats or new vendor devices introduce proprietary extensions, the maintenance cost of the rule base increases dramatically and is prone to missed detections. Deep packet inspection-based scanning methods, when handling out-of-order fragments, cannot obtain the correct byte stream context. The scanning window is typically limited to the limited field of view of a single fragment, easily truncating pattern strings across fragment boundaries and thus missing message boundaries. This problem is particularly prominent in industrial data acquisition scenarios with dense concurrent sessions. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a semantically aware DPU fine-grained message reassembly method and system. By directly extracting protocol length elements, sequence confirmation elements, and payload digest elements from the original data packets through layered parsing and completing normalized encoding and semantic fingerprint mapping within the DPU, it can achieve real-time boundary identification and redundancy-free splicing of application layer messages in concurrent TCP streams without relying on preset protocol templates.

[0006] The above objectives can be achieved through the following approach:

[0007] A semantically aware DPU fine-grained message reassembly method includes: acquiring raw data packets received by the DPU and performing hierarchical parsing to generate a fragment set; extracting visible fields and payload visible fragments from each fragment in the fragment set, normalizing them, and generating a reassembled data element set; extracting and combining features from the reassembled data element set to generate a semantic clue set; performing fingerprinting mapping on each fragment based on the semantic clue set and outputting identifiers that distinguish concurrent messages to generate a semantic fingerprint set; constructing a fragment connection graph based on the semantic fingerprint set; performing constraint solving and boundary closure determination on the fragment connection graph to generate a reassembly path set; and performing fine-grained splicing and deduplication on the fragments according to the reassembly path set to generate a reassembled message stream.

[0008] Based on the same inventive concept, this invention also provides a semantically aware DPU fine-grained message reassembly system, characterized in that the system comprises: a fragment parsing module for acquiring the original data packets received by the DPU and performing layered parsing to generate a fragment set; an element extraction module for extracting the visible fields and payload visible fragments of each fragment in the fragment set, performing normalization, and generating a reassembled data element set; a clue extraction module for extracting and combining features from the reassembled data element set to generate a semantic clue set; a fingerprint extraction module for performing fingerprint mapping on each fragment based on the semantic clue set and outputting identifiers that can distinguish concurrent messages, generating a semantic fingerprint set; a graph construction module for constructing a fragment connection graph based on the semantic fingerprint set; a path generation module for performing constraint solving and boundary closure determination on the fragment connection graph to generate a reassembled path set; and a message reassembly module for performing fine-grained splicing and deduplication processing on the fragments according to the reassembled path set to generate a reassembled message stream.

[0009] Compared with the prior art, the present invention has the following advantages:

[0010] This invention solves the problem of data duplication in out-of-order and retransmission scenarios by identifying and suppressing conflict edges caused by retransmission fragmentation, and by combining overlay pruning operations to remove redundant data in overlapping sequence regions, thus ensuring the integrity and consistency of the reassembled message stream.

[0011] This invention utilizes semantic compatibility filtering and sequence continuity weight calculation to construct a weighted directed graph, and introduces a preset waiting condition in the boundary closure determination, which can accurately identify and extract independent application layer messages from concurrent and interleaved network data streams, thereby improving the accuracy of message reassembly.

[0012] This invention utilizes semantic compatibility filtering and sequence continuity weight calculation to construct a weighted directed graph, and introduces a preset waiting condition in the boundary closure determination, which can accurately identify and extract independent application layer messages from concurrent and interleaved network data streams, thereby improving the accuracy of message reassembly.

[0013] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the semantically aware DPU fine-grained message reassembly method according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the adaptive update curve of connection weights in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of the semantically aware DPU fine-grained message reassembly system according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Reference Figure 1 One embodiment of the present invention proposes a semantically aware DPU fine-grained message reassembly method. By directly extracting protocol length elements, sequence confirmation elements and payload digest elements from the original data packet through hierarchical parsing and completing normalization encoding and semantic fingerprint mapping within the DPU, it can achieve real-time boundary identification and redundancy-free splicing of application layer messages in concurrent TCP streams without relying on preset protocol templates.

[0020] The method described in this embodiment specifically includes:

[0021] S1. Obtain the raw data packets received by the DPU and perform layered parsing to generate a fragment set;

[0022] In one embodiment of the present invention, step S1 includes the following steps:

[0023] Obtain the link layer identifier and network layer identifier of the raw data packets received by the DPU, and generate data packet location information;

[0024] Identify the transport layer header corresponding to the data packet location information, extract sequence-related fields, and generate sequence elements;

[0025] The sequence elements are associated with the payload fragments of the original data packet, and the fragmentation direction is marked to generate a fragment set.

[0026] Specifically, DPU refers to a Data Processing Unit, a programmable dedicated hardware component capable of high-speed processing of network packets along the data path, undertaking the data parsing and forwarding tasks originally performed by the central processing unit. A raw data packet is a complete transmission unit in network communication, encapsulating control information and user data. Its structure, from the outside in, consists of the link layer header, network layer header, transport layer header, and payload fragment. The link layer identifier, extracted from the link layer header of the raw data packet, distinguishes data from different protocols or terminals on the same physical link; a typical entity is the Media Access Control address, or MAC address. The network layer identifier, extracted from the Internet Protocol header (IP header) of the raw data packet, distinguishes data from different terminals on the Internet; typical entities are the source IP address and destination IP address.

[0027] The processing begins with capturing a complete raw data packet from the DPU's receive buffer. Parsing starts at offset 0 from the packet's beginning. The first 14 bytes of the Ethernet frame header are read, extracting the 6-byte source MAC address and 6-byte destination MAC address, followed by the 2-byte frame type field. These 14 bytes of binary information are combined with the frame type field to form the link layer identifier. Next, the IP header is read from offset 14 bytes. The format is determined based on the IP header version field. If it's IPv4, the 20-byte standard header is read, extracting the 4-byte source IP address and 4-byte destination IP address, and recording the 1-byte protocol field. If it's IPv6, the 40-byte base header is read, extracting the 16-byte source IP address and 16-byte destination IP address, and recording the 1-byte next header field. The extracted IP address and protocol field are organized into a network layer identifier. The source MAC address and destination MAC address from the link layer identifier are concatenated with the source IP address, destination IP address, and protocol field from the network layer identifier to obtain the packet location information. This location information is represented as follows: tuples, where Source MAC address The destination MAC address The source IP address The destination IP address This is the protocol number.

[0028] After obtaining the data packet location information, based on the location information... Field filtering of transport layer protocols. If... A value of 6 indicates that the transport layer uses the Transmission Control Protocol (TCP). At this point, the total length of the network layer header is calculated based on the header length field of the IP header, thus locating the starting offset of the transport layer header. At the starting offset of the transport layer header, the following bytes are read sequentially: 2 bytes for the source port number, 2 bytes for the destination port number, 4 bytes for the sequence number, 4 bytes for the acknowledgment number, 2 bytes for the flags field, and 2 bytes for the window size field. The sequence-related fields refer to the control fields in the TCP protocol used to ensure ordered data delivery and reliable transmission; here, they specifically refer to the sequence number and acknowledgment number. The extracted sequence number and acknowledgment number are paired to generate sequence elements, represented as follows: ,in The sequence number is a 32-bit unsigned integer. The confirmation number is a 32-bit unsigned integer.

[0029] The payload fragment consists of all bytes starting from the position offset by 4 times the data offset field after the transport layer header and extending to the end of the data packet. This payload fragment can be any part of the user data and has a variable length. The fragment direction is a binary label indicating the flow direction of the fragment within the session. The rule for marking the fragment direction is: comparing the sending sequence number in the sequence element with the historical baseline sequence number, or based on the directional characteristics of the three-way handshake message during TCP connection establishment. In the flow table structure maintained locally by the DPU, the packet location information is used... The four-tuple is used as a flow identifier to retrieve the established session context. If the flow identifier's initiation direction matches the source of the current data packet, the fragmentation direction is marked as a sending direction with a value of 1; otherwise, it is marked as a receiving direction with a value of 0. Sequence elements, payload fragments, and fragmentation directions are associated and encapsulated to generate a fragment. Each fragment is structured as a three-tuple. ,in For sequence element tuples, The payload fragment is in the form of a byte array. The direction label is represented by a value of 1 or 0. After processing all the raw data packets in a single receive batch, the complete... The set formed is called the partitioned set, denoted as . , This represents the total number of fragments in this batch.

[0030] For example, the DPU captures a raw data packet from a physical port, and its binary content is parsed starting from the Ethernet frame header. The source MAC address is extracted as "00-0C-29-A1-B2-C3", the destination MAC address as "00-50-56-C0-00-08", and the frame type field is 0x0800, which is the link layer identifier. The IPv4 header is then parsed after the frame header, yielding the source IP address "192.168.10.5", the destination IP address "10.10.10.2", and the protocol field value as 6, which is the network layer identifier. These are then combined to generate the packet location information. .because Locate the TCP header and extract the send sequence number as 1001 and the acknowledgment number as 2002, then generate sequence elements. The calculated payload fragment length is 1460 bytes, and this portion of data is read completely. A query of the DPU flow table reveals that the packet originated from "192.168.10.5", matching the source IP, and the fragmentation direction is marked as 1. The fragment is then generated. The remaining 199 raw data packets received in the same batch are processed in the same way to form a fragment set containing 200 fragments.

[0031] S2. Extract the visible fields and visible payload fragments of each fragment in the fragment set, normalize them, and generate a recombined data element set;

[0032] In one embodiment of the present invention, step S2 includes the following steps:

[0033] Obtain the protocol type field and length-related fields for each fragment in the fragment set, and generate the protocol length element;

[0034] Obtain the sequence-related fields and confirmation-related fields of each segment in the segment set, establish the change relationship, and generate sequence confirmation elements;

[0035] Extract the visible payload fragments from each fragment in the fragment set and extract the byte pattern summary to generate payload summary elements;

[0036] The protocol length element, the sequence confirmation element, and the payload digest element are merged and normalized to generate a set of recombined data elements.

[0037] Specifically, each fragment in the fragment set generated in step S1 inherits all the header metadata and payload fragments of the original data packet. This metadata is temporarily stored in the extended descriptor of the fragment structure in the DPU's parsing pipeline. The protocol type field refers to the 1-byte protocol number in the IP header of the original data packet, used to identify the protocol type used by the transport layer. For TCP-based communication, its value is 6. The length-related fields consist of two parts: one is the 2-byte total length field in the IP header, indicating the number of bytes in the entire IP datagram; the other is the half-byte data offset field in the TCP header, indicating the length of the TCP header in 4-byte units. Utilizing the inherent relationships between the fields, the length of the transport layer payload can be extracted from the binary stream of the original data packet. This length is obtained by subtracting the IP header length from the total length field and then subtracting the TCP header length. The IP header length is determined by multiplying the 4-bit header length field of the IP header by 4 for IPv4, and is fixed at 40 bytes for IPv6. Combining the protocol type field with the calculated payload length generates the protocol length element, which is a two-dimensional vector, denoted as . ,in This is an integer value for the protocol type field, ranging from 0 to 255. The payload length is expressed in bytes, and its value range is typically no more than 1500, depending on the maximum transmission unit. The calculation is based on the formula. ,in Taken from the total length field of the IP header. This refers to the length of the IP header. This represents the TCP header length.

[0038] When obtaining the sequence-related and acknowledgment-related fields for each fragment, the sequence element tuple generated in step S1 is directly read from the fragment structure. The sequence-related field is the sending sequence number, and the acknowledgment-related field is the acknowledgment sequence number; both are 32-bit unsigned integers used in the TCP protocol to identify the position of the byte stream and the point of successful reception. Establishing the change relationship involves calculating the difference between the sending sequence number and the acknowledgment sequence number. This difference reflects the offset of the new data sent by the sender relative to the receiver's expected starting point, indicating the data stream's progress and potential interaction patterns. A sequence acknowledgment element is generated; this element is a three-dimensional vector, denoted as... ,in For the sequence number, To confirm the serial number, , is an integer value, and is negative when the sent sequence number is less than the acknowledgment sequence number.

[0039] When extracting the visible load segment for each segment, the visible load segment is defined as the prefix portion of the load segment that stably reflects the characteristics of the data content. Based on the load distribution sensitivity analysis conducted on 200 sets of industrial sensor measured data, the first 64 bytes of each load segment are selected as the visible segment. If the original load length is less than 64 bytes, it is padded with zero-value bytes to 64 bytes to ensure length consistency. Byte pattern summary extraction is accomplished by constructing a byte frequency histogram of this 64-byte visible segment. The construction process involves initializing a zero vector of length 256, traversing each byte of the visible segment, using its integer value as an index, incrementing the count value at the corresponding position by 1, and then dividing the entire vector by the effective number of bytes in the visible segment, i.e., 64, to obtain the normalized frequency distribution. The generated load summary element is a 256-dimensional vector, denoted as […]. ,in Represents byte value The frequency of occurrence in the visible segment satisfies The reason for setting the dimensions and normalization method in this way is that, through actual measurements of 200 sets of industrial sensors, it was found that a 64-byte prefix is ​​sufficient to capture the differences in byte distribution of messages from different application layers, and a 256-dimensional histogram can be efficiently implemented on the parallel computing unit of the DPU.

[0040] Before fusing the protocol length, sequence acknowledgment, and payload digest elements into a single high-dimensional vector, the significant differences in the numerical ranges of each dimension need to be addressed. The protocol type field takes values ​​between 0 and 255, the payload length can reach 65535, and the sequence number and acknowledgment number can reach [value missing]. The difference range is similar, while each dimension of the payload summary element is between 0 and 1. Direct concatenation would lead to numerical metric imbalance, so range normalization encoding is performed independently for the first two types of elements. That is, for each dimension of the protocol length element and each dimension of the sequence acknowledgment element, a preset global boundary value is used to map them to the 0-1 interval. Specifically, the protocol type field is normalized by dividing by 255; the payload length is normalized based on the typical Ethernet maximum transmission unit of 1500 bytes, divided by 1500 and truncated to 1; the transmission sequence number and acknowledgment number are modulo-divided. Divide the value after Normalize; difference First calculate its absolute value and sign bit, then divide the absolute value by... Normalization is performed, and the sign bit is stored separately in an additional dimension. After the above normalization, the normalized protocol length element vector is obtained. and normalized sequence confirmation element vector Load summary elements It is already between 0 and 1. The fusion operation is to... , and Sequentially concatenated to form a total dimension of The final vector of dimensions. Here, the dimensions are decomposed into: a 2-dimensional normalized protocol length element, and a normalized sequence confirmation element containing the normalized... Including the sign bit, there are 4 dimensions, and the payload summary elements have 256 dimensions. This 262-dimensional vector is the reconstructed data element. All the reconstructed data elements generated from the N fragments constitute the reconstructed data element set, denoted as . ,in It is the recombined data element vector corresponding to the i-th segment.

[0041] For example, the first fragment is taken from the fragment set generated in step S1. Its transmission sequence number is 1001, acknowledgment number is 2002, and payload length is 1460 bytes. Subtracting the header length from the total length field of the original data packet yields a payload length of 1460 bytes, which is also 1460 bytes. The protocol type field has a value of 6. (Protocol length element) Sequence confirmation elements The difference The first 64 bytes of the fragmented payload are extracted as the visible segment, and the payload summary elements are obtained after calculating the byte frequency histogram. For example, if byte 0x00 appears 3 times, then .right Normalization Dividing by 255 gives 0.02353. Dividing by 1500 gives 0.97333. (This is correct.) Normalization Divide by Get about , Similarly, absolute value Normalization by Get about The sign bit is set to -1. The fusion yields a reconstructed data element vector. Its dimension is 262. The remaining 199 fragments in this batch are processed sequentially to generate a reconstructed data element set. .

[0042] S3. Extract and combine features from the recombined data element set to generate a semantic clue set;

[0043] In one embodiment of the present invention, step S3 includes the following steps:

[0044] Obtain the protocol length element from the recombined data element set and identify the message boundary position to generate boundary candidate elements;

[0045] Obtain sequence confirmation elements from the recombined data element set and identify out-of-order and retransmission signs to generate transmission status elements;

[0046] By fusing the boundary candidate elements and the transmission state elements, a connectivity score between fragments is calculated to generate a semantic cue set.

[0047] Specifically, in the reconstructed data element set generated in step S2, each fragment corresponds to a 262-dimensional normalized vector, which is logically composed of three parts: a normalized protocol length element, a normalized sequence confirmation element, and a payload summary element. The processing flow begins by extracting the required portion from the vector by dimensional offset. Dimensions 0 to 1 correspond to the normalized protocol length element, denoted as... ,in This is the normalized protocol type field. The normalized load length; dimensions 2 to 5 correspond to the normalized sequence confirmation elements, denoted as ,in To normalize the transmission sequence number, To confirm the normalized sequence number, The normalized absolute value of the difference. This is the sign bit for the difference, with a value of 0 indicating non-negative and 1 indicating negative; dimensions 6 to 261 correspond to the payload summary elements. .

[0048] Message boundary location refers to the starting or ending point of an application layer message within a fragment's payload segment. A boundary candidate element is a marker indicating whether a fragment might carry a message boundary and the possible types of that boundary. The identification process utilizes the payload length from the normalized protocol length element. Combined with the protocol type field Expand. First, let's... Inverse normalization yields the original load length The calculation formula is: Round to the nearest byte. The judgment rule is: when... If the value is less than 1460 and the push flag (PSH flag) in the TCP header of the fragment is set to 1, the fragment is marked as containing a message boundary, with the boundary type being "push boundary". The PSH flag is a bit in the TCP header flag field, indicating that the sender requests that the data be delivered to the receiving application layer as soon as possible. When the value is less than 1460 and the PSH flag is not set, the boundary type is marked as "length boundary". When the value equals 1460, it typically indicates that the fragment fills the maximum segment size of the transport layer, the message boundary is not located there, and the boundary type is marked as "no boundary". Additionally, based on field measurements using 200 industrial sensors, it was observed that certain protocols carry a length prefix in the message header. If the original protocol number is mapped to a known self-describing protocol such as MQTT or Modbus TCP, then a further check is performed to see if the first two bytes of the payload fragment constitute a valid message length indicator. If satisfied ,in If a fixed length is appended to the header specified in the protocol, it is labeled as a "self-describing boundary". The generated boundary candidate features are triples for a single fragment, denoted as... , It is a boolean value; a true value indicates that the fragment contains the message boundary, and a false value indicates that it does not. It is a boundary type label, with values ​​that are enumerated: 0 represents no boundary, 1 represents push boundary, 2 represents length boundary, and 3 represents self-describing boundary. This is the boundary confidence score, which is assigned a real value between 0 and 1 based on the strictness of the match with the boundary recognition rule. For example, the confidence score for push boundary and self-describing boundary is set to 0.95, the confidence score for length boundary is set to 0.80, and the confidence score for no boundary is set to 0.10.

[0049] Out-of-order and retransmission indicators are based on the changing patterns of the sequence number space in the Transmission Control Protocol (TCP) to detect whether data packets have arrived out of order or have been duplicated. The process of generating transmission state elements starts with normalized sequence acknowledgments, first denormalizing to recover the original values. (Sequence number is also mentioned.) Confirm serial number absolute value of difference , sign bit The criteria for determining out-of-order delivery are defined as follows: For a set of fragments belonging to the same stream identifier, after sorting by capture timestamp, if the sequence number of the next fragment appears... The sequence number of the transmission is less than the previous fragment. ,Right now Then mark the fragment. There are signs of out-of-order delivery. The criteria for determining retransmission signs are defined as follows: if fragmentation... The sequence number of the transmission If the sequence number of a fragment is equal to that of any previously recorded fragment, and the payload length is also equal, then the fragment is marked. There are signs of retransmission. The transmission state element is a three-dimensional vector, denoted as... ,in This is a random order flag; a value of 0 indicates that the order is not random, and a value of 1 indicates that the order is random. This is the retransmission flag; a value of 0 indicates no retransmission, and a value of 1 indicates a retransmission. Sequence offset is defined as the ratio of the offset of the fragment's transmission sequence number relative to the stream's reference start sequence number to the maximum offset observed during transmission. The formula is as follows: ,in It is the initial sequence number of the TCP stream, taken from the initial sequence number field of the three-way handshake phase, with 1 added to the denominator to prevent division by zero.

[0050] Inter-fragment connectability score is a numerical measure of the probability that two fragments can form a continuous message segment. This score integrates multiple factors from boundary candidate elements and transport state elements. (Fragmentation settings are then defined.) With fragmentation For the pair to be evaluated, and the partitioning direction and Same. Connectivity score The calculation formula is:

[0051]

[0052] In the formula, The weighting coefficients are 0.3, 0.4, 0.15, and 0.15, respectively. The values ​​are determined by maximizing the reconstruction integrity rate through grid search on 200 sets of industrial sensor measured data. It is a boundary compatibility function; if fragmented... Boundary type If the push boundary or self-describing boundary indicates that the fragment is the end of the message, then the fragment... It can only be the starting fragment of the next message, at this time The value is 1; if fragmented The type is unbounded, indicating that the message is incomplete and needs to be continued; it is fragmented. The message origin should not be included; if fragmentation is used in this case... Boundary type If it is unbounded, then The value is 1 if it is not 0.3 otherwise; if it is fragmented If identified as out-of-order or retransmission, then Take 0 directly. It is a sequence continuity function, used to test for fragmentation. Does the sequence number of the sent data exactly correspond to the fragment? The sum of the transmission sequence number and the payload length, i.e., the expected sequence number. If the sequence space is exactly continuous, that is ,but The value is 1.0; if there is a sequence number gap, then... The value decays exponentially. If there is overlap, then... The value is directly set to 0. and These are obtained from transmission state elements to suppress the tendency to connect to retransmissions and out-of-order fragmentation.

[0053] The semantic clue set is defined as a set of structures composed of the boundary candidate elements, transport state elements, and the connectability scores between each fragment and all other fragments in the reconstructed data element set. For a set containing N fragments, the semantic clues of the i-th fragment... Depend on The composition, the overall set of semantic clues, is denoted as .

[0054] For example, the first and second fragments within the reorganized data feature set in step S2 are taken. The normalized payload length of fragment 1... Inverse normalization When the PSH flag is 0, the boundary type is determined to be no boundary. Boundary candidate elements The fragment 1, after inverse normalization, yields... No out-of-order delivery or retransmissions, stream reference number is 1001, maximum observed offset is 1460. Transmission status elements Normalized load length of slice 2 Inverse normalization With the PSH flag set to 1, the boundary type is determined to be a push boundary. Boundary candidate elements Fragment 2 Send Sequence Number The order was not disordered and the transmission was not repeated. Transmission status elements Calculate the connectability score from shard 1 to shard 2: Shard 1 is unbounded, shard 2 contains message boundaries and is not unbounded, according to the rules... Take 0.3; Expected sequence number Exactly equal to , Set the value to 1.0; Substituting into the formula, we get The semantic clues of fragment 1 include , And all connectivity scores with other fragments, with a connectivity score of 0.79 with fragment 2. These are calculated pairwise to form a semantic cue set containing 200 semantic cues.

[0055] S4. Based on the semantic clue set, perform fingerprinting mapping on each segment and output an identifier that can distinguish concurrent messages to generate a semantic fingerprint set;

[0056] In one embodiment of the present invention, step S4 includes the following steps:

[0057] Based on the set of semantic clues, the boundary candidate elements of the segments are located and encoded to generate boundary encoded elements.

[0058] The protocol-related features of the fragments are categorized and encoded to generate protocol coding elements;

[0059] The boundary coding elements and the protocol coding elements are combined and mapped by introducing a preset direction consistency constraint to generate a semantic fingerprint set.

[0060] Specifically, the semantic clue set generated in step S3 provides structured semantic clues for each slice, denoted as ,in It is a boundary candidate element triple. , It is a transmission state element triplet , This is the set of connectability scores between this fragment and all other fragments. Location coding transforms the qualitative labels and quantitative confidence scores in boundary candidate features into compact, spatially discriminative numerical codewords; the boundary-coded feature is this codeword. Boundary type labels in boundary candidate features. The values ​​are 0, 1, 2, and 3, representing no boundary, push boundary, length boundary, and self-describing boundary, respectively. Since the semantic positions corresponding to different boundary types are drastically different, one-hot encoding is used to... Expanded into a 4-dimensional vector, denoted as , among which when hour All other dimensions are 0. Boundary confidence. These are real numbers between 0 and 1, and their scalar values ​​are retained as confidence weighting factors. Boundary condition exists. It is a Boolean value, with true corresponding to the value 1 and false corresponding to the value 0. The location coding combines these three values ​​to generate boundary coding elements. Let be a 5-dimensional real vector, and the calculation formula is:

[0061] ,

[0062] in, There are markers for the boundary. For boundary confidence, The k-th dimension component of the boundary type label is one-hot encoded. The effect of this formula is: when a shard does not contain message boundaries. , The first dimension is 0, and the values ​​of all other dimensions are also set to zero, resulting in a zero vector; when the shard contains message boundaries... The first dimension retains the confidence level, while each type of dimension only carries confidence information at the matching label, and the rest are 0.

[0063] Protocol-related features refer to the set of features directly associated with the behavior of the transport and network layers, capable of distinguishing different message templates. Specifically, this refers to the unnormalized, inversely reconstructed original network and transport layer header information extracted from the reconstructed data element set, including the protocol type field, TCP header flag combinations, and specific byte patterns of the TCP option field. This information is used as descriptors for the fragment structure during step S2, accompanying the fragment flow, and is retrieved again in step S4. The process of generating protocol-encoded elements involves obtaining the integer value of the protocol type field. This is directly used as the first dimension feature. The TCP flag field is obtained; this field is 8 bits long and contains 6 valid control bits (FIN, SYN, RST, PSH, ACK, URG) and 2 reserved bits. The integer read from the flag field is normalized by dividing by 255. As the second feature, the TCP options field is located at the end of the TCP header and before the payload begins. Its length is variable, and a simplified cumulative hash value is calculated after extracting its complete byte sequence. The calculation formula is: ,in It is the kth byte of the TCP options field. This is the byte length of the option field. 31 is the prime multiplier, and taking the modulo 1024 limits the hash space to the interval between 0 and 1023. Dividing the hash value by 1023 normalizes it to the interval between 0 and 1, which is then used as the third dimension feature. The protocol encoding element is thus a three-dimensional vector, denoted as... .

[0064] The pre-defined directional consistency constraint is a rule abstracted from the network session model. It stipulates that in TCP bidirectional communication, consecutive data fragments in the same direction form a byte stream, while fragments in different directions are naturally separated. This constraint's role in the mapping phase is to ensure that fragments belonging to the same direction are assigned similar semantic fingerprints, while the fingerprints of fragments in different directions are spaced further apart. Formally, the directional consistency constraint is a binary variable multiplied by a discrete sign factor. The fragment direction is read from the fragment structure saved in step S1. A value of 1 indicates the sending direction, and a value of 0 indicates the receiving direction. A direction identifier is introduced when generating semantic fingerprints. This value is 1 in the sending direction and -1 in the receiving direction.

[0065] The boundary coding elements, protocol coding elements, and direction identifiers are combined and mapped. First, dimensional concatenation is performed to create the 5-dimensional boundary coding elements. With 3D protocol coding elements Connecting the beginning and end, they form an 8-dimensional intermediate vector. The specific dimensional arrangement of the intermediate vector is as follows: , , , , , , , .

[0066] This 8-dimensional intermediate vector is input into a confusion mapping function to generate a semantic fingerprint. The confusion mapping function employs a hashing method based on random projection, the core of which is a pre-defined 8x32 random projection matrix. Matrix elements From the standard normal distribution Mid-sample and fixed during the initialization phase, For row index, Column index. Projection calculation is matrix multiplication. This yields a 32-dimensional continuous vector. A sign function is then applied to each dimension of this continuous vector. Binarization is performed, mapping components greater than or equal to 0 to bits 1 and components less than 0 to bits 0, resulting in a 32-bit binary basic fingerprint. The direction identifier is then added to this. Encoded as the 33rd bit, when When the bit is 1, when At that time, the bit is 0. Ultimately, the semantic fingerprint is a 33-bit binary vector, denoted as... Perform the above mapping on N slices to generate a semantic fingerprint set. This semantic fingerprint encodes the message structure attributes and protocol behavior of fragments through boundary coding elements and protocol coding elements, and ensures the distinguishability of message fingerprint direction through directional consistency constraints, so that fragments belonging to different concurrent messages in the same session can naturally cluster in the Hamming space.

[0067] For example, this follows the first and second segments within the semantic clue set in step S3. The boundary candidate features of segment 1... The boundary has markers. Type tags Confidence level .because Boundary coding elements The protocol type field of fragment 1 In the TCP flags field, the ACK flag is set to 1 and the rest are 0, with an integer byte value of 16. The TCP options field is 12 bytes long, with each byte containing the following values: 2, 4, 5, 180, 1, 3, 3, 8, 0, 0, 0, 0. The cumulative hash is calculated as follows: This yields an integer value. Let's say the result is 718. Protocol coding elements .direction Direction identifier The intermediate vector is obtained by concatenation. The length is 8 dimensions, and it is consistent with the preset matrix. After multiplication and binarization, a direction bit 1 is appended to obtain a 33-bit semantic fingerprint. Assume its value is the binary sequence "10110…1". Boundary candidate features of partition 2. Boundary coding elements Its protocol type and flags are the same as fragment 1. , The option hash differs depending on the timestamp option update, assuming , Protocol coding elements Same direction Semantic fingerprints are obtained through projection mapping. The remaining 198 fragments are processed in this manner to generate a semantic fingerprint set containing 200 33-bit binary vectors.

[0068] S5. Construct a fragmented connection graph based on the semantic fingerprint set;

[0069] In one embodiment of the present invention, step S5 includes the following steps:

[0070] Obtain the semantic fingerprint set and filter semantically compatible fragment pairs to generate a candidate connection pair set;

[0071] Obtain the candidate connection pair set and calculate the sequence continuity weights by combining the sequence-related fields to generate a connection weight set;

[0072] A graph structure is constructed based on the set of candidate connection pairs and the set of connection weights, and boundary candidate features are marked to generate a piecewise connection graph.

[0073] Specifically, the semantic fingerprint set generated in step S4 Each semantic fingerprint It is a 33-bit binary vector, composed of 32 hash bits based on boundary and protocol feature projections and a 1-bit direction identifier bit. Semantic compatibility means that the semantic fingerprints of two fragments are sufficiently close in the vector space, indicating that they are highly consistent in message structure attributes, protocol behavior, and flow direction, and have the potential to constitute the same application layer message or adjacent messages. When filtering semantically compatible fragment pairs, all fragments in the fragment set that satisfy the condition are considered. orderly pair The algorithm iterates through the two semantic fingerprints and calculates the Hamming distance between them. This distance is defined as the total number of positions in two equal-length binary vectors where corresponding bit values ​​differ. The calculation formula is as follows:

[0074] ,

[0075] in the formula Semantic fingerprint The k-th bit takes the value 0 or 1; This is an indicator function; it takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Hamming distance. The value of is an integer ranging from 0 to 33.

[0076] Based on the statistical distribution of semantic fingerprints under different message boundaries and out-of-order scenarios in 200 sets of industrial sensor measured data, a compatibility threshold of 4 was set. This threshold was set based on the following: Hamming distances were calculated for different fragment pairs belonging to the same complete message and fragment pairs belonging to different messages in the measured data. 95% of the Hamming distances for the former were distributed between 0 and 4, while 95% of the Hamming distances for the latter were distributed between 5 and 33. Selecting 4 as the cutoff point resulted in a recall rate of 0.953 and a precision rate of 0.917 for the compatibility screening. Then determine the fragmentation. With fragmentation Semantic compatibility, the ordered pair Store in the candidate join pair set, denoted as An additional direction filtering condition is applied here. That is, only fragment pairs marked as having the same fragmentation direction in step S1 are considered, ensuring that candidate connections conform to the unidirectional flow characteristics of network sessions.

[0077] Obtain the set of candidate connection pairs Then, for each of the candidate connection pairs The sequence continuity weight is calculated by combining the sequence correlation field. The sequence correlation field refers to the transmission sequence number read directly from the fragment structure. and the load length of the segment These two fields have been extracted and stored in steps S1 and S2. The sequence continuity weight quantifies the degree of sequential connection between two fragments in the TCP byte stream space, and its calculation formula is as follows:

[0078] ,

[0079] in It is a fragment. The sequence number of the transmission. It is a fragment. The load length, The starting sequence number of the next expected segment; It is a fragment. The sequence number of the transmission; This is the upper limit for sequence number deviation tolerance, set to 4380. This value is three times the maximum transmission unit (1500 bytes) minus the header overhead, representing the typical payload limit. This setting is based on the fact that when the TCP protocol stack sends messages continuously, it may aggregate multiple small messages within the same sending window, and the sequence number jump range usually does not exceed the size of three complete segments. When fragmenting... The starting sequence number is exactly equal to the fragment number. When the sequence ends at the specified index, the sequence is perfectly continuous, and the weight is set to the maximum value of 1.0; when there are gaps or overlaps in the indexes, the weight decreases linearly, and if the deviation exceeds a certain threshold... The time weight is reduced to 0. For the set of candidate connection pairs... Each element in calculate This constitutes a set of connection weights, denoted as . .

[0080] A graph structure is a mathematical model consisting of nodes and edges connecting them; here, it's used to organize the connections between partitions. When constructing the graph structure, each partition in the partition set is treated as a node in the directed graph, with the node number corresponding to a partition index. Candidate join pair set Each ordered pair in As a node Pointing to node Directed edges. Set of connection weights. In This serves as the weight attribute for the corresponding edge. Furthermore, each node needs to label candidate boundary features, which have already been generated in step S3 and stored in the semantic cues. In the middle, specifically the triplet The way to mark it is to... As a node The attribute values ​​are attached to the graph nodes, allowing subsequent steps to identify which nodes correspond to the start and end boundaries of messages. The resulting fragmented connection graph is a weighted directed graph, formally defined as a triple. ,in It is a set of nodes. It is a directed edge set. It is an edge weight set, for each node Includes boundary candidate feature attributes This graph structure provides a complete topological space and numerical basis for the constraint solution and path search in step S6.

[0081] For example, continuing from the semantic fingerprint set in step S4, this set contains 200 33-bit semantic fingerprints. Take the semantic fingerprint of slice 1. semantic fingerprint of slice 2 Perform a compatibility check, assuming The binary sequence is "101100110101…1", which is 33 bits long. The sequence "101100110001…1" has 33 characters. A digit-by-digit comparison reveals that the 9th and 14th characters are different. (Hamming distance) Meanwhile, the fragmentation direction of both fragment 1 and fragment 2 is the transmission direction. If the order pairs are determined to be semantically compatible, they will be ordered. Including candidate connection pairs Then take fragment 1 and fragment 3, both of which are in the sending direction, but the Hamming distance is calculated to be 11, which exceeds the threshold of 4, and they are not included in the candidate connection pair set.

[0082] For candidate connection pairs Calculate the sequence continuity weight. The transmission sequence number of fragment 1. Load length The expected next sequence number is The sequence number of fragment 2. Exact match, weight If it is a candidate connection pair That is, fragment 1 and fragment 3, the sequence number of fragment 3 being sent. Expected sequence number 2461, deviation less than ,but After the weights of all candidate connection pairs are calculated, a set of connection weights is formed. .

[0083] When constructing the graph structure, node 1 marks the boundary candidate features. Node 2 mark Node 3 marker There is a directed edge from node 1 to node 2 with a weight of 1.0; there is also a directed edge from node 1 to node 3 with a weight of 0.7628. By constructing all 200 nodes and their edges in this way, a piecewise connected graph is formed. .

[0084] S6. Perform constraint solving and boundary closure determination on the segmented connection graph to generate a set of recombined paths;

[0085] In one embodiment of the present invention, step S6 includes the following steps:

[0086] Obtain the fragmented connection graph and identify conflicting edges containing retransmission indications to generate a conflict set;

[0087] A preset duplicate data suppression constraint is applied to the conflict set, and consistency filtering is performed to generate a consistent connection subgraph;

[0088] Perform boundary closure determination on the consistent connection subgraph and output the connection sequence that meets the preset waiting conditions to generate a set of recombined paths.

[0089] Specifically, the fragmented connection diagram output in step S5 It is a weighted directed graph, a set of nodes. Each node in A corresponding piece is assigned, and boundary candidate features are attached. Directed edge set Each edge consists of candidate connection pairs. Weights with sequence continuity Retransmission indications refer to the transmission state elements generated by the fragmentation process in step S3. Mid-relay mark A value of 1 indicates that the payload data of the fragment is completely duplicated by a previously existing fragment. Conflicting edges are edges in the fragment connection graph that are associated with retransmitted fragments and overlap in the sequence coverage area. If these edges are not removed, they will generate two or more parallel paths pointing to the same byte stream interval during graph search, disrupting the determinism of message reassembly.

[0090] The process of identifying conflicting edges containing signs of retransmission involves traversing all nodes in the fragmented connectivity graph. For each node... Read the retransmission flag from the attached transmission state elements. .like Mark this node as a retransmission node. For each retransmission node... Define its sequence coverage range ,in It is the sequence number extracted from the fragment structure. It is the load length. In the fragmented connection diagram, identify the retransmission node. All outgoing edges starting from the origin and retransmission nodes All incoming edges of the endpoint Add these edges to a temporary candidate set. Then examine this retransmission node. The corresponding original transmission fragment is a previously transmitted fragment that was not marked as a retransmission but has the exact same sequence number and payload length as the retransmission node. This original fragment node is denoted as... Its sequence coverage range satisfy and Comparison with The sequence spanning relationship represented by the edge with endpoint and the... An edge is an endpoint. If an edge exists... And there are edges This makes fragmentation The sequence number of the transmission simultaneously satisfies the condition from Departure and from The expected departure sequence number, i.e. Then the edge With edge To form a mutually exclusive relationship, the edges The edge is determined to be a conflict edge. If an edge exists... And nodes The expected next sequence number It is equal to It is also equal to Then the edge Edges on the corresponding original path Conflict will arise, and the border Edges are identified as conflict edges. All edges identified as conflict edges form a conflict set, denoted as . .

[0091] The predefined deduplication constraint is a set of rules that stipulates that in the path ultimately used to construct the message, each byte stream interval can only be provided by at most one fragment. The retransmitted fragment must form an exclusive choice with its original fragment, and neither can be retained simultaneously in the same reassembly path. The process of applying this constraint and performing consistency filtering is as follows: from the fragment connectivity graph... Remove all elements belonging to the conflict set. The edges are used to obtain the trimmed image. .exist In the middle, further examine each retransmission node. and its original node Since retransmitted fragments often arrive later than the original fragments, and the original fragments typically have a more complete connection context, the consistency filtering rule prioritizes retaining edges in which the original fragments are involved. For retransmitted nodes... If it is in If any outgoing or incoming edges still exist, then combine its outgoing edge set with the original node. Compare the outgoing edge sets; if any edge exists... In terms of weight and Identical and nodes Boundary candidate elements If true, it means that the edge may cause two identical boundary pieces to be connected. In this case, from Remove edges Similarly, handle incoming edges. After completing these removal operations, a consistent connected subgraph is obtained. It is the original image. A subgraph that contains no conflicting edges and satisfies the repetition suppression constraint.

[0092] Boundary closure determination is a process of searching for a complete directed path from the message start point to the message end point on a consistent connectivity subgraph and determining whether this path can form a closure semantically in the sequence. The message start node is defined as a node that satisfies the following condition: the node's boundary candidate elements... Middle Boundary Type That is, the shard contains message boundaries, and Meanwhile, this node is in the graph A message termination node is defined as a node that contains a message boundary and whose boundary type is either a push boundary or a self-describing boundary. Similarly, a message termination node is defined as a node that contains a message boundary and whose boundary type is either a push boundary or a self-describing boundary. The node.

[0093] Starting from each message originating node, perform a depth-first traversal along the directed edges of the consistent connected subgraph. During the traversal, maintain the cumulative weight sum of the current path. The cumulative weight sum is defined as the sum of the weights of all edges on the path divided by the number of edges in the path, i.e., the average weight, calculated using the following formula: , This represents the number of edges in the path. The preset waiting condition is a composite decision criterion used to determine whether a path can be terminated and output as a valid reassembled path. This condition requires that: the path reaches a message termination node; and the average weight of the path... Furthermore, if the path terminates after the node in the graph... There are also outgoing edges pointing to a certain node. Then you need to wait for the node to be checked. Whether it is the starting node of the next message, only the acknowledgment node. The boundary candidate elements are marked as the message origin, i.e. and Only if all three conditions are met is the current path allowed to terminate here. When a path satisfies all three conditions, the sequence of nodes on that path is recorded sequentially as a connection sequence. A connection sequence is an ordered list of nodes. ,in As the message originating node, The node is the message termination node. The adjacent nodes in the sequence have directed edges in the original graph with non-zero weights.

[0094] After traversing all message start nodes, the collected connection sequences that meet the conditions constitute the set of reconstructed paths, denoted as . , The number of independent recombination paths discovered. This set of recombination paths provides a clear sequence of piecewise assembly for the fine-grained splicing in step S7.

[0095] For example, a fragmented connection graph based on the example in step S5. The graph contains 200 nodes. Node 7 is marked as a transmission state element in step S3. Retransmission flag It is a retransmission node. Its sequence number is... Load length Coverage area The original shard node is node 1. , .picture There are edges in and edge The sequence number sent by node 2 is 2461. (Edge) With edge Constituting a conflict relationship, on the side Added to the conflict set After removing the edge, a consistency check is performed. The remaining edges of the retransmitted node 7 are evaluated to determine whether further removal is necessary. After pruning, a consistent connected subgraph is obtained. .exist In the diagram, node 1 is of the unbounded type. Not used as the starting node; candidate boundary features of node 3 Furthermore, there are no valid incoming edges, so it serves as the message starting node. An edge originating from node 3 points to node 4 with a weight of 0.95. Node 4 has a boundary type of no boundary. The edge continues along the path to node 5, until the boundary candidate element of node 6 is reached. This forms the push boundary and constitutes the message termination node. Path The average weight is 0.88, which is greater than 0.6. The outgoing edge of the terminating node 6 points to node 8, and the boundary candidate element of node 8 is the message starting point, satisfying the waiting condition. This connection sequence... The output is a reconstructed path. This process is repeated for all starting points, generating a set of reconstructed paths containing several connection sequences. .

[0096] S7. Perform fine-grained splicing and deduplication on the fragments according to the reassembly path set to generate a reassembled message stream.

[0097] In one embodiment of the present invention, step S7 includes the following steps:

[0098] Obtain the set of recombination paths and aggregate the fragmented payloads according to the connection sequence to generate an aggregated payload sequence;

[0099] Perform coverage clipping on the sequence coverage area in the aggregated load sequence and output non-overlapping loads to generate a deduplicated load sequence;

[0100] Based on the boundary candidate elements, the deduplication load sequence is finely segmented and spliced ​​and encapsulated to generate a recombined message stream.

[0101] Specifically, the set of recombination paths generated in step S6 In, each connection sequence It is an ordered list of nodes Each node corresponds to a fragment in the fragment set. Payload aggregation is the operation of concatenating the payload fragments carried by all fragments within a connection sequence into a continuous byte stream in sequence. For connection sequences... Traverse in node index order, starting from the first node. To the Kth node Read the payload fragment of each segment, and store the payload fragment in the segment structure generated in step S1. The payload byte array in the sequence. The aggregated payload sequence refers to a temporary byte sequence formed by concatenating the first and last payload fragments according to the order of fragmentation within the sequence, denoted as... Its construction method is as follows:

[0102]

[0103] symbol This represents the sequential concatenation operation of byte arrays. The length is equal to the sum of the lengths of each load segment, that is... , For fragmentation The load length. This aggregation operation has not yet taken into account the possible overlap of sequence numbers between fragments, therefore It may contain redundant bytes.

[0104] The sequence coverage area refers to the sequence number range occupied by each fragment in the TCP byte stream space. For fragments... This interval is , This is the sequence number for the fragment. Within a connection sequence, between adjacent fragments, the expected coverage endpoint of the previous fragment is... The starting point of the coverage of the next segment is When the coverage start point of the subsequent fragment is less than the coverage end point of the preceding fragment plus 1, that is... This indicates that the two fragments overlap in the sequence number space. The responsibility of the cover pruning is to detect and remove duplicate data introduced by sequence number overlap in the aggregated load sequence. The processing flow starts from the second fragment of the join sequence and checks the fragments sequentially. The first segment and the first The relationship between the fragments. Calculate the overlap length. :

[0105]

[0106] in the formula It is the first The right boundary of each segmented coverage area is the first uncovered sequence number. It is the first The starting sequence number of each fragment. If This indicates the first The first segment The byte and the first End of each fragment Each byte is repeated in the sequence number space. This is then used in the aggregated payload sequence. Positioning in the middle The load start position of each segment, relative to The offset of the starting point is Skip backwards from this offset. Byte, reserved Offset in each segmented load The following part. When the overlap length is equal to At that time, the first The entire payload of each segment is included within the coverage area of ​​the preceding segment, thus covering the entire first segment. The load of each fragment is removed from the aggregated load sequence. After performing overlay pruning on all adjacent fragment pairs, the remaining byte sequence is the non-overlapping load, which is concatenated into a deduplicated load sequence, denoted as . De-load sequence length .

[0107] Boundary candidate features are triples retained on each shard node in step S3. ,in There are markers for the boundary. For boundary type labels, Boundary confidence is established. Fine-grained segmentation involves splitting a continuous byte stream into independent application-layer messages based on the positions of nodes carrying boundaries in the connection sequence, within the deduplication payload sequence. The segmentation process traverses the connection sequence. Each node in Read its boundary candidate features. and If the position of the fragment in the deduplication load sequence is determined, then the corresponding position is identified as a message splitting point. The precise offset calculation of the splitting point requires tracing back to the termination boundary of the fragment in the deduplication load sequence. This is done along the cumulative offset of the deduplication load sequence, from the first fragment to the current fragment. So far, the effective payload length traversed, i.e., after subtracting the overlapping bytes of each adjacent pair along the way, gives the end of the current fragment payload at... The offset in, denoted as . The recursive calculation formula is as follows: Let ,for ,have , where defined All pieces that satisfy the boundary candidate feature determination criteria, their corresponding The values ​​form a list of split points. In the deduplication load sequence The byte stream is cut off at each split point offset, and the continuous byte segment between two adjacent split points constitutes a complete application layer message payload.

[0108] Concatenation and encapsulation assembles the segmented message payloads into a reconstructed message conforming to the input format of the analysis system. For each segmented message payload fragment, the TCP header information of the starting and ending fragments that the message traverses is extracted from the connection sequence, and a complete message description structure is reconstructed. This structure includes: source IP address, destination IP address, source port number, destination port number, and protocol type; these fields are obtained from the packet location information of the starting fragment; the message payload is the segmented byte segment; the message length is the byte segment length; the message direction is inherited from the fragmentation direction of the starting fragment; and the message sequence number is obtained by incrementing the message's order in the stream. Each segmented message is arranged in ascending order according to the sending sequence number of the starting fragment, and the encapsulated sequence is the reconstructed message stream.

[0109] For example, based on the set of recombined paths exemplified in step S6, one connection sequence is selected. It contains four fragments. Fragment 3 has a payload length of 730 bytes and a sequence number of 2461; fragment 4 has a payload length of 1460 bytes and a sequence number of 3191; fragment 5 has a payload length of 1460 bytes and a sequence number of 4651; and fragment 6 has a payload length of 500 bytes and a sequence number of 6111. Aggregating them in order yields the aggregated payload sequence. The total length is byte.

[0110] Check adjacent coverage: Patch 3 expected coverage endpoint is Fragment 4 starts at index 3191, with no overlap. The expected coverage endpoint of the 4-shard segment is... Fragment 5 starts at index 4651, with no overlap. The expected coverage endpoint for slice 5 is... Fragment 6 starts at index 6111, with no overlap. The path has no overlap, deduplication load sequence. and same.

[0111] Boundary candidate features for partition 3 , and , is the starting point for the division, offset Candidate features for the boundaries of segment 6 , is the endpoint of the segmentation. From offset 0 to The packet is segmented, and a 4150-byte message payload is extracted. The source IP "192.168.10.5", destination IP "10.10.10.2", source port 3321, destination port 502, and protocol type 6 are extracted from the packet location information of the initial fragment 3 and encapsulated into a reassembled message. This connection sequence generates one reassembled message.

[0112] If another connection sequence overlaps, for example, node 7 has a payload length of 1460 bytes and sends sequence number 1001; node 8 has a payload length of 1200 bytes and sends sequence number 2000. Fragment 7 covers the following area. 8-segment coverage area Overlap length The total length of the deduplicated load sequence is [bytes]. In the aggregated load sequence, fragment 7 (1460 bytes) comes first, followed by fragment 8 (1200 bytes). During pruning, 461 bytes are skipped from the beginning of fragment 8, retaining only the 739 bytes offset from 461 to 1199 of fragment 8. Bytes. Subsequent segmentation and encapsulation are performed according to the above rules. After processing all reassembly paths, a complete reassembled message stream is generated.

[0113] S8. The method further includes: obtaining the ambiguity decision results and waiting trigger results in the reorganization path set, and generating reorganization feedback information; extracting the semantic cue patterns that cause ambiguity from the reorganization feedback information and calculating the adjustment amount, and generating the semantic weight update amount; and updating the connection weight set using the semantic weight update amount.

[0114] Specifically, such as Figure 2As shown, step S6 generates two types of auxiliary decision records during boundary closure determination, which are output along with the recombined path set and persisted in the DPU's internal state buffer. Ambiguous decision results refer to situations where a node in a consistent connection subgraph has multiple outgoing edges pointing to different successor nodes, and the sequence continuity weights of these outgoing edges all exceed 0.6, causing branch selection during depth-first traversal when determining the unique next hop. In this case, the processing logic of step S6 will continue based on the edge with the higher average weight, but will record the unselected candidate edges and their context as ambiguous decision results. Waiting-to-trigger results refer to events where a path is temporarily suspended at a message termination node because it does not meet the preset waiting condition that the successor node must be the message starting point, and the determination is retried upon the arrival of a new fragment. This record includes the endpoint node index of the suspended path, the boundary candidate elements of the endpoint node, and the successor node index and boundary candidate elements of the successor node pointed to by the outgoing edges of the endpoint node.

[0115] The process of generating reorganized feedback information merges ambiguous decision results and waiting-to-trigger results into a structured feedback record. For each ambiguous decision record, the winning edge selected is extracted. With suppressed losing edge and the time node when the decision occurs Boundary candidate elements and nodes ,node Each candidate boundary feature. For each pending trigger record, extract the endpoint node of its pending path. Candidate boundary features of the endpoint node Successor nodes Candidate boundary features of successor nodes And the number of shards that the path traverses from start to finish. These records are organized into a list of feedback entries, each containing a type label ("ambiguous" or "waiting"), the relevant node index pair, the corresponding boundary candidate features, and path association data. This list is the reorganized feedback information, denoted as... .

[0116] Semantic cue patterns are feature combinations of boundary candidate elements extracted from reconstructed feedback information that contribute to decision bias in ambiguous decision-making or waiting-to-trigger scenarios. The extraction method involves reading items of type "ambiguity" from the feedback information. , and Construct a pattern vector Its composition is ,in This is the difference in sequence continuity weights between the losing and winning edges. For entries of type "waiting" in the feedback information, read... and Construct pattern vectors .

[0117] The objective of semantic weight update calculation is to apply inverse error correction to edges in the connection weight set that match ambiguous patterns. A pattern response function based on actual measurements from 200 industrial sensors is introduced when calculating the adjustment. The calculation formula is:

[0118] ,

[0119] in, The learning step size is set to 0.15. This value was selected by balancing convergence speed and weight stability through repeated recombination experiments in 200 sets of industrial sensors. It is the difference in weight between the losing edge and the winning edge, with a dimensionless unit. It is a scaling kernel function, defined as When the absolute value of the weight difference reaches 0.3 or above, the scaling factor saturates to 1. When the difference is small, the scaling factor shrinks linearly to avoid being overly sensitive to small differences. Pick The sign indicates that the weight of the losing edge is higher than that of the winning edge, and the weight of the losing edge needs to be lowered or the weight of the winning edge needs to be increased. The direction of adjustment is determined by the sign.

[0120] Locate the specific edges that need updating in the connection weight set based on the pattern vector. For ambiguous feedback, locate the connection weight set. Middle and winning side and the side of the unselected The corresponding two weight values. The update rule is: increase the weight of the winning edge. Reduce the weight of the unselected edges The updated weight values ​​are truncated at... Within the interval. For feedback that is pending, identify the endpoint node. To the successor node edge If the edge exists in the set of connection weights If it is in the middle, then its weight is reduced by a certain amount. This represents the penalty imposed on the tendency to cross-message connections for failing to complete message closure within a reasonable path length. If there are no edges in the connection weight set... ,and If the path length exceeds the preset upper limit of 8, then the edge will be... Adding an initial weight of 0.3 to the set of connection weights is a predictive edge insertion operation that provides a remedial path for potentially missed connections.

[0121] After adjusting the weights of all feedback items, connect the weight set. Some edge weights in the array have been modified. The set of modified connection weights is denoted as . When the next shard batch arrives and triggers the graph structure construction in step S5, the updated set of connection weights will be... The weights assigned directly to candidate connection pairs will cause the graph search preference to gradually tend toward connection patterns that stably constitute complete application layer messages in the actual network environment.

[0122] For example, the set of recombined paths generated in step S6 contains one ambiguous decision record. At node 4, there is an outgoing edge. and .side Sequence continuity weights ,side weight Step S6, the boundary closure determination, selects the edge ,side It becomes an unselected edge. The boundary candidate features of node 4 are... Node 8 is Node 9 is This record constitutes an ambiguous feedback entry. Calculate the weight difference. , , Adjustment amount Update the edge weight set. The weight is ,side The weight is After both are updated, they tend to reach equilibrium. If there is still a record waiting for feedback, with the endpoint node 5 suspended, the successor node being 10, and the path spanning 6 partitions, then the edge... If the original weight is 0.56, the penalty amount The updated weight is These adjusted weights are written back to the connection weight set. This is used for subsequent construction of the piecewise connection graph.

[0123] Based on the same inventive concept, such as Figure 3 As shown, the present invention also provides a semantically aware DPU fine-grained message reassembly system, characterized in that the system comprises:

[0124] The fragment parsing module is used to obtain the raw data packets received by the DPU and perform layered parsing to generate a fragment set;

[0125] The feature extraction module is used to extract the visible fields and visible payload fragments of each fragment in the fragment set, normalize them, and generate a reconstructed data feature set.

[0126] The clue extraction module is used to extract and combine features from the recombined data element set to generate a semantic clue set;

[0127] The fingerprint extraction module is used to perform fingerprinting mapping on each segment based on the semantic clue set and output an identifier that can distinguish concurrent messages, thereby generating a semantic fingerprint set;

[0128] The graph construction module is used to construct a fragmented connectivity graph based on the semantic fingerprint set;

[0129] The path generation module is used to perform constraint solving and boundary closure determination on the segmented connection graph to generate a set of recombined paths;

[0130] The message reassembly module is used to perform fine-grained splicing and deduplication on the fragments according to the reassembly path set, and generate a reassembled message stream.

[0131] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0132] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A semantically aware fine-grained message reassembly method for DPU, characterized in that, The method includes: The raw data packets received by the DPU are acquired and parsed hierarchically to generate a fragment set; Extract the visible fields and visible payload fragments of each fragment in the fragment set, normalize them, and generate a reconstructed data element set; Feature extraction and combination are performed on the recombined data element set to generate a semantic clue set; Based on the semantic clue set, fingerprinting is performed on each shard and an identifier that can distinguish concurrent messages is output to generate a semantic fingerprint set; Construct a fragmented connection graph based on the semantic fingerprint set; Perform constraint solving and boundary closure determination on the segmented connection graph to generate a set of recombined paths; According to the set of reassembly paths, the fragments are finely spliced ​​and deduplicated to generate a reassembled message stream.

2. The semantically aware DPU fine-grained message reassembly method according to claim 1, characterized in that, The process of acquiring the raw data packets received by the DPU and performing hierarchical parsing to generate a fragment set includes: Obtain the link layer identifier and network layer identifier of the raw data packets received by the DPU, and generate data packet location information; Identify the transport layer header corresponding to the data packet location information, extract sequence-related fields, and generate sequence elements; The sequence elements are associated with the payload fragments of the original data packet, and the fragmentation direction is marked to generate a fragment set.

3. The semantically aware DPU fine-grained message reassembly method according to claim 2, characterized in that, The step of extracting the visible fields and payload visible fragments of each fragment in the fragment set, normalizing them, and generating a reconstructed data element set includes: Obtain the protocol type field and length-related fields for each fragment in the fragment set, and generate the protocol length element; Obtain the sequence-related fields and confirmation-related fields of each segment in the segment set, establish the change relationship, and generate sequence confirmation elements; Extract the visible payload fragments from each fragment in the fragment set and extract the byte pattern summary to generate payload summary elements; The protocol length element, the sequence confirmation element, and the payload digest element are merged and normalized to generate a set of recombined data elements.

4. The semantically aware DPU fine-grained message reassembly method according to claim 3, characterized in that, The step of extracting and combining features from the recombined data element set to generate a semantic clue set includes: Obtain the protocol length element from the recombined data element set and identify the message boundary position to generate boundary candidate elements; Obtain sequence confirmation elements from the recombined data element set and identify out-of-order and retransmission signs to generate transmission status elements; By fusing the boundary candidate elements and the transmission state elements, a connectivity score between fragments is calculated to generate a semantic cue set.

5. The semantically aware DPU fine-grained message reassembly method according to claim 4, characterized in that, The step of performing fingerprinting mapping on each shard based on the semantic clue set and outputting identifiers that can distinguish concurrent messages, generating a semantic fingerprint set, includes: Based on the set of semantic clues, the boundary candidate elements of the segments are located and encoded to generate boundary encoded elements. The protocol-related features of the fragments are categorized and encoded to generate protocol coding elements; The boundary coding elements and the protocol coding elements are combined and mapped by introducing a preset direction consistency constraint to generate a semantic fingerprint set.

6. The semantically aware DPU fine-grained message reassembly method according to claim 5, characterized in that, The step of constructing a fragmented connection graph based on the semantic fingerprint set includes: Obtain the semantic fingerprint set and filter semantically compatible fragment pairs to generate a candidate connection pair set; Obtain the candidate connection pair set and calculate the sequence continuity weights by combining the sequence-related fields to generate a connection weight set; A graph structure is constructed based on the set of candidate connection pairs and the set of connection weights, and boundary candidate features are marked to generate a piecewise connection graph.

7. The semantically aware DPU fine-grained message reassembly method according to claim 6, characterized in that, The step of performing constraint solving and boundary closure determination on the piecewise connectivity graph to generate a set of recombined paths includes: Obtain the fragmented connection graph and identify conflicting edges containing retransmission indications to generate a conflict set; A preset duplicate data suppression constraint is applied to the conflict set, and consistency filtering is performed to generate a consistent connection subgraph; Perform boundary closure determination on the consistent connection subgraph and output the connection sequence that meets the preset waiting conditions to generate a set of recombined paths.

8. The semantically aware DPU fine-grained message reassembly method according to claim 7, characterized in that, The step of performing fine-grained splicing and deduplication on the fragments according to the reassembly path set to generate a reassembled message stream includes: Obtain the set of recombination paths and aggregate the fragmented payloads according to the connection sequence to generate an aggregated payload sequence; Perform coverage clipping on the sequence coverage area in the aggregated load sequence and output non-overlapping loads to generate a deduplicated load sequence; Based on the boundary candidate elements, the deduplication load sequence is finely segmented and spliced ​​and encapsulated to generate a recombined message stream.

9. The semantically aware DPU fine-grained message reassembly method according to claim 7, characterized in that, The method further includes: Obtain the ambiguous decision results and waiting trigger results from the set of reorganization paths, and generate reorganization feedback information; Extract the semantic cue patterns that lead to ambiguity from the reorganized feedback information and calculate the adjustment amount to generate the semantic weight update amount; The connection weight set is updated using the semantic weight update amount.

10. A semantically aware DPU fine-grained message reassembly system, characterized in that, The system includes: The fragment parsing module is used to obtain the raw data packets received by the DPU and perform layered parsing to generate a fragment set; The feature extraction module is used to extract the visible fields and visible payload fragments of each fragment in the fragment set, normalize them, and generate a reconstructed data feature set. The clue extraction module is used to extract and combine features from the recombined data element set to generate a semantic clue set; The fingerprint extraction module is used to perform fingerprinting mapping on each segment based on the semantic clue set and output an identifier that can distinguish concurrent messages, thereby generating a semantic fingerprint set; The graph construction module is used to construct a fragmented connectivity graph based on the semantic fingerprint set; The path generation module is used to perform constraint solving and boundary closure determination on the segmented connection graph to generate a set of recombined paths; The message reassembly module is used to perform fine-grained splicing and deduplication on the fragments according to the reassembly path set, and generate a reassembled message stream.

Images