An unknown protocol message format inference system based on incremental clustering

Abstract

This invention provides an unknown protocol message format inference system based on incremental clustering, relating to the field of network communication protocol analysis. It includes message cluster state management, incremental attribution, field boundary evidence updating, boundary divergence identification, local cluster reconstruction, and format inference units. The system maintains the main boundary path between anchor points based on stable anchor points, and performs variable-length field displacement compensation and structural matching on the boundary paths formed by newly added messages. When a continuously supported competing boundary path forms within the same anchor point interval that is structurally exclusive to the main boundary path, local format differentiation is confirmed, and the corresponding message cluster is locally reconstructed. Simultaneously, only historical messages related to the divergence interval are backtracked, thereby forming the unknown protocol message format corresponding to each message cluster. This invention can correct mixed clustering results during continuous message addition, reducing full re-clustering and repetitive format inference.

Description

Technical Field

[0001] This invention relates to the field of network communication protocol analysis, and more specifically to an unknown protocol message format inference system based on incremental clustering. Background Technology

[0002] With the increasing number of proprietary protocols, dedicated device protocols, and communication protocols lacking complete technical documentation in network communication systems, reconstructing protocol message formats from communication messages has become a crucial foundational step in protocol reverse engineering when protocol descriptions are unknown. Current message format inference typically requires identifying structurally similar message types from collected message samples, then determining fixed and variable regions and field boundaries through sequence alignment and byte change analysis, and further inferring field attributes such as length and data fields. Since different message types may share the same common header, similar message lengths, and a large number of identical bytes, classifying messages solely based on overall byte similarity can easily lead to grouping messages with different actual field structures into the same category.

[0003] In the prior art, CN102891852A discloses an automatic protocol format inference method based on message analysis. This method first segments network messages according to the displayable attributes of bytes, and then performs preliminary clustering based on the format sequences formed by the segments. Subsequently, it performs multi-sequence alignment on message samples with similar structures, obtaining the basic message structure through local sequence alignment, hierarchical clustering, and global sequence alignment. Based on this, it infers the interval field, data field, sequence number field, length field, and format identifier field according to the values ​​and variation characteristics of each field. This scheme can reduce the computational overhead caused by message formats participating in sequence alignment and improve the efficiency of message structure and field semantic inference.

[0004] However, the above approach, based on obtaining a sufficient number of message samples and conducting centralized analysis, still involves first forming message categories and then inferring structure and fields within existing categories. As the number of unknown protocol messages continues to increase, insufficient early samples may temporarily group messages with the same common structure but different internal field organization into the same message cluster. Even if new boundary differences emerge during subsequent field structure analysis, the existing processing method lacks a targeted mechanism to use these local field structure divergences to correct existing message clusters. Directly re-clustering new messages and all historical messages would further increase the overhead of repetitive distance calculations, structure comparisons, and format inference.

[0005] Therefore, existing technologies still need to address the following issues: how to continuously verify existing clustering results using the field structure information gradually obtained during the format inference process when unknown protocol messages continue to enter, and how to distinguish the boundary displacement caused by ordinary variable-length fields from the different field organization structures when real format differentiation occurs within the same message cluster, and adjust the clustering results only for the local messages where structural divergence actually occurs and related historical messages, thereby avoiding the continued impact of early erroneous clustering on subsequent format inference, while reducing the processing overhead of repeatedly performing clustering and format analysis on all historical messages. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings by proposing an unknown protocol message format inference system based on incremental clustering.

[0007] The present invention adopts the following technical solution:

[0008] An unknown protocol message format inference system based on incremental clustering is characterized by comprising a message cluster state management unit, an incremental attribution unit, a field boundary evidence update unit, a boundary divergence identification unit, a local cluster reconstruction unit, and a format inference unit.

[0009] The message cluster state management unit is used to maintain multiple unknown protocol message clusters and maintain stable anchor points for each message cluster as well as the main boundary path between adjacent stable anchor points.

[0010] The incremental attribution unit is used to determine the candidate attribution cluster of the new message based on the structural compatibility relationship between the new message and the stable anchor point and main boundary path of each message cluster.

[0011] The field boundary evidence update unit is used to obtain the boundary path formed between adjacent stable anchor points of the candidate belonging cluster by the newly added message.

[0012] The boundary divergence identification unit is used to identify the boundary path as a candidate alternative boundary path when the boundary path formed by the newly added message cannot be explained by the length change of the main boundary path and its existing variable length field, and to identify the candidate alternative boundary path as a competing boundary path when the candidate alternative boundary path between the same pair of stable anchors continuously obtains multiple message support and forms structural exclusion with the main boundary path, and to determine that the corresponding message cluster has undergone local format differentiation.

[0013] The local cluster reconstruction unit is used to perform local reconstruction of the packet clusters that have undergone local format differentiation based on the packets that support the main boundary path and the competing boundary path respectively, and to maintain the boundary path between the stable anchor point for each reconstructed packet cluster.

[0014] The format inference unit is used to generate the corresponding unknown protocol message format based on the stable anchor point and boundary path of each message cluster after reconstruction.

[0015] Furthermore, the message cluster state management unit is used to perform coarse-grained structure alignment on messages within the same message cluster, determine stable anchor points based on the repeated support of candidate structure positions by multiple messages, and limit the local analysis interval using adjacent stable anchor points.

[0016] Furthermore, the field boundary evidence update unit is used to determine the internal field boundary between adjacent stable anchor points and form a boundary path according to the order of the internal field boundaries;

[0017] When comparing boundary paths formed by different messages, it is determined whether the boundary paths belong to the same path based on the number of internal field boundaries, their arrangement order, their positional relationship after local alignment, and the relationship of field length changes.

[0018] Furthermore, the incremental attribution unit is used to mark newly added messages that form a structurally compatible relationship with the candidate attribution cluster as members to be confirmed;

[0019] The field boundary evidence update unit records the boundary path generated by the member to be confirmed as temporary path evidence. When the member to be confirmed can support the current main boundary path and has a higher degree of structural compatibility with the current message cluster than other message clusters, or when multiple members to be confirmed repeatedly support the same candidate alternative boundary path and form a consistent local structure, the corresponding temporary path evidence is converted into valid path evidence.

[0020] Furthermore, the boundary divergence identification unit is used to compensate for the displacement of the affected internal field boundaries based on the length change of the existing variable-length field when there is a difference in the position of the internal field boundaries between the newly added message and the current main boundary path.

[0021] If, after displacement compensation, the number of internal field boundaries, their arrangement order, or the field separation relationship still cannot match the main boundary path, the boundary path formed by the newly added message will be determined as a candidate alternative boundary path.

[0022] Furthermore, the boundary divergence identification unit is used to maintain the set of valid supporting members of the main boundary path, the set of valid supporting members of the candidate alternative boundary path, and the set of fuzzy path members, and to determine whether the candidate alternative boundary path satisfies the format differentiation condition based on the repeatability, persistence, internal consistency of supporting members, and structural exclusion relationship with the main boundary path of the candidate alternative boundary path.

[0023] Furthermore, the structural repulsion relationship is as follows:

[0024] The valid supporting members of the primary boundary path cannot form the critical internal field boundaries in the candidate alternative boundary path, and the valid supporting members of the candidate alternative boundary path cannot form the critical internal field boundaries in the primary boundary path.

[0025] The candidate alternative boundary path also satisfies that the number of internal field boundaries, their arrangement relationship, or their field organization relationship cannot be explained by the length changes of existing variable-length fields in the main boundary path.

[0026] Furthermore, the local cluster reconstruction unit is used to retain the packets that continuously support the main boundary path in the original packet cluster, form a new packet cluster with the packets that continuously support the competing boundary path, and make the new packet cluster inherit the stable anchor point and boundary path that it shares with the original packet cluster outside the boundary divergence interval, and maintain different main boundary paths respectively within the boundary divergence interval.

[0027] Furthermore, it also includes a pending message backtracking unit;

[0028] The message cluster state management unit is used to save the local path state corresponding to each stable anchor point interval for member messages.

[0029] The pending message backtracking unit is used to filter historical messages that have shown candidate path support, path ambiguity, lack of valid boundary evidence, or missing key internal field boundaries in the main boundary path within the boundary divergence interval after local cluster reconstruction, based on the local path status, and to re-determine the message cluster affiliation of the historical messages only for the boundary divergence interval.

[0030] Furthermore, the format inference unit is used to divide field regions based on stable field boundaries confirmed by field boundary evidence in each reconstructed message cluster, and to determine at least one field attribute among fixed fields, fixed-length variable fields, length fields, variable-length fields, and check fields based on the length changes, byte value changes, and inter-field relationships of each field region.

[0031] The beneficial effects achieved by this invention are:

[0032] This invention establishes stable anchor points and boundary paths between anchor points in message clusters, transforming the incremental clustering of unknown protocol messages from overall byte similarity judgment to local field structure relationship judgment. It utilizes variable-length field displacement compensation to distinguish between explainable field position changes and true boundary path differentiation. When competing boundary paths that mutually exclude the main boundary path and cannot be explained by the existing variable-length structure continuously form between the same pair of stable anchor points, this local boundary divergence is used as the basis for message format differentiation, and the corresponding message cluster is partially reconstructed. This allows the format inference results to correct the clustering problems generated in the early clustering process. During local reconstruction, only messages supporting different boundary paths are processed, and pending messages related to the current divergence interval are traced back based on historical local path states. This eliminates the need to re-cluster all historical messages and re-perform complete format inference, improving the accuracy of different message format differentiation while reducing redundant computation during continuous message analysis. This allows the message cluster structure and protocol format to gradually converge as new messages are added.

[0033] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall system structure of the present invention;

[0035] Figure 2 This is a schematic diagram of the incremental clustering format inference process of the present invention;

[0036] Figure 3 This is a schematic diagram of boundary path divergence under the stable anchor point constraint of the present invention.

[0037] Figure 4 This is a schematic diagram illustrating the local cluster reconstruction and historical message backtracking of the present invention. Detailed Implementation

[0038] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0039] Example 1:

[0040] This embodiment provides an unknown protocol message format inference system based on incremental clustering, combined with... Figure 1 The system comprises a message cluster state management unit, an incremental attribution unit, a field boundary evidence update unit, a boundary divergence identification unit, a local cluster reconstruction unit, a pending message backtracking unit, and a format inference unit. For unknown protocol messages to be analyzed, the system maintains multiple message clusters. Each message cluster is associated with and stores member message indexes and format states that change as member messages are added. The format states include stable anchor points, local analysis intervals defined by adjacent stable anchor points, main boundary paths within each local analysis interval, candidate alternative boundary paths, supporting member sets corresponding to each boundary path, and fuzzy member sets whose attribution cannot be determined under the current path structure. Message cluster member relationships and format states change synchronously, enabling the format inference results to continuously participate in subsequent message attribution and, when hidden format divergences are detected within the current message cluster, the message cluster structure is adjusted in reverse.

[0041] After the system starts running, several initial message clusters are formed based on the message length, stable byte segments, repeating byte patterns, and local sequence similarity among existing unknown protocol messages. These initial message clusters are only used to establish the starting state for incremental analysis and are not required to accurately distinguish all actual message formats given the limited number of initial samples. Messages with the same common header, many common stable bytes, or similar length relationships can temporarily enter the same initial message cluster. Subsequent local field structure evidence formed by newly added messages continues to verify the current clustering results.

[0042] After the initial message cluster is formed, coarse-grained structural alignment is performed on the messages within the cluster. Coarse-grained structural alignment uses stable byte segments that can be directly and repeatedly matched, repeating byte patterns, and local sequence matching results to establish several candidate structural correspondence positions between different messages. This stage does not require stable anchor points as a prerequisite, nor does it require that field boundaries have been identified. Its role is to provide initial reference coordinates for the formation of stable anchor points.

[0043] Candidate positions generated by coarse-grained structure alignment The message cluster state management unit records the number of valid messages that can support the candidate location as a repeatable structure location point. And record the total number of valid messages participating in the current judgment. Thus, the anchor support rate of the candidate position is obtained:

[0044]

[0045] When the number of valid supports reaches the set minimum sample size, the anchor support rate reaches the set stability threshold, and the deviation of the corresponding position in each supporting message after coarse-grained alignment does not exceed the allowable range, the candidate position is determined as a stable anchor point. The allowable position deviation is used to absorb small-range movements caused by local insertion, variable-length fields, and local scaling of the message, while unconstrained drift of the candidate position in the message is not allowed.

[0046] Once stable anchor points are formed, subsequent field structure analysis no longer relies on the initial coarse-grained alignment results, but instead uses local fine-grained alignment constrained by stable anchor points. This establishes a processing relationship of "coarse-grained alignment generates candidate positions, candidate positions are repeatedly verified to form stable anchor points, and stable anchor points are then used for subsequent local fine-grained alignment," avoiding the stable anchor point generation process and the local alignment process from being mutually dependent.

[0047] Stable anchor points are not limited to confirmed field separator positions. Even if the fields on either side of a given position are not yet definitively determined, a stable anchor point can be used as long as it can be reliably and repeatedly located across multiple messages within the same message cluster, and the local structural relationships before and after that position remain stable. Stable anchor points primarily serve to locate local structures; they are only output as part of the final field boundary when subsequent field boundary evidence confirms that the anchor point also corresponds to a field separator position.

[0048] Combination Figure 3 Adjacent stable anchor points and They collectively define a local analysis interval. Within this local analysis interval, the field boundary evidence update unit analyzes byte changes, length changes, and the correlations between different positions. When the stability of bytes before and after a certain position is consistently different, different value correlations are formed on both sides of the position, or different length change patterns are observed on both sides of the position, internal field boundary evidence is generated at the corresponding position.

[0049] A structural transition generated by a single message only forms temporary boundary evidence and does not directly change the current stable format state of the message cluster. After multiple messages that have been confirmed to belong to the current message cluster repeatedly support corresponding internal boundaries within the same local interval, they are then processed according to these internal boundaries from the stable anchor point. To the stable anchor point The order in which they are arranged forms the boundary path.

[0050] Set message The actual location of the pre-stabilized anchor point within the local analysis interval is: The actual location of the subsequent stabilizing anchor point is The actual positions of its internal boundaries are as follows: The boundary path formed by the message within the local analysis interval is then represented as:

[0051]

[0052] To eliminate the impact of variations in local message length on absolute position, local reference coordinates are established for each internal boundary relative to the preceding and following stable anchor points:

[0053]

[0054] The system stores the actual position of the internal boundary in the original message and the mapped local reference position. The positions of the 6th byte, 12th byte, 18th byte, etc., mentioned later all represent the reference positions after local alignment. The actual byte positions in the original message are allowed to change within a small range under the constraint of stable anchor points.

[0055] Whether two messages form the same local boundary path is not determined directly by whether the absolute positions of all internal boundaries are completely identical. The system first determines whether there is an interpretable displacement caused by an identified variable-length field between the two messages, and then performs a path consistency comparison.

[0056] When a variable-length field already exists in the current main boundary path, and one or more internal boundaries following the variable-length field in a new message are displaced, the system compensates for the position of the affected subsequent internal boundaries based on the change in length of the variable-length field in the new message. Let the original variable-length field length be... The length of this field in the newly added message becomes A certain subsequent internal boundary actually occurred The synchronous displacement is then mapped back to the reference position when no length change occurs, and then compared with the corresponding internal boundary in the main boundary path.

[0057] Only after the displacement compensation of the known variable-length fields is completed, does the system compare the number of internal boundaries, the order of the boundaries, the local reference position deviation of the corresponding boundaries, and the length relationship between adjacent boundaries of the two paths. When the number of internal boundaries is the same, the order is consistent, the position deviation of the corresponding boundaries after compensation is within the allowable range, and there is no structural contradiction in the length variation relationship between the fields, the two local boundary sequences are classified into the same boundary path.

[0058] The boundary path that is continuously supported by multiple valid member messages and currently has the highest level of support is designated as the primary boundary path for that local analysis interval. Other boundary paths that have not yet met the stability requirements but have been repeatedly supported by multiple messages are designated as candidate alternative boundary paths. The system stores the valid supporting members for each boundary path separately and stores temporary path evidence separately for messages that have not yet completed path confirmation.

[0059] Combination Figure 2When a new unknown protocol message enters the system, the incremental attribution unit reads the currently saved stable anchor points and local boundary path states of each message cluster to determine which message cluster's existing structure can interpret the new message. The structural compatibility judgment focuses on whether the new message can maintain existing stable anchor points, whether it can support the main boundary path between adjacent stable anchor points, whether the existing variable-length field states can explain the boundary displacements generated by the new message, and whether the new message forms an internal structure within a certain stable anchor point interval that cannot be explained by the current main boundary path.

[0060] The changes in byte values ​​identified as variable payload regions reduce their impact on attribution determination, while stable anchors and the internal boundary organization between anchors increase their impact on attribution determination. Even if two messages have a large number of different bytes in their data payloads, they can still be classified into the same message cluster as long as the local boundary paths remain consistent. Even if two messages have a long common prefix, if one message consistently forms different internal field organization relationships between a pair of stable anchors, the structural difference will be preserved and will not be canceled out by the similarity of common bytes.

[0061] Once a new message establishes a clear top-level structural compatibility relationship with a message cluster, it is initially treated as a member to be confirmed within that message cluster. The internal boundary and path information formed by the member to be confirmed are initially recorded as temporary evidence and are not immediately used for statistics on stable anchor support rate, main boundary path support rate, or competing path support rate, thereby preventing a single erroneously attributed message from directly changing the current message cluster format state.

[0062] If a message to be confirmed fully supports the current main boundary path and its structural compatibility with the current message cluster is consistently higher than that of other message clusters, the message is converted into a confirmed message, and its temporary path evidence becomes valid path evidence. If multiple messages to be confirmed repeatedly form the same candidate alternative boundary path and can form a stable local structure with each other, the temporary alternative path evidence corresponding to these messages can be collectively converted into valid candidate path evidence. If a message to be confirmed is subsequently better interpreted by other message clusters, its temporary evidence formed in the current message cluster is revoked and does not participate in the current cluster's stable state statistics.

[0063] If a new message can be interpreted by multiple message clusters, and the structural compatibility differences between the different message clusters are insufficient to form a reliable attribution, the message is recorded as a low-confidence message, and its path evidence remains in a temporary state. If a new message cannot be reasonably interpreted by any stable anchor point and boundary path of any existing message cluster, it is stored in the unclassified message set.

[0064] If unclassified messages subsequently form duplicate stable structural locations and identical or alignable boundary paths, and the number of supporting members meets the initial clustering requirements, then a new initial message cluster is established using these messages. The new initial message cluster generates candidate structural locations, stable anchor points, and local boundary paths in the same way as other initial message clusters.

[0065] After a new message enters the candidate message cluster, the local analysis intervals are checked one by one based on the current stable anchor point of the message cluster. For local intervals that can support the current main boundary path, only the temporary or valid support status and field length range of the corresponding path are updated. When a local interval that cannot be interpreted by the current main boundary path is found, the system does not re-analyze the entire message, but instead determines the nearest previous stable anchor point and subsequent stable anchor point surrounding the conflict area, and defines the interval between the two anchor points as the boundary divergence interval.

[0066] For example, a message cluster forms stable anchor points near the 6th and 20th bytes of the local reference coordinates, and its main boundary path is:

[0067]

[0068] A newly added message maintains a stable structure around bytes 6 and 20, but after performing known variable-length field displacement compensation, it still cannot support the internal boundary around byte 8, and forms new internal boundary evidence around bytes 10 and 14. Therefore, bytes 6 to 20 are identified as the boundary divergence interval corresponding to this message, and a candidate alternative path is formed.

[0069]

[0070] Instead of treating bytes 8, 10, and 14 as separate exceptions, the system records that another complete internal boundary path has appeared between the same pair of stable anchors.

[0071] When a new message subsequently generates a structural difference between bytes 6 and 20, the boundary divergence identification unit first determines whether the stable anchor points before and after it are consistent with the existing divergence interval, then performs variable-length field displacement compensation, and then determines whether it supports the existing candidate alternative path based on the number of internal boundaries, boundary order, local position after compensation, and field length relationship.

[0072] If a new message has the same number and order of internal boundaries as the main boundary path, and the compensated internal boundaries can be restored to the corresponding positions of the main path, then the message continues to support the original main boundary path. Only when the new message still exhibits a new number of internal boundaries, a different boundary arrangement, or a field separation relationship that the original main path cannot explain after the known variable-length field displacement compensation is completed, will it be retained as an independent candidate alternative path.

[0073] For example, the original main boundary path is:

[0074]

[0075] The corresponding boundary in the newly added message is increased due to the increase in the length of the preceding field. And moved to After length compensation, it can be restored to its original state. If the new message still returns to its original path after compensation:

[0076]

[0077] and and If the transitions correspond to stable field structures, then the structure is considered to be unexplainable by changes in the length of the original single internal boundary and is retained as a candidate alternative path.

[0078] After candidate alternative boundary paths are formed, the system maintains a set of valid support members for each primary boundary path. Valid support set of candidate alternative paths and path fuzzy member set The path ambiguity member is not a member that simultaneously and completely supports both the main path and the candidate alternative path. Rather, it is a member that cannot be excluded from either path temporarily due to insufficient message length, partial truncation, lack of key boundary evidence, or insufficient existing information.

[0079] Enter Messages not included in the valid support count for the main boundary path or candidate alternative paths are not considered positive evidence of a mutually exclusive relationship between two paths. As the stable anchor points and boundary path states further converge, fuzzy members that can explicitly support one of the paths will be counted from [the previous list of paths]. Messages that remain undetermined will be transferred to the corresponding set of valid supporting members; messages that are still undetermined will remain in an ambiguous state.

[0080] The boundary divergence identification unit continuously assesses the repeatability, persistence, internal consistency, and path exclusion relationships of candidate alternative paths. Path repeatability indicates whether different valid members form the same or compensated, locally aligned, and consistent internal boundary sequence between the same pair of stable anchors; path persistence indicates whether the structure continues to receive valid support in subsequent incoming messages; path internal consistency indicates whether the length relationship, byte changes, and boundary correspondence between adjacent fields can form a stable pattern after the messages supporting the path are divided according to the path.

[0081] Path exclusion relationships are used to determine whether the primary boundary path and candidate alternative paths truly represent two different formats. Most members in the process still cannot form the critical internal boundary in the candidate alternative path after length compensation. Most members in the path also consistently fail to form the key internal boundary in the main boundary path, and the path is an fuzzy member set. When the proportion is not high enough to make the two paths indistinguishable, the main boundary path and the candidate alternative path are considered to have a structurally exclusive relationship.

[0082] Path mutual exclusion does not require that all packets can be immediately included. or The presence of a small number of ambiguous members will not prevent format differentiation confirmation, as long as the two groups of valid supporting members form stable and mutually uninterpretable local structures, and the proportion of ambiguous members is below the set maximum allowable range.

[0083] Once a candidate alternative path reaches a minimum number of valid supporting members, maintains a minimum valid support ratio within a continuously increasing message window, meets the set consistency requirements within the path, and forms a structural exclusion relationship with the main boundary path, the candidate alternative path is converted into a competing boundary path. Competing boundary paths also require exclusion based on variable-length fields; that is, the number of internal boundaries, boundary arrangement, or field organization of the path cannot be explained by changes in the length of existing variable-length fields in the main boundary path. After meeting the above conditions, it is confirmed that there are two independent field organization structures between the same pair of stable anchor points, and the current message cluster experiences format differentiation in this local interval.

[0084] The criteria for format differentiation are limited to stable anchor points that generate mutually exclusive boundary paths. Stable structures outside the boundary divergence interval, already supported by two groups of valid members, are not included in this format differentiation assessment. Figure 4 The local cluster reconstruction unit effectively supports member sets by reading the main boundary path. Competition boundary path effectively supports member set and path fuzzy member set And adjust the current message cluster accordingly.

[0085] Continuous and stable support for the main boundary path Members remain in the original message set, providing continued and stable support for contention boundary paths. Members form a new message cluster. Outside the boundary divergence interval, the stable anchor points and boundary paths jointly supported by the two groups of members are directly inherited by the new message cluster. Inside the boundary divergence interval, the two message clusters maintain their respective main boundary paths. The new message cluster does not infer the format from the entire message again, but inherits the common stable structure that has not diverged in the original message cluster, and only establishes independent format states for the diverged local intervals.

[0086] Taking the aforementioned byte 6 to byte 20 range as an example, the original message cluster continues to be maintained:

[0087]

[0088] Maintenance of newly formed message clusters:

[0089]

[0090] Structures that have not differentiated and have stabilized before byte 6 and after byte 20 are not re-inferred, thus ensuring that the scope of local cluster reconstruction is consistent with the actual location where format differentiation occurs.

[0091] To accurately locate historical messages requiring re-evaluation during subsequent format differentiation, the message cluster state management unit saves local path state records for each member message within each stable anchor point interval after the member message first enters the message cluster. For member messages... Its local path state record is represented as follows:

[0092]

[0093] in, and This indicates the stable anchor point identifiers before and after the corresponding local analysis interval. This indicates the boundary path identifier currently corresponding to this message. This indicates the current path support status.

[0094] The path support status includes at least primary path support, candidate path support, path ambiguity, and no valid boundary evidence. A message that fully supports the current primary boundary path is recorded as having primary path support; a message that forms an unstable alternative boundary structure is recorded as having candidate path support; a message that cannot rule out two paths due to missing key boundaries is recorded as having path ambiguity; and a message that cannot form a valid structure judgment due to insufficient length, truncation, or insufficient local information is recorded as having no valid boundary evidence.

[0095] The local path state generated by a pending member is initially saved as a temporary state. Once the message becomes a confirmed member or its alternative path evidence forms a stable repetition relationship with other messages, the corresponding temporary state becomes a valid state. When a message is subsequently reassigned to another message cluster, its temporary evidence in the original cluster is revoked to prevent incorrect assignment from remaining as valid boundary evidence in the original message cluster.

[0096] When a member message participates in subsequent format state updates, its local path state record is updated synchronously. If a message was initially recorded as having an ambiguous path, but as the boundary path state converges and clearly supports the main path, its corresponding state is changed to "main path supported." If a message subsequently forms a stable candidate alternative path with other members, its state is updated to "candidate path supported." Thus, the local path state record can preserve the sources of uncertainty that historical members generated within different stable anchor point intervals.

[0097] When a pair of stable anchor points After a competing boundary path is confirmed and a local cluster reconstruction occurs, the pending message backtracking unit retrieves historical members related to the current boundary divergence interval based on the local path status record. Historical members that have been marked as stable main path supporters in this interval and have never formed a critical boundary conflict do not participate in backtracking; historical messages that were previously recorded as candidate path supporters, had ambiguous paths, lacked valid boundary evidence, or had missing critical boundaries of the main path in this anchor point interval are included in the current backtracking set.

[0098] The system does not add a message to the current backtracking set simply because it exists in an uncertain state in other local intervals. Even if a historical message has a low overall confidence level, if its uncertainty stems from a field region between another pair of stable anchors, and it consistently supports the main path within the current divergence interval, it will not participate in this local reconstruction. This ensures that the historical backtracking range corresponds to the actual local structure where format divergence occurred.

[0099] Historical messages entering the backtracking set are only re-evaluated for local structure within the current boundary divergence interval; the entire message format inference is not re-executed. Messages that fully support the original main boundary path remain in the original message cluster; messages that fully support competing boundary paths migrate to the new message cluster; and messages that still cannot form sufficient evidence of local structure remain as fuzzy members.

[0100] For example, the system currently maintains a message cluster containing 68 acknowledgment members. This message cluster has formed a stable structure around bytes 2, 4, 6, and 18, and a main boundary path is formed between bytes 6 and 18.

[0101]

[0102] Subsequent incoming messages consistently supported the two stable anchor points at bytes 6 and 18, but after completing the displacement compensation for the existing variable-length fields, they still could not support the internal boundary at byte 12, and repeatedly formed internal boundaries near bytes 8 and 14, gradually forming candidate alternative paths:

[0103]

[0104] Initially, the first one or two messages exhibiting this structure only generate temporary candidate path evidence, without directly altering the message cluster structure. As subsequent unconfirmed messages repeatedly form the same local boundary paths, and these messages can form stable field structures with each other, the corresponding temporary evidence transforms into valid supporting evidence for candidate alternative paths, thus solidifying the candidate alternative paths. The set gradually takes shape.

[0105] support The confirmed members consistently form stable field separators around byte 12, but even after completing the same local alignment and length compensation, a complete separator relationship between bytes 8 and 14 cannot be formed; Support The confirmed members consistently fail to form a single internal boundary at byte 12, and instead form two stable structural transitions near bytes 8 and 14 respectively. Some messages, due to insufficient length or unusual local field values, cannot reliably distinguish between the two paths, and these messages enter... The number of valid supports for any boundary path is not included.

[0106] The system continues to check whether candidate alternative paths can be explained by variable-length fields in the original main path. Assume there is a variable-length field between bytes 6 and 12 in the original message; increasing the length of this field can cause subsequent boundary positions to shift synchronously. The system compensates for the relevant subsequent boundaries in the new message according to the actual change in the length of this field. After compensation, if the new message only shows the 12th byte boundary returning to its corresponding reference position, it is still classified as part of the original main path; however, if the current new message, after compensation, still stably forms two independent internal boundaries, bytes 8 and 14, and independent and repeating field change relationships exist between bytes 8 and 14, and between bytes 14 and 18, respectively, it cannot be explained by the length shift of the original single 12th byte boundary.

[0107] The number of valid supporting members for candidate alternative paths reaches a set minimum, remains stable within a continuously increasing message window, and forms a consistent local structure after being divided according to the 8th and 14th byte boundaries. and Most of the valid members in the group reject the other path, and The proportion did not exceed the allowed ambiguity range, thus the candidate alternative path was converted into a competitive boundary path, and format differentiation was confirmed in the 6th to 18th byte interval.

[0108] Local cluster reconstruction units will continue to support of Members will remain in the original message set and will continue to provide support. of Members form a new message cluster. The format state, which is stable before byte 6 and after byte 18 and supported by both groups of members, is inherited by both message clusters. Members are not currently required to be assigned to any message cluster.

[0109] The system then reads historical local path status records, retrieving only historical messages that were marked as candidate path support, path ambiguity, lack of valid boundary evidence, or missing critical boundary at byte 12 in the 6th to 18th byte range. Assuming there are 7 historical messages with local uncertainty in the original message cluster, with 5 of them having uncertainty in the 6th to 18th byte range and the other 2 having uncertainty in other fields after byte 18, then only the aforementioned 5 messages will be backtracked in this instance.

[0110] The aforementioned five historical messages were re-compared with the two now-stable local boundary paths. This ensures complete support. The historical messages have been migrated to the new message set, which can fully support... Historical messages are retained in the original message cluster, and messages that still cannot form reliable evidence of local boundaries remain ambiguous. Two other historical messages whose uncertainties are unrelated to the current boundary divergence interval are not included in this processing.

[0111] After the partial reconstruction is completed, the two message clusters will use their respective local main boundary paths for the assignment of subsequent new messages. Even if subsequent messages continue to share the same common message header, their format assignment can be determined based on which boundary path they can support after length compensation and local alignment in the 6th to 18th byte range.

[0112] If subsequent new messages only change the internal values ​​of existing fields or expand the length range of a confirmed variable-length field in the main boundary path, then the corresponding field status and allowed length range are updated, without changing the current message cluster structure. If subsequent new messages continuously form a candidate alternative path between another pair of stable anchors that cannot be explained by existing variable-length fields and forms a stable structure that excludes the current main path, then only the boundary path verification and local cluster reconstruction are repeated for the new local analysis interval, without recalculating other local intervals that have reached a stable state.

[0113] After the stable anchor points and main boundary paths of each message cluster gradually converge, the format inference unit divides the field regions based on the boundary positions confirmed by field boundary evidence. Stable anchor points that only serve a local structural location function and have not yet been confirmed as field separator positions are not directly output as the final field boundaries. Fixed field candidates are formed when the field length is fixed and the byte value remains stable; fixed-length variable field candidates are formed when the field length is fixed but the value varies in different messages; length field and variable-length field candidates are formed when the field length varies with the message and maintains a stable correspondence with the length values ​​of other fields; and check field candidates are formed when the fields at the end of the message change with the content of the preceding message and maintain a stable check relationship.

[0114] Candidate boundary paths that have not yet reached stability are kept in the format state of the corresponding message cluster and are not forced as the final field partitioning result. Through the above system, the update objects of incremental clustering include not only message membership relationships, but also stable anchors bound to message clusters, boundary paths between anchors, and the path support state of members in each local interval. New messages use existing format states to determine candidate affiliation, and their temporary structural evidence only participates in stable state updates after member confirmation is completed; boundary path judgment first performs known variable-length field displacement compensation, and then performs path structure comparison; path mutual exclusion judgment only relies on valid members that can clearly support different paths, and does not mistakenly regard ambiguous members with insufficient evidence as supporting two paths simultaneously; when two boundary paths that are continuously supported, cannot be explained by the same variable-length structure, and are mutually exclusive are formed between the same pair of stable anchors, the system confirms the hidden format differentiation within the scope of this local interval, and only reconstructs members that support different paths, while using pre-saved local path state records to locate historical messages that need to be re-judged, so that the format inference results continuously correct the incremental clustering results in reverse, while avoiding reprocessing all historical messages and all stable field regions due to changes in local field structure.

[0115] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A system for inferring the format of an unknown protocol message based on incremental clustering, characterized in that, It includes a message cluster status management unit, an incremental attribution unit, a field boundary evidence update unit, a boundary divergence identification unit, a local cluster reconstruction unit, and a format inference unit; The message cluster state management unit is used to maintain multiple unknown protocol message clusters and maintain stable anchor points for each message cluster as well as the main boundary path between adjacent stable anchor points. The incremental attribution unit is used to determine the candidate attribution cluster of the new message based on the structural compatibility relationship between the new message and the stable anchor point and main boundary path of each message cluster. The field boundary evidence update unit is used to obtain the boundary path formed between adjacent stable anchor points of the candidate belonging cluster by the newly added message. The boundary divergence identification unit is used to identify the boundary path as a candidate alternative boundary path when the boundary path formed by the newly added message cannot be explained by the length change of the main boundary path and its existing variable length field, and to identify the candidate alternative boundary path as a competing boundary path when the candidate alternative boundary path between the same pair of stable anchors continuously obtains multiple message support and forms structural exclusion with the main boundary path, and to determine that the corresponding message cluster has undergone local format differentiation. The local cluster reconstruction unit is used to perform local reconstruction of the packet clusters that have undergone local format differentiation based on the packets that support the main boundary path and the competing boundary path respectively, and to maintain the boundary path between the stable anchor point for each reconstructed packet cluster. The format inference unit is used to generate the corresponding unknown protocol message format based on the stable anchor point and boundary path of each message cluster after reconstruction.

2. The unknown protocol message format inference system based on incremental clustering as described in claim 1, characterized in that, The message cluster state management unit is used to perform coarse-grained structure alignment on messages within the same message cluster, determine stable anchor points based on the repeated support of candidate structure positions by multiple messages, and limit the local analysis interval using adjacent stable anchor points.

3. The unknown protocol message format inference system based on incremental clustering as described in claim 2, characterized in that, The field boundary evidence update unit is used to determine the internal field boundary between adjacent stable anchor points and form a boundary path according to the order of the internal field boundaries. When comparing boundary paths formed by different messages, it is determined whether the boundary paths belong to the same path based on the number of internal field boundaries, their arrangement order, their positional relationship after local alignment, and the relationship of field length changes.

4. The unknown protocol message format inference system based on incremental clustering as described in claim 3, characterized in that, The incremental attribution unit is used to mark newly added messages that form a structurally compatible relationship with the candidate attribution cluster as members to be confirmed; The field boundary evidence update unit records the boundary path generated by the member to be confirmed as temporary path evidence. When the member to be confirmed can support the current main boundary path and has a higher degree of structural compatibility with the current message cluster than other message clusters, or when multiple members to be confirmed repeatedly support the same candidate alternative boundary path and form a consistent local structure, the corresponding temporary path evidence is converted into valid path evidence.

5. The unknown protocol message format inference system based on incremental clustering as described in claim 4, characterized in that, The boundary divergence identification unit is used to compensate for the displacement of the affected internal field boundaries based on the length change of the existing variable-length fields when there is a difference in the position of the internal field boundaries between the new message and the current main boundary path. If, after displacement compensation, the number of internal field boundaries, their arrangement order, or the field separation relationship still cannot match the main boundary path, the boundary path formed by the newly added message will be determined as a candidate alternative boundary path.

6. The unknown protocol message format inference system based on incremental clustering as described in claim 5, characterized in that, The boundary divergence identification unit is used to maintain the set of valid supporting members of the main boundary path, the set of valid supporting members of the candidate alternative boundary path, and the set of fuzzy path members, and to determine whether the candidate alternative boundary path satisfies the format differentiation condition based on the repeatability, persistence, internal consistency of supporting members, and structural exclusion relationship with the main boundary path of the candidate alternative boundary path.

7. The unknown protocol message format inference system based on incremental clustering as described in claim 6, characterized in that, The structural exclusion relationship is as follows: The valid supporting members of the primary boundary path cannot form the critical internal field boundaries in the candidate alternative boundary path, and the valid supporting members of the candidate alternative boundary path cannot form the critical internal field boundaries in the primary boundary path. The candidate alternative boundary path also satisfies that the number of internal field boundaries, their arrangement relationship, or their field organization relationship cannot be explained by the length changes of existing variable-length fields in the main boundary path.

8. The unknown protocol message format inference system based on incremental clustering as described in claim 7, characterized in that, The local cluster reconstruction unit is used to retain packets that continuously support the main boundary path in the original packet cluster, form a new packet cluster from packets that continuously support the competing boundary path, and enable the new packet cluster to inherit the stable anchor point and boundary path shared with the original packet cluster outside the boundary divergence interval, and maintain different main boundary paths respectively within the boundary divergence interval.

9. The unknown protocol message format inference system based on incremental clustering as described in claim 8, characterized in that, It also includes a pending message backtracking unit; The message cluster state management unit is used to save the local path state corresponding to each stable anchor point interval for member messages. The pending message backtracking unit is used to filter historical messages that have shown candidate path support, path ambiguity, lack of valid boundary evidence, or missing key internal field boundaries in the main boundary path within the boundary divergence interval after local cluster reconstruction, based on the local path status, and to re-determine the message cluster affiliation of the historical messages only for the boundary divergence interval.

10. The unknown protocol message format inference system based on incremental clustering as described in claim 9, characterized in that, The format inference unit is used to divide field regions based on stable field boundaries confirmed by field boundary evidence in each reconstructed message cluster, and to determine at least one field attribute among fixed fields, fixed-length variable fields, length fields, variable-length fields and check fields based on the length changes, byte value changes and inter-field relationships of each field region.

Images