Binary protocol message clustering method and system based on regional division keyword positioning
By identifying fixed and non-fixed offset regions in binary protocols and combining dual probabilistic constraints to infer keyword fields, this method solves the problem of insufficient clustering accuracy in binary protocols by existing methods, and achieves more efficient and accurate protocol message clustering.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Chinese People's Liberation Army Cyberspace Force Information Engineering University
- Filing Date
- 2025-12-10
- Publication Date
- 2026-05-08
AI Technical Summary
Existing binary protocol reverse clustering methods lack accuracy and versatility when dealing with variable-length fields, non-fixed offsets, and flexible value spaces. In particular, when there are structural differences between fixed and non-fixed offset regions in binary protocols, existing methods are prone to introducing errors and redundancy, leading to a decrease in clustering accuracy.
By identifying fixed and non-fixed offset regions in binary protocol messages, semantic detection rules are used for segmentation, and keyword fields are inferred by combining dual probabilistic constraints. A two-stage method combining clustering constraints and self-constraints is used to generate high-precision keyword fields for clustering.
It effectively reduces clustering bias caused by alignment errors, lowers redundant calculations and misjudgment rate, and improves the accuracy and efficiency of clustering, especially performing excellently when dealing with complex binary protocols.
Smart Images
Figure CN121997084A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of binary protocol reverse analysis technology, and in particular to a binary protocol message clustering method and system based on region partitioning keyword positioning. Background Technology
[0002] In existing protocol reverse engineering methods based on network traffic, message type clustering is a core foundational task. Correct clustering provides a prerequisite for subsequent message structure analysis and semantic inference. Currently, mainstream clustering methods mainly fall into the following three categories: (1) Alignment-based clustering methods. These methods draw on multiple sequence alignment (MSA) techniques in bioinformatics, calculating similarity scores between messages through byte- or character-level alignment, thereby constructing a similarity matrix and completing clustering. Typical examples include PI (Beddoe, 2004), PEXT (Shevrtalov et al., 2007), Netzob (Bossert et al., 2014), and ScriptGen (Leita et al., 2005). The basic assumption of these methods is that messages of the same type should have high similarity in field values. However, this assumption does not always hold true in binary protocols, especially when field offsets are not fixed or when variable-length fields exist. In addition, the MSA algorithm introduces redundant gaps, causing the loss of key field position information in fixed offset regions (such as message headers), thus affecting the clustering accuracy. On the other hand, MSA has a high time complexity (e.g., the optimized progressive alignment algorithm has a complexity of O(K). 2 *n 2 The computational cost is significant when dealing with large-scale datasets.
[0003] (2) Token-based clustering methods. These methods rely on predefined delimiters or n-grams to segment messages into tokens and cluster them based on frequently occurring token values. Typical tools include Discover (Cui et al., 2007), ProDecoder (Wang et al., 2012), and Veritas (Wang et al., 2011). Although these methods perform well in text protocols, binary protocols lack explicit syntactic structures and have highly flexible value spaces, posing significant challenges to their accuracy and time complexity. Especially when dealing with binary data without delimiters, the ambiguity of token partitioning further reduces the clustering effect.
[0004] (3) Keyword-based clustering methods. These methods generate candidate keyword fields by aligning messages and derive keywords using probabilistic constraints to achieve clustering, revealing the root cause of message format differences. Representative works include Netplier (Ye et al., 2021) and MDIplier (Liang et al., 2024). Although these methods have theoretical advantages, their implementation still heavily relies on global or local MSA for alignment, failing to fully consider the structural differences between fixed offset regions (FOR) and non-fixed offset regions (NFOR) in binary messages. For example, Netplier and MDIplier may disrupt the fixed offset characteristics of keyword fields during alignment by inserting redundant gaps, leading to incorrect clustering, such as... Figure 1 As shown, MSA disrupts the fixed offset attribute of the "function code" field during alignment, causing it to be unable to be correctly identified, thus leading to incorrect clustering (such as...). Figure 2 (b) shown).
[0005] Furthermore, while existing methods such as ProInfer (Guo et al., 2025) attempt to infer keywords through statistical probability, their field partitioning strategies are uncertain, and they assume that all keywords have fixed offset characteristics, making them ineffective in scenarios where keywords are located in NFOR (such as the DHCP protocol). On the other hand, while deep learning methods (such as CNNPre and LSTM-FCN) perform well in some scenarios, their applicability in reverse engineering of unknown protocols is limited due to issues such as the flexibility of binary protocol design and high computational overhead. Summary of the Invention
[0006] To address the shortcomings of existing protocol reverse clustering methods, which rely on specific structural assumptions and exhibit insufficient versatility and decreased accuracy when dealing with variable-length fields, non-fixed offsets, and flexible value spaces common in binary protocols, this invention proposes a binary protocol message clustering method and system based on region partitioning and keyword localization. By identifying the fixed offset region (FOR) and the non-fixed offset region (NFOR) in the message, candidate keyword fields are generated by partitioning, and high-precision binary protocol message clustering is achieved by combining dual probabilistic constraint inference.
[0007] In a first aspect, the present invention provides a binary protocol message clustering method based on region partitioning keyword location, comprising: Step 1: Extract application layer data from network traffic and preprocess it to obtain a set of binary protocol messages; Step 2: Based on semantic detection rules, horizontally divide the binary protocol message structure in the binary protocol message set to determine the fixed offset region set and the non-fixed offset region set; Step 3: Extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set using different strategies to obtain a candidate set of keyword fields; Step 4: By combining clustering constraints and self-constraints, a two-stage inference is performed on the candidate set of keyword fields to obtain the true keyword fields; Step 5: Cluster the binary protocol message set based on the real keyword field value.
[0008] Furthermore, in step 2, the semantic detection rules include constant field detection rules, serial number detection rules, timestamp detection rules, sparse value detection rules, address detection rules, and checksum detection rules.
[0009] Furthermore, step 2 specifically includes: The minimum message length of the binary protocol message set is calculated as the scanning range; Traverse the offset positions within the scanning range of each binary protocol message; Collect message fragments at the current offset position of each binary protocol message. If any message fragment meets one of the rules in the semantic detection rules, add the right boundary of the message fragment to the hit record offset set and detect the next offset position. If none of the message fragments at the current offset position of each binary protocol message meet the semantic detection rules, then detect the next offset position. After traversing the offset positions within the scanning range of each binary protocol message, the maximum value in the set of hit record offsets is taken as the final boundary. Based on the final boundary, each binary protocol message is divided into a fixed offset region and a non-fixed offset region, resulting in a set of fixed offset regions and a set of non-fixed offset regions.
[0010] Furthermore, step 3 specifically includes: for a fixed offset region, extracting candidate keyword fields by excluding fields whose semantic features do not match the keyword fields; For non-fixed offset regions, the TLV structure pattern is found using pattern matching methods to generate candidate keyword fields.
[0011] Furthermore, the extraction of candidate keyword fields by excluding fields whose semantic features do not match the keyword fields for a fixed offset region specifically includes: Identify semantic regions within the fixed offset region that conform to the semantic detection rules; Delete the sparse value field region in the semantic region to obtain the highly variable field region; The high-variability field region in the fixed offset region is deleted to obtain the undetected region; The undetected region is merged with the sparse value field region to generate a candidate scan sequence; Traverse the offset positions in the candidate scan sequence, filter out the offset positions that meet the preset candidate length through modulo operation, and verify the continuity of the filtered offset positions. If the verification is successful, use the field corresponding to the current offset position as the candidate keyword field.
[0012] Furthermore, for non-fixed offset regions, the method of searching for TLV structure patterns using pattern matching to generate candidate keyword fields specifically includes: The minimum message length in the set of non-fixed offset regions is taken as the maximum compatible processing length; For each message in the set of non-fixed offset regions, within the maximum compatible processing length of each message, the sliding window method is used to gradually scan and extract the type T field and length L field of the TLV structure; Based on the T field and the L field, verify whether the TLV structure is complete and the encoding is valid. If the verification is successful, mark the T field as a candidate keyword field.
[0013] Furthermore, in step 4, the first stage inference includes: calculating the posterior probability of each candidate keyword field based on clustering constraints, and sorting the candidate keyword fields from high to low according to the posterior probability; wherein the clustering constraints include message similarity constraints, remote coupling constraints, structural consistency constraints, and dimensional constraints.
[0014] Furthermore, in step 4, the self-constraint includes bit usage constraints and position constraints; The second stage of inference includes: calculating the keyword field probability of the top N candidate keyword fields in the first stage of inference ranking results based on bit usage constraints and position constraints, and selecting the keyword field with the highest keyword field probability as the real keyword field; The formula for calculating the probability of the keyword field is as follows: In the formula, in, This indicates the candidate keyword field. This indicates that the candidate keyword field is the actual keyword field. The indication bit uses constrained probability. Represents the probability of positional constraints. Represents the posterior probability. express, express.
[0015] Furthermore, the bit usage constraint probability is calculated using the following formula: In the formula, in, Represents Euclidean distance. Indicates the maximum distance. Indicates the candidate keyword field number The probability of bit usage. Indicates the candidate keyword field number The theoretical probability of using a bit. Indicates the most significant bit of the candidate keyword field. This indicates the use of a valid bit with a probability of 1; the position constraint probability is calculated using the following formula: in, This indicates the offset of the candidate keyword field.
[0016] Secondly, the present invention provides a binary protocol message clustering system based on region partitioning keyword positioning, comprising: The preprocessing module is used to extract application layer data from network traffic for preprocessing to obtain a set of binary protocol messages; The region partitioning module is used to horizontally partition the binary protocol message structure in the binary protocol message set based on semantic detection rules, and determine the fixed offset region set and the non-fixed offset region set. The candidate keyword field generation module is used to extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set using different strategies, so as to obtain a candidate set of keyword fields; The true keyword field generation module is used to perform a two-stage inference on the candidate set of keyword fields by combining clustering constraints and self-constraints to obtain the true keyword fields. The binary protocol message clustering module is used to cluster the binary protocol message set based on the real keyword field value.
[0017] The beneficial effects of this invention are as follows: The binary protocol message clustering method provided by this invention reduces clustering bias caused by alignment errors by combining a semantic rule base to identify the boundaries between fixed and non-fixed offset regions of binary protocol messages. It also reduces redundant calculations and avoids the high computational overhead of MSA by extracting candidate keyword fields for different regions. Furthermore, it effectively reduces the misjudgment rate of keyword fields by selecting candidate keywords through clustering constraints and self-constraints. Attached Figure Description
[0018] Figure 1 This is a diagram illustrating the impact of MSA on keyword recognition. Figure 2 Figure (a) shows the correct clustering result based on the function code, and Figure (b) shows the incorrect clustering result based on the length. Figure 3 This is a flowchart illustrating a binary protocol message clustering method based on region partitioning and keyword location. Figure 4 A schematic diagram of fixed offset region FOR and non-fixed offset region NFOR; Figure 5 A flowchart illustrating the method for dividing fixed offset regions into non-fixed offset regions; Figure 6 A flowchart illustrating the process of extracting candidate keyword fields from a fixed offset region; Figure 7 A flowchart illustrating the process of extracting candidate keyword fields from non-fixed offset regions; Figure 8 To utilize the method provided by the present invention for Figure 4 A schematic diagram illustrating the analysis results of the example; Figure 9 This is an architecture diagram of a binary protocol message clustering system based on region partitioning and keyword location. Figure 10 A graph showing the result of generating keyword candidate numbers using four methods. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0020] like Figure 3 As shown in the figure, an embodiment of the present invention provides a binary protocol message clustering method based on region partitioning keyword location, comprising: Step 1: Extract application layer data from network traffic and preprocess it to obtain a set of binary protocol messages.
[0021] Specifically, the binary protocol message set consists of pcap packets captured by tcpdump or tshark analyzers. These packets contain complete protocol layer data, while unknown protocols requiring clustering reside at the application layer. During the preprocessing stage, application layer data is extracted from the pcap packets. Furthermore, to obtain more information on the interactions of unknown protocols, information such as source IP address, destination IP address, source port, destination port, and timestamps are extracted from the network and transport layers to classify the messages. After preprocessing, the binary protocol data and their interaction information are obtained.
[0022] Step 2: Based on semantic detection rules, horizontally divide the binary protocol message structure in the binary protocol message set to determine the fixed offset region FOR set and the non-fixed offset region NFOR set.
[0023] Specifically, such as Figure 4 As shown, the structure of binary protocol messages typically follows a "FOR followed by NFOR" pattern. FOR is the fixed offset region at the beginning of the message (such as the protocol header), and NFOR is the subsequent non-fixed offset region (such as variable-length options). That is, the keyword fields of binary protocol messages exhibit different characteristics in the FOR and NFOR regions. To effectively utilize these characteristics to generate high-quality keyword candidate fields, this step requires accurately identifying the FOR-NFOR boundary and using this boundary to horizontally divide the binary protocol message, implementing different keyword field generation strategies in these two regions.
[0024] Step 3: Use different strategies to extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set respectively, to obtain the keyword field candidate set.
[0025] Specifically, fixed offset regions and non-fixed offset regions exhibit different characteristics. After dividing the regions, different strategies are needed to extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set respectively.
[0026] Step 4: By combining clustering constraints and self-constraints, a two-stage inference is performed on the candidate set of keyword fields to obtain the true keyword fields.
[0027] Specifically, the candidate keyword fields extracted in step 3 are merged into a complete candidate keyword field list. A two-stage probabilistic inference is used to combine the clustering constraints and self-constraints of the keyword fields to fit the characteristics of the keyword fields. The self-constraints include bit usage constraints and position constraints.
[0028] Step 5: Cluster the binary protocol message set based on the actual keyword field value.
[0029] The binary protocol message clustering method provided in this invention divides the binary protocol message into regions, extracts candidate keyword fields using different strategies based on the different offset distribution characteristics of different regions, performs efficient and accurate two-stage inference on the candidate keyword fields to obtain the real keyword fields, and uses the real keyword fields to cluster the binary protocol message.
[0030] Based on the above embodiments, this invention provides specific steps for dividing the binary protocol message structure.
[0031] First, the parameters and concepts involved in the embodiments of the present invention will be explained.
[0032] Basic symbols and definitions: message sets Each message Treated as a byte array. Field Represented as a closed interval ,in It is the starting offset (based on 0). It is the end offset (satisfying) ).
[0033] Fixed offset field: ,if It is a field A fixed offset, then It is a fixed offset field.
[0034] area: It is a fixed offset field that satisfies the arrangement of consecutive fields. initial position .
[0035] The NFOR section is the part of the message that does not include the FOR section.
[0036] Semantic detection rules: Each rule In semantic rule base Used to detect message sets In offset interval The semantics within, where This is the starting offset. This is the ending offset.
[0037] Detection function: for rules The detection function is CheckRule. ,in This function determines Do all message fragments satisfy the rules? The corresponding semantics are determined, and either True or False is returned.
[0038] Boundary identification theorem: set up .
[0039] if Then FOR-NFOR boundary Otherwise, .
[0040] According to the boundary identification theorem, the prerequisite for identifying the FOR-NFOR boundary is the accurate identification of explicit semantics. Semantic detection rules include constant field detection rules, sequence number detection rules, timestamp detection rules, sparse value detection rules, address detection rules, and checksum detection rules.
[0041] Constant fields: Represent fixed-value fields with zero entropy and consistency across messages (e.g., protocol version number).
[0042] in, represents Shannon entropy, and ≡ represents cross-message equivalence. Its input is a byte slice of arbitrary length, and the output is an offset and a fixed value in hexadecimal format.
[0043] Serial number: Detects a fixed-step increment counter with no value wrapping (e.g., transaction ID).
[0044] in, It is step size and It is a byte length (1-4). Input: A 1-4 byte slice. Output: Step, type label, offset.
[0045] Timestamp: Verifies the time encoding within the capture window (supports time zone offset).
[0046] in and Input: A 4 / 8-byte slice plus the capture time range. Output: Offset and encoding specification.
[0047] Sparse values: Identify non-zero fields that are not being fully utilized (e.g., protocol flags).
[0048] Input: A 1-2 byte slice. Output: Offset and utilization.
[0049] Address: Detect client-server address pairs using field alternation patterns.
[0050] in, It is the cross-correlation coefficient with a threshold of -0.8. Input: Adjacent 2-4 byte slices. Output: Offset and field width.
[0051] Checksum: Verification algorithm checksum field (e.g., CRC, XOR).
[0052] in, Includes algorithms such as CRC-16 and XOR-8, and This is the range of message data. Input: A 1-4 byte slice plus the data range. Output: Algorithm name and offset.
[0053] like Figure 5 As shown, step 2 specifically includes: Step 2.1: Calculate the minimum message length of the binary protocol message set as the scan range; Specifically, calculating the minimum message length Establish an effective scanning range.
[0054] Step 2.2: Traverse the offset positions within the scanning range of each binary protocol message; Specifically, the outer loop iterates through each byte offset. .
[0055] Step 2.3: Collect message fragments at the current offset position of each binary protocol message. If any message fragment meets one of the semantic detection rules, add the right boundary of the message fragment to the set of hit record offsets and detect the next offset position. If none of the message fragments at the current offset position of each binary protocol message meet the semantic detection rules, then detect the next offset position. Specifically, the inner loop applies all semantic rules. Perform multidimensional validation. For each Yes, the algorithm dynamically collects message fragments. ,in This is expressed as the rule-specific field length, and then validated. To satisfy the constraints. After successful verification, the right boundary... Recorded in In the middle, and immediately terminate the loop to force the first match priority.
[0056] Step 2.4: After traversing the offset positions within the scanning range of each binary protocol message, take the maximum value in the set of hit record offsets as the final boundary; Specifically, when the NFLO region begins after the last valid FOR field, the final boundary is calculated as follows: the NFLO region in At that time, the value of the NOR region is: .
[0057] Step 2.5: Divide each binary protocol message into fixed offset region and non-fixed offset region according to the final boundary to obtain a set of fixed offset regions and a set of non-fixed offset regions.
[0058] It is understood that by dividing the region into fixed offset and non-fixed offset regions through this embodiment, the final goal can be achieved. Time complexity, where .
[0059] Based on the above embodiments, this invention provides specific implementation methods for different strategies of extracting candidate keyword fields for fixed offset regions and non-fixed offset regions.
[0060] like Figure 6 As shown, for a fixed offset region, candidate keyword fields are extracted by excluding fields whose semantic features do not match the keyword fields. Specifically, this includes: Step 3.11: Identify semantic regions within the fixed offset region that conform to the semantic detection rules; Specifically, this involves identifying semantic regions that conform to semantic detection rules. (Non-keyword fields, such as timestamps).
[0061] Step 3.12: Delete the sparse value field region in the semantic region to obtain the highly variable field region; Specifically, sparse value fields From semantic regions ( Figure 6 Excluding from (as shown in Figure 3), we obtain the high-variability field region. .
[0062] Step 3.13: Delete the high-variance field regions in the fixed offset region to obtain the undetected region; Specifically, the high-variability field region From fixed offset area Delete and construct undetected areas. ( Figure 6 (As shown in Bank of China 4).
[0063] Step 3.14: Merge the undetected regions with the sparse value field regions to generate candidate scan sequences; Specifically, undetected areas Merging with the sparse field initiation point to form a candidate scan sequence ( Figure 6 (As shown in Bank of China 5).
[0064] Step 3.15: Traverse the offset positions in the candidate scan sequence, filter out the offset positions that meet the preset candidate length through the modulo operation, and verify the continuity of the filtered offset positions. After the verification is passed, use the field corresponding to the current offset position as the candidate keyword field.
[0065] Specifically, traversing positions Through the modulus operation (mod) () Figure 6 Bank of China 8) Filter length-aligned offsets and use ISCONTINUOUS Function verification The continuity of candidates ensures the interval It contains valid protocol fields discovered by the semantic region. This method implements... The time complexity.
[0066] The generation of keyword candidates in NFOR differs from the generation of keyword field candidates in the FOR region, as fields in the NFOR region are not naturally aligned. Therefore, embodiments of this invention employ a dynamic method to find potential keyword field candidates. Due to the efficiency requirements of binary protocols, their data is generally compactly arranged. Because binary protocols lack explicit delimiters found in text-based protocols, variable-length fields in binary protocols require a specific pattern to indicate when the end occurs, and this pattern typically contains key information indicating the message type. For example, a common serialization pattern is the LV pattern, which uses the value in the L field to represent the length of the V field. Another common pattern is the TLV pattern, which provides specific type information about the value before the LV pattern. In the LV pattern, the V field is a variable-length field without type information and is therefore unsuitable as a keyword field. However, after the T and V are associated in the TLV pattern, they can be used as a whole to indicate a change in the value space of a specific type, and thus can be used as a keyword field candidate. Embodiments of this invention discover the TLV pattern in the NFOR region through pattern matching and use TV as a combination to extract candidate keyword fields. Figure 7 As shown, for non-fixed offset regions, the TLV structure pattern is searched using a pattern matching method to generate candidate keyword fields, specifically including: Step 3.21: Use the minimum message length in the set of non-fixed offset regions as the maximum compatible processing length; Specifically, calculate the maximum compatible processing length for all messages. ( Figure 7 Bank of China (4), to ensure global consistency.
[0067] Step 3.22: For each message in the non-fixed offset region set, within the maximum compatible processing length of each message, use the sliding window method to scan and extract the type T field and length L field of the TLV structure step by step; Specifically, for each message ( Figure 7 Bank of China (BOC) 5), using a sliding window method to progressively identify potential TLV structures, specifically, using the ParseBytes function to extract the type field ( Figure 7 Bank of China (8) and length field ( Figure 7 Bank of China (9).
[0068] Step 3.23: Verify whether the TLV structure is complete and the encoding is valid based on the T field and L field. If the verification is successful, mark the T field as a candidate keyword field.
[0069] Specifically, the data integrity constraints are first verified by adding the starting offset of the T field to the total length of the T field and the L field. Figure 7 Bank of China (10-12) ensures no field overflow, then passes through the ValidateTLV semantic checker ( Figure 7 Bank of China (14) confirms whether the TLV structure is a valid encoding. When a valid TLV structure is detected ( Figure 7 When in line 17, the TLV structure (i.e. Figure 7 In ) and the T field Recorded in collection ( Figure 7 The pattern segment in Bank of China 17) was used to initiate duplicate detection. Figure 7 (Bank of China 18-20) The boundary is expanded by continuously expanding backward. For repetition counts ≥ 1, the start and end positions are stored in a set. ( Figure 7 In Bank of China (24), the offset advancement adopts two modes: successful verification will trigger a jump to the new structural starting point ( Figure 7 Bank of China 24), while failures are handled using a single-byte step method ( Figure 7 Bank of China 26). Final output set It contains all detected TLV structure schemas and their type identifiers, and uses the T field as a candidate key field, while the collection... Repeatedly record the boundaries of the TLV structure sequence. Variable naming and subscripting. The symbols must be consistent.
[0070] Based on the above embodiments, this embodiment of the invention provides specific implementation steps for the two-stage inference of the candidate set of keyword fields by combining clustering constraints and self-constraints in step 4.
[0071] The first stage of inference includes: calculating the posterior probability of each candidate keyword field based on clustering constraints, and sorting the candidate keyword fields from high to low according to the posterior probability; where the clustering constraints (Ye, Y., Zhang, Z., Wang, F., Zhang, X., Xu, D., 2021. NetPlier: ProbabilisticNetwork Protocol ReverseEngineering from Message Traces, in: Proceedings 2021 Network and DistributedSystem Security Symposium, Internet Society, Virtual.URL) include message similarity constraints, remote coupling constraints, structural consistency constraints, and dimensional constraints.
[0072] Specifically, embodiments of the present invention will not use the keyword field ranked first in the first stage as the true keyword field, because simply relying on the posterior probability of clustering constraint features may still lead to incorrect keyword inference. For example, when using clustering constraints as Figure 1 When inferring keyword fields from messages, the constraint satisfaction based on the actual keyword field "functioncode" is lower than the constraint satisfaction based on the 3 offset bytes. This is an illusion based solely on clustering constraints. Keyword fields that appear to better fit the constraints are not actually the true keyword fields.
[0073] The second stage of inference includes: calculating the keyword field probability of the top N candidate keyword fields in the first stage inference ranking results based on bit usage constraints and position constraints, and selecting the keyword field with the highest keyword field probability as the real keyword field.
[0074] This invention models the uncertainty in keyword recognition as a joint distribution of observed and latent variables, where the observed variables are the probability of bit usage constraint and the probability of position constraint. and latent variables This indicates whether a candidate job title is a keyword field. The probability of clustering constraint inference is used as the prior probability, i.e. and .
[0075] Normalized inference. Based on the two constraints mentioned above, we set a Bit Usage factor and a position factor to measure the degree of matching between the candidate field and these two constraints.
[0076] Assume conditional independence of the observed factors: Substitute the prior The formula for calculating the probability of the keyword field is as follows: In the formula, in, This indicates the candidate keyword field. This indicates that the candidate keyword field is the actual keyword field. The indication bit uses constrained probability. Represents the probability of positional constraints. Represents the posterior probability. express, express.
[0077] Bit usage constraints. In binary protocols, the significant bits of a field appear more frequently in lower positions, similar to how shorter numbers are more common in decimal systems. As fields controlling message types, the space of significant values for keyword fields is typically limited in most protocols. Considering protocol scalability requirements, significant bits should be in lower positions of the byte value space to leave redundant space in higher positions. Specifically, for each candidate field, all its values are iterated and the position of the most significant bit (MSB) is determined. For example, if the candidate field... Given three binary values {00100100, 00000010, 00010000}, the first value, 00100100, effectively uses bits [0, 5] (from least significant bit 0 to most significant bit 5), while the other two values use bits [0, 1] and [0, 4] respectively. Therefore, the field is determined... The MSB is 5. Next, calculate each bit. (in The probability of use from 0 to MSB Defined as The proportion of values in the field. The probability of using the constraint is then calculated using the following formula: In the formula, in, Represents Euclidean distance. Indicates the maximum distance. Indicates the candidate keyword field number The probability of bit usage. Indicates the candidate keyword field number The theoretical probability of using a bit. Indicates the most significant bit of the candidate keyword field. This indicates that a valid bit with a probability of 1 is used. When = 0 (complete match): ;when (At maximum difference): .
[0078] Location Constraints. In binary protocols, enabling heterogeneous devices to process messages efficiently requires locating keyword fields as quickly as possible after receiving a message for subsequent message parsing. This consideration is also reflected in the location distribution of keyword fields. That is, in binary protocols, keywords are more likely to be located within a fixed FOR (Forward or Forward) sequence, and the offset of this position is generally small, allowing for rapid keyword identification. This embodiment of the invention utilizes this feature to further infer the true keyword. The location constraint probability is calculated using the following formula: in, This indicates the offset of the candidate keyword field.
[0079] like Figure 8 As shown, Figure 4 The clustering results in the example were generated by the method of this invention based on a 12-offset field. It can be understood that the second stage performs a new round of inference on the top-ranked domains from the first stage inference results to improve accuracy. To this end, embodiments of this invention introduce two observed self-constraining relationships: bit usage constraints and positional constraints.
[0080] like Figure 9 As shown, this embodiment of the invention also provides a binary protocol message clustering system based on region partitioning keyword location, including: The preprocessing module is used to extract application layer data from network traffic for preprocessing to obtain a set of binary protocol messages; The region partitioning module is used to horizontally partition the binary protocol message structure in the binary protocol message set based on semantic detection rules, and determine the fixed offset region set and the non-fixed offset region set. The candidate keyword field generation module is used to extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set using different strategies, so as to obtain a candidate set of keyword fields; The true keyword field generation module is used to perform a two-stage inference on the candidate set of keyword fields by combining clustering constraints and self-constraints to obtain the true keyword fields. The binary protocol message clustering module is used to cluster the binary protocol message set based on the real keyword field value.
[0081] To verify the effectiveness of the method provided by this invention, embodiments of this invention compare the method of this invention with state-of-the-art methods by generating candidate keyword fields, clustering, and performing performance comparisons with datasets of different sizes and protocols.
[0082] 1. Dataset The datasets used in this embodiment of the invention include multiple binary protocols. Each protocol consists of 1000 messages, and three different datasets with sizes of 100, 500, and 1000 were set up to evaluate the effectiveness of the method of the present invention. These include two industrial control system protocols, DNP3 and Modbus, and two typical proprietary protocols, Mavlink and HART. _ IP, along with six other common network protocols: NTP, SMB, SMB2, TFTP, BGP, and DHCP.
[0083] 2. Cluster Evaluation Evaluation Metrics. This embodiment of the invention uses homogeneity and completeness to evaluate the clustering results. Homogeneity means that all messages in each cluster belong to the same type of message, while completeness means that all messages of a given type are assigned to the same cluster. Homogeneity is expressed as... Completeness is represented as . Indicates the total number of messages. Representation type The number of messages, Represents a cluster The number of messages, Represents a cluster Medium type The number of messages, and the entropy of the actual type distribution are: Where |𝑇| represents the total number of true types. Given that the clustering results are known, the uncertainty of the message type is: The entropy of the true clustering, denoted as , and the conditional entropy of the clustering of a given type, denoted as , are defined symmetrically. Then, for... <s:1>The calculations for 𝑐are are as follows: when <s:1>When = 1, it represents complete homogeneity, meaning each cluster contains only one category. When = 1, it represents complete completeness, meaning the same category is assigned to the same cluster. To comprehensively consider the influence of these two scores, their harmonic mean, i.e., the v measure, is also introduced: Clustering Results. In the experiments, the method of this invention was compared with Netplier, MDIplier, Proinfer, and NEMETYL on eight protocols. On average, the method of this invention performed well, with uniformity of 0.959, completeness of 0.941, and v-measure of 0.949. In comparison, Netplier scores (0.700, 0.893, 0.761), Proinfer scores (0.934, 0.883, 0.904), and FieldHunter scores (0.784, 0.905, 0.810). The clustering results of different methods are shown below. Figure 10 As shown, the method of this invention correctly identified keywords in all seven protocols, thus achieving 100% in all three metrics. This is precisely the natural advantage of keyword-based clustering. An exception is the NTP protocol, where the three metrics of the method of this invention are 0.67, 0.53, and 0.59, respectively. The reason is that the method of this invention uses bytes as the smallest unit when identifying keywords, while the NTP protocol uses the 5th-7th bits of the first byte to represent information. This makes it difficult for the method of this invention to accurately locate it. However, the method of this invention still matches Netplier, MDIplier, and Proinfer in all three metrics and outperforms NEMETYL.
[0084] The other four methods perform similarly to the method of this invention on Modbus and SMB protocols, but still fall short on the other six protocols, particularly in terms of homogeneity. For example, in the SMB protocol, Netplier, MDIplier, and Proinfer score 0.28, while in the BGP protocol, NEMETYL scores 0.54. Netplier and MDIplier use MSA alignment, which generates too many keyword field candidates, introducing unnecessary interference to keyword probability inference. This results in low scores for their keyword-based clustering results. Proinfer performs simple candidate field filtering before inferring keywords, excluding sequence numbers and LV patterns, but still generates a relatively large number of candidate fields. Furthermore, Proinfer introduces a strong assumption that all keywords have a fixed offset, which does not align with the case of keywords located in the NFOR region in the DHCP protocol. NEMETYL employs a message similarity-based clustering method, which differs from keyword-based clustering methods, and therefore its clustering results are also unsatisfactory.
[0085] 3. Evaluation of FOR-NFOR boundary reasoning The segmentation between the FOR and NFOR regions directly affects the generation of candidate protocol keyword fields. To evaluate the semantic-based boundary recognition capability of the method of this invention, boundary inference was performed on datasets of 100, 500, and 1000 messages described above, respectively, to characterize the impact of data size on the results. Table 1 presents the detailed evaluation results.
[0086] Table 1. Boundary inference results of the method of the present invention on different datasets. The method of this invention can accurately infer the position of delimiters in Modbus, NTP, and BGP protocols. For DHCP and SMB2 protocols, the error values are +5 and +4 bytes, respectively. In different datasets for the SMB protocol, the error is ±3 bytes. For a dataset of 100 messages for DNP3 and TFTP, errors of +2 and +1 bytes occur, respectively. One reason for these errors is the limited diversity of the sample data. More importantly, the inference method aims to maximize the range of FOR to find more naturally aligned regions, as clearly described earlier.
[0087] While these minor errors may cause slight variations in the range of the FOR and NFOR regions, they do not affect the strategy for generating candidate fields in both regions. For example, fields in the For field identified in the DHCP protocol remain perfectly aligned, and the actual keyword field is still included in the keyword field candidates. Therefore, the delimiters identified in the eight protocols are considered valid within a tolerable error range.
[0088] 4. Evaluation of Keyword Reasoning Keyword-based protocol message clustering fundamentally relies on the accurate inference of true keywords. The constraints used for inference in the method of this invention are entirely derived from values in the message data. Therefore, the size and diversity of the available data directly affect the final result. To evaluate the ability of the method of this invention to infer keywords under different data scales, eight protocols were tested on datasets of 100, 500, and 1000 messages, respectively. The test results are shown in Table 2.
[0089] Table 2. Probability inference performance of the method of the present invention on different datasets. In Table 2, the first column represents the test protocol cases, the second column represents the number of messages, and the third column represents the ranking of the inference results. The ranking decreases from left to right, and the candidate probability for each field is listed (Client). _ (Server). The field index is in offset bytes. The first field is the keyword found using the method of this invention. The last column shows the position of the true keyword field as described in the official documentation. The results show that RP-KClust performs well on datasets of 500 and 1000 messages. The probability of the true keyword field ranks first among 7 protocols. On a dataset of 100 messages, the probability of the true keyword field ranks first among 4 protocols. This indicates that the number of messages directly affects the probability value of candidate fields, and when the probability rankings of candidate fields are close, it is easy to cause changes in the ranking. For example, in the SMB2 protocol, the true keyword is 16. _ 16 ranks second out of 100 messages, which is an incorrect result, but very close to first place. In 500 and 1000 messages, 16... _ Ranked first with 16, the probability value increased significantly. Similar situations occurred in the DNP3 and SMB protocols. These findings indicate that the inference accuracy of the method described in this invention will significantly improve when the dataset contains more valid information. This is a predictable result, as it conforms to the general rules of protocol reverse engineering based on network trajectories. It should be noted that because the method of this invention filters out fixed values when generating the FOR candidate field, and the TFTP protocol has a simple structure, only one true keyword field is generated as a candidate field for this protocol.
[0090] This embodiment compares and analyzes the method of the present invention with four previous studies using the same dataset. Considering accuracy and efficiency, comparative experiments were conducted using a dataset of 1000 messages containing more information. Due to the uncertainty of ProInfer's field segmentation method, the ProSeg method recommended in its paper was chosen. The final comparison results are shown in Table 3.
[0091] Table 3. Comparison of keywords inferred by different methods Through comparative analysis, the method of this invention can accurately identify the location of keyword fields in seven protocols, significantly outperforming other methods. Netplier and MDIplier only identify keyword fields in three protocols. Because they do not perform semantic filtering during candidate field generation, only considering the clustering constraints of keyword fields, they introduce excessive interference into keyword inference. When processing DNP3, they determine the length field, which is closely related to the correct clustering result, as the keyword field. Similarly, they make incorrect inferences when processing DHCP, BGP, and SMB2 protocols. ProInfer excludes the length field when generating keyword fields, correctly inferring the keyword for DNP3, but makes an error when inferring the DHCP protocol because its field segmentation method does not correctly delineate the boundaries of the keyword field. NEMETYL makes inference errors when processing DNP3, BGP, and SMB2 protocols. The processing steps and their inference methods heavily rely on sequence comparison results; therefore, when processing messages containing many fields with varying values, the inference results are uncertain, affecting the correctness of the results.
[0092] 5. Performance Evaluation The number of candidate fields generated directly affects the performance of the keyword inference stage. One of the main advantages of the method in this invention is that it can minimize the number of candidate fields without using a global multi-sequence alignment method. Figure 10 The table shows the number of keyword candidates generated by four methods: the method of this invention, Netplier, MDIplier, and ProInfer. Across eight protocols, the method of this invention generated an average of 9.625 candidate keywords. Netplier and MDIplier generated an average of 17.75 keywords, significantly more than the method of this invention. This is related to their conservative, unfiltered generation strategies. In the worst case, they generate candidate data byte-by-byte. ProInfer generated an average of 16.125 keywords, still significantly higher than the method of this invention. Although it employs a filtering strategy, it fails to incorporate rich semantics. Because NEETYL does not determine keywords through candidates, which is significantly different from other methods, it is not included in the performance evaluation.
[0093] The performance of the method of this invention was tested on a server with a 6-core CPU (Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz) and 64 GB of main memory. Considering that the MSA used by Netplier and MDIplier is significantly different from the method of this invention, and that the uncertainty of ProInfer's field segmentation method affects its performance, Netplier and MDIplier were used as a reference.
[0094] The average time and memory overhead of the method in this invention are 75.64 (1.64 + 74.02) seconds and 63.68 (9.77 + 53.91) MB, respectively. The average time and memory overhead of Netplier are 915.37 seconds and 527.54 MB, respectively, and that of MDIplier is 922.63 seconds and 528.17 MB, respectively. The overhead of the method in this invention is significantly lower than that of Netplier and MDIplier. The main reason is that the MSA algorithm requires comparing all messages at once, resulting in high time complexity. For messages of length *k*, using the optimized progressive multi-sequence alignment algorithm, the time complexity is *r(k / k). 2 ∗𝑛 2 When messages are long and vary significantly, the overhead is very high. For example, Netplier takes over 3939.58 seconds to process DHCP, while the method of this invention only takes 227(3.23+223.77) seconds. This demonstrates that the method of this invention has a significant performance advantage when processing large-scale datasets.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A binary protocol message clustering method based on region partitioning keyword location, characterized in that, include: Step 1: Extract application layer data from network traffic and preprocess it to obtain a set of binary protocol messages; Step 2: Based on semantic detection rules, horizontally divide the binary protocol message structure in the binary protocol message set to determine the fixed offset region set and the non-fixed offset region set; Step 3: Extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set using different strategies to obtain a candidate set of keyword fields; Step 4: By combining clustering constraints and self-constraints, a two-stage inference is performed on the candidate set of keyword fields to obtain the true keyword fields; Step 5: Cluster the binary protocol message set based on the real keyword field value.
2. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 1, characterized in that, In step 2, the semantic detection rules include constant field detection rules, serial number detection rules, timestamp detection rules, sparse value detection rules, address detection rules, and checksum detection rules.
3. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 2, characterized in that, Step 2 specifically includes: The minimum message length of the binary protocol message set is calculated as the scanning range; Traverse the offset positions within the scanning range of each binary protocol message; Collect message fragments at the current offset position of each binary protocol message. If any message fragment meets one of the rules in the semantic detection rules, add the right boundary of the message fragment to the hit record offset set and detect the next offset position. If none of the message fragments at the current offset position of each binary protocol message meet the semantic detection rules, then detect the next offset position. After traversing the offset positions within the scanning range of each binary protocol message, the maximum value in the set of hit record offsets is taken as the final boundary. Based on the final boundary, each binary protocol message is divided into a fixed offset region and a non-fixed offset region, resulting in a set of fixed offset regions and a set of non-fixed offset regions.
4. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 2, characterized in that, Step 3 specifically includes: for a fixed offset region, extracting candidate keyword fields by excluding fields whose semantic features do not match the keyword fields; For non-fixed offset regions, the TLV structure pattern is found using pattern matching methods to generate candidate keyword fields.
5. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 4, characterized in that, For a fixed offset region, the extraction of candidate keyword fields by excluding fields whose semantic features do not match the keyword fields specifically includes: Identify semantic regions within the fixed offset region that conform to the semantic detection rules; Delete the sparse value field region in the semantic region to obtain the highly variable field region; The high-variability field region in the fixed offset region is deleted to obtain the undetected region; The undetected region is merged with the sparse value field region to generate a candidate scan sequence; Traverse the offset positions in the candidate scan sequence, filter out the offset positions that meet the preset candidate length through modulo operation, and verify the continuity of the filtered offset positions. If the verification is successful, use the field corresponding to the current offset position as the candidate keyword field.
6. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 4, characterized in that, For non-fixed offset regions, the method of searching for TLV structure patterns using pattern matching to generate candidate keyword fields specifically includes: The minimum message length in the set of non-fixed offset regions is taken as the maximum compatible processing length; For each message in the set of non-fixed offset regions, within the maximum compatible processing length of each message, the sliding window method is used to gradually scan and extract the type T field and length L field of the TLV structure; Based on the T field and the L field, verify whether the TLV structure is complete and the encoding is valid. If the verification is successful, mark the T field as a candidate keyword field.
7. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 1, characterized in that, In step 4, the first stage of inference includes: calculating the posterior probability of each candidate keyword field based on clustering constraints, and sorting the candidate keyword fields from high to low according to the posterior probability; wherein the clustering constraints include message similarity constraints, remote coupling constraints, structural consistency constraints, and dimensional constraints.
8. The binary protocol message clustering method based on region partitioning keyword positioning according to claim 7, characterized in that, In step 4, the self-constraint includes bit usage constraints and position constraints; The second stage of inference includes: calculating the keyword field probability of the top N candidate keyword fields in the first stage of inference ranking results based on bit usage constraints and position constraints, and selecting the keyword field with the highest keyword field probability as the real keyword field; The formula for calculating the probability of the keyword field is as follows: In the formula, in, This indicates the candidate keyword field. This indicates that the candidate keyword field is the actual keyword field. The indication bit uses constrained probability. Represents the probability of positional constraints. Represents the posterior probability. express, express.
9. A binary protocol message clustering method based on region partitioning keyword location according to claim 8, characterized in that, The probability of using the bit constraint is calculated using the following formula: In the formula, in, Represents Euclidean distance. Indicates the maximum distance. Indicates the candidate keyword field number The probability of bit usage. Indicates the candidate keyword field number The theoretical probability of using a bit. Indicates the most significant bit of the candidate keyword field. This indicates the use of a valid bit with a probability of 1; the position constraint probability is calculated using the following formula: in, This indicates the offset of the candidate keyword field.
10. A binary protocol message clustering system based on region partitioning keyword positioning, characterized in that, include: The preprocessing module is used to extract application layer data from network traffic for preprocessing to obtain a set of binary protocol messages; The region partitioning module is used to horizontally partition the binary protocol message structure in the binary protocol message set based on semantic detection rules, and determine the fixed offset region set and the non-fixed offset region set. The candidate keyword field generation module is used to extract candidate keyword fields from the fixed offset region set and the non-fixed offset region set using different strategies, so as to obtain a candidate set of keyword fields; The true keyword field generation module is used to perform a two-stage inference on the candidate set of keyword fields by combining clustering constraints and self-constraints to obtain the true keyword fields. The binary protocol message clustering module is used to cluster the binary protocol message set based on the real keyword field value.