A protocol reverse parsing method based on multi-feature extraction and semantic simplification

Abstract

The present invention discloses a protocol reverse parsing method based on multi-feature extraction and semantic simplification, which classifies and extracts features in protocol messages and divides them into fixed features, changing features and associated features. On the basis of extracting these three types of features, a targeted semantic simplification strategy is further introduced. For fixed features, by merging similar semantic fields, duplicate information is reduced and the simplicity of feature expression is optimized. For changing features, a feature pattern processing strategy is adopted to uniformly summarize the changing rules of increasing and decreasing features, and separate the fixed part and the changing part of fixed-length optional features to ensure the integrity of the changing information while reducing dimensions and redundancy. For associated features, the logical relationship analysis between fields is used to dig out hidden field associations, thereby improving the expressiveness and accuracy of the feature library. Through this series of semantic simplification processes, the efficiency of feature extraction is significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of cyberspace security technology, and in particular to a protocol reverse parsing method based on multi-feature extraction and semantic simplification. Background Art

[0002] With the rapid development of network communication technologies, the structure and complexity of protocol messages are increasing, posing new challenges for applications such as network security, traffic monitoring, protocol analysis, and intrusion detection. Accurately parsing message field information is crucial for these applications, especially when dealing with unknown or private protocols. The key to message field parsing lies in analyzing message data to infer the meaning, location, length, and interrelationships of fields. This technology is particularly important in areas such as reverse engineering, network forensics, and abnormal traffic analysis. However, existing message parsing technologies have various limitations and shortcomings when dealing with modern complex protocols.

[0003] Currently, commonly used message parsing techniques include manual analysis, static analysis, dynamic analysis, and machine learning-based methods. Manual analysis relies on expert experience and knowledge, resulting in inefficiencies and error-proneness. This is particularly true when dealing with complex or large-scale messages, making it difficult to meet automation requirements. Static analysis is typically based on known protocol formats, making it less applicable to unknown or frequently changing protocols. Furthermore, static analysis cannot handle encrypted or obfuscated messages. While dynamic analysis can infer some field information by observing data changes during protocol interactions, its effectiveness is significantly reduced when message content is encrypted or obfuscated. Furthermore, dynamic analysis is susceptible to data noise when processing high-dimensional messages, leading to inaccurate results. For scenarios that rely on interactive behavior, the dynamic analysis capabilities of static messages are also very limited.

[0004] Machine learning-based parsing methods offer some promise for automation, but they typically require large amounts of labeled data for model training. This labeling process is time-consuming and subjective, which can affect the model's generalization. Furthermore, the complexity of deep learning models leads to high demands on computing resources, potentially creating performance bottlenecks in real-time traffic parsing. Furthermore, machine learning methods are prone to generating redundant and high-dimensional features when processing complex packets, further complicating parsing.

[0005] Existing feature extraction methods also have limitations when dealing with complex messages. The feature extraction process easily generates a large number of redundant features, resulting in excessively high data dimensionality and increasing the computational burden of the parsing process. This is particularly true when dealing with correlations between message fields, which lack effective automated simplification mechanisms. Furthermore, current feature simplification and merging methods often rely on manual adjustments, making it difficult to maintain parsing accuracy while improving efficiency.

[0006] In order to solve these shortcomings, a new message parsing method is urgently needed that can reduce feature redundancy, improve parsing accuracy and efficiency, and better adapt to the current complex network environment and protocol structure. Summary of the Invention

[0007] In order to solve the problems existing in the prior art, the present invention proposes a protocol reverse parsing method based on multi-feature extraction and semantic simplification.

[0008] The technical solutions of the present invention are as follows:

[0009] A protocol reverse parsing method based on multi-feature extraction and semantic simplification, comprising:

[0010] Step 1) Using a network analysis tool to read PCAP packets of a known protocol, extracting raw message data, dividing the raw message data into fields, identifying and extracting field information, and saving the fields and their corresponding values, positions, lengths, and semantic information to form protocol data;

[0011] Step 2) extracting fixed features, changing features, and associated features from the protocol data, and storing them in a fixed feature library for storing fixed features of fields, a changing feature library for storing changing features of fields, and an associated feature library for storing dependent features between fields, respectively;

[0012] Step 3) semantically simplifying the fixed features, changing features, and associated features, removing redundant information, and extracting discriminative semantic features;

[0013] Step 4) constructing a protocol feature template based on the semantically simplified semantic features; the protocol feature template includes a basic structure corresponding to the fixed features, a change range or rules corresponding to the changing features, and a derivation logic corresponding to the associated features;

[0014] Step 5) Based on the similarity matching result between the new message data and the protocol feature template, the new message data is identified and parsed, and the protocol feature template is dynamically updated.

[0015] Furthermore, the specific method of step 1) includes:

[0016] Step 1-1) Use a network analysis tool to read the pcap data packet of a known protocol, extract the original message data, and store the original message data as a raw byte stream;

[0017] Steps 1-2) Parse the original message data, dividing the fields in the message according to the protocol format. Based on the protocol documentation or known standards, identify the location, length, and type of each field and determine the field boundaries. For each field, read its data content and record the actual value, starting position, and length of the field in the data packet.

[0018] Steps 1-3) Extract the semantic information of each field based on the type, value, and position of the field according to the protocol document or known standards;

[0019] Steps 1-4) save the value, position, length, and semantic information of the extracted fields to form a complete field data record.

[0020] Furthermore, the fixed features in the fixed feature library are field attributes with fixed positions in the protocol message and values ​​that do not change with time or data packets, including version numbers and site identifiers.

[0021] Furthermore, the change features in the change feature library are field attributes in which the values ​​in the protocol message change with the data packet or time and have specific rules, including serial number, timestamp, IP address and MAC address.

[0022] Furthermore, the dependency features in the association feature library are field attributes in the protocol message that have a logical or statistical dependency relationship between fields, including checksum and data length.

[0023] Furthermore, the specific method of step 2) includes:

[0024] Step 2-1) For protocol data with clear fixed features, scan multiple data packets to confirm that the version number field in the fixed feature is consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of the field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value; for the site identifier in the fixed feature, match the field with the known site ID pattern or site code, locate and extract the field, and merge it into the feature vector f l =[FP,FL,FV,site ID];

[0025] Step 2-2) For protocol data with unclear fixed features, input multiple data packet samples and use the comprehensive indicators of fixed fields to I Determine the position of those fields in the protocol header that do not change with time, data flow, or packet content, and evaluate the consistency of the field by calculating the standard deviation of the field position. If the position of the field in different samples changes little, it is considered to be a fixed field, and the standard deviation of the field position is . The calculation formula is as follows:

[0026] ;

[0027] in, m is the number of fields,n is the sample size, FP ij It is in i In the data packet j The field value of each position, is the mean of the field positions, defining a consistency index C , used to quantify the consistency of fields, consistency indicators C The calculation formula is as follows: ;

[0028] in, is a small constant that prevents division by zero errors when the standard deviation is zero;

[0029] Step 2-3) Get the field content of the current message data position FP, and improve the search efficiency by using the bad character rule and good suffix rule; build a bad character table to record the last occurrence position of each character in the pattern; build a good suffix table to record the matching information of the suffix part of the pattern; calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of fixed fields I = w 1 C + w 2Pattern Score , w 1 and w 2 are the 1st and 2nd weight factors respectively;

[0030] Step 2-4) Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection; perform frequency analysis on the position of each field in the data packet, and set the first i The location is , the frequency of the field appearing at this position is ;like , then the field can be considered as a fixed position; i The value of the field v i Perform change detection and calculate its rate of change r ( v i ),like , then the field value is almost unchanged and can be determined as a fixed value; determine the field feature at the message data position FP as a fixed feature, and obtain the field length FL and field value FV at this position, expressed as a feature vector f i =[FP,FL,FV,Label] and stored in the fixed feature library DB f =[f v , f l ,…, f i ]middle;

[0031] Step 2-5) For the fields with clear change characteristics in the protocol, for the sequence number field, by detecting the continuous field values ​​in the data packet, find its increasing or decreasing pattern, extract the field and merge it into the feature vector d s =[FP, FL, FV, sequence number]; for the IP address and MAC address fields, locate these fields in the data packet through the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors respectively. d I =[FP,FL,FV,IP address] and d M =[FP,FL,FV,MAC address];

[0032] Step 2-6) For fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change pattern, including field value difference analysis and time series analysis; first, calculate the variance of the field value :

[0033] ;

[0034] in, FV ij It is i In the data packet j The field value of each position, is the average of the field values, m is the number of fields, n is the number of packets;

[0035] Step 2-7) Calculate the rate of change of field values ​​in adjacent packets , the calculation formula is as follows:

[0036] ;

[0037] in, FV ij It is i In the data packet j The field value of each position, FV (i+1)j It is i+1 In the data packet j The field value of each position;

[0038] Calculate the comprehensive index of difference V 1: ;

[0039] in, Indicates the difference in field values. The smaller the variance, the more stable the field. It is the average value of the rate of change, indicating the average change in the field value; s 1 and s 2 are the first and second weight factors respectively;

[0040] Step 2-8) For the sequence of field values, use the method of autocorrelation time series analysis to identify the pattern; the autocorrelation coefficient V 2 is calculated using the following formula:

[0041] ;

[0042] in, k is the lag step length, FV ij It is i In the data packet j The field value of each position, m is the number of fields, n is the number of packets;

[0043] Step 2-9) Combine the variance analysis and autocorrelation analysis of field values ​​to create a composite index V To evaluate whether a field is a fixed field:

[0044] ;

[0045] in, and are the 1st and 2nd weight factors respectively;

[0046] Determine the field value FV as the change feature value of the field, determine the field position FP and length FL based on this, and express the extracted potential change features as feature vectors d i =[FP,FL,FV,Label], and finally stored in the change feature library DB d =[ d s , d I , d M ,…, d i ]middle;

[0047] Step 2-10) For protocol data with clear correlation features, first analyze the field structure in the data packet to locate field pairs that may have dependencies; for the checksum field, extract the field and its corresponding data content segment, apply the CRC calculation algorithm and verify the match between the checksum field and the calculation result, and merge it into the feature vector For the data length field, locate the field that identifies the data length in the protocol, extract the field value and calculate the actual data length, verify the consistency of the length field value and the actual data length, and merge it into the feature vector middle;

[0048] Step 2-11) For protocol data with unclear correlation features, correlation analysis is used to explore its features. First, the covariance between protocol fields is calculated. Covariance can measure whether two fields have a linear correlation. When the covariance is close to zero, the two fields are independent. When the covariance is far from zero, it indicates that there is a correlation between the fields. Cov ( FV X , FV Y ) is calculated as follows:

[0049] ;

[0050] in, m is the number of fields, n is the number of packets, and The fields are FV X In the i In the data packet j The value of the field position, field FV Y In the i-th packet k The value of the field position; and The fields are FV X and fields FV Y The mean of

[0051] Step 2-12) Calculate the conditional entropy of the protocol field. The conditional entropy measures the uncertainty of another field when the value of one field is given. H ( FV Y | FV X ) is calculated as follows:

[0052] ;

[0053] in, Is a field FV X and fields FV Y Simultaneous value and The joint probability of is in the field Under the conditions, the field The conditional probability of

[0054] Step 2-13) Calculate the mutual information of the protocol fields. Mutual information measures the amount of information shared between two fields. I ( FV X ; FV Y )The formula is as follows:

[0055] ;

[0056] in, Is a field FV X and fields FV Y The joint probability distribution of and The fields are FV X and fields FV Y The marginal probability distribution of

[0057] Step 2-14) Combine the covariance, conditional entropy and mutual information calculated above to create a comprehensive index A to evaluate the degree of correlation between fields; comprehensive index A The calculation is as follows:

[0058] ;

[0059] in, Cov ( FV X , FV Y ) is the covariance between the two fields; H ( FV Y | FV X ) is a field FV Y In a given field FV X Conditional entropy when ; I ( FV X ; FV Y ) is the mutual information, indicatingFV X and FV Y The amount of information shared between 、 、 They are the 1st, 2nd and 3rd weight factors, and their sum is 1;

[0060] Determine the field value FV as the field's change feature value, based on which the field position FP and length FL are determined, and the extracted potential correlation features are expressed as feature vectors r i =[ FP ,FL,FV,Label], and finally stored in the associated feature library middle.

[0061] Furthermore, the specific method of step 3) includes:

[0062] 3-1) For fixed features, multiple fields with the same semantics are directly merged to form a new feature vector;

[0063] 3-2) For changing features, semantic simplification is divided into two parts, one for increasing and decreasing features and the other for fixed-length optional features. First, for increasing and decreasing features, further identification and confirmation of their change patterns are performed. After identifying these features, features with the same change pattern are merged. Features that appear as increasing or decreasing sequences are simplified into a unified range or pattern representation to reduce redundant information. After the merging is complete, a simplified feature vector is generated and stored in the changing feature library to replace the original feature vector.

[0064] 3-3) For fixed-length optional features, simplification is achieved by separating the fixed part from the variable part. The fixed-length optional features are divided into fixed fields and variable fields, corresponding to the fixed prefix part and the variable suffix part respectively. The fixed prefix part is extracted and stored in the fixed feature library to preserve the basic properties of the feature. For the variable suffix part, further merging and simplification are performed, and suffix parts with the same change pattern or logic are merged into a unified suffix pattern representation and stored in the variable feature library. Finally, the processed fixed prefix part and the simplified variable suffix part are combined to generate the final simplified feature vector, which is stored in the feature library instead of the original feature vector.

[0065] 3-4) For the correlation features, enter the set of field pairs extracted through correlation analysis L 2, which contains all the identified pairwise related field pairs in the protocol data packets; for these field pairs, use them as a basis to construct three-item sets and higher-order multi-item sets;

[0066] 3-5) Using field pairsL 2. Each pair of fields ( f 1 , f 2 ), construct 3-item sets and calculate the support of these combinations sup({ f 1, f 2, f 3}), where A pair of fields in , and From the Fields collection F Other fields in ; support is calculated as follows: ;

[0067] in, Indicates that the field combination { f 1, f 2, f 3} the number of packets, D Represents the total number of data packets; if the support of a 3-item set is greater than or equal to the threshold , then retain the item set and add it to the frequent 3-item set; use the frequent 3-item set to construct the 4-item set, continue to calculate the support and filter out the frequent 4-item set; and so on, until no new frequent item sets are generated.

[0068] Furthermore, the specific method of step 4) includes:

[0069] Step 4-1) Build a template based on the three semantically simplified features in the feature library; extract the simplified fixed features from the feature library and use them as the basic structure of the template;

[0070] Step 4-2) For the changing features, define the corresponding change range or rules for the corresponding fields, allowing their values ​​to be dynamically adjusted according to actual conditions;

[0071] Step 4-3) For the associated features, the complex relationships between fields are reduced to clear inference logic. By integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed.

[0072] Step 4-4) Fuse the feature information of fixed features, changing features, and associated features to form an integrated feature vector

[0073] ,

[0074] This feature vector combines all the feature results proposed above; then, download the public pcap data sample set and calculate the corresponding feature vector V for each field s of the pcap package in these sample sets. s, and embed these feature vectors into a 16-dimensional space to form a template field feature vector space.

[0075] Furthermore, the specific method of step 5) includes:

[0076] When a new message arrives, it is matched with the established protocol feature template and the similarity distance is calculated: First, follow the field information in the protocol feature template and extract the content of each field in the message according to the position relationship V i Then, the template field closest to the field information in the feature vector space of the protocol feature template field is obtained according to the following calculation formula: ;

[0077] Where S is the template field value space, Indicates the absolute value of the 16-dimensional feature vector calculated by the field to be analyzed. The cosin similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s is calculated. By searching for the minimum value, if the result of the value is less than 5%, the semantics of the field to be analyzed is determined to be the semantics of the corresponding template field. Otherwise, the field is determined to be a new semantics, and the feature vector of the field is embedded into the feature space. The feature vector of the feature space is continuously enriched and improved to realize the optimization and update of the template field, thereby completing the recognition and parsing of new message data and the dynamic update of the protocol feature template.

[0078] Furthermore, the network analysis tool is Wireshark.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] The present invention provides a protocol reverse parsing method based on multi-feature extraction and semantic simplification, which processes and semantically simplifies message data of three types of features: first, the features in the protocol message are classified and extracted, and are divided into fixed features, changing features and associated features. On the basis of extracting these three types of features, a targeted semantic simplification strategy is further introduced. For fixed features, by merging similar semantic fields, duplicate information is reduced and the simplicity of feature expression is optimized. For changing features, a feature pattern processing strategy is adopted to uniformly summarize the changing rules of increasing and decreasing features, and separate the fixed part and the changing part of the fixed-length optional feature to ensure the integrity of the changing information, while reducing dimensions and redundancy. For associated features, the logical relationship analysis between fields is used to dig out hidden field associations, thereby improving the expressiveness and accuracy of the feature library. Through this series of semantic simplification processes, the efficiency of feature extraction is significantly improved and the accuracy of protocol parsing is enhanced.

[0081] The method of the present invention is based on a feature generation and matching mechanism of a template: by establishing a feature template, efficient parsing of the protocol is achieved. The template is not only constructed through a semantically simplified feature library, covering the refined expression of three types of features, but also can quickly match and parse new messages. The establishment of the template is based on the abstraction of fixed, changing and associated features, and these features are integrated into a standardized template structure according to the logical order of the protocol message. During use, the system will directly perform field matching according to the template, introduce fixed fields, changing fields and associated fields, and achieve efficient message parsing. This matching mechanism based on feature templates not only improves the speed of protocol parsing, but also has good scalability and can adapt to complex and changeable protocol scenarios.

[0082] This method uses covariance, conditional entropy, and mutual information to quantify inter-field dependencies, automatically identifying relevant features such as checksums and data lengths, and improving parsing accuracy. It supports reverse parsing of unknown protocols and is particularly suitable for encrypted or obfuscated messages, meeting the real-time needs of network security and traffic monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is an overall flow chart of a protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention.

[0084] Figure 2 This invention discloses a protocol reverse parsing method based on multi-feature extraction and semantic simplification, which includes field feature extraction and database building process.

[0085] Figure 3 The present invention is a protocol reverse parsing method based on multi-feature extraction and semantic simplification, which illustrates the simplification of functional code semantics. DETAILED DESCRIPTION

[0086] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art fall within the scope defined by the claims attached to this application.

[0087] Example 1:

[0088] The present invention provides a protocol reverse parsing method based on multi-feature extraction and semantic simplification, comprising:

[0089] Step 1) Use a network analysis tool to read PCAP packets of a known protocol, extract the original message data, divide the original message data into fields, identify and extract field information, and save the fields and their corresponding values, positions, lengths, and semantic information to form protocol data, which can be saved in a CSV file.

[0090] Step 2) Based on the field division, fixed features, changing features, and associated features are extracted from the protocol data and stored in a fixed feature library for storing fixed features of the field, a changing feature library for storing changing features of the field, and an associated feature library for storing dependent features between fields.

[0091] Fixed features: used to quickly locate the protocol header structure and identify the protocol type (such as the version number to distinguish HTTP / 1.1 from HTTP / 2).

[0092] Change characteristics: Supports real-time traffic analysis and detects abnormal behavior (such as abnormal sequence number jumps that may indicate an attack).

[0093] Dependency features: Improve the accuracy of protocol reversal, such as verifying the correctness of parsing results through checksums.

[0094] Step 3) Simplify the semantics of fixed features, changing features, and associated features, remove redundant information, and extract representative and discriminative semantic features. This process aims to improve the expressiveness of features, making them more concise and informative, thereby better supporting subsequent protocol analysis.

[0095] Step 4) Based on the semantically simplified semantic features, a protocol feature template is constructed. The protocol feature template includes the basic structure corresponding to the fixed features, the change range or rules corresponding to the changing features, and the derivation logic corresponding to the associated features. This template will serve as the basic framework for protocol reverse engineering and is used to identify and parse message data of similar protocols.

[0096] Step 5) Based on the similarity matching results between the new message data and the protocol feature template, the new message data is identified and parsed, and the protocol feature template is dynamically updated.

[0097] In an optional embodiment, the specific method of step 1) includes:

[0098] Step 1-1) Use a network analysis tool to read the pcap data packet of a known protocol, extract the original message data, and store the original message data as a raw byte stream;

[0099] Steps 1-2) Parse the original message data, dividing the fields in the message according to the protocol format. Based on the protocol documentation or known standards, identify the location, length, and type of each field and determine the field boundaries. For each field, read its data content and record the actual value, starting position, and length of the field in the data packet.

[0100] Steps 1-3) Extract the semantic information of each field based on the protocol document or known standards, combining the field type, value, and position. For example, a field may represent a version number, identifier, data content, etc. Fields need to be classified and labeled according to the semantic definition of the protocol.

[0101] Steps 1-4) Save the extracted field values, positions, lengths, and semantic information into a CSV file to form a complete field data record for subsequent analysis and processing.

[0102] In an optional embodiment, the fixed features in the fixed feature library are field attributes with fixed positions in the protocol message and values ​​that do not change with time or data packets, including version numbers and site identifiers.

[0103] The change features in the change feature library are field attributes in the protocol message whose values ​​change with data packets or time and have specific rules, including sequence numbers, timestamps, IP addresses, and MAC addresses.

[0104] The dependent features in the correlation feature library are field attributes in the protocol message that have logical or statistical dependencies between fields, including checksums and data lengths.

[0105] In an optional embodiment, the specific method of step 2) includes:

[0106] Step 2-1) For protocol data with clear fixed features, scan multiple data packets to confirm that the version number field in the fixed feature is consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of the field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value; for the site identifier in the fixed feature, match the field with the known site ID pattern or site code, locate and extract the field, and merge it into the feature vector f l =[FP,FL,FV,site ID];

[0107] Step 2-2) For protocol data with unclear fixed features, input multiple data packet samples and use the comprehensive indicators of fixed fields to I Determine the position of fields in the protocol header that do not change with time, data flow, or packet content. These fields usually have fixed offsets and lengths and are clearly specified in the protocol document or standard. The consistency of the fields is evaluated by calculating the standard deviation of the field position. If the position of a field in different samples changes very little, it is considered to be a fixed field. The standard deviation of the field position is The calculation formula is as follows: ;

[0108] in, m is the number of fields, n is the sample size, FP ij It is in i In the data packet j The field value of each position, is the mean of the field positions, defining a consistency index C , used to quantify the consistency of the field. This indicator can set the threshold based on the standard deviation. The consistency indicator C The calculation formula is as follows: ;

[0109] in, is a small constant to prevent division by zero errors when the standard deviation is zero; if C A higher value indicates better consistency in field positions;

[0110] Steps 2-3) Obtain the content of the field at position FP in the current message data and improve search efficiency by applying the bad character rule and good suffix rule. Build a bad character table to record the last occurrence of each character in the pattern. Build a good suffix table to record the matching information of the suffix part of the pattern. Utilize the bad character rule and good suffix rule to skip unnecessary comparisons and improve matching efficiency. Calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of fixed fields I = w 1 C + w 2Pattern Score , w 1 and w 2 are the 1st and 2nd weight factors respectively;

[0111] Step 2-4) Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection; perform frequency analysis on the position of each field in the data packet, and set the first i The location is , the frequency of the field appearing at this position is .like , then the field can be considered as a fixed position; i The value of the field Perform change detection and calculate its rate of change r ( v i ),like , then the field value is almost unchanged and can be determined as a fixed value; determine the field feature at the message data position FP as a fixed feature, and obtain the field length FL and field value FV at this position, expressed as a feature vector f i =[FP,FL,FV,Label] and stored in the fixed feature library DB f =[ f v , f l ,…, f i ]middle;

[0112] Step 2-5) For the fields with clear change characteristics in the protocol, for the sequence number field, by detecting the continuous field values ​​in the data packet, find its increasing or decreasing pattern, extract the field and merge it into the feature vector d s =[FP, FL, FV, sequence number]; for the IP address and MAC address fields, locate these fields in the data packet through the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors respectively. d I =[FP,FL,FV,IP address] and d M =[FP,FL,FV,MAC address];

[0113] Step 2-6) For fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change pattern, including field value difference analysis and time series analysis; by analyzing the difference of field values ​​in multiple data packets, we can identify fields with change patterns. First, calculate the variance of the field value If the variance of a field is large, it means that its value changes greatly, which may contain some kind of change pattern: ;

[0114] in, FV ij It is i In the data packet j The field value of each position, is the average of the field values, m is the number of fields, n is the number of packets;

[0115] Step 2-7) Calculate the rate of change of field values ​​in adjacent packets If the field value shows a certain rate of change, it may have a certain change pattern. The calculation formula is as follows: ;

[0116] in, FV ij It is i In the data packet j The field value of each position, FV (i+1)j It is i+1 In the data packet j The field value of each position;

[0117] Calculate the comprehensive index of difference V 1: ;

[0118] in, Indicates the difference in field values. The smaller the variance, the more stable the field. It is the average value of the rate of change, indicating the average change in the field value; a smaller rate of change indicates that the change in the field value is more stable. s 1 and s 2 are the first and second weight factors, which are adjusted according to actual conditions;

[0119] Step 2-8) For the sequence of field values, use the autocorrelation time series analysis method to identify patterns; autocorrelation can be used to measure the similarity between field values ​​at different time points, and then identify whether there are regular changes; the autocorrelation coefficient V 2 is calculated using the following formula: ;

[0120] in, k is the lag step length, FV ij It is i In the data packet j The field value of each position, m is the number of fields, n is the number of packets;

[0121] Step 2-9) Combine the variance analysis and autocorrelation analysis of field values ​​to create a composite index V To evaluate whether a field is a fixed field: ;

[0122] in, and They are the first and second weight factors, which are adjusted according to actual conditions;

[0123] Determine the field value FV as the change feature value of the field, determine the field position FP and length FL based on this, and express the extracted potential change features as feature vectors d i=[FP,FL,FV,Label], and finally stored in the change feature library DB d =[ d s , d I , d M ,…, d i ]middle;

[0124] Step 2-10) For protocol data with clear correlation features, first analyze the field structure in the data packet to locate field pairs that may have dependencies; for the checksum field, extract the field and its corresponding data content segment, apply the CRC calculation algorithm and verify the match between the checksum field and the calculation result, and merge it into the feature vector For the data length field, locate the field that identifies the data length in the protocol, extract the field value and calculate the actual data length, verify the consistency of the length field value and the actual data length, and merge it into the feature vector All extracted correlation features have been verified multiple times to ensure their consistency and effectiveness;

[0125] Step 2-11) For protocol data with unclear correlation features, correlation analysis is used to explore its features. First, the covariance between protocol fields is calculated. Covariance can measure whether two fields have a linear correlation. When the covariance is close to zero, the two fields are independent. When the covariance is far from zero, it indicates that there is a correlation between the fields. Cov ( FV X , FV Y ) is calculated as follows: ;

[0126] in, m is the number of fields, n is the number of packets, and The fields are FV X In the i In the data packet j The value of the field position, field FV Y In the i-th packet k The value of the field position; and The fields are FV X and fields FV Y If the covariance is large, there is a strong correlation between the two fields;

[0127] Step 2-12) Calculate the conditional entropy of the protocol fields. Conditional entropy measures the uncertainty of a field given the value of another field. The smaller the conditional entropy, the better one field can predict the other field, indicating a strong correlation between the fields. H ( FV Y | FV X ) is calculated as follows: ;

[0128] in, Is a field FV X and fields FV Y Simultaneous value and The joint probability of is in the field Under the conditions, the field The conditional probability of

[0129] Step 2-13) Calculate the mutual information of the protocol fields. Mutual information measures the amount of information shared between two fields and indicates the degree of dependence between them. The larger the mutual information, the stronger the dependence between the fields. I ( FV X ; FV Y )The formula is as follows:

[0130] ;

[0131] in, yes FV X and fields FV Y The joint probability distribution of . and The fields are FV X and fields FV Y The marginal probability distribution of

[0132] Step 2-14) Combine the covariance, conditional entropy, and mutual information calculated above to create a comprehensive indicator A to evaluate the degree of association between fields. The comprehensive indicator is calculated as follows:

[0133] ;

[0134] in, Cov ( FVX , FV Y ) is the covariance between the two fields. The larger the covariance, the stronger the correlation. H ( FV Y | FV X ) is a field FV Y In a given field FV X The conditional entropy is smaller when the conditional entropy is smaller. FV X and FV Y The stronger the dependency; I ( FV X ; FV Y ) is the mutual information, indicating FV X and FV Y The amount of information shared between 、 、 They are the 1st, 2nd and 3rd weight factors, and their sum is 1;

[0135] Determine the field value FV as the field's change feature value, based on which the field position FP and length FL are determined, and the extracted potential correlation features are expressed as feature vectors r i =[ FP ,FL,FV,Label], and finally stored in the associated feature library middle.

[0136] In an optional embodiment, the specific method of step 3) includes:

[0137] 3-1) For fixed features, multiple fields with the same semantics are directly merged to form a new feature vector. For example, the features extracted from the original version number are: [0x01,"version number"], [0x02,"version number"], [0x03,"version number"], which can be simplified to: [{0x01,0x02,0x03},"version number"];

[0138] 3-2) For change features, semantic simplification is divided into two parts, which are respectively for simplifying increasing and decreasing features and fixed-length optional features; first, for the increasing and decreasing features that have been extracted, the first step is to further identify and confirm the change pattern of these features; increasing and decreasing features are usually manifested as continuous changes in numerical values ​​or timestamps, such as the field values ​​in multiple data packets showing an increasing or decreasing trend. After identifying these features, features with the same change pattern are merged. Specifically, for features that appear as increasing or decreasing sequences, they are simplified into a unified range or pattern representation to reduce redundant information. For example, if a feature appears as a numerical sequence [1,2,3] in multiple messages, it can be simplified to the form of "1-3" or "increasing sequence"; after the merging is completed, a simplified feature vector is generated and stored in the change feature library to replace the original feature vector. This feature vector is used to represent the features in all data packets that contain this increasing and decreasing pattern;

[0139] 3-3) For the identified fixed-length optional features, simplification is achieved by separating the fixed part and the variable part; such features usually contain a fixed prefix part and a variable suffix part, such as the function code field in the protocol. The fixed-length optional features (such as function code features) are divided into fixed fields and variable fields, corresponding to the fixed prefix part and the variable suffix part respectively; the fixed part is usually the first half of the feature, which represents the basic attributes of the feature; the variable part is the suffix part, which represents the variable content of the feature. The fixed prefix part is extracted and stored in the fixed feature library to retain the basic attributes of the feature; for the variable suffix part, further merging and simplification are performed, and the suffix parts with the same change pattern or logic are merged into a unified suffix pattern representation, and stored in the variable feature library; finally, the processed fixed prefix part and the simplified variable suffix part are combined to generate the final simplified feature vector, and the simplified feature vector is stored in the feature library instead of the original feature vector;

[0140] 3-4) For correlation features, we input the set of field pairs extracted through correlation analysis, which contains all identified pairwise correlated field pairs in the protocol data packets. We use these field pairs as a basis to construct three-item sets and higher-order multi-item sets.

[0141] 3-5) On this basis, use the field L 2. Each pair of fields ( f 1 , f 2 ), construct 3-item sets (three-item combinations) and calculate the support of these combinations sup({ f 1, f 2, f 3}), where A pair of fields in , andf 3 comes from other fields in the field set F; the support is calculated as follows: ;

[0142] in, Indicates that the field combination { f 1, f 2, f 3} the number of packets, D Represents the total number of data packets; if the support of a 3-item set is greater than or equal to the threshold , then retain the item set and add it to the frequent 3-item set; use the frequent 3-item set to construct the 4-item set, continue to calculate the support and filter out the frequent 4-item set; and so on, until no new frequent item sets are generated.

[0143] In an optional embodiment, the specific method of step 4) includes:

[0144] Step 4-1) Build a template based on the three semantically simplified features in the feature library. Extract the simplified fixed features from the feature library. These features represent the parts whose positions and contents remain unchanged in all messages and serve as the basic structure of the template.

[0145] Step 4-2) For the changing features, define the corresponding change range or rules for the corresponding fields, allowing their values ​​to be dynamically adjusted according to actual conditions;

[0146] Step 4-3) For associated features, the complex relationships between fields are reduced to clear inference logic, such as inferring the content of subsequent fields based on the values ​​of certain preceding fields. By integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed.

[0147] Step 4-4) Fuse the feature information of fixed features, changing features, and associated features to form an integrated feature vector , the feature vector combines all the feature results proposed above; then, download the public pcap data sample set from https: / / wiki.wireshark.org / SampleCaptures / website, and calculate the corresponding feature vector V for each field s of the pcap package in these sample sets s , and embed these feature vectors into a 16-dimensional space to form a template field feature vector space.

[0148] In an optional embodiment, the specific method of step 5) includes:

[0149] When a new message arrives, it is matched with the established protocol feature template and the similarity distance is calculated: First, follow the field information in the protocol feature template and extract the content of each field in the message according to the position relationship V i Then, the template field closest to the field information in the feature vector space of the protocol feature template field is obtained according to the following calculation formula: ;

[0150] Where S is the template field value space, Indicates the absolute value of the 16-dimensional feature vector calculated by the field to be analyzed. The cosin similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s is calculated. By searching for the minimum value, if the result of the value is less than 5%, the semantics of the field to be analyzed is determined to be the semantics of the corresponding template field. Otherwise, the field is determined to be a new semantics, and the feature vector of the field is embedded into the feature space. The feature vector of the feature space is continuously enriched and improved to realize the optimization and update of the template field, thereby completing the recognition and parsing of new message data and the dynamic update of the protocol feature template.

[0151] Example 2:

[0152] like Figure 1 and 2 As shown, the present invention proposes a protocol reverse parsing method based on multi-feature extraction and semantic simplification, including: reading a known protocol pcap package, using a network analysis tool to read the pcap data packet of the known protocol, and extracting the original message data.

[0153] In an optional embodiment, the network analysis tool is Wireshark.

[0154] In an optional embodiment, the message is divided into fields, the field information is identified and extracted, and the fields and their corresponding values, positions, lengths, and semantic information are saved in a CSV file;

[0155] In an optional embodiment, based on the field segmentation, fixed features are extracted from the protocol data and stored in a corresponding fixed feature library, including:

[0156] A1. For the version number field, by scanning multiple data packets, confirm that the field value at this position is consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of this field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value. For site identification, match the field with the known site ID pattern or site code, locate and extract the field, and merge it into the feature vector fl =[FP,FL,FV,site ID];

[0157] A2. For protocols that have not yet clearly defined fixed features, input multiple data packet samples and use the comprehensive indicators of fixed fields to I Determine the position of fields in the protocol header that do not change over time, data flow, or packet content. These fields usually have fixed offsets and lengths and are clearly specified in the protocol document or standard. Assess the consistency of fields by calculating the standard deviation of field positions. If the position of a field varies little across different samples, it can be considered a fixed field. Field Position Standard Deviation The calculation formula is as follows: ;

[0158] in, m is the number of fields, n is the sample size, FP ij It is in i In the data packet j The field value of each position, is the mean of the field positions. Define a consistency index C , used to quantify the consistency of the field. This metric can be thresholded based on the standard deviation: ;in, is a small constant that prevents division by zero errors when the standard deviation is zero. C A higher value indicates better consistency in field positions;

[0159] A3. Obtain the current FP field content and improve search efficiency using the "bad character rule" and "good suffix rule." Build a bad character table to record the last occurrence of each character in the pattern. Build a good suffix table to record matching information for the suffix portion of the pattern. Use the bad character rule and good suffix rule to skip unnecessary comparisons and improve matching efficiency. Calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of fixed fields I = w 1 C + w 2Pattern Score , w 1 and w 2 is the weighting factor.

[0160] A4. Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection. Perform frequency analysis on the position of each field in the data packet and calculate the position Frequency ,like , then the field can be considered as a fixed position. For each field value Perform change detection and calculate its rate of change r ( v i ),like , then the field value is almost unchanged and can be determined as a fixed value. Determine the field feature at the FP position as a fixed feature, and obtain the field length FL and field value FV at this position, expressed as a feature vector f i =[FP,FL,FV,Label] and stored in the fixed feature library DB f =[ f v , f l ,…, f i ]middle;

[0161] A5. For the sequence number field, by detecting the continuous field values ​​in the data packet, find the pattern of its increasing or decreasing, extract the field and merge it into the feature vector d s =[FP, FL, FV, sequence number]. For the IP address and MAC address fields, locate these fields in the data packet through the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors respectively. d I =[FP,FL,FV,IP address] and d M =[FP,FL,FV,MAC address];

[0162] A6. For fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change pattern, including field value difference analysis and time series analysis. By analyzing the difference of field values ​​in multiple data packets, we can identify fields with change patterns. First, calculate the variance of the field value. If the variance of a field is large, it means that its value changes greatly, which may contain some kind of change pattern: ;

[0163] in, FV ij It is i In the data packet j The field value of each position, is the average of the field values, m is the number of fields, n is the number of packets;

[0164] A7. Calculate the rate of change of field values ​​in adjacent data packets If the field value shows a certain rate of change, it may have a certain change pattern, and the calculation formula is as follows: ;in, FV ij It is i In the data packet j The field value of each position, FV (i+1)j It is i+1 In the data packet j The field value for each position.

[0165] Calculate the comprehensive index of difference V 1: ;

[0166] in, Indicates the difference in field values. The smaller the variance, the more stable the field. It is the average value of the rate of change, indicating the average change in the field value. A smaller rate of change indicates that the field value changes more steadily. s 1 and s 2 is the weight factor, which is adjusted according to actual conditions;

[0167] In an optional embodiment, based on the field segmentation, the change features are extracted from the protocol data and stored in the corresponding change feature library, including:

[0168] B1. For the sequence of field values, use the autocorrelation time series analysis method to identify patterns. Autocorrelation can be used to measure the similarity between field values ​​at different time points, and then identify whether there are regular changes. The autocorrelation coefficient is calculated using the following formula: ;in, k is the lag step length, FV ij It is i In the data packet j The field value of each position, m is the number of fields, n is the number of packets;

[0169] B2. Combine the difference analysis and autocorrelation analysis of field values ​​to create a comprehensive indicator to evaluate whether a field is a fixed field: ;in, and is the weight factor, which is adjusted according to actual conditions.

[0170] Determine FV as the change feature value of the field, based on which the field position FP and length FL are determined, and the extracted potential change features are expressed as feature vectorsd i =[FP,FL,FV,Label], and finally stored in the change feature library DB d =[ d s , d I , d M ,…, d i ]middle;

[0171] In an optional embodiment, based on the field segmentation, the associated features are extracted from the protocol data and stored in the corresponding associated feature library, including: C1. For the associated features, first, the field structure in the data packet is analyzed to locate the field pairs that may have dependencies. For the checksum field, the field and its corresponding data content segment are extracted, the CRC calculation algorithm is applied and the matching of the checksum field and the calculation result is verified, and then the result is merged into the feature vector. For the data length field, locate the field that identifies the data length in the protocol, extract the field value and calculate the actual data length, verify the consistency of the length field value and the actual data length, and merge it into the feature vector All extracted correlation features have been verified multiple times to ensure their consistency and effectiveness;

[0172] C2. For related fields that are difficult to extract through empirical features, correlation analysis is used to mine their features. First, the covariance between the protocol fields is calculated. Covariance can measure whether there is a linear correlation between two fields. When the covariance is close to zero, the two fields are independent; when the covariance is far from zero, it indicates that there is a correlation between the fields. Cov ( FV X , FV Y )The formula is as follows: ;

[0173] in, m is the number of fields, n is the number of packets, and The fields are FV X In the i In the data packet j The value of the field position, field FV Y In the data packet k The value of the field position. and The fields are FV X and fieldsFV Y If the covariance is large, there is a strong correlation between the two fields;

[0174] C3. Calculate the conditional entropy of the protocol fields. Conditional entropy measures the uncertainty of another field when the value of one field is given. The smaller the conditional entropy, the better one field can predict the other field, indicating a strong correlation between the fields. H ( FV Y | FV X )The formula is as follows: ;

[0175] in, Is a field FV X and fields FV Y Simultaneous value and The joint probability of . is in the field Under the conditions, the field The conditional probability of

[0176] C4. Calculate the mutual information of the protocol fields. Mutual information can measure the amount of information shared between two fields and indicate the degree of dependence between them. The larger the mutual information, the stronger the dependence between the fields. I ( FV X ; FV Y ) is calculated as follows: ;in, Is a field FV X and fields FV Y The joint probability distribution of . and The fields are FV X and fields FV Y The marginal probability distribution of

[0177] C5. Combine the covariance, conditional entropy and mutual information calculated above to create a comprehensive index A To evaluate the degree of correlation between fields. Comprehensive indicators A The calculation is as follows:

[0178] ;

[0179] in, Cov (FV X , FV Y ) is the covariance between the two fields. The larger the covariance, the stronger the correlation. H ( FV Y | FV X ) is a field FV Y In a given field FV X The conditional entropy is smaller when the conditional entropy is smaller. FV X and FV Y The stronger the dependency.

[0180] Determine FV as the changing feature value of the field, based on which the field position FP and length FL are determined, and the extracted potential correlation features are expressed as feature vectors r i =[ FP ,FL,FV,Label], and finally stored in the associated feature library middle.

[0181] In an optional embodiment, semantic simplification is performed on the fixed features, variable features, and associated features in the feature library to remove redundant information and extract more representative and discriminative semantic features, including:

[0182] D1. For fixed features, multiple fields with the same semantics are directly merged to form a new feature vector. For example, the features extracted from the original version number are: [0x01,"version number"], [0x02,"version number"], [0x03,"version number"], and after simplification: [{0x01,0x02,0x03},"version number"];

[0183] D2. For change features, semantic simplification is divided into two parts, which are used to simplify increasing and decreasing features and fixed-length optional features respectively. First, for the increasing and decreasing features that have been extracted, the first step of the method is to further identify and confirm the change patterns of these features. Increasing and decreasing features are usually manifested as continuous changes in numerical values ​​or timestamps, such as field values ​​in multiple data packets showing an increasing or decreasing trend. After identifying these features, the method merges features with the same change pattern. Specifically, for features that appear as increasing or decreasing sequences, they are simplified into a unified range or pattern representation to reduce redundant information. For example, if a feature appears as a numerical sequence [1,2,3] in multiple messages, it can be simplified to the form of "1-3" or "increasing sequence". After the merging is completed, a simplified feature vector is generated and stored in the change feature library to replace the original feature vector. This feature vector is used to represent the features in all data packets that contain the increasing and decreasing pattern;

[0184] D3. For the identified fixed-length optional features, the method simplifies them by separating the fixed part and the variable part. This type of feature usually contains a fixed prefix part and a variable suffix part, such as the function code field in the protocol. The function code feature is divided into a fixed field and a variable field. The fixed part is usually the first half of the feature, which represents the basic attributes of the feature; the variable part is the suffix part, which represents the variable content of the feature. The method extracts the fixed prefix part and stores it in the fixed feature library to retain the basic attributes of the feature. For the variable suffix part, further merging and simplification are performed, and the suffix parts with the same change pattern or logic are merged into a unified suffix pattern representation, and stored in the variable feature library. Finally, the processed fixed part and the simplified variable suffix part are combined to generate the final simplified feature vector, and the simplified feature vector is stored in the feature library instead of the original feature vector;

[0185] D4. For the associated features, input the field pair set extracted through correlation analysis. L 2, which contains all the identified pairwise related field pairs in the protocol data packets. For these field pairs, use them as a basis to construct three-item sets and higher-order multi-item sets;

[0186] D5. On this basis, use the field L 2. Each pair of fields ( f 1 , f 2 ), construct three-item combinations (3-item sets) and calculate the support of these combinations sup({ f 1, f 2, f 3}), where A pair of fields in , and f 3 from the fields collectionF Other fields in . The support is calculated as follows: ;in, Indicates that the field combination { f 1, f 2, f 3} the number of packets, D If the support of a 3-item set is greater than or equal to the threshold , then retain the itemset and add it to the frequent 3-itemsets. Use the frequent 3-itemsets to construct 4-itemsets, continue to calculate the support and filter out the frequent 4-itemsets. And so on, until no new frequent itemsets are generated.

[0187] In an optional embodiment, a matching template is established based on the semantically simplified protocol field features, and the template is matched against the new protocol to achieve rapid reverse analysis, including:

[0188] E1. Build a template based on the three semantically simplified features in the feature library. Extract the simplified fixed features from the feature library. These features represent the parts whose positions and contents remain unchanged in all messages and serve as the basic structure of the template.

[0189] E2. For changing features, define corresponding change ranges or rules for the corresponding fields, allowing their values ​​to be dynamically adjusted according to actual conditions;

[0190] E3. For related features, the complex relationships between fields are reduced to clear inference logic, such as inferring the content of subsequent fields based on the values ​​of certain preceding fields. By integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed;

[0191] E4. Fuse the feature information of fixed features, changing features, and associated features to form an integrated feature vector , the feature vector combines all the feature results proposed above, then downloads the public pcap data sample set from https: / / wiki.wireshark.org / SampleCaptures / website, and calculates the corresponding feature vector V for each field s of the pcap package in these sample sets. s , and embed these feature vectors into a 16-dimensional space to form a template field feature vector space;

[0192] E5. When a new message arrives, the system will match it with the established template and calculate the similarity distance: first, follow the field information in the template and extract the content of each field in the message according to the position relationship V i, and then the template field closest to the field information in the template field feature vector space is obtained according to the following calculation formula: ; In the above formula, S is the template field value space, Indicates the absolute value of the 16-dimensional feature vector calculated by the field to be analyzed. It is the cosin similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s. By searching for the minimum value, if the result of this value is less than 5%, the semantics of the field to be analyzed is determined to be the semantics of the corresponding template field. Otherwise, the field is determined to be a new semantics, and the feature vector of the field is embedded in the feature space. The feature vector of the feature space is continuously enriched and improved to achieve optimization and update of the template field.

[0193] Example 3:

[0194] like Figure 3 As shown, a protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention is schematically illustrated for semantic simplification of function codes, including:

[0195] 1. Merge multiple fixed fields with the same semantics into a new feature vector. For example, multiple fields [0x01,"Read Operation"], [0x02,"Read Operation"], [0x03,"Read Operation"] in the original function code can be merged into [{0x01,0x02,0x03},"Read Operation"].

[0196] 2. Identify fields in the function code that are incremented or decremented (such as serial numbers or counters) and combine fields with the same change pattern into a range or pattern representation. For example, the numerical sequence [1,2,3] can be simplified to "1-3" or "increasing sequence".

[0197] 3. The function code field is divided into a fixed prefix and a variable suffix. The fixed prefix indicates the basic attributes of the function (such as the operation type), and the variable suffix indicates the variable content (such as parameters or addresses). For example, the function codes [0xA1, 0x01] and [0xA1, 0x02] can be simplified to [0xA1, {0x01, 0x02}].

[0198] 4. Merge fixed fields to generate simplified fixed feature vectors. Classify variable fields, extract increasing and decreasing features and fixed-length optional features, and simplify them separately. Store the simplified feature vectors in the feature library, replacing the original features.

[0199] Example 4:

[0200] This example implements the protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention through simulation experiments. All experiments in this example are completed on a test host running the 64-bit Ubuntu 20.04 operating system. The host is equipped with an Intel Core i7-11700 CPU with a main frequency of 2.50GHz and a memory capacity of 32GB. The experimental program is written in Python. The core module relies on Scapy for PCAP packet parsing and uses Pandas for structured management of field information. At the same time, the frequent itemset mining algorithm is implemented with the help of the MLXTEND library, including:

[0201] 1. We selected multiple public protocol datasets, including actual packet capture files for typical protocols such as HTTP, FTP, and DNS, and simulated TCP sessions in a self-constructed environment to supplement specific field variation scenarios. The data was standardized and formatted uniformly.

[0202] 2. Assume that the set of all fields obtained after field division in the protocol message is F ={ f 1, f 2,…, f n}, where each field f i Represents an independent semantic unit in the protocol, such as the length field, function code, etc.

[0203] 3. Identify the correlation between field pairs through mutual information, conditional entropy, etc. to form a field pair set . Each pair ( f i , f j ) meet a certain correlation threshold, and it is preliminarily believed that there is a potential dependency relationship.

[0204] 4. Based on L 2, construct three-item combinations (3-item sets) and calculate their frequencies in the dataset. , its support is defined as follows: ;in, I k Indicates inclusion k Field combination of fields; a k Indicates the number selected from all fields; D ( I k ) indicates that all packets contain the field , ,..., A subset of packets; is the number of data packets in this subset; is the total number of packets in the complete protocol data set; sup( I k ) is a field combination I k The support of is used to measure the representativeness of the combination in the entire data set.

[0205] 5. Set based on experience of protocol type and field combination complexity is between 0.05 and 0.15, is the preset minimum support threshold. When the support of a certain item set satisfies , then I k Consider all frequent itemsets as k Item sets are combined into frequent item set sets .by (frequent 3-item sets) as the basis, construct candidate 4-item sets through pairwise connection operations , repeat the support calculation and screening process in sequence, recursively until (No new frequent itemsets are added).

[0206] 6. Finally form and output multi-order frequent field combinations , which contains the stable co-occurrence relationships between multiple fields in the protocol, k represents the itemset starting from , K Indicates the maximum size of the itemset, Indicates all frequent k Itemset.

[0207] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A protocol reverse parsing method based on multi-feature extraction and semantic simplification, characterized in that: include: Step 1) Using a network analysis tool to read PCAP packets of a known protocol, extracting raw message data, dividing the raw message data into fields, identifying and extracting field information, and saving the fields and their corresponding values, positions, lengths, and semantic information to form protocol data; Step 2) extracting fixed features, changing features, and associated features from the protocol data, and storing them in a fixed feature library for storing fixed features of fields, a changing feature library for storing changing features of fields, and an associated feature library for storing dependent features between fields, respectively; the fixed features in the fixed feature library are field attributes with fixed positions in the protocol message and values ​​that do not change with time or data packets, including version numbers and site identifiers; the changing features in the changing feature library are field attributes in the protocol message whose values ​​change with data packets or time and have specific rules, including sequence numbers, timestamps, IP addresses, and MAC addresses; the dependent features in the associated feature library are field attributes in the protocol message that have logical or statistical dependencies between fields, including checksums and data lengths; Step 3) semantically simplifying the fixed features, changing features, and associated features, removing redundant information, and extracting discriminative semantic features. The specific method of step 3) includes: 3-1) For fixed features, multiple fields with the same semantics are directly merged to form a new feature vector; 3-2) For changing features, semantic simplification is divided into two parts, one for increasing and decreasing features and the other for fixed-length optional features. First, for increasing and decreasing features, further identification and confirmation of their change patterns are performed. After identifying these features, features with the same change pattern are merged. Features that appear as increasing or decreasing sequences are simplified into a unified range or pattern representation to reduce redundant information. After the merging is complete, a simplified feature vector is generated and stored in the changing feature library to replace the original feature vector. 3-3) For fixed-length optional features, simplification is achieved by separating the fixed part from the variable part. The fixed-length optional features are divided into fixed fields and variable fields, corresponding to the fixed prefix part and the variable suffix part respectively. The fixed prefix part is extracted and stored in the fixed feature library to preserve the basic properties of the feature. For the variable suffix part, further merging and simplification are performed, and suffix parts with the same change pattern or logic are merged into a unified suffix pattern representation and stored in the variable feature library. Finally, the processed fixed prefix part and the simplified variable suffix part are combined to generate the final simplified feature vector, which is stored in the feature library instead of the original feature vector. 3-4) For the correlation features, enter the set of field pairs extracted through correlation analysis L 2, which contains all the identified pairwise related field pairs in the protocol data packets; for these field pairs, use them as a basis to construct three-item sets and higher-order multi-item sets; 3-5) Using field pairs L 2. Each pair of fields ( f 1 , f 2 ), construct 3-item sets and calculate the support of these combinations sup({ f 1, f 2, f 3}), where A pair of fields in , and From the Fields collection F Other fields in ; support is calculated as follows: ; in, Indicates that the field combination { f 1, f 2, f 3} the number of packets, D Represents the total number of data packets; if the support of a 3-item set is greater than or equal to the threshold , then retain the item set and add it to the frequent 3-item set; use the frequent 3-item set to construct the 4-item set, continue to calculate the support and filter out the frequent 4-item set; and so on, until no new frequent item sets are generated; Step 4) constructing a protocol feature template based on the semantically simplified semantic features; the protocol feature template includes a basic structure corresponding to the fixed features, a change range or rules corresponding to the changing features, and a derivation logic corresponding to the associated features; Step 5) Based on the similarity matching result between the new message data and the protocol feature template, the new message data is identified and parsed, and the protocol feature template is dynamically updated.

2. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 1 is characterized in that: The specific methods of step 1) include: Step 1-1) Use a network analysis tool to read the pcap data packet of a known protocol, extract the original message data, and store the original message data as a raw byte stream; Steps 1-2) Parse the original message data, dividing the fields in the message according to the protocol format. Based on the protocol documentation or known standards, identify the location, length, and type of each field and determine the field boundaries. For each field, read its data content and record the actual value, starting position, and length of the field in the data packet. Steps 1-3) Extract the semantic information of each field based on the type, value, and position of the field according to the protocol document or known standards; Steps 1-4) save the value, position, length, and semantic information of the extracted fields to form a complete field data record.

3. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 2 is characterized in that: The specific methods of step 2) include: Step 2-1) For protocol data with clear fixed features, scan multiple data packets to confirm that the version number field in the fixed feature is consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of the field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value; for the site identifier in the fixed feature, match the field with the known site ID pattern or site code, locate and extract the field, and merge it into the feature vector f l =[FP,FL,FV,site ID]; Step 2-2) For protocol data with unclear fixed features, input multiple data packet samples and use the comprehensive indicators of fixed fields to I Determine the position of those fields in the protocol header that do not change with time, data flow, or packet content, and evaluate the consistency of the field by calculating the standard deviation of the field position. If the position of the field in different samples changes little, it is considered to be a fixed field, and the standard deviation of the field position is . The calculation formula is as follows: ； in, m is the number of fields, n is the sample size, FP ij It is in i In the data packet j The field value of each position, is the mean of the field positions, defining a consistency index C , used to quantify the consistency of fields, consistency indicators C The calculation formula is as follows: ; in, is a small constant that prevents division by zero errors when the standard deviation is zero; Step 2-3) Get the field content of the current message data position FP, and improve the search efficiency by using the bad character rule and good suffix rule; build a bad character table to record the last occurrence position of each character in the pattern; build a good suffix table to record the matching information of the suffix part of the pattern; calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of fixed fields I = w 1 C + w 2Pattern Score , w 1 and w 2 are the 1st and 2nd weight factors respectively; Step 2-4) Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection; perform frequency analysis on the position of each field in the data packet, and set the first i The location is , the frequency of the field appearing at this position is ;like , then the field can be considered as a fixed position; i The value of the field v i Perform change detection and calculate its rate of change r ( v i ),like , then the field value is almost unchanged and can be determined as a fixed value; determine the field feature at the message data position FP as a fixed feature, and obtain the field length FL and field value FV at this position, expressed as a feature vector f i =[FP,FL,FV,Label] and stored in the fixed feature library DB f =[ f v , f l ,…, f i ]middle; Step 2-5) For the fields with clear change characteristics in the protocol, for the sequence number field, by detecting the continuous field values ​​in the data packet, find its increasing or decreasing pattern, extract the field and merge it into the feature vector d s =[FP, FL, FV, sequence number]; for the IP address and MAC address fields, locate these fields in the data packet through the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors respectively. d I =[FP,FL,FV,IP address] and d M =[FP,FL,FV,MAC address]; Step 2-6) For fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change pattern, including field value difference analysis and time series analysis; first, calculate the variance of the field value : ； in, FV ij It is i In the data packet j The field value of each position, is the average of the field values, m is the number of fields, n is the number of packets; Step 2-7) Calculate the rate of change of field values ​​in adjacent packets , the calculation formula is as follows: ； in, FV ij It is i In the data packet j The field value of each position, FV (i+1)j It is i+1 In the data packet j The field value of each position; Calculate the comprehensive index of difference V 1: ; in, Indicates the difference in field values. The smaller the variance, the more stable the field. It is the average value of the rate of change, indicating the average change in the field value; s 1 and s 2 are the first and second weight factors respectively; Step 2-8) For the sequence of field values, use the method of autocorrelation time series analysis to identify the pattern; the autocorrelation coefficient V 2 is calculated using the following formula: ； in, k is the lag step length, FV ij It is i In the data packet j The field value of each position, m is the number of fields, n is the number of packets; Step 2-9) Combine the variance analysis and autocorrelation analysis of field values ​​to create a composite index V To evaluate whether a field is a fixed field: ； in, and are the 1st and 2nd weight factors respectively; Determine the field value FV as the change feature value of the field, determine the field position FP and length FL based on this, and express the extracted potential change features as feature vectors d i =[FP,FL,FV,Label], and finally stored in the change feature library DB d =[ d s , d I , d M ,…, d i ]middle; Step 2-10) For protocol data with clear correlation features, first analyze the field structure in the data packet to locate field pairs that may have dependencies; for the checksum field, extract the field and its corresponding data content segment, apply the CRC calculation algorithm and verify the match between the checksum field and the calculation result, and merge it into the feature vector For the data length field, locate the field that identifies the data length in the protocol, extract the field value and calculate the actual data length, verify the consistency of the length field value and the actual data length, and merge it into the feature vector middle; Step 2-11) For protocol data with unclear correlation features, correlation analysis is used to explore its features. First, the covariance between protocol fields is calculated. Covariance can measure whether two fields have a linear correlation. When the covariance is close to zero, the two fields are independent. When the covariance is far from zero, it indicates that there is a correlation between the fields. Cov ( FV X , FV Y ) is calculated as follows: ； in, m is the number of fields, n is the number of packets, and The fields are FV X In the i In the data packet j The value of the field position, field FV Y In the i-th packet k The value of the field position; and The fields are FV X and fields FV Y The mean of Step 2-12) Calculate the conditional entropy of the protocol field. The conditional entropy measures the uncertainty of another field when the value of one field is given. H ( FV Y | FV X ) is calculated as follows: ； in, Is a field FV X and fields FV Y Simultaneous value and The joint probability of is in the field Under the conditions, the field The conditional probability of Step 2-13) Calculate the mutual information of the protocol fields. Mutual information measures the amount of information shared between two fields. I ( FV X ; FV Y )The formula is as follows: ； in, Is a field FV X and fields FV Y The joint probability distribution of and The fields are FV X and fields FV Y The marginal probability distribution of Step 2-14) Combine the covariance, conditional entropy and mutual information calculated above to create a comprehensive index A to evaluate the degree of correlation between fields; comprehensive index A The calculation is as follows: ； in, Cov ( FV X , FV Y ) is the covariance between the two fields; H ( FV Y | FV X ) is a field FV Y In a given field FV X Conditional entropy when ; I ( FV X ; FV Y ) is the mutual information, indicating FV X and FV Y The amount of information shared between 、 、 They are the 1st, 2nd and 3rd weight factors, and their sum is 1; Determine the field value FV as the field's change feature value, based on which the field position FP and length FL are determined, and the extracted potential correlation features are expressed as feature vectors r i =[ FP ,FL,FV,Label], and finally stored in the associated feature library middle.

4. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 3 is characterized in that: The specific methods of step 4) include: Step 4-1) Build a template based on the three semantically simplified features in the feature library; extract the simplified fixed features from the feature library and use them as the basic structure of the template; Step 4-2) For the changing features, define the corresponding change range or rules for the corresponding fields, allowing their values ​​to be dynamically adjusted according to actual conditions; Step 4-3) For the associated features, the complex relationships between fields are reduced to clear inference logic. By integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed. Step 4-4) Fuse the feature information of fixed features, changing features, and associated features to form an integrated feature vector ， This feature vector combines all the feature results proposed above; then, download the public pcap data sample set and calculate the corresponding feature vector V for each field s of the pcap package in these sample sets. s , and embed these feature vectors into a 16-dimensional space to form a template field feature vector space.

5. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 4 is characterized in that: The specific methods of step 5) include: When a new message arrives, it is matched with the established protocol feature template and the similarity distance is calculated: First, follow the field information in the protocol feature template and extract the content of each field in the message according to the position relationship V i Then, the template field closest to the field information in the feature vector space of the protocol feature template field is obtained according to the following calculation formula: ; Where S is the template field value space, Indicates the absolute value of the 16-dimensional feature vector calculated by the field to be analyzed. The cosin similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s is calculated. By searching for the minimum value, if the result of the value is less than 5%, the semantics of the field to be analyzed is determined to be the semantics of the corresponding template field. Otherwise, the field is determined to be a new semantics, and the feature vector of the field is embedded into the feature space. The feature vector of the feature space is continuously enriched and improved to realize the optimization and update of the template field, thereby completing the recognition and parsing of new message data and the dynamic update of the protocol feature template.

6. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to any one of claims 1 to 5, characterized in that: The network analysis tool is Wireshark.

Images