Protocol reverse analysis method based on multi-feature extraction and semantic simplification

Abstract

The invention discloses a protocol reverse analysis method based on multi-feature extraction and semantic simplification, and the method comprises the steps: carrying out the classification extraction of features in a protocol message, and dividing the features into fixed features, variable features and associated features; on the basis of extracting the three types of features, a targeted semantic simplification strategy is further introduced. For fixed features, repeated information is reduced by combining similar semantic fields, and the conciseness of feature expression is optimized. For change features, a feature modeling processing strategy is adopted, the change rule of progressive increase and progressive decrease features is uniformly summarized, and a fixed part and a change part of a fixed-length selectable feature are separately processed, so that the integrity of change information is ensured, and meanwhile, the dimensionality and redundancy are reduced. And for associated features, hidden field association is mined by applying logic relationship analysis between fields, so that the expressive force and accuracy of the feature library are improved. Through the series of semantic simplification processing, the feature extraction efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cyberspace security, and particularly relates to a protocol reverse parsing method based on multi-feature extraction and semantic simplification. Background Art

[0002] With the rapid development of network communication technology, the structure and complexity of protocol messages are increasing day by day, bringing new challenges to application scenarios such as network security, traffic monitoring, protocol analysis, and intrusion detection. Accurately parsing the field information in the message is crucial for these application scenarios. Especially when dealing with unknown or unpublished protocols, the key to message field parsing lies in inferring the meaning, position, length, and their mutual relationships of the fields by analyzing the message data. This technology is particularly important in fields such as reverse engineering, network forensics, and abnormal traffic analysis. However, existing message parsing technologies have various limitations and deficiencies when facing modern complex protocols.

[0003] Currently, commonly used message parsing technologies include manual analysis, static analysis, dynamic analysis, and machine learning-based methods. Manual analysis relies on the experience and knowledge of experts, has the problems of low efficiency and easy errors, and is difficult to meet the automation requirements especially when facing complex or large-scale messages. Static analysis is usually based on known protocol formats, and its applicability is greatly reduced for unknown or frequently changing protocols. In addition, static analysis cannot handle the situation of message encryption or obfuscation. Although dynamic analysis can infer some field information by observing the data changes in protocol interactions, its effect significantly decreases when the message content is encrypted or obfuscated. At the same time, dynamic analysis is vulnerable to data noise when dealing with high-dimensional messages, resulting in inaccurate results. For scenarios that rely on interaction behavior, the dynamic analysis ability of static messages is also very limited.

[0004] The machine learning-based parsing method brings some hope for automation, but usually requires a large amount of labeled data for model training. The labeling process is time-consuming and highly subjective, which is likely to affect the generalization effect of the model. In addition, the complexity of deep learning models leads to high requirements for computing resources, and there may be performance bottlenecks especially in real-time traffic parsing. At the same time, machine learning methods are prone to generating redundant features and high-dimensional features when dealing with complex messages, further increasing the complexity of parsing.

[0005] Existing feature extraction methods also have limitations when facing complex messages. A large number of redundant features are easily generated during the feature extraction process, resulting in high data dimensions and increasing the computational burden of the parsing process. Especially when dealing with the correlation relationships of message fields, there is a lack of effective automatic simplification mechanisms. In addition, current feature simplification and merging mostly rely on manual adjustment, and it is difficult to improve efficiency while maintaining parsing accuracy.

[0006] To address these deficiencies, there is an urgent need for a new message parsing method that can reduce feature redundancy, improve parsing accuracy and efficiency, and better adapt to the current complex network environment and protocol structure. Summary of the Invention

[0007] To solve the problems existing in the prior art, the present invention proposes a protocol reverse parsing method based on multi-feature extraction and semantic simplification.

[0008] The technical solution of the present invention is as follows: A protocol reverse parsing method based on multi-feature extraction and semantic simplification, comprising: Step 1) Use a network analysis tool to read the PCAP data packets of a known protocol, extract the original message data, perform field division on the original message data, identify and extract field information, and save the field and its corresponding value, position, length, and semantic information to form protocol data; Step 2) Extract fixed features, variable features, and associated features from the protocol data, and store them respectively in a fixed feature library for storing the fixed features of fields, a variable feature library for storing the variable features of fields, and an associated feature library for storing the dependency features between fields; Step 3) Simplify the semantics of the fixed features, variable features, and associated features, remove redundant information, and refine the semantic features with distinctiveness; Step 4) Based on the semantically simplified semantic features, construct a protocol feature template; the protocol feature template includes a basic structure corresponding to the fixed features, a variation range or rule corresponding to the variable features, and a derivation logic corresponding to the associated features; Step 5) Identify and parse the new message data and dynamically update the protocol feature template based on the similarity matching result between the new message data and the protocol feature template.

[0009] Further, the specific method of Step 1) includes: Step 1-1) Use a network analysis tool to read the pcap data packets of a known protocol, extract the original message data, and store the original message data as an original byte stream; Step 1-2) Parse the original message data, divide each field in the message according to the protocol format, identify the position, length, and type of each field according to the protocol document or known standard, and determine the boundary of the field; for each field, read its data content and record the actual value, starting position, and length of the field in the data packet; Step 1-3) Extract the semantic information of each field according to the protocol document or known standard, in combination with the type, value, and position of the field; Step 1-4): Save the values, positions, lengths, and semantic information of the extracted fields to form a complete field data record.

[0010] Further, the fixed features in the fixed feature library are field attributes in the protocol packet with fixed positions and values that do not change over time or with data packets, including version numbers and site identifiers.

[0011] Further, the variable features in the variable feature library are field attributes in the protocol packet whose values change with data packets or time and have specific rules, including sequence numbers, timestamps, IP addresses, and MAC addresses.

[0012] Further, the dependent features in the association feature library are field attributes in the protocol packet with logical or statistical dependencies between fields, including checksum and data length.

[0013] Further, the specific method of step 2) includes: Step 2-1): For the protocol data with clear fixed features, for the version number field in the fixed features, by scanning multiple data packets, confirm that the field values at this position are consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of this field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value; for the site identifier in the fixed features, match the field with the known site ID pattern or site code, locate and extract this field, and merge it into the feature vector f l =[FP, FL, FV, site identifier]; Step 2-2): For the protocol data with unclear fixed features, input multiple data packet samples, and through the comprehensive index of fixed fields I Determine the positions of the fields in the protocol header that do not change with time, data flow, or data packet content. Evaluate the consistency of the fields by calculating the standard deviation of the field positions. If the position changes of the fields in different samples are very small, then it is considered a fixed field, and the standard deviation of the field position The calculation formula is as follows: ; Among them, m Is the number of fields, n Is the number of samples, FP ij Is at the i In the j Field value at the position, Is the mean of the field positions, and a consistency index C Is defined to quantify the consistency of the fields, and the consistency indexC The calculation formula is as follows: ; Wherein, is a very small constant to prevent a division-by-zero error when the standard deviation is zero; Step 2-3) Obtain the field content at the current message data position FP, and improve the search efficiency through the bad character rule and the good suffix rule; construct a bad character table to record the last occurrence position of each character in the pattern; construct a good suffix table to record the matching information of the suffix part of the pattern; calculate the matching success rate Pattern Score =(number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of the fixed field I = w 1 C + w 2Pattern Score , w 1 and w 2 are the first and second weight factors respectively; Step 2-4) Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection; perform frequency analysis on the position of each field in the data packet. Let the i th position be , and the frequency of the field appearing at this position be ; if , then this field can be considered as a fixed position; perform change detection on the value i of the v i th field, calculate its change rate r ( v i ), if , then the value of this field hardly changes and can be determined as a fixed value; determine that the field feature at the message data position FP is a fixed feature, and obtain the field length FL and field value FV at this position, expressed as the feature vector f i =[FP, FL, FV, Label], and store it in the fixed feature library DB f = f v , f l ,…, f i ; Step 2-5) For the fields with clearly changing features in the protocol, for the sequence number field, by detecting the consecutive field values in the data packet, find its increasing or decreasing pattern, extract this field and merge it into the feature vector d s=[FP, FL, FV, serial number]; for the IP address and MAC address fields, locate these fields in the data packet according to the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors respectively d I =[FP, FL, FV, IP address] and d M =[FP, FL, FV, MAC address]; Step 2-6) For the fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change rules therein, including the difference analysis and time series analysis of the field values; first, calculate the variance of the field values : ; wherein, FV ij is the field value at the i th position in the j th data packet, is the average value of the field values, m is the number of fields, n is the number of data packets; Step 2-7) Calculate the change rate of the field values in adjacent data packets , and the calculation formula is as follows: ; wherein, FV ij is the field value at the i th position in the j th data packet, FV (i+1)j is the field value at the i+1 th position in the j th data packet; Calculate the comprehensive difference index V 1: ; wherein, represents the difference of the field values, and the smaller the variance, the more stable the field; is the average value of the change rates, representing the average change situation of the field values; s 1 and s 2 are the first and second weight factors respectively; Step 2-8) For the sequence of field values, adopt the method of autocorrelation time series analysis to identify the rules; the autocorrelation coefficient V 2 is calculated by the following formula: ; wherein, kis the lag step size, FV ij is the i field value at the j th position in the m th data packet, n is the number of fields, is the number of data packets; V Step 2-9) combines the differential analysis and autocorrelation analysis of field values to create a comprehensive metric ; Among them, and are the first and second weight factors respectively; Determine the field value FV as the change characteristic value of the field, and accordingly determine the field position FP and length FL, and represent the extracted potential change characteristics as a feature vector d i = [FP, FL, FV, Label], and finally store it in the change characteristic library DB d = d s , d I , d M ,…, d i ; Step 2-10) For the protocol data with clear associated features, first, analyze the field structure in the data packet to locate the field pairs that may have a dependency relationship; for the checksum field, extract the field and its corresponding data content segment, apply the checksum algorithm CRC to calculate and verify the matching of the checksum field with the calculation result, and merge it into the feature vector ; for the data length field, locate the field in the protocol that identifies the data length, extract the field value and calculate the actual data length, verify the consistency of the length field value with the actual data length, and merge it into the feature vector ; Step 2-11) For the protocol data whose associated features have not been clearly identified, use correlation analysis to mine its features; first calculate the covariance between protocol fields, and the covariance can measure whether two fields have a linear correlation; when the covariance is close to zero, the two fields are independent; when the covariance is far from zero, it indicates that there is an association between the fields; the covariance Cov ( FV X , FV Y ) calculation formula is as follows: ; Among them, mis the number of fields, n is the number of data packets, and are respectively the values of the field FV X at the i th field position in the j th data packet, and the field FV Y is the value of the field at the k th field position in the i-th data packet; and are respectively the means of the fields F X and the field F Y ; Step 2 - 12) Calculate the conditional entropy of the protocol fields. The conditional entropy measures the uncertainty of another field given the value of one field; the conditional entropy H ( FV Y | FV X ) is calculated by the following formula: ; where, is the joint probability that the fields FV X and the field FV Y take values and simultaneously; is the conditional probability of the field given the field ; Step 2 - 13) Calculate the mutual information of the protocol fields. The mutual information measures the amount of information shared between two fields, and the mutual information I ( FV X ; FV Y ) is calculated by the following formula: ; where, is the joint probability distribution of the fields FV X and the field FV Y ; and are respectively the marginal probability distributions of the fields FV X and the field FV Y ; Step 2-14) Combine the covariance, conditional entropy, and mutual information calculated above to create a comprehensive metric A to evaluate the degree of association between fields; the comprehensive metric A is calculated as follows: ; where Cov ( FV X , FV Y ) is the covariance between two fields; H ( FV Y | FV X ) is the conditional entropy of field FV Y given field FV X ; I ( FV X ; FV Y ) is the mutual information, representing the amount of information shared between FV X and FV Y ; , , are the first, second, and third weight factors respectively, and their sum is 1; Determine the field value FV as the change characteristic value of the field, and accordingly determine the field position FP and length FL, and represent the extracted potential association features as a feature vector r i = FP , FL, FV, Label], and finally store it in the association feature library .

[0014] Furthermore, the specific method of step 3) includes: 3-1) For fixed features, directly merge multiple fields with the same semantics to form a new feature vector; 3-2) For change features, semantic simplification is divided into two parts, and simplification processing is carried out for increasing / decreasing features and fixed-length optional features respectively; first, for increasing / decreasing features, further identify and confirm the change patterns of these features; after identifying these features, merge the features with the same change pattern, and for features showing an increasing or decreasing sequence, simplify them into a unified range or pattern representation to reduce redundant information; after the merging is completed, generate the simplified feature vector and store it in the change feature library to replace the original feature vector; 3-3) For fixed-length optional features, simplify them by separating the fixed part and the variable part; divide the fixed-length optional features into fixed fields and variable fields, corresponding to the fixed prefix part and the variable suffix part respectively; extract and store the fixed prefix part in the fixed feature library to retain the basic attributes of the features; for the variable suffix part, further merge and simplify it, merge the suffix parts with the same variation pattern or logic into a unified suffix pattern representation, and store it in the variable feature library; finally, combine the processed fixed prefix part and the simplified variable suffix part to generate the final simplified feature vector, and replace the original feature vector and store it in the feature library; 3-4) For associated features, input the set of field pairs extracted through correlation analysis L 2, which contains all the identified pairwise-related field pairs in the protocol data packet; for these field pairs, use them as the basis to construct three-item sets and higher-order multi-item sets; 3-5) Utilize the field pairs L 2 in each pair of fields( f 1 , f 2 ) to construct 3-item sets and calculate the support of these combinations sup({ f 1, f 2, f 3}), where, is a pair of fields in, and comes from other fields in the field set F ; the support calculation method is as follows: ; Among them, represents the number of data packets containing the field combination { f 1, f 2, f 3}, D represents the total number of data packets; if the support of a certain 3-item set is greater than or equal to the threshold , then retain this item set and add it to the frequent 3-item sets; use the frequent 3-item sets to construct 4-item sets, continue to calculate the support and filter out the frequent 4-item sets; and so on until no new frequent item sets are generated.

[0015] Furthermore, the specific method of step 4) includes: Step 4-1) Build a template based on the three semantically simplified features in the feature library; extract the simplified fixed features from the feature library and use them as the basic structure in the template; Step 4-2) For variable features, define corresponding variation ranges or rules for the corresponding fields, allowing their values to be dynamically adjusted according to the actual situation; Step 4-3) For associated features, reduce the complex relationships between fields to explicit derivation logics; by integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed. Step 4-4) Integrate the feature information of fixed features, variable features, and associated features to form an integrated feature vector. , This feature vector combines all the previously proposed feature results; then, download the publicly available pcap data sample set, and calculate the corresponding feature vector V for each field s in the pcap packets of these sample sets. s , and embed these feature vectors into a 16-dimensional space to form a template field feature vector space.

[0016] Furthermore, the specific method of step 5) includes: When a new packet arrives, match it with the established protocol feature template and calculate the similarity distance: First, follow the field information in the protocol feature template and extract the content V of each field in the packet according to the positional relationship. i Then, obtain the template field closest to the field information in the protocol feature template field feature vector space according to the following calculation formula: ; In the formula, S is the value space of the template field, represents the absolute value of the 16-dimensional feature vector calculated for the field to be analyzed, is the cosine similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s, and by searching for the minimum value, if the result of this value is less than 5%, it is determined that the semantics of the field to be analyzed is the semantics of the corresponding template field, otherwise it is determined that the field is a new semantics, and the feature vector of the field is embedded into the feature space to continuously enrich and improve the feature vectors in the feature space, realizing the optimization and update of the template field, thereby completing the recognition and parsing of the new packet data and the dynamic update of the protocol feature template.

[0017] Furthermore, the network analysis tool is Wireshark.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a protocol reverse parsing method based on multi-feature extraction and semantic simplification. This method processes and simplifies the semantics of message data with three types of features: First, the features in the protocol message are classified and extracted, and are respectively divided into fixed features, variable features, and associated features. On the basis of extracting these three types of features, a targeted semantic simplification strategy is further introduced. For fixed features, by merging similar semantic fields, redundant information is reduced, and the conciseness of feature expression is optimized. For variable features, a feature pattern processing strategy is adopted to uniformly summarize the variation rules of increasing and decreasing features, and the fixed part and the variable part of the fixed-length optional features are separated for processing to ensure the integrity of the variable information while reducing the dimension and redundancy. For associated features, by analyzing the logical relationships between fields, hidden field associations are mined, enhancing the expressiveness and accuracy of the feature library. Through this series of semantic simplification processes, the efficiency of feature extraction is significantly improved, and the accuracy of protocol parsing is enhanced.

[0019] The method of the present invention is based on a template-based feature generation and matching mechanism: By establishing a feature template, efficient parsing of the protocol is achieved. This template is not only constructed through a semantically simplified feature library, covering refined expressions of three types of features, but also capable of quickly matching and parsing new messages. The establishment of the template is based on the abstraction of fixed, variable, and associated features, and these features are integrated into a standardized template structure according to the logical order of the protocol message. During use, the system directly performs field matching according to the template, deducing fixed fields, variable fields, and associated fields to achieve efficient message parsing. This template-based matching mechanism not only improves the speed of protocol parsing but also has good scalability and can adapt to complex and changing protocol scenarios.

[0020] The method of the present invention uses covariance, conditional entropy, and mutual information to quantify the dependence relationship between fields, automatically identifies associated features such as checksum and data length, and improves parsing accuracy. The method of the present invention supports reverse parsing of unknown protocols, especially suitable for encrypted or obfuscated messages, meeting the real-time requirements of network security and traffic monitoring. Brief Description of the Drawings

[0021] Figure 1 It is the overall flowchart of a protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention.

[0022] Figure 2 It is the field feature extraction and library building process of a protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention.

[0023] Figure 3 It is the semantic simplification schematic of function codes of a protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention. Detailed Embodiment

[0024] The present invention will be further illustrated below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent modifications made by those skilled in the art fall within the scope defined by the appended claims of this application.

[0025] Embodiment 1: A protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention includes: Step 1) Use a network analysis tool to read the PCAP data packets of a known protocol, extract the original message data, perform field division on the original message data, identify and extract field information, and save the field together with its corresponding value, position, length, and semantic information to form protocol data, which can be saved to a CSV file; Step 2) Based on the field division, extract fixed features, variable features, and associated features from the protocol data, and store them respectively in a fixed feature library for storing the fixed features of the fields, a variable feature library for storing the variable features of the fields, and an associated feature library for storing the dependency features between the fields; Fixed features: Used to quickly locate the protocol header structure and identify the protocol type (such as distinguishing HTTP / 1.1 and HTTP / 2 by version number).

[0026] Variable features: Support real-time traffic analysis and detect abnormal behaviors (such as abnormal jumps in sequence numbers may indicate an attack).

[0027] Associated features: Improve the accuracy of protocol reverse engineering. For example, verify the correctness of the parsing result through a checksum.

[0028] Step 3) Semantically simplify the fixed features, variable features, and associated features, remove redundant information, and refine representative and distinguishable semantic features; this process aims to improve the expression ability of the features, make them more concise and informative, so as to better support subsequent protocol analysis; Step 4) Based on the semantically simplified semantic features, construct a protocol feature template; the protocol feature template includes the basic structure corresponding to the fixed features, the variable range or rules corresponding to the variable features, and the derivation logic corresponding to the associated features. This template will serve as the basic framework for protocol reverse engineering and be used to identify and parse the message data of similar protocols.

[0029] Step 5) Identify and parse the new message data and dynamically update the protocol feature template based on the similarity matching result between the new message data and the protocol feature template.

[0030] In an optional embodiment, the specific method of step 1) includes: Step 1-1) Use a network analysis tool to read pcap packets of known protocols, extract the original message data, and store the original message data as an original byte stream; Step 1-2) Parse the original message data, divide each field in the message according to the protocol format, identify the position, length, and type of each field according to the protocol document or known standard, and determine the field boundaries; for each field, read its data content and record the actual value, start position, and length of the field in the data packet; Step 1-3) According to the protocol document or known standard, combine the type, value, and position of the field to extract the semantic information of each field; for example, the field may represent a version number, identifier, data content, etc., and the field needs to be classified and labeled according to the semantic definition of the protocol; Step 1-4) Save the extracted field values, positions, lengths, and semantic information to a CSV file to form a complete field data record for subsequent analysis and processing.

[0031] In an optional embodiment, the fixed features in the fixed feature library are field attributes in the protocol message that are fixed in position and whose values do not change with time or data packets, including version numbers and site identifiers.

[0032] The variable features in the variable feature library are field attributes in the protocol message whose values change with data packets or time and have specific rules, including sequence numbers, timestamps, IP addresses, and MAC addresses.

[0033] The dependent features in the association feature library are field attributes in the protocol message that have logical or statistical dependencies between fields, including checksums and data lengths.

[0034] In an optional embodiment, the specific method of step 2) includes: Step 2-1) For the protocol data with clear fixed features, for the version number field in the fixed features, by scanning multiple data packets, confirm that the field value at this position remains consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of this field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value; for the site identifier in the fixed features, match the field with the known site ID pattern or site code, locate and extract this field, and merge it into the feature vector f l =[FP, FL, FV, site identifier]; Step 2-2) For the protocol data whose fixed features are not yet clear, input multiple data packet samples, through the comprehensive index of fixed fields IDetermine the positions of the fields in the protocol header that do not change with time, data flow, or packet content. These fields usually have fixed offsets and lengths and are clearly specified in the protocol document or standard. Evaluate the consistency of the fields by calculating the standard deviation of the field positions. If the positions of the fields vary little among different samples, then it is considered a fixed field. The standard deviation of the field positions is calculated as follows: ; where m is the number of fields, n is the number of samples, FP ij is the value of the field at the i th position in the j th packet, is the mean of the field positions. Define a consistency index C , which is used to quantify the consistency of the field. This index can set a threshold based on the standard deviation. The consistency index C is calculated as follows: ; where is a very small constant to prevent division by zero error when the standard deviation is zero; if C has a high value, it indicates good consistency of the field positions; Step 2 - 3) Obtain the field content at the current message data position FP, and improve the search efficiency through the bad character rule and the good suffix rule; construct a bad character table to record the last occurrence position of each character in the pattern. Construct a good suffix table to record the matching information of the suffix part of the pattern; use the bad character rule and the good suffix rule to skip unnecessary comparisons and improve the matching efficiency. Calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of the fixed field I = w 1 C + w 2Pattern Score , w 1 and w 2 are the first and second weight factors respectively; Step 2 - 4) Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection; perform frequency analysis on the positions of each field in the packet. Let the i th position be , and the frequency of the field appearing at this position be . If , then it can be considered that the field is at a fixed position; for the value of the i th field Perform change detection and calculate its change rate r ( v i ) If , then the value of this field changes little and can be determined as a fixed value; determine that the field feature at the message data position FP is a fixed feature, and obtain the field length FL and field value FV at this position, expressed as a feature vector f i = [FP, FL, FV, Label], and store it in the fixed feature library DB f = f v , f l ,…, f i ; Step 2-5) For the fields with clear change features in the protocol, for the sequence number field, by detecting the consecutive field values in the data packet, find its increasing or decreasing pattern, extract this field and merge it into the feature vector d s = [FP, FL, FV, sequence number]; for the IP address and MAC address fields, locate these fields in the data packet through the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors d I = [FP, FL, FV, IP address] and d M = [FP, FL, FV, MAC address]; Step 2-6) For the fields with unknown change features in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change pattern therein, including the differential analysis of field values and time series analysis; by analyzing the differences in field values among multiple data packets, identify the fields with change patterns. First, calculate the variance of the field values , if the variance of a certain field is large, it indicates that its value changes greatly and may contain a certain change pattern: ; Among them, FV ij is the field value at the i th position in the j th data packet, is the average value of the field values, m is the number of fields, n is the number of data packets; Step 2-7) Calculate the change rate of field values in adjacent data packets , if the field values show a certain change rate, then it may have a certain change pattern, and the calculation formula is as follows: ; Among them, FV ij is the field value at the i th position in the j th data packet, FV (i+1)j is the field value at the i+1 th position in the j th data packet; Calculate the comprehensive difference index V 1: ; Among them, represents the difference of the field value. The smaller the variance, the more stable the field; is the average value of the change rate, indicating the average change of the field value; a smaller change rate indicates that the change of the field value is relatively stable. s 1 and s 2 are the first and second weight factors respectively, which are adjusted according to the actual situation; Step 2 - 8) For the sequence of field values, use the method of autocorrelation time series analysis to identify patterns; autocorrelation can be used to measure the similarity between field values at different time points, and then identify whether there are regular changes; the autocorrelation coefficient V 2 is calculated by the following formula: ; Among them, k is the lag step, FV ij is the field value at the i th position in the j th data packet, m is the number of fields, n is the number of data packets; Step 2 - 9) Combine the difference analysis and autocorrelation analysis of field values to create a comprehensive index V to evaluate whether the field is a fixed field: ; Among them, and are the first and second weight factors respectively, which are adjusted according to the actual situation; Determine that the field value FV is the change characteristic value of the field, and accordingly determine the field position FP and length FL, and represent the extracted potential change characteristics as a feature vector d i = [FP, FL, FV, Label], and finally store it in the change characteristic library DB d = d s , d I ,d M ,…, d i ; Step 2-10) For the protocol data with clearly associated features, first, analyze the field structure in the data packet to locate the field pairs that may have dependencies; for the checksum field, extract the field and its corresponding data content segment, apply the CRC check algorithm to calculate and verify the matching of the checksum field with the calculation result, and merge it into the feature vector ; for the data length field, locate the field in the protocol that identifies the data length, extract the field value and calculate the actual data length, verify the consistency between the length field value and the actual data length, and merge it into the feature vector ; all the extracted associated features are verified multiple times to ensure their consistency and effectiveness; Step 2-11) For the protocol data with unclear associated features, use correlation analysis to mine its features; first calculate the covariance between protocol fields, and the covariance can measure whether two fields have linear correlation; when the covariance is close to zero, the two fields are independent; when the covariance is far from zero, it indicates that there is an association between the fields; the covariance Cov ( FV X , FV Y ) is calculated as follows: ; where m is the number of fields, n is the number of data packets, and are the values of field FV X at the i th position of the j th field in the FV Y th data packet, and the value of field k at the and are the means of field F X and field F Y respectively; if the covariance is large, there is a strong association between the two fields; Step 2-12) Calculate the conditional entropy of protocol fields. The conditional entropy measures the uncertainty of another field given the value of one field; the smaller the conditional entropy, the better one field can predict another field, indicating a strong association between the fields. The conditional entropy H ( FV Y | FVX ) The calculation formula is as follows: ; Wherein, is the FV X and the FV Y joint probability of taking values simultaneously and ; is the conditional probability of the field under the condition of the Step 2-13) Calculate the mutual information of the protocol fields. The mutual information measures the amount of information shared between two fields and represents the degree of dependence between them. The greater the mutual information, the stronger the dependence between the fields. The mutual information I ( FV X ; FV Y ) The formula is as follows: ; Wherein, is FV X and the FV Y joint probability distribution. and are respectively the FV X and the FV Y marginal probability distributions of the Step 2-14) Combine the covariance, conditional entropy, and mutual information calculated above to create a comprehensive index A to evaluate the degree of association between the fields. The calculation method of the comprehensive index is as follows: ; Wherein, Cov ( FV X , FV Y ) is the covariance between two fields. The greater the covariance, the stronger the association. H ( FV Y | FV X ) is the conditional entropy of the FV Y field FV X given the FV X and FV Y The smaller the conditional entropy, the stronger the dependence betweenI ( FV X ; FV Y ) is the mutual information, representing FV X and FV Y the amount of information shared between; , , are the first, second, and third weight factors respectively, and the sum is 1; Determine the field value FV as the change characteristic value of the field, and accordingly determine the field position FP and length FL, and represent the extracted potential association characteristics as a feature vector r i = FP , FL, FV, Label] and finally store it in the association feature library in.

[0035] In an optional embodiment, the specific method of step 3) includes: 3-1) For fixed features, directly merge multiple fields with the same semantics to form a new feature vector; for example, the features extracted from the original version number are: [0x01, "version number"], [0x02, "version number"], [0x03, "version number"], and after simplification: [{0x01, 0x02, 0x03}, "version number"]; 3-2) For variable features, semantic simplification is divided into two parts, and simplification processing is performed for increasing / decreasing features and fixed-length optional features respectively; first, for the increasing / decreasing features that have been extracted, the first step is to further identify and confirm the change patterns of these features; increasing / decreasing features usually show continuous changes in numerical values or timestamps, such as the field values showing an increasing or decreasing trend in multiple data packets. After identifying these features, merge the features with the same change pattern. Specifically, for features showing an increasing or decreasing sequence, simplify them into a unified range or pattern representation to reduce redundant information. For example, if a feature shows a numerical sequence [1, 2, 3] in multiple messages, it can be simplified into the form of "1-3" or "increasing sequence"; after the merge is completed, generate the simplified feature vector and store it in the variable feature library to replace the original feature vector. This feature vector is used to represent the features in all data packets containing this increasing / decreasing pattern; 3-3) For the identified fixed-length optional features, simplify them by separating the fixed part and the variable part; such features usually contain a fixed prefix part and a variable suffix part, such as the function code field in a protocol. Divide the fixed-length optional feature (such as the function code feature) into a fixed field and a variable field, corresponding to the fixed prefix part and the variable suffix part respectively; the fixed part is usually the first half of the feature, representing the basic attributes of the feature; the variable part is the suffix part, representing the variable content of the feature. Extract and store the fixed prefix part in the fixed feature library to retain the basic attributes of the feature; for the variable suffix part, further merge and simplify it, merge the suffix parts with the same variation pattern or logic into a unified suffix pattern representation, and store it in the variable feature library; finally, combine the processed fixed prefix part and the simplified variable suffix part to generate the final simplified feature vector, and replace the original feature vector and store it in the feature library; 3-4) For the associated features, input the set of field pairs extracted through correlation analysis, which contains all the identified pairwise related field pairs in the protocol data packet; for these field pairs, use them as a basis to construct three-item sets and higher-order multi-item sets; 3-5) On this basis, use the field pairs L in f 1 , f 2 ) to construct 3-item sets (three-item combinations) and calculate the support degree sup({ f 1, f 2, f 3}), where is a pair of fields in f , and 3 comes from other fields in the field set F; the support degree calculation method is as follows: where represents the number of data packets containing the field combination{ f 1, f 2, f 3}, D represents the total number of data packets; if the support degree of a certain 3-item set is greater than or equal to the threshold , then retain this item set and add it to the frequent 3-item set; use the frequent 3-item set to construct 4-item sets, continue to calculate the support degree and screen out the frequent 4-item sets; and so on, until no new frequent item sets are generated.

[0036] In an optional embodiment, the specific method of step 4) includes: Step 4-1) Build a template based on the three features that have been semantically simplified in the feature library; extract the simplified fixed features from the feature library, which represent the parts that remain unchanged in position and content in all messages, and use them as the basic structure in the template; Step 4-2) For the variable features, define corresponding variable ranges or rules for the corresponding fields, allowing their values to be adjusted dynamically according to the actual situation; Step 4-3) For the associated features, reduce the complex relationships between fields to clear derivation logics, such as deriving the content of subsequent fields based on the values of certain previous fields; by integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed.

[0037] Step 4-4) Integrate the feature information of the fixed features, variable features, and associated features to form an integrated feature vector , which combines all the feature results proposed above; then, download the publicly available pcap data sample set from the website https: / / wiki.wireshark.org / SampleCaptures / , and calculate the corresponding feature vector V for each field s in the pcap packets in these sample sets s , and embed these feature vectors into a 16-dimensional space to form a template field feature vector space.

[0038] In an optional embodiment, the specific method of step 5) includes: When a new message arrives, match it with the established protocol feature template and calculate the similarity distance: first, follow the field information in the protocol feature template and extract the content V of each field in the message according to the positional relationship i , and then obtain the template field closest to the field information in the protocol feature template field feature vector space according to the following calculation formula: ; In the formula, S is the value space of the template field, represents the absolute value of the 16-dimensional feature vector calculated for the field to be analyzed, is the cosine similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s, and by searching for the minimum value, if the result of this value is less than 5%, it is determined that the semantics of the field to be analyzed is the semantics of the corresponding template field, otherwise it is determined that the field is a new semantics, and the feature vector of the field is embedded into the feature space, continuously enriching and improving the feature vectors in the feature space, realizing the optimization and update of the template field, thereby completing the recognition and parsing of the new message data and the dynamic update of the protocol feature template.

[0039] Embodiment 2: As Figure 1 and2 As shown in 2 , a protocol reverse parsing method based on multi-feature extraction and semantic simplification proposed by the present invention includes: reading a known protocol pcap packet, using a network analysis tool to read the pcap data packet of the known protocol, and extracting the original message data.

[0040] In an optional embodiment, the network analysis tool is Wireshark.

[0041] In an optional embodiment, the message is divided into fields, the field information is identified and extracted, and the field and its corresponding value, position, length, and semantic information are saved to a CSV file; In an optional embodiment, based on the field segmentation, fixed features are extracted from the protocol data and stored in the corresponding fixed feature library, including: A1. For the version number field, by scanning multiple data packets, it is confirmed that the field value at this position remains consistent in all data packets, and then its fixed offset position in the data packet is located. Then, the relevant information of this field is extracted and merged into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value. For the site identifier, match the field with the known site ID pattern or site encoding, locate and extract this field, and merge it into the feature vector f l =[FP, FL, FV, site identifier]; A2. For protocols with undetermined fixed features, input multiple data packet samples, and through the comprehensive index of fixed fields I Determine the positions of those fields in the protocol header that do not change with time, data stream, or data packet content. These fields usually have fixed offsets and lengths and are clearly specified in the protocol document or standard. Evaluate the consistency of the fields by calculating the standard deviation of the field positions. If the position changes of the field in different samples are very small, then it can be considered a fixed field. The standard deviation of the field position The calculation formula is as follows: ; Among them, m is the number of fields, n is the number of samples, FP ij is at the i th position in the j th data packet, is the mean value of the field positions. Define a consistency index C , used to quantify the consistency of the field. This index can set a threshold based on the standard deviation: ; Among them, is a very small constant to prevent division by zero errors when the standard deviation is zero. If C has a high value, it indicates better consistency in the field positions;

[0042] A3. Obtain the field content of the current FP and improve the search efficiency through the "bad character rule" and "good suffix rule". Construct a bad character table to record the last occurrence position of each character in the pattern. Construct a good suffix table to record the matching information of the suffix part of the pattern. Utilize the bad character rule and good suffix rule to skip unnecessary comparisons and improve the matching efficiency. Calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of the fixed field I = w 1 C + w 2Pattern Score , w 1 and w 2 are weight factors.

[0043] A4. Ensure the accuracy of the above fixed field feature extraction through frequency analysis and value change detection. Conduct frequency analysis on the positions of each field in the data packet and calculate the frequency of the position , if , then this field can be considered as a fixed position. Conduct change detection on the value of each field and calculate its change rate r ( v i ), if , then the value of this field hardly changes and can be determined as a fixed value. Determine that the field feature at the FP position is a fixed feature, and obtain the field length FL and field value FV at this position, represented as a feature vector f i = [FP, FL, FV, Label], and store it in the fixed feature library DB f = f v , f l ,…, f i ; A5. For the serial number field, by detecting the consecutive field values in the data packet, find its increasing or decreasing pattern, extract this field and merge it into the feature vector d s=[FP, FL, FV, serial number]. For the IP address and MAC address fields, locate these fields in the data packet according to the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors respectively d I =[FP, FL, FV, IP address] and d M =[FP, FL, FV, MAC address]; A6. For the fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change rules therein, including the differential analysis and time series analysis of the field values. Identify the fields with change rules by analyzing the differences in the field values in multiple data packets. First, calculate the variance of the field values , if the variance of a certain field is large, it indicates that its value changes greatly and may contain some change rules: ; Among them, FV ij is the field value at the i th position in the j th data packet, is the average value of the field values, m is the number of fields, n is the number of data packets; A7. Calculate the change rate of the field values in adjacent data packets . If the field values show a certain change rate, they may have some change rules. The calculation formula is as follows: ; Among them, FV ij is the field value at the i th position in the j th data packet, FV (i+1)j is the field value at the i+1 th position in the j th data packet.

[0044] Calculate the comprehensive difference index V 1: ; Among them, represents the difference of the field values. The smaller the variance, the more stable the field. is the average value of the change rate, indicating the average change of the field values. A smaller change rate indicates that the change of the field values is more stable. s 1 and s 2 are weight factors, which are adjusted according to the actual situation; In an alternative embodiment, based on field segmentation, variation features are extracted from protocol data and stored in a corresponding variation feature library, including: B1. For the sequence of field values, the method of autocorrelation time series analysis is used to identify patterns. Autocorrelation can be used to measure the similarity between field values at different time points, and thus identify whether there are regular variations. The autocorrelation coefficient is calculated by the following formula: ; where k is the lag step, FV ij is the i th field value at the j th position in the m th data packet, n is the number of fields, is the number of data packets; ; where and are weight factors, which are adjusted according to the actual situation.

[0045] Determine FV as the variation feature value of the field, thereby determining the field position FP and length FL, and represent the extracted potential variation features as a feature vector d i = [FP, FL, FV, Label], and finally store it in the variation feature library DB d = d s , d I , d M , …, d i ; In an alternative embodiment, based on field segmentation, association features are extracted from protocol data and stored in a corresponding association feature library, including: C1. For association features, first, analyze the field structure in the data packet to locate field pairs that may have dependencies. For the checksum field, extract the field and its corresponding data content segment, apply the checksum algorithm CRC to calculate and verify the matching of the checksum field with the calculation result, and merge it into the feature vector . For the data length field, locate the field in the protocol that identifies the data length, extract the field value and calculate the actual data length, verify the consistency of the length field value with the actual data length, and merge it into the feature vector . All the extracted association features are verified multiple times to ensure their consistency and effectiveness; C2. For the associated fields that are difficult to extract through empirical features, correlation analysis is used to mine their features. First, calculate the covariance between protocol fields. Covariance can measure whether two fields have a linear correlation. When the covariance is close to zero, the two fields are independent; when the covariance is far from zero, it indicates that there is an association between the fields. Covariance Cov ( FV X , FV Y ) The formula is as follows: ; Wherein, m is the number of fields, n is the number of data packets, and are respectively the values of field FV X at the i th data packet and the j th field position, and the value of field FV Y at the k th field position in the th data packet. and F X are respectively the means of field F Y and field C3. Calculate the conditional entropy of the protocol fields. Conditional entropy measures the uncertainty of another field given the value of one field. The smaller the conditional entropy, the better one field can predict another field, indicating a strong association between the fields. Conditional entropy H ( FV Y | FV X ) The formula is as follows: ; Wherein, is the joint probability that field FV X and field FV Y take values and simultaneously. is the conditional probability of field under the condition of field ; C4. Calculate the mutual information of the protocol fields. Mutual information can measure the amount of information shared between two fields, indicating the degree of dependence between them. The larger the mutual information, the stronger the dependence between the fields. Mutual information I ( FV X; FV Y ) The calculation formula is as follows: ; where, is the field FV X and the field FV Y 's joint probability distribution. and are respectively the marginal probability distributions of the field FV X and the field FV Y ; C5. Combine the covariance, conditional entropy, and mutual information obtained from the above calculations to create a comprehensive metric A to evaluate the degree of association between fields. The comprehensive metric A is calculated as follows: ; where, Cov ( FV X , FV Y ) is the covariance between two fields. The greater the covariance, the stronger the correlation. H ( FV Y | FV X ) is the conditional entropy of the field FV Y given the field FV X . The smaller the conditional entropy, the stronger the dependence between FV X and FV Y .

[0046] Determine FV as the change characteristic value of the field. Accordingly, determine the field position FP and length FL, and represent the extracted potential association characteristics as the feature vector r i = FP , FL, FV, Label] and finally store it in the association feature library .

[0047] In an alternative embodiment, semantic simplification is performed on the fixed features, change features, and association features in the feature library to remove redundant information and refine more representative and distinguishable semantic features, including: D1. For fixed features, directly merge multiple fields with the same semantics to form a new feature vector. For example, the features extracted from the original version number are: [0x01, "version number"], [0x02, "version number"], [0x03, "version number"]. After simplification: [{0x01, 0x02, 0x03}, "version number"]; D2. For variable features, semantic simplification is divided into two parts, which are processed separately for increasing / decreasing features and fixed-length optional features. First, for the increasing / decreasing features that have been extracted, the first step of the method is to further identify and confirm the change patterns of these features. Increasing / decreasing features usually show continuous changes in numerical values or timestamps, such as the field values showing an increasing or decreasing trend in multiple data packets. After identifying these features, the method merges the features with the same change pattern. Specifically, for features showing an increasing or decreasing sequence, they are simplified into a unified range or pattern representation to reduce redundant information. For example, if a feature shows a numerical sequence [1, 2, 3] in multiple messages, it can be simplified into the form of "1 - 3" or "increasing sequence". After the merging is completed, a simplified feature vector is generated and stored in the variable feature library to replace the original feature vector. This feature vector is used to represent the features in all data packets containing this increasing / decreasing pattern; D3. For the identified fixed-length optional features, the method simplifies them by separating the fixed part and the variable part. Such features usually contain a fixed prefix part and a variable suffix part, such as the function code field in a protocol. The function code feature is divided into a fixed field and a variable field. The fixed part is usually the first half of the feature, representing the basic attributes of the feature; the variable part is the suffix part, representing the variable content of the feature. The method extracts and stores the fixed prefix part in the fixed feature library to retain the basic attributes of the feature. For the variable suffix part, further merging and simplification are carried out, and the suffix parts with the same change pattern or logic are merged into a unified suffix pattern representation and stored in the variable feature library. Finally, the processed fixed part and the simplified variable suffix part are combined to generate the final simplified feature vector and replace the original feature vector and store it in the feature library; D4. For associated features, the set of field pairs L extracted through correlation analysis of the input, which contains all the identified pairwise-related field pairs in the protocol data packets. Using these field pairs as a basis, three-item sets and higher-order multi-item sets are constructed; D5. On this basis, using each pair of fields L in f 1 , f 2 ), construct three-item combinations (3-item sets) and calculate the support sup({f 1, f 2, f 3}), where a pair of fields in, and f 3 comes from other fields in the field set F . The support calculation method is as follows: ; where represents the number of data packets containing the field combination { f 1, f 2, f 3}, D represents the total number of data packets. If the support of a certain 3-item set is greater than or equal to the threshold , then retain this item set and add it to the frequent 3-item sets. Use the frequent 3-item sets to construct 4-item sets, continue to calculate the support and filter out the frequent 4-item sets. And so on, until no new frequent item sets are generated.

[0048] In an optional embodiment, a matching template is established according to the protocol field characteristics simplified by semantics, and the new protocol is template-matched to achieve fast reverse analysis, including: E1. Build a template based on three features that have been semantically simplified in the feature library. Extract the simplified fixed features from the feature library, which represent the parts that are always unchanged in position and content in all packets, and use them as the basic structure in the template; E2. For variable features, define corresponding variable ranges or rules for the corresponding fields, allowing their values to be dynamically adjusted according to the actual situation; E3. For associated features, reduce the complex relationships between fields to clear derivation logics, such as deriving the content of subsequent fields based on the values of certain preamble fields. By integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed; E4. Fuse the feature information of fixed features, variable features, and associated features to form an integrated feature vector , this feature vector combines all the feature results proposed above. Then, download the publicly available pcap data sample set from the website https: / / wiki.wireshark.org / SampleCaptures / , and calculate the corresponding feature vector V s for each field s in the pcap packets in these sample sets, and embed these feature vectors into a 16-dimensional space to form a template field feature vector space; E5. When a new packet arrives, the system will match it with the established template and calculate the similarity distance: First, follow the field information in the template and extract the content V of each field in the packet according to the positional relationship i, and then the template field with the closest distance to the field information in the template field feature vector space is obtained according to the following calculation formula: ; in the above formula, S is the value space of the template field, represents the absolute value of the 16-dimensional feature vector calculated from the field to be analyzed, is the cosine similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s. By searching for the minimum value, if the result of this value is less than 5%, it is determined that the semantics of the field to be analyzed is the semantics of the corresponding template field; otherwise, it is determined that the field is a new semantics, and the feature vector of the field is embedded into the feature space to continuously enrich and improve the feature vectors in the feature space, realizing the optimization and update of the template field.

[0049] Embodiment 3: As Figure 3 shown, a protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention, for the semantic simplification schematic of the function code, includes: 1. Merge multiple fixed fields with the same semantics into a new feature vector. For example, multiple fields [0x01, "read operation"], [0x02, "read operation"], [0x03, "read operation"] in the original function code can be merged into [{0x01, 0x02, 0x03}, "read operation"].

[0050] 2. Identify the fields (such as serial numbers or counters) that show increment or decrement in the function code, and merge the fields with the same change pattern into a range or pattern representation. For example, simplify the numerical sequence [1, 2, 3] to "1-3" or "increasing sequence".

[0051] 3. Divide the function code fields into a fixed prefix and a variable suffix. The fixed prefix represents the basic attributes of the function (such as the operation type), and the variable suffix represents the variable content (such as parameters or addresses). For example, the function codes [0xA1, 0x01] and [0xA1, 0x02] can be simplified to [0xA1, {0x01, 0x02}].

[0052] 4. Merge the fixed fields to generate a simplified fixed feature vector. Classify the changing fields, extract the increment and decrement features and the fixed-length optional features, and simplify them respectively. Store the simplified feature vectors in the feature library to replace the original features.

[0053] Embodiment 4: In this embodiment, the protocol reverse parsing method based on multi-feature extraction and semantic simplification of the present invention is implemented in the form of a simulation test. All the tests in this example are completed on a test host running the 64-bit Ubuntu 20.04 operating system, which is equipped with an Intel Core i7-11700 CPU with a main frequency of 2.50 GHz and a memory capacity of 32 GB. The test program is written in the Python language. The core module depends on scapy for PCAP packet parsing, uses pandas for structured management of field information, and at the same time uses the mlxtend library to implement the specific execution of the frequent item set mining algorithm, including: 1. Select multiple public protocol data sets, including actual capture files of typical protocols such as HTTP, FTP, and DNS, and simulate TCP sessions through a self-constructed environment to supplement specific field mutation scenarios. After standardization processing, the data format is unified.

[0054] 2. Let the set of all fields obtained after field division processing in the protocol message be F ={ f 1, f 2,…, f n}, where each field f i represents an independent semantic unit in the protocol, such as a length field, a function code, etc.

[0055] 3. Identify the correlation relationship between field pairs through mutual information, conditional entropy, etc., and form a field pair set . Each pair ( f i , f j ) meets a certain correlation threshold and is initially considered to have a potential dependency relationship.

[0056] 4. Based on the field pairs in L 2, construct three-item combinations (3-itemsets) and calculate their frequencies of occurrence in the dataset. For any field combination , its support is defined as follows: ; where, I k represents a field combination containing k fields; a k represents the numbers selected from all fields; D ( I k ) represents the subset of data packets that simultaneously contain the fields , ,... ; is the number of data packets in the subset; is the total number of data packets in the complete protocol dataset; sup( I k ) is the field combination I k support, which is used to measure the representativeness of this combination in the entire dataset.

[0057] 5. Set according to the protocol type and the experience of field combination complexity to be between 0.05 and 0.15, is the preset minimum support threshold. When the support of an itemset satisfies , then I k is regarded as a frequent itemset. Combine all frequent k itemsets into a frequent itemset set . Based on (frequent 3-itemset), construct candidate 4-itemsets through pairwise connection operations , and repeat the support calculation and screening process in turn, recursively, until (no new frequent itemset) is reached.

[0058] 6. Finally, form and output multi-level frequent field combinations , which contains the stable co-occurrence relationship between multiple fields in the protocol, k represents the itemset starting from K represents the maximum value of the itemset size, represents all frequent k itemsets.

[0059] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A protocol reverse parsing method based on multi-feature extraction and semantic simplification, characterized in that Including: Step 1) Use a network analysis tool to read PCAP data packets of known protocols, extract the original message data, perform field division on the original message data, identify and extract field information, and save the field, its corresponding value, position, length, and semantic information to form protocol data; Step 2) Extract fixed features, variable features, and associated features from the protocol data, and store them respectively in a fixed feature library for storing fixed features of fields, a variable feature library for storing variable features of fields, and an associated feature library for storing dependency features between fields; Step 3) Simplify the semantics of the fixed features, variable features, and associated features, remove redundant information, and refine distinguishable semantic features; Step 4) Based on the semantically simplified semantic features, construct a protocol feature template; the protocol feature template includes a basic structure corresponding to the fixed features, a variable range or rule corresponding to the variable features, and a derivation logic corresponding to the associated features; Step 5) Identify and parse new message data and dynamically update the protocol feature template based on the similarity matching result between the new message data and the protocol feature template.

2. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 1, characterized in that The specific method of Step 1) includes: Step 1-1) Use a network analysis tool to read pcap data packets of known protocols, extract the original message data, and store the original message data as an original byte stream; Step 1-2) Parse the original message data, divide each field in the message according to the protocol format, identify the position, length, and type of each field according to the protocol document or known standard, and determine the boundary of the field; for each field, read its data content and record the actual value, start position, and length of the field in the data packet; Step 1-3) According to the protocol document or known standard, combine the type, value, and position of the field to extract the semantic information of each field; Step 1-4) Save the extracted value, position, length, and semantic information of the field to form a complete field data record.

3. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 2, characterized in that, The fixed features in the fixed feature library are field attributes with fixed positions and values that do not change over time or data packets in the protocol message, including version number, site identifier.

4. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 3, characterized in that The variable features in the variable feature library are field attributes with values that change over data packets or time and have specific rules in the protocol message, including sequence number, timestamp, IP address, and MAC address.

5. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 4, wherein The dependency features in the associated feature library are field attributes with logical or statistical dependency relationships between fields in the protocol message, including checksum, data length.

6. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 5, characterized in that The specific method of Step 2) includes: Step 2-1) For the protocol data with clear fixed features, for the version number field in the fixed features, by scanning multiple data packets, confirm that the field value at this position remains consistent in all data packets, and then locate its fixed offset position in the data packet. Then, extract the relevant information of this field and merge it into the feature vector f v =[FP, FL, FV, version number], where FP is the message data position, FL is the message data length, and FV is the message data value; for the site identifier in the fixed features, match the field with the known site ID pattern or site code, locate and extract this field, and merge it into the feature vector f l =[FP, FL, FV, site identifier]; Step 2-2) For protocol data whose fixed features are not yet clear, input multiple packet samples, and through the comprehensive index of fixed fields I Determine the positions of the fields in the protocol header that do not change with time, data stream, or packet content. Evaluate the consistency of the fields by calculating the standard deviation of the field positions. If the positions of the fields change very little in different samples, then it is considered a fixed field. The standard deviation of the field positions The calculation formula is as follows: ； Among them, m is the number of fields, n is the number of samples, FP ij is the field value at the i -th position in the j -th data packet, is the mean of the field positions, and a consistency index C is defined to quantify the consistency of the fields. The calculation formula of the consistency index C is as follows: ; wherein, is a very small constant to prevent a division-by-zero error when the standard deviation is zero; Step 2-3) Obtain the field content at the current message data position FP, and improve the search efficiency through the bad character rule and the good suffix rule; construct a bad character table to record the last occurrence position of each character in the pattern; construct a good suffix table to record the matching information of the suffix part of the pattern; calculate the matching success rate Pattern Score = (number of successful matches / total number of matching attempts) * 100%, and calculate the comprehensive index of the fixed field I = w 1 C + w 2Pattern Score , w 1 and w 2 are the first and second weight factors respectively; Step 2-4) Ensure the accuracy of the above-mentioned fixed-field feature extraction through frequency analysis and value change detection; perform frequency analysis on the positions of each field in the data packet. Let the i th position be , and the frequency of the field appearing at this position be ; if , then this field can be considered as a fixed position; perform change detection on the value i of the v i th field, calculate its change rate r ( v i ), if , then the value of this field hardly changes and can be determined as a fixed value; determine that the field feature at the message data position FP is a fixed feature, and obtain the field length FL and field value FV at this position, expressed as a feature vector f i = [FP, FL, FV, Label], and store it in the fixed feature library DB f = f v , f l , …, f i ; Step 2-5) For the fields with clearly defined changing features in the protocol, for the serial number field, by detecting the consecutive field values in the data packet, find its increasing or decreasing pattern, extract the field and merge it into the feature vector d s =[FP, FL, FV, serial number]; for the IP address and MAC address fields, locate these fields in the data packet according to the position definition in the protocol standard or the known address format, identify their specific values, and obtain the feature vectors d I =[FP, FL, FV, IP address] and d M =[FP, FL, FV, MAC address]; Step 2-6) For the fields with unknown change characteristics in the protocol, use statistical and pattern recognition methods to analyze the field value sequence to identify the change patterns therein, including the differential analysis and time series analysis of the field values; First, calculate the variance of the field values : ； in, FV ij It is i In the data packet j The field value of the position, is the average of the field values, m is the number of fields, n is the number of packets; Step 2-7) Calculate the change rate of the field values in adjacent data packets , and the calculation formula is as follows: ； Among them, FV ij is the i field value at the j th position in the FV (i+1)j th data packet, i+1 and j is the field value at the th position in the th data packet; Calculate the comprehensive difference index V 1: ; Among them, represents the difference of field values. The smaller the variance, the more stable the field; is the average value of the change rate, indicating the average change of field values; s 1 and s 2 are the first and second weight factors respectively; Step 2-8) For the sequence of field values, an autocorrelation time series analysis method is used to identify the pattern; the autocorrelation coefficient V 2 is calculated by the following formula: ； in, k is the lag step length, FV ij It is i In the data packet j The field value of the position, m is the number of fields, n is the number of packets; Step 2-9) Create a comprehensive indicator by combining the differential analysis and autocorrelation analysis of field values V to evaluate whether the field is a fixed field: ； Among them, and are the first and second weight factors respectively; Determine that the field value FV is the change feature value of the field, and accordingly determine the field position FP and length FL, and represent the extracted potential change features as a feature vector d i =[FP, FL, FV, Label], and finally store it in the change feature library DB d = d s , d I , d M ,…, d i in; Step 2-10): For the protocol data with clearly associated features, first, analyze the field structure in the data packet to locate the field pairs that may have dependencies; for the checksum field, extract the field and its corresponding data content segment, apply the CRC check algorithm to calculate and verify the matching of the checksum field with the calculation result, and merge it into the feature vector ; for the data length field, locate the field in the protocol that identifies the data length, extract the field value and calculate the actual data length, verify the consistency between the length field value and the actual data length, and merge it into the feature vector ; Step 2-11): For protocol data whose associated features are not yet clear, perform correlation analysis to mine its features; First, calculate the covariance between protocol fields. Covariance can measure whether two fields have linear correlation; When the covariance is close to zero, the two fields are independent; When the covariance is far from zero, it indicates that there is an association between the fields. Covariance Cov ( FV X , FV Y ) The calculation formula is as follows: ； Among them, m is the number of fields, n is the number of data packets, and are respectively the value of the FV X th field at the i th position in the j th data packet, and the value of the FV Y th field at the k th position in the i-th data packet; and are respectively the average values of the F X field and the F Y field; Step 2-12) Calculate the conditional entropy of the protocol fields. The conditional entropy measures the uncertainty of another field given the value of one field; the conditional entropy H ( FV Y | FV X ) The calculation formula is as follows: ； Among them, is the field FV X and the field FV Y take values simultaneously and of the joint probability; is the conditional probability of the field under the condition of the field ; Step 2-13) Calculate the mutual information of protocol fields. Mutual information measures the amount of information shared between two fields. The mutual information I ( FV X ; FV Y ) The formula is as follows: ； Among them, is the field FV X and the field FV Y of the joint probability distribution; and are respectively the field FV X and the field FV Y of the marginal probability distribution; Step 2-14) Combine the covariance, conditional entropy, and mutual information calculated above to create a comprehensive metric A to evaluate the degree of association between fields; the comprehensive metric A The calculation method is as follows: ； Among them, Cov ( FV X , FV Y ) is the covariance between two fields; H ( FV Y | FV X ) is the conditional entropy of field FV Y given field FV X ; I ( FV X ; FV Y ) is the mutual information, indicating FV X and FV Y the amount of information shared between; 、 、 are the first, second, and third weight factors respectively, and the sum is 1; Determine that the field value FV is the change characteristic value of the field, and accordingly determine the field position FP and the length FL, and represent the extracted potential association features as a feature vector r i = FP , FL, FV, Label], and finally store it in the association feature library inside.

7. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 6, characterized in that, The specific method of Step 3) includes: 3-1) For the fixed features, directly merge multiple fields with the same semantics to form a new feature vector; 3-2) For variable features, semantic simplification is divided into two parts, which are respectively processed for increasing / decreasing features and fixed-length optional features; First, for increasing / decreasing features, further identification is carried out to confirm the change patterns of these features; After identifying these features, features with the same change pattern are merged. For features presented as increasing or decreasing sequences, they are simplified into a unified range or pattern representation to reduce redundant information; After the merging is completed, the simplified feature vector is generated and stored in the variable feature library to replace the original feature vector; 3-3) For fixed-length optional features, simplification is achieved by separating the fixed part and the variable part; The fixed-length optional features are divided into a fixed field and a variable field, corresponding to the fixed prefix part and the variable suffix part respectively; The fixed prefix part is extracted and stored in the fixed feature library to retain the basic attributes of the feature; For the variable suffix part, further merging and simplification are carried out, and the suffix parts with the same change pattern or logic are merged into a unified suffix pattern representation and stored in the variable feature library; Finally, the processed fixed prefix part and the simplified variable suffix part are combined to generate the final simplified feature vector, which replaces the original feature vector and is stored in the feature library; 3-4) For associated features, input the set of field pairs extracted through correlation analysis L 2, which contains all the identified pairwise correlated field pairs in the protocol data packet; for these field pairs, use them as a basis to construct three-item sets and higher-order multi-item sets; 3 - 5) Use field pairs L For each pair of fields in f 1 , f 2 ), construct 3 - item sets and calculate the support sup({ f 1, f 2, f 3}), where is a pair of fields in and F is from other fields in the field set ; The support calculation method is as follows: Among them, represents the number of data packets containing the field combination { f 1, f 2, f 3}, D represents the total number of data packets; if the support of a certain 3-item set is greater than or equal to the threshold , then retain this item set and add it to the frequent 3-item set; use the frequent 3-item set to construct 4-item sets, continue to calculate the support and filter out the frequent 4-item sets; and so on until no new frequent item sets are generated.

8. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 7, wherein The specific method of step 4) includes: Step 4-1) Build a template based on the three features that have been semantically simplified in the feature library; Extract the simplified fixed features from the feature library and use them as the basic structure in the template; Step 4-2) For variable features, define corresponding change ranges or rules for the corresponding fields to allow their values to be dynamically adjusted according to the actual situation; Step 4-3) For associated features, reduce the complex relationships between fields to clear derivation logics; By integrating and simplifying these three types of features into a high-level semantic structure, the final template framework is formed; Step 4-4) Integrate the feature information of fixed features, variable features, and associated features to form an integrated feature vector ， This feature vector combines all the previously proposed feature results; then, download the publicly available pcap data sample set, and calculate the corresponding feature vector V for each field s of the pcap packets in these sample sets respectively s and embed these feature vectors into a 16-dimensional space to form a template field feature vector space 9. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to claim 8, wherein The specific method of step 5) includes: When a new message arrives, it is matched with the established protocol feature template, and the similarity distance is calculated: First, follow the field information in the protocol feature template, and extract the content V of each field in the message according to the positional relationship i , and then obtain the template field with the closest distance in the protocol feature template field feature vector space according to the following calculation formula: ; where S is the value space of the template field, represents the absolute value of the 16-dimensional feature vector calculated from the field to be analyzed, is the cosine similarity value between the feature vector of the field to be analyzed and the feature vector of the template field s. By searching for the minimum value, if the result of this value is less than 5%, it is determined that the semantics of the field to be analyzed is the semantics of the corresponding template field; otherwise, it is determined that the field has a new semantics, and the feature vector of this field is embedded into the feature space to continuously enrich and improve the feature vectors in the feature space, realizing the optimization and update of the template field. Thus, the recognition and parsing of new message data and the dynamic update of the protocol feature template are completed.

10. The protocol reverse parsing method based on multi-feature extraction and semantic simplification according to any one of claims 1 to 9, characterized in that, The network analysis tool is Wireshark.

Images