Network protocol format reverse analysis method and device based on collaborative evolution algorithm

Abstract

The present invention is a method and device for reverse analysis of network protocol formats based on a co-evolutionary algorithm. The main purpose is to provide a protocol reverse tool based on network protocol traffic, which can analyze the protocol format without the need for protocol specifications or binary codes or programs that implement the protocol. The specific steps include pre-processing the traffic through a private protocol data extraction module and a message transmission direction identification module, and then handing the pre-processed traffic to a pre-clustering module for grouping. Then, each group of traffic is input into a protocol format extraction module constructed based on a multiple sequence alignment algorithm and a probabilistic statistical algorithm to obtain multiple groups of initial protocol specification seeds. Finally, a multi-population co-evolutionary algorithm is constructed on the basis of the two populations to optimize the obtained protocol specifications. This method comprehensively utilizes the results of multiple sequence alignment and probabilistic statistical algorithms, effectively improving the accuracy of reverse analysis of protocol specifications.

Description

Technical Field

[0001] The present invention belongs to the technical field of information security, and provides a network protocol format reverse analysis method and device based on a collaborative evolutionary algorithm. Background Art

[0002] With the rapid development of technologies such as the Internet of Things (IoT), the Internet of Vehicles (IoV), and industrial control networks in recent years, a large number of proprietary protocols have emerged in network traffic within these fields. These protocols lack publicly available format documentation and source code. More importantly, many proprietary protocols have security vulnerabilities in their designs, and due to their closed-source nature, their security cannot be guaranteed. This situation limits conventional analytical methods for protocol vulnerabilities, network behavior analysis, network monitoring, and botnet detection. Therefore, protocol reverse engineering typically focuses on network traces or binary programs that implement the protocol.

[0003] The former protocol reverse engineering method based on network traces has broad application prospects, but it suffers from a slight lack of accuracy and cannot process encrypted traffic. The latter is feasible for analyzing encrypted traffic, but it is generally difficult to obtain binary programs that implement the protocol. The method and device described in this invention belong to the category of protocol reverse engineering methods based on network traces. Existing methods in this field can be divided into two categories: those based on multiple sequence alignment and those based on the principles of probability and statistics. Methods based on multiple sequence alignment include PI, Netzob, Netplier, etc. Methods based on the principles of probability and statistics include Biprominer, ProDecoder, ProGraph, etc.

[0004] The PI tool proposed in 2004 marked the beginning of the field of traffic-based protocol reversal. With the increase of private traffic in recent years, this field has gradually become a research hotspot. Among them, protocol reversal based on sequence alignment is an important branch, and new results are constantly emerging. These methods usually compare according to the characteristics of byte changes, and use reward and punishment functions to guide the construction of a scoring matrix. While continuing the ideas of tools such as PI, research methods such as Netzob and Netplier have proposed some innovations, such as the optimization of reward and punishment functions, the optimization of error judgment mechanisms, and the judgment of specific character semantics, such as the forward and reverse combination of FD fields, delimiters and lengths, and the introduction of posterior probabilities to assist in keyword judgment. However, this type of method also has some shortcomings: some methods introduce too much prior knowledge of the protocol; in addition to keyword judgment, there are many errors in the division of the functional segment format boundaries;

[0005] The Biprominer and ProDecoder algorithms use statistical methods to find keywords. Biprominer was developed for binary protocols. ProDecoder, based on Biprominer, has been modified and improved for text protocols. ProGraph groups the target protocol and then constructs a graph model to analyze the correlation between the groups. Methods based on probabilistic statistics typically only focus on the frequency of occurrence of key bytes or bits, ignoring the alignment between fields.

[0006] Methods based on multiple sequence alignment typically rely on known protocol samples or reference sequences, inferring the structure and function of the target protocol by aligning its sequence data with the known sequence. This method can quickly and accurately identify parts with high similarity, but its accuracy may be limited in the case of highly variable protocols or unknown structures.

[0007] In contrast, methods based on probabilistic statistics are more flexible. They can discover patterns and regularities hidden in protocol traffic data through statistical analysis and model inference. This approach requires no prior knowledge and can effectively infer complex protocol structures, but it can be computationally complex when processing large amounts of data. Summary of the Invention

[0008] The present invention aims to solve the problem of how to reversely deduce the protocol format by analyzing network protocol traffic in the absence of protocol specifications or binary codes implementing the protocol.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention provides a reverse analysis method for a private network protocol format, comprising the following steps:

[0011] Step 1. Packet Sequence Collection and Preprocessing: Collect the protocol traffic to be analyzed and capture the message data packets, which are the request traffic packet sequence and the response traffic packet sequence. Use network packet analysis tools to perform preliminary analysis on the original traffic message data packets to extract the payload and the underlying known protocol information.

[0012] Step 2. Private protocol traffic clustering: Using hierarchical clustering, the payloads obtained in step 1 are clustered using the underlying known protocol information obtained in step 1 to obtain clustering results. In the clustering results, payloads with similar structure, content, and underlying known protocol information are classified into the same class.

[0013] Step 3. Protocol specification seed generation: The clustering results of step 2 are processed based on multiple sequence alignment and format specification analysis methods based on probabilistic statistics to generate two types of protocol specification seeds;

[0014] Step 4. Use a multi-population co-evolution algorithm: Use the two types of protocol specification seeds obtained in step 3 as two populations, perform sub-population optimization, population crossover, and seed mutation to achieve multi-population co-evolution and obtain a network protocol format that combines the advantages of the two types of network protocol format inverse methods.

[0015] In the above technical solution, step 1 includes the following steps:

[0016] Step 1.1: Collect protocol traffic packets: Use tools like tcpdump or Tshark to capture network traffic and save it as a PCAP file. The data in the PCAP file is stored in binary packets. Each packet contains the network data captured within a certain period of time, as well as related metadata, including timestamp, source IP address, destination IP address, and protocol type.

[0017] Step 1.2, separation of unknown protocol data: Use scapy, pyshark, and dpkt tools to separate the payload of the protocol to be analyzed from the data packet;

[0018] Step 1.3: Parsing known protocols: Use scapy, pyshark, and dpkt tools to parse known protocol traffic, extract the address, port number, and protocol packet transmission direction information, and classify the traffic packets into two groups: client and server.

[0019] In the above technical solution, the specific steps of step 2 are:

[0020] Step 2.1: Construct a similarity matrix: Calculate the similarity between message packets to construct a symmetrical similarity matrix.

[0021] Step 2.1.1: The similarity is calculated using the Needleman-Wunsch algorithm, with a matching score of 1, a mismatch score of -1, and a gap penalty of 0.

[0022] Step 2.1.2: Normalize the similarity score by dividing the total score by the length of the longer message sequence in the comparison.

[0023] Step 2.1.3, construct an n×n symmetric matrix with < <X 11 , X 12 ,...,X 1n >, <X 21 , X 22 ,...,X2n >..., <X n1 , X n2 ,...,X nn >> indicates that X ij The distance from the i-th message sequence to the j-th message sequence, n is the number of message sequences, X ii =0,X ij =X ji ,

[0024] Step 2.1.4. Store the similarity matrix in the file Sequence_Distance.txt as text.

[0025] Step 2.2, perform hierarchical clustering: Based on the constructed similarity matrix, perform hierarchical clustering based on the UPGMA algorithm and construct a phylogenetic tree according to the cluster hierarchy;

[0026] Step 2.2.1. Initially, each message sequence is considered as an initial cluster;

[0027] Step 2.2.2: Merge the two closest clusters. The distance between the new cluster and the other clusters is the average distance between its subclasses and the other clusters.

[0028] Repeat steps 2.2.1-2.2.2 until only one cluster remains, and construct a phylogenetic tree based on the cluster hierarchy.

[0029] Step 2.3, grouping private protocol message sequences: Based on the phylogenetic tree constructed by hierarchical clustering, iteratively try different positive integer k values ​​to divide the protocol message sequences into 2 k There are k=2 or k=3 classes, and each class of messages is taken as a group to form an original input for constructing a protocol format seed.

[0030] In the above technical solution, step 3 specifically includes:

[0031] Step 3.1: Generate seed based on the protocol specification of multiple sequence alignment to form the first population;

[0032] Step 3.1.1, use the protocol reverse engineering tool based on multiple sequence alignment to obtain the protocol specification, and build a protocol specification seed generator based on multiple sequence alignment based on the multiple sequence alignment tool mafft,

[0033] Step 3.1.2: Process long sequences and use the L-INS-i algorithm to handle highly divergent sequences;

[0034] Step 3.1.3: For short sequences or sequences with large differences, use the G-INS-i algorithm.

[0035] Step 3.1.4: During the multiple sequence alignment, the identifier "-" is used to fill the gaps generated during the alignment process;

[0036] Step 3.1.5. Save the output in FASTA format, with each line corresponding to a message sequence;

[0037] Step 3.1.6. Save the final result to the file Output_MSA_Oneline.txt.

[0038] Step 3.2: Generate protocol specification seeds based on the probability and statistics principle to form the second population;

[0039] Step 3.2.1: For binary packet sequences, use the Biprominer method to extract cells from the packet sequence and label them.

[0040] Step 3.2.2: Build a state transition model and select the project with the highest probability as the reversed protocol format;

[0041] Step 3.2.3: For text-type packet sequences, use the ProDecoder method and use keywords and transition probabilities as features.

[0042] Step 3.2.4: Perform hierarchical clustering and then use the Needleman-Wunsch algorithm to perform sequence alignment to infer the message format.

[0043] In the above technical solution, step 4 comprises the following steps:

[0044] Step 4.1, population division: The protocol specification seeds generated based on multiple sequence alignment and probability statistics principles are divided into two populations, namely the multiple sequence population and the probability statistics population;

[0045] Step 4.2, Subpopulation Optimization: Each subpopulation uses a genetic algorithm to independently perform the optimization process. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. To perform the subpopulation optimization process, the protocol specification seed needs to be evaluated for adaptability. This is done by applying the protocol specification seed to the sample and observing whether the partitioning results conform to the protocol design principles. Specifically, after applying the protocol specification to the sample, each field is gradually tested. If a field has limited variation across all samples, it is considered a successfully partitioned field. Finally, the proportion of successfully partitioned fields to the total number of fields is calculated to obtain the protocol specification seed score.

[0046] Step 4.3, Population Crossover: Use a sorting algorithm to sort the protocol specification seeds in each group separately, select the top 10% of seeds in each group as excellent seeds, and replace the bottom 10% of seeds in another group with each other, so as to achieve information sharing between different populations and promote the process of exploring the global optimal solution;

[0047] Step 4.4, seed mutation: In order to search for the global optimal solution, new seeds need to be generated for exploration. This process is completed by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields.

[0048] Step 4.5, multi-population coevolution: Repeat the above process until the seed fitness converges.

[0049] The present invention also provides a network protocol format reverse analysis device based on a collaborative evolutionary algorithm, comprising the following modules:

[0050] Acquisition unit: collects the protocol traffic to be analyzed and captures message data packets, which are request traffic packet sequences and response traffic packet sequences. The network data packet analysis tool is used to perform preliminary analysis on the message data packets of the original traffic to extract the payload and the underlying known protocol information.

[0051] Clustering unit: clusters the payloads using the underlying known protocol information through hierarchical clustering to obtain clustering results. In the clustering results, payloads with similar structure, content, and underlying known protocol information will be classified into the same class.

[0052] The seed generation unit processes the clustering results based on multiple sequence alignment and format specification analysis methods based on probability statistics to generate two types of protocol specification seeds;

[0053] The evolutionary algorithm unit uses the two types of protocol specification seeds obtained as two populations, performs sub-population optimization, population crossover, and seed mutation to achieve multi-population collaborative evolution, and obtains a network protocol format that combines the advantages of the two types of network protocol format inverse methods.

[0054] In the above solution, the acquisition unit includes:

[0055] Protocol traffic packet collection module: Uses tcpdump and Tshark tools to capture network traffic and saves it as a PCAP file. The data in the PCAP file is stored in binary packets. Each packet contains the network data captured within a certain period of time and related metadata, including timestamp, source IP address, destination IP address, and protocol type.

[0056] Unknown protocol data separation module: Use scapy, pyshark, and dpkt tools to separate the payload of the protocol to be analyzed from the data packet;

[0057] Known protocol parsing module: Use scapy, pyshark, and dpkt tools to parse known protocol traffic, extract the address, port number, and protocol data packet transmission direction information, and classify the traffic packets into two groups: client and server.

[0058] In the above solution, the clustering unit specifically includes:

[0059] The similarity matrix construction module constructs a symmetrical similarity matrix by calculating the similarity between message data packets;

[0060] The similarity calculation method uses the Needleman-Wunsch algorithm, setting the match score to 1, the mismatch score to -1, and the gap penalty to 0;

[0061] The similarity score is normalized by dividing the total score by the length of the longer of the compared message sequences;

[0062] Construct an n×n symmetric matrix with <<X 11 , X 12 ,...,X 1n >, <X 21 , X 22 ,...,X 2n >..., <X n1 , X n2 ,...,X nn >> indicates that X ij The distance from the i-th message sequence to the j-th message sequence, n is the number of message sequences, X ii =0,X ij =X ji ,

[0063] Store the similarity matrix in the file Sequence_Distance.txt as text type;

[0064] The secondary clustering execution module performs hierarchical clustering based on the constructed similarity matrix and constructs a phylogenetic tree according to the hierarchical structure of the clusters. Specifically,

[0065] Initially, each message sequence is considered as an initial cluster, and the two closest clusters are merged. The distance from the new cluster to other clusters is the average distance from its subclass to other clusters.

[0066] Repeat until only one cluster remains, and construct a phylogenetic tree based on the hierarchy of clusters;

[0067] The private protocol message sequence grouping module divides the protocol message sequence into 2 groups according to the phylogenetic tree constructed by hierarchical clustering and iteratively tries different positive integer k values. k There are k=2 or k=3 classes, and each class of messages is taken as a group to form an original input for constructing a protocol format seed.

[0068] In the above solution, the seed generation unit specifically includes:

[0069] The multiple sequence alignment protocol specification seed generation module generates the first population based on the multiple sequence alignment protocol specification seed, which specifically includes the following steps:

[0070] Use the protocol reverse engineering tool based on multiple sequence alignment to obtain the protocol specification, and build a protocol specification seed generator based on multiple sequence alignment based on the multiple sequence alignment tool mafft.

[0071] To process long sequences, the L-INS-i algorithm is used to handle highly divergent sequences;

[0072] To handle short sequences or sequences with large differences, the G-INS-i algorithm is used;

[0073] During the multiple sequence alignment, the identifier “-” was used to fill the gaps generated during the alignment process;

[0074] The output results are saved in FASTA format, with each line corresponding to a message sequence;

[0075] The final result is saved in the file Output_MSA_Oneline.txt;

[0076] The protocol specification seed generation module based on the probability and statistics principle generates the protocol specification seed based on the probability and statistics principle to form the second population, which specifically includes the following steps:

[0077] For binary packet sequences, the Biprominer method is used to extract cells from the packet sequence and label them.

[0078] Build a state transition model and select the project with the highest probability as the reverse-engineered protocol format;

[0079] For text-type package sequences, the ProDecoder method is used, with keywords and transition probabilities as features;

[0080] Hierarchical clustering was performed, and then sequence alignment was performed using the Needleman-Wunsch algorithm to infer the message format.

[0081] In the above solution, the evolutionary algorithm unit specifically includes:

[0082] The population division module divides the protocol specification seeds generated based on multiple sequence alignment and probability statistics into two populations, namely the multiple sequence population and the probability statistics population;

[0083] In the subpopulation optimization module, each subpopulation uses a genetic algorithm to independently perform the optimization process. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. To execute the subpopulation optimization process, the protocol specification seed needs to be evaluated for adaptability. This is done by applying the protocol specification seed to the sample and observing whether the partitioning results conform to the protocol design principles. Specifically, after applying the protocol specification to the sample, each field is gradually tested. If a field has limited variation across all samples, it is considered a successfully partitioned field. Finally, the proportion of successfully partitioned fields to the total number of fields is counted to obtain the protocol specification seed score.

[0084] The population crossover module uses a sorting algorithm to sort the protocol specification seeds in each group, selects the top 10% of seeds in each group as excellent seeds, and replaces the bottom 10% of seeds in another group with each other, thus sharing information between different populations and promoting the process of exploring the global optimal solution;

[0085] Seed mutation module: In order to search for the global optimal solution, new seeds need to be generated for exploration. This process is completed by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields.

[0086] The multi-population co-evolution module is used to repeat the process of subpopulation optimization, population crossover and seed mutation until the seed fitness converges.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] First, a preliminary analysis was conducted during the preprocessing phase. This fully utilized the information provided by the underlying known protocols to extract fields such as port numbers and addresses, thereby improving the accuracy of subsequent hierarchical clustering and the quality of protocol specification seed generation.

[0089] Second, a multi-population co-evolutionary algorithm combines the advantages of methods based on multiple sequence alignment and probabilistic statistics to improve the accuracy of protocol reverse engineering. By simultaneously considering information from sequence alignment and probabilistic statistics, it not only improves the accuracy of identifying known protocol structures, but also better handles protocols with unknown structures or large variations. The multi-population co-evolutionary algorithm can exchange information and experience between different populations, guiding the search process towards a more comprehensive and accurate direction, thereby improving the overall performance and effectiveness of protocol reverse engineering.

[0090] Third, the present invention utilizes readily available network protocol traces for reverse engineering of network protocols, eliminating the need for more difficult-to-obtain protocol public documentation, protocol source code, or protocol binary files. Therefore, the present invention can perform protocol reverse engineering in restricted environments, i.e., without program source code or protocol specification documentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is the overall architecture diagram of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0093] The purpose of the present invention is to provide a method for reverse analysis of private network protocol formats, which can fully utilize the advantages of both methods to perform fine-grained and accurate format field boundary determination. By simultaneously considering sequence alignment and probability statistics information, not only can the recognition accuracy of known protocol structures be improved, but also protocols with unknown structures or large variations can be better handled. Multi-population co-evolutionary algorithms can exchange information and experience between different populations, guiding the search process towards a more comprehensive and accurate direction, thereby improving the overall performance and effect of protocol reverse analysis.

[0094] This method uses progressive multiple sequence alignment and probabilistic statistical methods to obtain an initial protocol specification seed. Subsequently, a multi-population coevolutionary algorithm is used to optimize protocol boundary determination, refine the granularity of field format boundary determination, and improve boundary determination accuracy. The analysis process does not require code or server and client binary implementations of the protocol, thus enabling reverse engineering of the protocol based on network traces within the aforementioned constraints.

[0095] A network protocol format reverse analysis method based on a co-evolutionary algorithm. The method involves parsing the underlying protocol in the data collection and preprocessing phase. The method uses the UPGMA algorithm to cluster and group protocol packets in the clustering phase. In the protocol specification seed generation phase, the method based on the multiple sequence alignment tool MAFFT and the probabilistic statistics-based Biprominer and ProDecoder methods are used to generate protocol specification seeds. In the multi-population collaboration phase, the method uses a multi-population co-evolutionary algorithm to cross-fuse the solutions obtained from the multiple sequence alignment and probabilistic statistics methods to co-evolve a protocol specification with finer granularity and higher accuracy. The method specifically includes the following steps:

[0096] S1. Collect packet sequences and preprocess them; use tools such as tcpdump and nmap to collect the protocol traffic to be analyzed. These tools help capture traffic packet sequences, including request and response traffic packet sequences. The collected traffic packet sequences constitute the input traffic set, providing basic data for subsequent analysis. Afterwards, use tools such as scapy, pyshark, and dpkt to extract the data to be analyzed and perform preliminary analysis. Based on the relationship between protocol interaction behavior and traffic packet sequences, extract and preliminarily identify private protocol information such as the packet transmission direction;

[0097] S2. Clustering of private protocol traffic: Clustering private protocol traffic and grouping traffic of the same type into the same group. This helps subsequent protocol format inference tools based on multiple sequence alignment and format inference tools based on probabilistic statistics to generate protocol specification seeds and divide populations in the co-evolution of multiple populations accordingly.

[0098] S3. Protocol specification seed generation. Two types of protocol specification seeds are generated based on multiple sequence alignment and probability statistics, respectively, as two populations. The multiple sequence alignment-based method is used to analyze the grouped protocol traffic packets, and each group generates a protocol specification seed based on multiple sequence alignment. This seed serves as an individual in the subsequent co-evolutionary algorithm. The seeds of all groups are collectively referred to as a population. Protocol specification seed generation based on probability statistics principles: The multiple sequence alignment-based method is used to analyze the grouped protocol traffic packets, and each group generates a protocol specification seed based on probability statistics principles. This seed serves as an individual in the subsequent co-evolutionary algorithm. The seeds of all groups are collectively referred to as a population.

[0099] In the above scheme, step 1 includes the following specific steps:

[0100] 1.1) Protocol Traffic Packet Collection: Use tools such as tcpdump and Tshark to capture network traffic and save it as a PCAP file. Data in a PCAP file is stored in binary packets. Each packet contains the network data captured over a specific period of time, along with associated metadata such as timestamp, source IP address, destination IP address, and protocol type. The data is stored in a txt format, using the matrix format: {{Sequence number 1, private protocol traffic packet 1}; {Sequence number 2, private protocol traffic packet 2}; ...; {Sequence number N, private protocol traffic packet N}}

[0101] 1.2) Separate unknown protocol data: Use tools such as scapy, pyshark, and dpkt to separate the payload of the protocol to be analyzed from the data packet. Create a new file named Payload.hex containing only the payload of the protocol to be analyzed in the order of the private protocol traffic packets, with each traffic line represented in hexadecimal.

[0102] 1.3) Parsing known protocols. Use tools such as scapy, pyshark, and dpkt to parse known protocol traffic and extract information such as the address, port number, and protocol packet transmission direction. Based on this information, classify the traffic packets into two groups: client and server. Save them as Server_Basic.txt and Client_Baisc.txt, respectively.

[0103] In the above technical solution, the specific steps of step 2 are:

[0104] 2.1) Construct a similarity matrix: Calculate the similarity between each packet pair to construct a similarity matrix. The similarity is calculated using the Needleman-Wunsch algorithm, setting the match score to 1, the mismatch score to -1, and the gap penalty to 0. Finally, the data needs to be normalized by dividing the total score by the length of the longer packet sequence. Finally, a symmetric matrix is ​​constructed with < <X 11 , X 12 ,...,X 1n >, <X 21 , X 22 ,...,X 2n >..., <Xn1,X n2 ,...,X nn >> indicates that X ij The distance from the i-th message sequence to the j-th message sequence, n is the number of message sequences. According to the definition of the distance matrix, X ii =0,X ij =X ji The results will be stored in the file Sequence_Distance.txt as text.

[0105] 2.2) Perform hierarchical clustering; based on the constructed similarity matrix, perform hierarchical clustering based on the UPGMA algorithm. Before describing how to construct a phylogenetic tree using the UPGMA hierarchical clustering algorithm, it is stipulated that the following symbols are used to describe the elements in the phylogenetic tree: V: represents a node set, including internal nodes and leaf nodes. E: represents an edge set, including edges connecting nodes. T = (V, E): represents a phylogenetic tree, where T is the name of the tree, V is a node set, and E is an edge set. v: represents a node in the tree, which can be an internal node or a leaf node. e: represents an edge in the tree, connecting two nodes. ρ(v): represents the parent node of node v, or empty if v is the root node. λ(v): represents the length of the edge from the root node to the node v, usually representing the evolutionary distance or branch length. Initially, each message sequence is regarded as an initial cluster, namely v i After that, merge the two most recent v i , v j, generate a new parent node u so that ρ(v i )=u,ρ(v i )=u, update λ(u), delete v from V i , v j Repeat the above two steps until ||V|| = 1. Construct a phylogenetic tree based on the hierarchical clustering process.

[0106] 2.3) Grouping private protocol message sequences; Based on the phylogenetic tree constructed by hierarchical clustering, iteratively try different positive integer k values ​​and divide the protocol message sequences into 2 k There are classes, usually k = 2 or k = 3. Each class of messages is taken as a group and constitutes the original input for constructing the protocol format seed.

[0107] In the above technical solution, step 3 specifically includes:

[0108] 3.1) Generation of protocol specification seeds based on multiple sequence alignment; use the protocol reverse engineering tool based on multiple sequence alignment to obtain the protocol specification. Construct a protocol specification seed generator based on multiple sequence alignment based on the multiple sequence alignment tool mafft. When processing longer sequences, especially when processing highly divergent sequences (such as protein sequences), the L-INS-i algorithm is used. When processing shorter sequences and sequences with greater inter-sequence differences, the G-INS-i algorithm is used. During the multiple sequence alignment process, the identifier "-" is used to fill the gaps generated during the alignment process. The output results are saved in FASTA format, but need to be adjusted to a format where one line of results corresponds to one message sequence. The final results are saved in the file Output_MSA_Oneline.txt.

[0109] 3.2) Seed generation based on probabilistic statistics: For binary packet sequences, the Biprominer method is used to extract cells from the packet sequence and label them. Finally, a state transition model is constructed, and the item with the highest probability is selected as the protocol format for reverse engineering. For text packet sequences, the ProDecoder method is used. Keywords and transition probabilities are used as features for hierarchical clustering. After clustering, the Needleman-Wunsch algorithm is used for sequence alignment to infer the message format.

[0110] In the above technical solution, step 4 comprises the following steps:

[0111] 4.1) Population division: The protocol specification seeds generated based on multiple sequence alignment and probability statistics principles are divided into two populations, namely the multiple sequence population and the probability statistics population.

[0112] 4.2) Subpopulation optimization: Each subpopulation uses a genetic algorithm to perform the optimization process independently. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. In order to perform the subpopulation optimization process, the protocol specification seed needs to be evaluated for adaptability. Due to the lack of real protocol specifications, it is impossible to directly evaluate the quality of the protocol specification seed. This solution evaluates the quality of the seed by applying the protocol specification seed to the sample and observing whether the division result conforms to the protocol design principles. Specifically, after applying the protocol specification to the sample, each field is gradually tested. If a field has limited changes in all samples, it is considered to be a successfully divided field. Finally, the proportion of successfully divided fields to the total number of fields is counted to obtain the score of the protocol specification seed.

[0113] 4.3) Population Crossover: Use a sorting algorithm to sort the protocol specification seeds within each group. Select the top 10% of seeds in each group as excellent seeds and replace the bottom 10% of seeds in the other group. This process enables information sharing between different populations and promotes the search for the global optimal solution.

[0114] 4.4) Seed Mutation: To search for the global optimal solution, new seeds need to be generated for exploration. This process is accomplished by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields.

[0115] 4.5) Multiple populations co-evolve; repeat the above process until the seed fitness converges.

[0116] In the above technical solution, the extraction granularity of the protocol specification reaches the byte level, and it combines the advantages of two types of methods based on multiple sequence alignment and based on probability statistics, with stronger analysis capabilities and higher accuracy.

[0117] The present invention also provides a network protocol format reverse analysis device based on a collaborative evolutionary algorithm, comprising the following modules:

[0118] Acquisition unit: collects the protocol traffic to be analyzed and captures message data packets, which are request traffic packet sequences and response traffic packet sequences. The network data packet analysis tool is used to perform preliminary analysis on the message data packets of the original traffic to extract the payload and the underlying known protocol information.

[0119] Clustering unit: clusters the payloads using the underlying known protocol information through hierarchical clustering to obtain clustering results. In the clustering results, payloads with similar structure, content, and underlying known protocol information will be classified into the same class.

[0120] The seed generation unit processes the clustering results based on multiple sequence alignment and format specification analysis methods based on probability statistics to generate two types of protocol specification seeds;

[0121] The evolutionary algorithm unit uses the two types of protocol specification seeds obtained as two populations, performs sub-population optimization, population crossover, and seed mutation to achieve multi-population collaborative evolution, and obtains a network protocol format that combines the advantages of the two types of network protocol format inverse methods.

[0122] In the above solution, the acquisition unit includes:

[0123] Protocol traffic packet collection module: Uses tcpdump and Tshark tools to capture network traffic and saves it as a PCAP file. The data in the PCAP file is stored in binary packets. Each packet contains the network data captured within a certain period of time and related metadata, including timestamp, source IP address, destination IP address, and protocol type.

[0124] Unknown protocol data separation module: Use scapy, pyshark, and dpkt tools to separate the payload of the protocol to be analyzed from the data packet;

[0125] Known protocol parsing module: Use scapy, pyshark, and dpkt tools to parse known protocol traffic, extract the address, port number, and protocol data packet transmission direction information, and classify the traffic packets into two groups: client and server.

[0126] In the above solution, the clustering unit specifically includes:

[0127] The similarity matrix construction module constructs a symmetrical similarity matrix by calculating the similarity between message data packets;

[0128] The similarity calculation method uses the Needleman-Wunsch algorithm, setting the match score to 1, the mismatch score to -1, and the gap penalty to 0;

[0129] The similarity score is normalized by dividing the total score by the length of the longer of the compared message sequences;

[0130] Construct an n×n symmetric matrix with < <X 11 , X 12 ,...,X 1n >,<X 21 , X 22 ,...,X 2n >..., <X n1 , X n2 ,...,X nn>> indicates that X ij The distance from the i-th message sequence to the j-th message sequence, n is the number of message sequences, X ii =0,X ij =X ji ,

[0131] Store the similarity matrix in the file Sequence_Distance.txt as text type;

[0132] The secondary clustering execution module performs hierarchical clustering based on the constructed similarity matrix and constructs a phylogenetic tree according to the hierarchical structure of the clusters. Specifically,

[0133] Initially, each message sequence is considered as an initial cluster, and the two closest clusters are merged. The distance from the new cluster to other clusters is the average distance from its subclass to other clusters.

[0134] Repeat until only one cluster remains, and construct a phylogenetic tree based on the hierarchy of clusters;

[0135] The private protocol message sequence grouping module divides the protocol message sequence into 2 groups according to the phylogenetic tree constructed by hierarchical clustering and iteratively tries different positive integer k values. k There are k=2 or k=3 classes, and each class of messages is taken as a group to form an original input for constructing a protocol format seed.

[0136] In the above solution, the seed generation unit specifically includes:

[0137] The multiple sequence alignment protocol specification seed generation module generates the first population based on the multiple sequence alignment protocol specification seed, which specifically includes the following steps:

[0138] Use the protocol reverse engineering tool based on multiple sequence alignment to obtain the protocol specification, and build a protocol specification seed generator based on multiple sequence alignment based on the multiple sequence alignment tool mafft.

[0139] To process long sequences, the L-INS-i algorithm is used to handle highly divergent sequences;

[0140] To handle short sequences or sequences with large differences, the G-INS-i algorithm is used;

[0141] During the multiple sequence alignment, the identifier “-” was used to fill the gaps generated during the alignment process;

[0142] The output results are saved in FASTA format, with each line corresponding to a message sequence;

[0143] The final result is saved in the file Output_MSA_Oneline.txt;

[0144] The protocol specification seed generation module based on the probability and statistics principle generates the protocol specification seed based on the probability and statistics principle to form the second population, which specifically includes the following steps:

[0145] For binary packet sequences, the Biprominer method is used to extract cells from the packet sequence and label them.

[0146] Build a state transition model and select the project with the highest probability as the reverse-engineered protocol format;

[0147] For text-type package sequences, the ProDecoder method is used, with keywords and transition probabilities as features;

[0148] Hierarchical clustering was performed, and then sequence alignment was performed using the Needleman-Wunsch algorithm to infer the message format.

[0149] In the above solution, the evolutionary algorithm unit specifically includes:

[0150] The population division module divides the protocol specification seeds generated based on multiple sequence alignment and probability statistics into two populations, namely the multiple sequence population and the probability statistics population;

[0151] In the subpopulation optimization module, each subpopulation uses a genetic algorithm to independently perform the optimization process. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. To execute the subpopulation optimization process, the protocol specification seed needs to be evaluated for adaptability. This is done by applying the protocol specification seed to the sample and observing whether the partitioning results conform to the protocol design principles. Specifically, after applying the protocol specification to the sample, each field is gradually tested. If a field has limited variation across all samples, it is considered a successfully partitioned field. Finally, the proportion of successfully partitioned fields to the total number of fields is counted to obtain the protocol specification seed score.

[0152] The population crossover module uses a sorting algorithm to sort the protocol specification seeds in each group, selects the top 10% of seeds in each group as excellent seeds, and replaces the bottom 10% of seeds in another group with each other, thus sharing information between different populations and promoting the process of exploring the global optimal solution;

[0153] Seed mutation module: In order to search for the global optimal solution, new seeds need to be generated for exploration. This process is completed by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields.

[0154] The multi-population co-evolution module is used to repeat the process of subpopulation optimization, population crossover and seed mutation until the seed fitness converges.

[0155] Example

[0156] The solution does not introduce DHCP prior knowledge, so the DHCP protocol can be regarded as a proprietary protocol for this solution and device.

[0157] Collection and preprocessing phase. Use a packet capture tool on the core switch to collect 1,000 request and response traffic packets. In the preprocessing phase, extract the application layer protocol traffic, namely the protocol packet sequence payload. Save it in hexadecimal format as Payload.hex based on the sender of the packet. Then, parse the known underlying protocol. Record information such as the IP address and port number, and use this to determine the packet's direction. Classify the traffic packets into two groups: client and server. Save them as Server_Basic.txt and Client_Baisc.txt, respectively.

[0158] Clustering stage. Based on the constructed similarity matrix, perform hierarchical clustering based on the UPGMA algorithm. Initially, each message sequence is regarded as an initial cluster. Afterwards, merge the two closest clusters, and the distance from the new cluster to the other clusters is the average distance from its subclasses to other clusters. Repeat the above two steps until only one cluster remains. Construct a phylogenetic tree based on the clustering hierarchy. Group private protocol message sequences; Based on the phylogenetic tree constructed by hierarchical clustering, divide the protocol message sequences into 2 k There are three classes, so k=3.

[0159] Protocol specification seed generation phase. Protocol specification seed generation based on multiple sequence alignment: The protocol specification is obtained using a protocol reverse engineering tool based on multiple sequence alignment. A protocol specification seed generator based on multiple sequence alignment is constructed based on the multiple sequence alignment tool mafft. Observing the characteristics of DHCP traffic, which is generally long sequences, the L-INS-i algorithm is used. During the multiple sequence alignment process, the identifier "-" is used to fill the gaps generated during the alignment process. The output results are saved in FASTA format, but need to be adjusted to a format where one line of results corresponds to one message sequence. The final result is saved in the file Output_MSA_Oneline.txt. Seed generation based on probabilistic statistics: Observing the characteristics of DHCP traffic, it is believed that DHCP is a text protocol, so the ProDecoder method is used. Hierarchical clustering is performed using keywords and transition probabilities as features. After clustering, the Needleman-Wunsch algorithm is used to perform sequence alignment to infer the message format.

[0160] Multi-population co-evolution stage. The protocol specification seeds generated based on multiple sequence alignment and probability statistics principles are divided into two populations, called the multi-sequence population and the probability statistics population. Sub-population optimization: Each sub-population uses a genetic algorithm to independently perform the optimization process. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. Population crossover: Use the quick sort algorithm to sort the protocol specification seeds in each group separately, select the top 10% of seeds in each group as excellent seeds, and replace the bottom 10% of seeds in the other group with each other. This process enables information sharing between different populations and promotes the exploration of the global optimal solution. Seed mutation: In order to search for the global optimal solution, new seeds need to be generated for exploration. This process is completed by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields. Multi-population co-evolution: Repeat the above process until the seed fitness converges.

[0161] In the above technical solution, the extraction granularity of the protocol specification reaches the byte level, and it combines the advantages of two types of methods based on multiple sequence alignment and based on probability statistics, with stronger analysis capabilities and higher accuracy.

[0162] The present invention also provides a network protocol format reverse analysis device based on a collaborative evolutionary algorithm, comprising the following modules:

[0163] Data collection module: uses network sniffing tools to collect network traffic, and identifies and filters out irrelevant network traffic;

[0164] Preprocessing module: Analyzes network traffic metadata, including packet size, timestamp, etc. Analyzes underlying known protocols and extracts information such as IP addresses and port numbers;

[0165] Message similarity calculation module: calculates the similarity between the collected message sequences and outputs it in matrix form.

[0166] Clustering module: Initializes each message into a class, uses message similarity as the distance between initial classes, and clusters and groups protocol traffic;

[0167] Multiple sequence alignment protocol specification seed generation module: generates protocol specification seeds based on the multiple sequence alignment method. All protocol specification seeds generated based on the multiple sequence alignment method constitute the multiple sequence alignment population;

[0168] Probabilistic statistical protocol specification seed generation module: generates protocol specification seeds based on probabilistic statistical methods. All protocol specification seeds generated based on probabilistic statistical methods constitute a probabilistic statistical population.

[0169] Seed evaluation module: Based on the partitioning effect of the application protocol specification on the original traffic, the fitness of the protocol specification seed is evaluated, and the top 10% and bottom 10% individuals of each population are marked.

[0170] Subpopulation evolution module: The subpopulation evolves independently in isolation from other populations to produce protocol specification seeds with higher fitness.

[0171] Population cross module: provides a mechanism for exchanging information between populations and searches for the optimal solution that combines the advantages of both populations.

[0172] Seed mutation module: Standardizes seeds through operator mutation protocols such as merging, splitting, and modifying to discover potentially better seeds.

[0173] In summary, the technical solution of the present invention has the following characteristics:

[0174] 1. Collection and preprocessing:

[0175] Technical means: Use network packet analysis tools such as tcpdump and Tshark to capture network traffic and save it as a PCAP format file.

[0176] Technical issues: Private network protocol formats lack public documentation or implementation code, making direct analysis difficult.

[0177] Effect achieved: Through collection and preprocessing, the payload and underlying known protocol information are extracted to provide basic data for subsequent analysis.

[0178] Advantages over existing technologies: No need to rely on difficult-to-obtain protocol specifications or codes, reverse analysis can be performed only through network traffic.

[0179] 2. Private protocol traffic clustering:

[0180] Technical means: Use hierarchical clustering algorithms, such as UPGMA, to cluster messages based on the similarity of the message sequences.

[0181] Technical problem: How to effectively distinguish and group similar message sequences in a large amount of private protocol traffic.

[0182] Effect: Through clustering, similar message sequences are divided into the same group, which helps in the subsequent generation of protocol specification seeds.

[0183] Advantages over existing technologies: The use of hierarchical clustering algorithms improves the accuracy of grouping and reduces the possibility of misclassification.

[0184] 3. Protocol specification seed generation:

[0185] Technical means: Based on multiple sequence alignment and probability statistics principles, the clustering results are processed to generate protocol specification seeds.

[0186] Technical problem: How to extract the standard format of the protocol from the packetized traffic.

[0187] Effect achieved: Generates a seed that can reflect the characteristics of the protocol format, providing key information for subsequent protocol reverse analysis.

[0188] Advantages over existing technologies: Combining multiple sequence alignment and probability statistics improves the accuracy of seed generation and is applicable to different types of protocols.

[0189] 4. Multi-population co-evolution algorithm:

[0190] Technical means: Use genetic algorithms to optimize the generated seeds and promote information exchange and co-evolution among different populations.

[0191] Technical issue: How to further improve the accuracy and efficiency of protocol reverse analysis.

[0192] Effect achieved: Through multi-population co-evolution, the protocol specification seed is optimized and the accuracy and efficiency of reverse analysis are improved.

[0193] Advantages over existing technologies: It combines the advantages of different methods to achieve more comprehensive protocol format analysis, especially suitable for complex or highly variable protocols.

[0194] The above are only representative embodiments of the present invention in many specific application scopes and do not constitute any limitation on the protection scope of the present invention. Any technical solutions formed by transformation or equivalent replacement fall within the scope of protection of the present invention.

Claims

1. A network protocol format reverse analysis method based on a co-evolutionary algorithm, characterized by: The following steps are involved: Step 1. Packet Sequence Collection and Preprocessing: Collect the protocol traffic to be analyzed and capture the message data packets, which are the request traffic packet sequence and the response traffic packet sequence. Use network packet analysis tools to perform preliminary analysis on the original traffic message data packets to extract the payload and the underlying known protocol information. Step 2. Private protocol traffic clustering: Using hierarchical clustering, the payloads obtained in step 1 are clustered using the underlying known protocol information obtained in step 1 to obtain clustering results. In the clustering results, payloads with similar structure, content, and underlying known protocol information are classified into the same class. Step 3. Protocol specification seed generation: The clustering results of step 2 are processed based on multiple sequence alignment and format specification analysis methods based on probabilistic statistics to generate two types of protocol specification seeds; Step 4. Use a multi-population co-evolution algorithm: Use the two types of protocol specification seeds obtained in step 3 as two populations, perform sub-population optimization, population crossover, and seed mutation to achieve multi-population co-evolution and obtain a network protocol format that combines the advantages of the two types of network protocol format inverse methods.

2. A network protocol format reverse analysis method based on a collaborative evolutionary algorithm according to claim 1, characterized in that: Step 1 includes the following steps: Step 1.1: Collect protocol traffic packets: Use tools like tcpdump or Tshark to capture network traffic and save it as a PCAP file. The data in the PCAP file is stored in binary packets. Each packet contains the network data captured within a certain period of time, as well as related metadata, including timestamp, source IP address, destination IP address, and protocol type. Step 1.2, separation of unknown protocol data: Use scapy, pyshark, and dpkt tools to separate the payload of the protocol to be analyzed from the data packet; Step 1.3: Parsing known protocols: Use scapy, pyshark, and dpkt tools to parse known protocol traffic, extract the address, port number, and protocol packet transmission direction information, and classify the traffic packets into two groups: client and server.

3. The network protocol format reverse analysis method based on the collaborative evolution algorithm according to claim 1 is characterized in that: Step 2: Step 2.1: Construct a similarity matrix: Calculate the similarity between message packets to construct a symmetrical similarity matrix. Step 2.1.1: The similarity is calculated using the Needleman-Wunsch algorithm, with a matching score of 1, a mismatch score of -1, and a gap penalty of 0. Step 2.1.2: Normalize the similarity score by dividing the total score by the length of the longer message sequence in the comparison. Step 2.1.3, construct an n×n symmetric matrix with < <X 11 , X 12 ,…,X 1n >, <X 21 , X 22 ,…,X 2n >…, <X n1 , X n2 ,…,X nn >> indicates that X ij The distance from the i-th message sequence to the j-th message sequence, n is the number of message sequences, X ii =0,X ij =X ji , Step 2.1.

4. Store the similarity matrix in the file Sequence_Distance.txt as text. Step 2.2, perform hierarchical clustering: Based on the constructed similarity matrix, perform hierarchical clustering based on the UPGMA algorithm and construct a phylogenetic tree according to the cluster hierarchy; Step 2.2.

1. Initially, each message sequence is considered as an initial cluster; Step 2.2.2: Merge the two closest clusters. The distance from the new cluster to the other clusters is the average distance from the subclasses of the new cluster to the other clusters. Repeat steps 2.2.1-2.2.2 until only one cluster remains, and construct a phylogenetic tree based on the cluster hierarchy. Step 2.3, grouping private protocol message sequences: Based on the phylogenetic tree constructed by hierarchical clustering, iteratively try different positive integer k values ​​to divide the protocol message sequences into 2 k There are k=2 or k=3 classes, and each class of messages is taken as a group to form an original input for constructing a protocol specification seed.

4. The network protocol format reverse analysis method based on the collaborative evolution algorithm according to claim 1 is characterized in that: Step 3 specifically includes: Step 3.1: Generate seed based on the protocol specification of multiple sequence alignment to form the first population; Step 3.1.1, use the protocol reverse engineering tool based on multiple sequence alignment to obtain the protocol specification, and build a protocol specification seed generator based on multiple sequence alignment based on the multiple sequence alignment tool mafft, Step 3.1.2: Process long sequences and use the L-INS-i algorithm to handle highly divergent sequences; Step 3.1.3: For short sequences or sequences with large differences, use the G-INS-i algorithm. Step 3.1.4: During the multiple sequence alignment process, the identifier "-" is used to fill the gaps generated during the alignment process; Step 3.1.

5. Save the output in FASTA format, with each line corresponding to a message sequence; Step 3.1.

6. Save the final result to the file Output_MSA_Oneline.txt. Step 3.2: Generate protocol specification seeds based on the probability and statistics principle to form the second population; Step 3.2.1: For binary packet sequences, use the Biprominer method to extract cells from the packet sequence and label them. Step 3.2.2: Build a state transition model and select the project with the highest probability as the reversed protocol format; Step 3.2.3: For text-type packet sequences, use the ProDecoder method and use keywords and transition probabilities as features. Step 3.2.4: Perform hierarchical clustering and then use the Needleman-Wunsch algorithm to perform sequence alignment to infer the message format.

5. The network protocol format reverse analysis method based on the collaborative evolution algorithm according to claim 1 is characterized in that: Step 4 The steps are: Step 4.1, population division: The protocol specification seeds generated based on multiple sequence alignment and probability statistics principles are divided into two populations, namely the multiple sequence population and the probability statistics population; Step 4.2, Subpopulation Optimization: Each subpopulation uses a genetic algorithm to independently perform the optimization process. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. To perform the subpopulation optimization process, the protocol specification seed needs to be evaluated for adaptability. This is done by applying the protocol specification seed to the sample and observing whether the partitioning results conform to the protocol design principles. Specifically, after applying the protocol specification to the sample, each field is gradually tested. If a field has limited variation across all samples, it is considered a successfully partitioned field. Finally, the proportion of successfully partitioned fields to the total number of fields is calculated to obtain the protocol specification seed score. Step 4.3, Population Crossover: Use a sorting algorithm to sort the protocol specification seeds in each group separately, select the top 10% of seeds in each group as excellent seeds, and replace the bottom 10% of seeds in another group with each other, so as to achieve information sharing between different populations and promote the process of exploring the global optimal solution; Step 4.4, seed mutation: In order to search for the global optimal solution, new seeds need to be generated for exploration. This process is completed by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields. Step 4.5, multi-population coevolution: Repeat the above process until the seed fitness converges.

6. A network protocol format reverse analysis device based on a collaborative evolutionary algorithm, characterized in that: Includes the following modules: Acquisition unit: collects the protocol traffic to be analyzed and captures message data packets, which are request traffic packet sequences and response traffic packet sequences. The network data packet analysis tool is used to perform preliminary analysis on the message data packets of the original traffic to extract the payload and the underlying known protocol information. Clustering unit: clusters the payloads using the underlying known protocol information through hierarchical clustering to obtain clustering results. In the clustering results, payloads with similar structure, content, and underlying known protocol information will be classified into the same class. The seed generation unit processes the clustering results based on multiple sequence alignment and format specification analysis methods based on probability statistics to generate two types of protocol specification seeds; The evolutionary algorithm unit uses the two types of protocol specification seeds obtained as two populations, performs sub-population optimization, population crossover, and seed mutation to achieve multi-population collaborative evolution, and obtains a network protocol format that combines the advantages of the two types of network protocol format inverse methods.

7. The network protocol format reverse analysis device based on the collaborative evolution algorithm according to claim 6 is characterized in that: The acquisition unit includes: Protocol traffic packet collection module: Uses tcpdump and Tshark tools to capture network traffic and saves it as a PCAP file. The data in the PCAP file is stored in binary packets. Each packet contains the network data captured within a certain period of time and related metadata, including timestamp, source IP address, destination IP address, and protocol type. Unknown protocol data separation module: Use scapy, pyshark, and dpkt tools to separate the payload of the protocol to be analyzed from the data packet; Known protocol parsing module: Use scapy, pyshark, and dpkt tools to parse known protocol traffic, extract the address, port number, and protocol data packet transmission direction information, and classify the traffic packets into two groups: client and server.

8. The network protocol format reverse analysis device based on the collaborative evolution algorithm according to claim 6 is characterized in that: The clustering units specifically include: The similarity matrix construction module constructs a symmetrical similarity matrix by calculating the similarity between message data packets; The similarity calculation method uses the Needleman-Wunsch algorithm, setting the match score to 1, the mismatch score to -1, and the gap penalty to 0; The similarity score is normalized by dividing the total score by the length of the longer of the compared message sequences; Construct an n×n symmetric matrix with < <X 11 , X 12 ,…,X 1n >, <X 21 , X 22 ,…,X 2n >…, <X n1 , X n2 ,…,X nn >> indicates that X ij The distance from the i-th message sequence to the j-th message sequence, n is the number of message sequences, X ii =0,X ij =X ji , Store the similarity matrix in the file Sequence_Distance.txt as text type; The secondary clustering execution module performs hierarchical clustering based on the constructed similarity matrix and constructs a phylogenetic tree according to the hierarchical structure of the clusters. Specifically, Initially, each message sequence is considered as an initial cluster, and the two closest clusters are merged. The distance from the new cluster to other clusters is the average distance from the subclasses of the new cluster to other clusters. Repeat until only one cluster remains, and construct a phylogenetic tree based on the hierarchy of clusters; The private protocol message sequence grouping module divides the protocol message sequence into 2 groups according to the phylogenetic tree constructed by hierarchical clustering and iteratively tries different positive integer k values. k There are k=2 or k=3 classes, and each class of messages is taken as a group to form an original input for constructing a protocol specification seed.

9. The network protocol format reverse analysis device based on the collaborative evolution algorithm according to claim 6, characterized in that: The seed generation unit specifically includes: The multiple sequence alignment protocol specification seed generation module generates the first population based on the multiple sequence alignment protocol specification seed, which specifically includes the following steps: Use the protocol reverse engineering tool based on multiple sequence alignment to obtain the protocol specification, and build a protocol specification seed generator based on multiple sequence alignment based on the multiple sequence alignment tool mafft. To process long sequences, the L-INS-i algorithm is used to handle highly divergent sequences; To handle short sequences or sequences with large differences, the G-INS-i algorithm is used; During the multiple sequence alignment process, the identifier "-" is used to fill the gaps generated during the alignment process; The output results are saved in FASTA format, with each line corresponding to a message sequence; The final result is saved in the file Output_MSA_Oneline.txt; The protocol specification seed generation module based on the probability and statistics principle generates the protocol specification seed based on the probability and statistics principle to form the second population, which specifically includes the following steps: For binary packet sequences, the Biprominer method is used to extract cells from the packet sequence and label them. Build a state transition model and select the project with the highest probability as the reverse-engineered protocol format; For text-type package sequences, the ProDecoder method is used, with keywords and transition probabilities as features; Hierarchical clustering was performed, and then sequence alignment was performed using the Needleman-Wunsch algorithm to infer the message format.

10. The network protocol format reverse analysis device based on the collaborative evolution algorithm according to claim 8, characterized in that: The evolutionary algorithm unit specifically includes: The population division module divides the protocol specification seeds generated based on multiple sequence alignment and probability statistics into two populations, namely the multiple sequence population and the probability statistics population; In the subpopulation optimization module, each subpopulation uses a genetic algorithm to independently perform the optimization process. Optimization stops when the fitness reaches a threshold or the maximum number of iterations is reached. To perform the subpopulation optimization process, the protocol specification seed needs to be evaluated for adaptability. This is done by applying the protocol specification seed to the sample and observing whether the partitioning results conform to the protocol design principles. Specifically, after applying the protocol specification to the sample, each field is gradually tested. If a field has limited variation across all samples, it is considered a successfully partitioned field. Finally, the proportion of successfully partitioned fields to the total number of fields is counted to obtain the protocol specification seed score. The population crossover module uses a sorting algorithm to sort the protocol specification seeds in each group, selects the top 10% of seeds in each group as excellent seeds, and replaces the bottom 10% of seeds in another group with each other, thus sharing information between different populations and promoting the process of exploring the global optimal solution; Seed mutation module: In order to search for the global optimal solution, new seeds need to be generated for exploration. This process is completed by mutating the seeds. Mutation operations include merging fields, splitting fields, and modifying fields. The multi-population co-evolution module is used to repeat the process of subpopulation optimization, population crossover and seed mutation until the seed fitness converges.

Images