A message format extraction method based on Needleman-Wunsch and N-gram

Abstract

A message format extraction method based on Needleman-Wunsch and N-gram is proposed to optimize the low computational efficiency and insufficient integration with wireless protocol features of the traditional NW algorithm. The sequence alignment technology is studied in depth to improve the relevant algorithms and improve the computational efficiency of sequence alignment. The NW algorithm is optimized for the message data processing part through the n-gram algorithm idea, and the NWN algorithm is proposed to improve the accuracy of message classification. It solves the problem that the sequence similarity algorithm often has large deviations in the analysis of the protocol message format in the communication protocol analysis of the wireless sensor network due to incomplete protocol message data samples or single message format.

Description

Technical Field

[0001] The invention relates to the field of RFID message information processing, and in particular to a message format extraction method based on Needleman-Wunsch and N-gram. Background Art

[0002] One of the main goals in protocol reverse analysis is to extract the protocol message format. The message format is an abstract description of the protocol message sequence, which mainly includes the semantic information, structural information and format information of each domain in the message. For example, the protocol message format is described using ABNF, which is convenient for machine storage and restoration of the message structure tree. This document is mainly based on the dynamic binary analysis method, which can obtain the domain boundary of the message format. The format tree construction is relatively easy. However, in the analysis of wireless communication protocols, since the message boundary cannot be directly judged, the realization of message format tree extraction still needs further research. At present, the domain structure extraction is mostly achieved by sequence alignment method. For example, PI and Prodecoder both use NW algorithm for message alignment. The sequences are grouped by calculating the similarity relationship between different sequences. The computational complexity is related to the sequence length. Genetic algorithm shows good performance in long sequence alignment, but the traditional NW algorithm has problems such as low computational efficiency and insufficient integration with wireless protocol features. Therefore, it is necessary to study sequence alignment technology in depth, improve related algorithms, and improve the computational efficiency of sequence alignment.

[0003] In the message classification stage, its accuracy usually depends on its sequence similarity algorithm. However, in the communication protocol analysis of wireless sensor networks, there are often problems with large deviations in the results of analyzing the protocol message format due to incomplete protocol message data samples or single message formats. For example, the NW algorithm usually requires a high degree of similarity within the data to achieve better results. For situations such as message data that have fixed patterns in the structure and large differences in data content, the final classification results are not ideal. The idea of ​​the n-gram algorithm is to establish the possibility of the order of words through the similarity in words or sentences to perform classification and prediction functions, but there are several major problems in applying the n-gram algorithm to RFID messages:

[0004] (1) The data size of RFID message information is much smaller than that of natural language corpus, and the message length in protocol data is generally short. It is common to see a large number of messages within 6 bytes in the data.

[0005] (2) RFID messages are transmitted in hexadecimal, in which there are fixed boundaries between feature information and between feature information and data information, making the data information at adjacent locations less correlated or completely unrelated. Therefore, when the n-gram algorithm is directly used in these places, the accuracy of subsequent data prediction will be greatly reduced.

[0006] (3) In the communication process of RFID messages, a large amount of repetitive data may appear, such as a lot of UID information when establishing a link. Since this information has a specific meaning for the entire message but does not have a linguistic meaning, it will not produce ideal results when it is treated as a word in natural language processing. Summary of the invention

[0007] Aiming at the shortcomings of the multiple sequence alignment algorithm in reverse analysis, this paper combines the n-gram algorithm and the NW algorithm, optimizes the NW algorithm's processing of the message data part through the n-gram algorithm idea, and proposes a message format extraction method (NWN algorithm) based on Needleman-Wunsch and N-gram, which improves the accuracy of message classification. First, the protocol is based on the abstract description method of the message format tree, which can reflect the domain structure characteristics of the protocol. Secondly, the study uses multiple sequence alignment technology to generate a message format tree, and proposes the NWN algorithm to improve the efficiency of the sequence alignment algorithm in the sequence frame structure analysis.

[0008] A message format extraction method based on Needleman-Wunsch and N-gram includes the following steps:

[0009] Step S1, using the N-gram algorithm to classify the sequence data to be processed, generating corresponding N-gram sets and frequencies for calculating the similarity between two sequences;

[0010] Step S2, calculating the similarity, screening the threshold of the related sequences, and obtaining the frequent sequences after screening;

[0011] Step S3: Use the correlation analysis method to perform secondary screening, further divide the message data and frame it, so as to obtain and infer the internal information of the message.

[0012] Furthermore, step S1 includes the following sub-steps:

[0013] S1.1, N-gram generation process is to divide the labeled sequence, generate the corresponding N-gram set, and count the occurrence frequency;

[0014] S1.2. Use recursion to solve the similarity score between two gene sequences and save it in a two-dimensional matrix;

[0015] In the process of filling the score matrix, the score of the current value element Mi,j is obtained by iterating the formula (1). The score of the current element Mi,j in formula (1) is obtained by calculating the max function;

[0016]

[0017] Where '-' represents the space filled in the Needleman-Wunsch algorithm; σ represents the penalty function. Suppose two sequences S = s 1 s 2 …s n and T = t 1 t 2 …t m If S i =T j , where S i is the i-th character in sequence S, T j is the jth character in the sequence T, then the value of the diagonal element plus the penalty value when the sequence is matched is assigned to the current element M i,j .

[0018] Furthermore, in step S2, the generalized Jaccard coefficient is calculated according to formula (2) as the similarity

[0019]

[0020] Where n is the total number of N-grams in the two strings, p i ,q i It is the frequency of the occurrence of the i-th n-gram in the two strings. The value range of the obtained similarity Sim is 0 to 1, that is, 0≤Sim≤1.

[0021] Furthermore, the larger the Sim value, the higher the similarity between the two character strings, which means that the possibility that the two pieces of message data belong to the same format is greater; conversely, the smaller the Sim value, the lower the similarity between the character strings, and the greater the possibility that the two pieces of message data belong to different formats.

[0022] Furthermore, in order to obtain more comprehensive protocol format information, step S3 requires semantic extraction while performing multiple sequence alignment on message fragments, wherein the correlation analysis method includes the following sub-steps:

[0023] S3.1, split the protocol message data into fixed lengths, then perform XOR operations on all messages and count them into prime groups;

[0024] S3.2, then calculate the Hamming weight of each column in the matrix formed by its array, that is, the number of 1s in each column;

[0025] S3.3, if the total number of 1s is greater than a certain threshold, it is preliminarily determined that there is a header, trailer or control keyword at that position; if the number of 1s is less than a certain threshold, it is determined that the message sequence segmentation is incorrect; by repeatedly adjusting the message length segmentation results, calculating the occurrence of each bit in the message, inferring the true message length, and thus completing the preliminary segmentation. The threshold value range is between 0 and 1. Under normal circumstances, depending on the data collection situation, in order to merge similar messages and avoid incorrect classification, the threshold is usually selected at 0.95 or above.

[0026] Furthermore, the method proposes an N-gram multiple sequence alignment technology in combination with the characteristics of RFID protocol byte stream, firstly, the message data is reasonably segmented by N-gram technology, and the threshold for screening related sequences is found by using the generalized Jaccard similarity coefficient, and frequent sequences are obtained after screening; secondary screening is performed using related information methods, and the message data is further divided and framed, so as to obtain and infer the internal information of the message.

[0027] The beneficial effects achieved by the present invention are:

[0028] (1) Optimize the traditional NW algorithm, which has problems such as low computational efficiency and insufficient integration with wireless protocol features. Conduct in-depth research on sequence alignment technology, improve related algorithms, and improve the computational efficiency of sequence alignment.

[0029] (2) The NW algorithm is optimized for the message data processing by using the n-gram algorithm, and the NWN algorithm is proposed to improve the accuracy of message classification. This solves the problem that in the message classification stage, the sequence similarity algorithm often has large deviations in the analysis of the protocol message format due to incomplete protocol message data samples or single message format in the communication protocol analysis of wireless sensor networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the NWN algorithm described in an embodiment of the present invention.

[0031] Figure 2 It is part of the hexadecimal sample data described in the embodiment of the present invention.

[0032] Figure 3 It is a data diagram of corresponding fitness values ​​under different l values ​​described in the embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0034] The invention proposes an n-gram multiple sequence alignment technology in combination with the characteristics of RFID protocol byte stream, etc. Firstly, the message data is reasonably segmented by the n-gram technology, and the threshold for screening related sequences is found by using the generalized Jaccard similarity coefficient, and the frequent sequences are obtained after screening; then the relevant information method is used for secondary screening, and the message data is further divided and framed, so as to obtain and infer the internal information of the message.

[0035] Currently, sequence alignment methods are mostly used to implement domain structure extraction. For example, PI and Prodecoder both use the NW algorithm for message alignment. Sequences are grouped by calculating the similarity relationship between different sequences. The computational complexity is related to the sequence length. Genetic algorithms show good performance in long sequence alignment, but traditional NW algorithms have problems such as low computational efficiency and insufficient integration with wireless protocol features. Therefore, it is necessary to conduct in-depth research on sequence alignment technology, improve related algorithms, and improve the computational efficiency of sequence alignment.

[0036] In the traditional multiple sequence alignment algorithm, after the character string is aligned according to the feature, the similarity value is obtained according to the data matching degree on the corresponding position, but the accuracy of the detection may change due to the structure of the data, the algorithm and the selection of parameters. The N-gram algorithm is used to improve the sequence matching algorithm, which can effectively eliminate the influence of the difference in the different protocol formats in the data on the result, and is conducive to improving the accuracy of the sequence alignment. Therefore, the present invention adopts the N-gram algorithm to classify the sequence data to be processed, generates the corresponding N-gram set and frequency, and is used to calculate the similarity between two sequences, so as to classify.

[0037] The message format extraction method based on Needleman-Wunsch and N-gram (NWN algorithm) of the present invention comprises the following steps:

[0038] 1. Use the N-gram algorithm to classify the sequence data to be processed, generate the corresponding N-gram set and frequency, and calculate the similarity between two sequences; 2. Calculate the threshold of similarity to filter related sequences, and obtain frequent sequences after screening; 3. Use the relevant information method to perform secondary screening, further divide and frame the message data, so as to obtain and infer the internal information of the message.

[0039] Multiple sequence alignment algorithm based on n-gram

[0040] (1) N-gram generation process

[0041] The N-gram generation process is to divide the labeled sequence, generate the corresponding N-gram set, and count the frequency of occurrence. The steps are as follows:

[0042] First, the meaning of gram must be determined. When applied to natural language text, gram usually represents a single character, letter, Chinese character or word, etc. The gram in this embodiment is a subsequence composed of several bytes of data (two-bit hexadecimal data).

[0043] Select the N value. The size of the N value determines the set into which the sequence is divided. The choice of the N value will greatly affect the similarity calculation results. A too small N value will cause the original structure to be ignored. When N is 1, the tag sequence will be divided into independent single tags, which is equivalent to a simple word frequency count and will lose the meaning and advantage of n-gram. A too large N value will make the similarity between texts too small and unable to be distinguished. Take the sequence 0055D407FF in the table as an example:

[0044] Table of n-gram generation results for different N values

[0045]

[0046]

[0047] The present embodiment subsequently conducted experiments to prove that the multiple sequence alignment effect is best when the value of N is 3.

[0048] (2) Improvement of Needleman-Wunsch algorithm

[0049] The basic idea of ​​the Needleman-Wunsch algorithm is to use a recursive form to solve the similarity score between two gene sequences and save it in a two-dimensional matrix; then, based on the obtained two-dimensional cost matrix, use the backtracking technique to find the best result of comparing the two sequences. The implementation of the Needleman-Wunsch algorithm is mainly divided into three aspects: the first is to initialize the two-dimensional score matrix according to the prescribed penalty rules; the second is to calculate and fill the score matrix; the third is to use the backtracking method to find the best path for the comparison, that is, the best result of the sequence comparison. Through the analysis of the three aspects of its implementation, it can be known that in the Needleman-Wunsch algorithm, the most time-consuming is to calculate and fill the matrix. For this reason, this embodiment mainly improves the calculation and filling of the score matrix in the second step, thereby reducing the time complexity. This section is mainly based on the Needleman-Wunsch algorithm, discussing how to improve it to reduce its time complexity, thereby improving the Needleman-Wunsch algorithm and improving its operating efficiency.

[0050] The specific idea of ​​improving the Needleman-Wunsch algorithm: After research, it was found that the most time-consuming step of the algorithm is to calculate and fill the cost matrix. When calculating the cost matrix, each Mi,j element needs to go through two layers of nested loops to calculate its score. Moreover, in the nested loops, it is also necessary to calculate the maximum value of the three score values ​​of the diagonal element Mi-1,j-1, the vertical element Mi-1,j, and the horizontal element Mi,j-1. Therefore, if the calculation statements in the nested loops can be reduced, the time complexity of the algorithm can be reduced, thereby making the Needleman-Wunsch algorithm run faster. In the process of filling the score matrix, the score of the current value element Mi,j can be obtained by the following iterative formula, where '-' represents the space filled in the Needleman-Wunsch algorithm process, and σ represents the penalty function:

[0051]

[0052] From the above formula, we can see that the score of the current element Mi,j is obtained by calculating the max function. The max function is obtained by calculating the diagonal element Mi-1,j-1 (plus the penalty function σ(S i ,T j ) value), the vertical element M i-1,j (plus penalty function σ('-',T j ) and the horizontal element M i,j-1 Add the penalty function σ(S i ,'-')) is the maximum value. Therefore, the element M i,j The score is the maximum value calculated by the max function. In this way, if we want to solve the element M i,j The value of M requires three operations. First, the total score of the scores in the three directions plus the penalty values ​​of each direction is calculated. Then the maximum score among the scores in the three directions is solved. Finally, the maximum score is assigned to the element value M. i,j .

[0053] After analyzing the Needleman-Wunsch algorithm, it is found that if two sequences S = s 1 s 2 …s n and T = t 1 t 2 …t m If S i =T j (Where S i is the i-th character in sequence S, T jis the jth character in the sequence T), then we can directly assign the value of the diagonal element plus the penalty value when the sequence is matched to the current element M. i,j , this saves the need to calculate the maximum value of the diagonal element Mi-1,j-1, the vertical element Mi-1,j and the horizontal element Mi,j-1, that is, to solve the max function.

[0054] The improved Needleman-Wunsch algorithm implementation process is as follows: Figure 1 The process includes the following steps: (1) reading in the sequence to be compared; (2) preliminarily grouping the sequences according to their length; (3) calculating the similarity between a sequence and the classified sequences in the group and comparing them with the threshold; (4) if the similarity is greater than the threshold, classifying the sequence into a new category; (5) if the similarity is less than the threshold, classifying the sequence into the new category; (6) determining whether all sequences have been classified, if not, returning to (3) and performing the classification process again; (7) completing the classification of all sequences and outputting the grouping results.

[0055] 1. Semantic Extraction

[0056] Based on the principle of correlation analysis algorithm, this embodiment adopts a method of estimating message length through correlation analysis. In the first step, the protocol message data is segmented according to a fixed length, and then all messages are subjected to XOR operation and counted into prime groups. Then, for the matrix formed by its array, the Hamming weight of each column in the matrix (i.e., the number of 1s in each column) is calculated. If the frequency of 1 is greater than the threshold, it can be preliminarily determined that there may be a header, a tail, or a control keyword at this position. If the number of 1s is less than the threshold, it can be determined that the message sequence segmentation is incorrect. By repeatedly adjusting the message length segmentation results and calculating the occurrence of each bit in the message message, the true message length can be inferred, thereby completing the preliminary segmentation. The threshold value range is between 0 and 1. Under normal circumstances, depending on the data collection situation, in order to merge similar messages and avoid incorrect classification, the threshold is usually selected at 0.95 or above.

[0057] Next, we will introduce the process of the correlation analysis algorithm, where the same or same operation is represented by the operator symbol ⊙:

[0058] First, the protocol message data is preliminarily grouped according to the length. If a group of messages with a length of l has r rows, a matrix A can be formed, and the form of A is:

[0059]

[0060] Then, we can perform bitwise XOR operation on one row of data and the other rows to get the result row vector. The elements in each row represent the correlation coefficients at different positions in the two sequences, represented by c:

[0061]

[0062] Then for any, 1≤m≤l, find the correlation coefficient [c(ij) 1 c(ij) 2 …c(ij) l ], then there is a correlation coefficient matrix C:

[0063]

[0064] Finally, sum each column of the matrix C to get the number of fields in the column that are the same as those in other columns. By traversing the matrix, we can find the maximum value cmax and the position k of the column where the value is located. The maximum value cmax above represents the maximum similarity of the elements at the position k.

[0065]

[0066] In order to evaluate the accuracy of the segmentation of this method, the classification fitness δ is defined, and the calculation formula is as follows:

[0067]

[0068] By continuously changing the value of l, the corresponding classification fitness δ can be obtained, and a distribution graph of l and δ can be obtained.

[0069] by Figure 2 is the hexadecimal sample data, where the keyword is "0x5555" and the last "0x00", then the corresponding fitness value distribution under different l values ​​is as follows Figure 3 As shown, it can be seen that when the segmentation is correct, that is, when the l value is selected appropriately, a higher fitness can be obtained in the sample data.

[0070] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A message format extraction method based on Needleman-Wunsch and N-gram, characterized by: The method comprises the following steps: Step S1, using the N-gram algorithm to classify the sequence data to be processed, generating corresponding N-gram sets and frequencies for calculating the similarity between two sequences; Step S1 includes the following sub-steps: S1.1, N-gram generation process is to divide the labeled sequence, generate the corresponding N-gram set, and count the occurrence frequency; S1.

2. Use recursion to solve the similarity score between two gene sequences and save it in a two-dimensional matrix; In the process of filling the score matrix, the current value element M i,j The score is obtained by iterating the formula (1), where the current element M is i,j The score is obtained by finding the max function; Where '-' represents the space filled in the Needleman-Wunsch algorithm; σ represents the penalty function. Suppose two sequences S = s1s2…s n and T = t1t2…t m If S i =T j , where S i is the i-th character in sequence S, T j is the jth character in the sequence T, then the value of the diagonal element plus the penalty value when the sequence is matched is assigned to the current element M i,j ; Step S2, calculating the similarity, screening the threshold of the related sequences, and obtaining the frequent sequences after screening; Step S3: Use the correlation analysis method to perform secondary screening, further divide the message data and frame it, so as to obtain and infer the internal information of the message.

2. The message format extraction method based on Needleman-Wunsch and N-gram according to claim 1, characterized in that: In step S2, the generalized Jaccard coefficient is calculated according to formula (2) as the similarity: Where n is the total number of N-grams in the two strings, p i ,q i It is the frequency of the occurrence of the i-th n-gram in the two strings. The value range of the obtained similarity Sim is 0 to 1, that is, 0≤Sim≤1.

3. The message format extraction method based on Needleman-Wunsch and N-gram according to claim 2, characterized in that: The larger the Sim value, the higher the similarity between the two character strings, which means that the possibility that the two pieces of message data belong to the same format is greater; conversely, the smaller the Sim value, the lower the similarity between the character strings, and the greater the possibility that the two pieces of message data belong to different formats.

4. The message format extraction method based on Needleman-Wunsch and N-gram according to claim 1, characterized in that: In step S3, in order to obtain more comprehensive protocol format information, multiple sequence alignment of message fragments is performed while semantic extraction is also required, wherein the correlation analysis method includes the following sub-steps: S3.1, split the protocol message data into fixed lengths, then perform XOR operations on all messages and count them into prime groups; S3.2, then calculate the Hamming weight of each column in the matrix formed by its array, that is, the number of 1s in each column; S3.3, if the total number of 1s is greater than a certain threshold, it is preliminarily determined that there is a header, a trailer, or a control keyword at that position; If the number of occurrences of 1 is less than a certain threshold, it is determined that the message sequence segmentation is incorrect; By repeatedly adjusting the message length segmentation results, the occurrence of each bit in the message is calculated, and the actual message length is inferred, thereby completing the preliminary segmentation.

5. The message format extraction method based on Needleman-Wunsch and N-gram according to claim 1, characterized in that: In step S3.3, the threshold value varies according to the data collection situation. In order to merge similar messages and avoid misclassification, the value range is between 0 and 1.

Images