A method for character string plaintext recognition based on feature information calculation

Abstract

The present application belongs to the field of plaintext recognition. It is a method for plaintext recognition of strings based on feature information calculation. After obtaining and parsing a certain amount of plaintext strings and ciphertext strings, the judgment thresholds of the plaintext strings under five dimensions are obtained by multi-dimensional analysis and training of the strings, which are information entropy threshold, average distance threshold, conversion probability threshold, readability threshold and continuous segment threshold. After determining the priority of each threshold, the string is respectively calculated for information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio to obtain a calculated score, and then the corresponding calculated score of each priority is compared with the threshold in order of priority to determine whether it is a plaintext string, and the plaintext string that has been judged is marked, thereby improving the judgment efficiency. In this way, the string features are analyzed from multiple angles to improve the accuracy of plaintext recognition.

Description

Technical Field

[0001] The present application belongs to the field of plaintext recognition, and in particular to a method for character string plaintext recognition based on feature information calculation. Background Art

[0002] Plain text transmission in the network is an important cause of security issues such as data leakage, data tampering, traffic hijacking, and phishing attacks. The secure transmission of sensitive data is an important part of network security technology, but the plain text transmission of sensitive data is a very common vulnerability. Attackers can obtain these unencrypted sensitive data through scanners or manual penetration. Once attackers obtain this data, they can use this information to enter the system as a legitimate user, and the threat caused is undoubtedly very large.

[0003] In order to prevent such situations from happening, it is necessary to encrypt the plaintext data transmitted over the network, which can effectively prevent attackers from capturing and intercepting it. The purpose of this solution is to detect whether the network data stream is encrypted. However, the existing methods are inaccurate in identifying unencrypted plaintext, and it is urgent to develop a new and reliable plaintext identification method. Summary of the invention

[0004] The purpose of this application is to provide a method for character string plaintext recognition based on feature information calculation to solve the problem that traditional plaintext recognition is inaccurate for unknown encryption type character strings.

[0005] The technical solution of the present application is: a method for character string plaintext recognition based on feature information calculation, comprising:

[0006] Obtain a certain amount of plaintext strings and ciphertext strings, parse the strings to be identified, and obtain the password fields contained in the URL respectively; and train the parsed password fields to determine the identification judgment items: information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment, and character type continuous segment ratio; obtain the information entropy threshold, average distance threshold, conversion probability threshold, readability threshold, and continuous segment threshold of the string respectively;

[0007] Determine the priority of the recognition judgment item, calculate the information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio of the character string, and obtain the calculation score; compare the first priority calculation score with the threshold corresponding to the first priority in order of priority to determine whether it is a plaintext string, and if so, mark the string; and compare the second priority calculation score of the remaining string with the corresponding threshold to determine whether it is a plaintext string, and mark the plaintext string determined in this round, and repeat this process until all five string recognition judgment items are completely judged;

[0008] The plaintext and ciphertext strings identified are counted according to the marking results.

[0009] Preferably, the thresholds corresponding to the string recognition judgment items include a first threshold and a second threshold, the first threshold is used for plaintext string judgment, and the second threshold is used for ciphertext string judgment. After judging whether a string is a plaintext string, if not, the calculated score of the string is compared with the second threshold to judge whether the string is a ciphertext string. If so, it is also marked and not judged in the next round; the accuracy of the values ​​of the first threshold and the second threshold is judged as follows: the probability of being a plaintext string or a ciphertext string is 99%. .

[0010] Preferably, the method for determining the threshold and priority of the identification judgment item is:

[0011] A clustering interval and step size are set for each individual recognition judgment item. Within the clustering interval, the value that reaches the specified accuracy and has the highest recall is selected as the threshold of the recognition judgment item according to the set step size; then the recall rates corresponding to each recognition judgment item are sorted from large to small, and the recognition judgment item with the largest recall rate is given the first priority, and so on.

[0012] Preferably, the method for obtaining the average distance calculation score of adjacent character strings is:

[0013] Determine the types of adjacent characters in the adjacent character string, set different reward and penalty coefficients for different adjacent character types, and determine the character length in the character string;

[0014] The calculation score of the string is obtained by the following distance formula:

[0015]

[0016] In the formula, v i is the ASCII value of the i-th character, where v i+1 is the ASCII value of the i+1th character, n is the length of the string, α i，i+1 is the reward and penalty coefficient, and j represents the group.

[0017] Preferably, the method for obtaining the character type conversion probability calculation score is:

[0018] The switching frequency of character types in all strings is counted, and then the ratio of the switching frequency to the non-switching frequency of adjacent characters in the string is used as the calculation score ratio of the character type conversion probability. The calculation formula is:

[0019]

[0020] In the formula, m different is the switching frequency between adjacent characters in the string, m same It is the frequency of non-switching between adjacent characters in a string.

[0021] Preferably, the method for obtaining the character string readability judgment calculation score is:

[0022] Get a certain amount of English dictionaries and pinyin, use the hidden Markov model to count the probability of subsequence occurrence of characters in the string, and calculate the score of string readability judgment based on the subsequence occurrence probability:

[0023]

[0024] Where p(sub) is the probability of a subsequence occurring, calculated using a simplified hidden Markov model, N is the maximum subsequence length, and L is the string length;

[0025] When P is less than a certain value, the calculation score of the string readability judgment is:

[0026]

[0027] Preferably, the method for obtaining the character type continuous judgment proportion calculation score is:

[0028] Count the number of all character segments and the number of continuous segments in the string. The criterion for determining a continuous character segment is: if three adjacent characters are of the same type, the calculation score for the continuous character type determination ratio is:

[0029]

[0030] Where, d 连续 is the number of consecutive segments in the string, and d is the number of all segments in the string. Preferably, the method for obtaining the information entropy calculation score is:

[0031] By counting the length of the corresponding string and the probability of any character appearing in the string, the calculation score corresponding to the information entropy is:

[0032]

[0033] Where n is the set length of the string, f i The frequency of occurrence of the character in the string.

[0034] The present application discloses a method for plaintext string recognition based on feature information calculation. After obtaining and parsing a certain amount of plaintext strings and ciphertext strings, the judgment thresholds of plaintext strings under five dimensions are obtained by multi-dimensional analysis and training of the strings, which are information entropy threshold, average distance threshold, conversion probability threshold, readability threshold and continuous segment threshold. After determining the priority of each threshold, the string is respectively calculated for information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio to obtain a calculated score. Then, the corresponding calculated score of each priority is compared with the threshold in order of priority to determine whether it is a plaintext string, and the plaintext string that has been judged is marked. In the next round of judgment, the string that has been judged as plaintext is removed, thereby improving the judgment efficiency. In this way, the string features are analyzed from multiple angles to improve the accuracy of plaintext recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution provided by the present application, the following is a brief introduction to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present application.

[0036] Figure 1 This is a schematic diagram of the overall process of this application;

[0037] Figure 2 This is a schematic diagram of the method for calculating the average distance between adjacent characters in this application;

[0038] Figure 3 This is a schematic diagram of the calculation of the proportion of continuous segments of character types in this application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] A string plaintext recognition method based on feature information calculation, such as Figure 1 As shown, the following steps are included:

[0041] Step S100, parsing the string and training

[0042] Get a certain amount of plaintext strings and ciphertext strings, such as 10,000 plaintext strings and 20,000 ciphertext strings, and parse the above number of strings by means of keyword matching to obtain the password fields contained in the URLs. By parsing the password fields, the features in the strings can be summarized for subsequent analysis.

[0043] The password fields that have been parsed are trained to determine the recognition judgment items: information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and proportion of continuous fragments of character type; the information entropy threshold, average distance threshold, conversion probability threshold, readability threshold and continuous fragment threshold of the string are obtained respectively.

[0044] The method for determining the threshold of the recognition judgment item is as follows: a clustering interval and a step length are set for each individual recognition judgment item, and within the clustering interval, the value that reaches the specified accuracy and has the highest recall is selected according to the set step length as the threshold of the recognition judgment item.

[0045] In order to improve the efficiency of judgment, a first threshold and a second threshold are set respectively. The first threshold is used for plaintext string judgment, and the second threshold is used for ciphertext string judgment. The accuracy of the values ​​of the first threshold and the second threshold is determined as follows: the probability of being a plaintext string or a ciphertext string is 99%.

[0046] That is, when a character string is determined to be a plain text character string by the first threshold, the accuracy is 99%; when a character string is determined to be a cipher text character string by the second threshold, the accuracy is 99%.

[0047] After a character string meets the requirements of the first threshold or the second threshold of any recognition judgment item, no subsequent judgment is performed. Therefore, after passing the judgment of the first threshold and the second threshold respectively, the number of character strings that need to be judged subsequently will be greatly reduced.

[0048] Step S200, judging whether the string is plain text or cipher text

[0049] Determine the priority of the identification judgment items. The priority is determined as follows:

[0050] According to the recall rate corresponding to each recognition judgment item, they are sorted from large to small, and the recognition judgment item with the largest recall rate is taken as the first priority, and so on.

[0051] Assume that there are 10,000 plaintext strings and 200,000 ciphertext strings, the information entropy corresponding threshold recalls 500, the average distance between adjacent characters recalls 4,000, the character type conversion probability can exclude 100,000 ciphertexts, the string readability judgment can recall 2,000, and the character type continuous segment ratio can recall 300. The corresponding recall rates are 0.005, 0.4, 0.5, 0.2, and 0.003, so the priority judgment order is: character conversion probability, average distance between characters, readability, information entropy, and character type continuous segment ratio.

[0052] The information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio of the string are calculated respectively to obtain the calculation score; the first priority calculation score is compared with the threshold corresponding to the first priority in order of priority to determine whether it is a plaintext string. If so, the string is marked; if not, the calculation score of the string is compared with the second threshold to determine whether the string is a ciphertext string. If so, it is also marked and the next round of judgment is not performed. For example, the plaintext string marked in the first round of judgment is 4000, and the ciphertext string marked is 6000. In this way, 10,000 strings are no longer judged in the next round, which effectively improves efficiency.

[0053] The next round of judgment will be performed only for the character strings whose calculated scores are between the first threshold and the second threshold.

[0054] The remaining character strings are compared with the corresponding threshold value by calculating the second priority score to determine whether they are plain text character strings, and the plain text character strings determined in this round are marked. This process is repeated until all five character string recognition determination items are completely determined.

[0055] Generally, all plaintext and ciphertext strings can be marked after 5 rounds of judgment. If there are still a few strings that have not been judged, manual judgment and marking are sufficient.

[0056] The following is an explanation of the specific scoring method for the identification judgment item:

[0057] Average distance between adjacent strings

[0058] Combination Figure 2 , the average distance between adjacent characters is calculated as:

[0059]

[0060] where v i is the ASCII value of the i-th character, where v i+1 is the ASCII value of the i+1th character, and n is the length of the string.

[0061] The ASCII distance of adjacent characters is obtained by first averaging within a group and then averaging between groups. It is affected by the length of the string. The longer the string is, the greater the distance value is. In order to reduce the impact of the string length on the distance calculation, the calculation is also done in a grouping manner. The formula for grouping calculation is:

[0062]

[0063] In the formula, j represents the group.

[0064] Since the plain text mainly consists of lowercase letters, numbers, and uppercase letters, we have set a penalty for the distance between different types of characters. For example, the penalty coefficient for the distance between characters of the same type is 0.6, the penalty coefficient for the distance between lowercase characters and uppercase letters is 1.0, the penalty coefficient for the distance between uppercase and lowercase letters and numbers is 1.1, the penalty coefficient for the distance between uppercase letters and symbols is 1.6, and the others are 2.0.

[0065] The calculation score of the string based on the reward and penalty coefficient is obtained by the following distance formula:

[0066]

[0067] In the formula, α i，i+1 It is the reward and punishment coefficient.

[0068] Character type conversion probability

[0069] Count the switching frequencies of character types in all strings. Normally, the switching frequency of characters in a string of plain text is not very high. On the contrary, after encryption, the ciphertext is closer to a random number. Therefore, the plaintext or ciphertext of the string can be judged more accurately by calculating the switching frequency.

[0070] Then, the ratio of the switching frequency to the non-switching frequency of adjacent characters in the string is used as the calculation score ratio of the character type conversion probability. The calculation formula is:

[0071]

[0072] In the formula, m different is the switching frequency between adjacent characters in the string, m same It is the frequency of non-switching between adjacent characters in a string.

[0073] Preferably, since the switching frequency between characters of the same type is relatively high, the first threshold of the character type conversion probability ratio is defined as 0.87.

[0074] String readability judgment

[0075] Get a certain amount of English dictionaries and pinyin, and count the jump probability between strings. Because there is no hidden state, there is no emission probability. If there is no jump, it means there is a subsequence. The probability of occurrence of the n-order subsequence of characters in the string is counted through a simplified variant n-order hidden Markov model, and the calculation score of the string readability judgment is calculated through the subsequence occurrence probability:

[0076]

[0077] In the formula, p(sub) is the probability of occurrence of a subsequence, calculated using a simplified hidden Markov model, N is the maximum subsequence length, and L is the string length. L as the denominator can mainly eliminate the influence of length as much as possible. By finding all n-order subsequences in the string and calculating the probability of occurrence, the recognition effect is greatly improved.

[0078] When P is less than a certain value, such as 10-12, it can be set according to actual needs. The discrimination is not enough, so it is finally converted into a form of entropy. The calculation score of the string readability judgment is:

[0079]

[0080] Experiments have shown that if P<8, the probability that the string is plaintext is >99%.

[0081] Character type continuous judgment ratio

[0082] The ciphertext obtained after the data flow is encrypted has a strong randomness. In the ciphertext, there are few continuous fragments. This application judges the ciphertext from the continuity level of characters. For a character string with a larger proportion of continuous fragments, the probability that it is plaintext is higher.

[0083] Count the number of all character segments and the number of continuous segments in the string. The criteria for determining a continuous character segment are: three adjacent characters of the same type, combined with Figure 3 , then the calculation score of the character type continuous judgment ratio is:

[0084]

[0085] Where, d 连续 is the number of consecutive segments in the string, and d is the number of all segments in the string.

[0086] All character strings are detected, and the first threshold of the continuous character type ratio is taken as 0.6, that is, if the continuous segment is greater than 0.6, the character string is considered to be plain text.

[0087] Information Entropy

[0088] The encrypted string is highly random, very similar to a completely random string (if the internationally accepted encryption method is used, it can be considered that this well-encrypted data stream is indistinguishable from a completely random data stream), which means that the encrypted string exhibits greater uncertainty and confusion than the plaintext string, and information entropy can reflect the purity of a string. It can be considered that the higher the purity of the string, the greater the possibility that the string is plaintext.

[0089] By counting the length of the corresponding string and the probability of any character appearing in the string, the calculation score corresponding to the information entropy can be obtained as:

[0090]

[0091] Where n is the set length of the string, f i The frequency of occurrence of the character in the string.

[0092] It is worth noting that the size of entropy is not only related to the purity of the characters in the string, but also affected by the length of the string. It is clear that within a certain length range (when the length difference is not large), the greater the entropy, the higher the degree of disorder of the string. In order to solve the impact of length on entropy, the global entropy is adjusted to group entropy here. The specific adjustment is shown in the following formula:

[0093]

[0094] For the last group of characters that is less than 8 bits, if the length is greater than 4, the entropy of the group is retained and calculated; if it is less than or equal to 4, it is discarded.

[0095] In one embodiment, all character strings are tested, and the proportion of plain text in the results with a first entropy value <1.4 is >99%.

[0096] Step S300: Count the identified plaintext character strings and ciphertext character strings according to the marking results.

[0097] After obtaining and parsing a certain amount of plaintext strings and ciphertext strings, this application obtains the judgment thresholds of plaintext strings under five dimensions through multi-dimensional analysis and training of the strings, namely, information entropy threshold, average distance threshold, conversion probability threshold, readability threshold and continuous segment threshold. After determining the priority of each threshold, the string is respectively calculated for information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio to obtain a calculated score, and then the corresponding calculated score of each priority is compared with the threshold in order of priority to determine whether it is a plaintext string, and the plaintext string that has been judged is marked, and the string that has been judged as plaintext is removed in the next round of judgment, thereby improving the judgment efficiency. In this way, the string features are analyzed from multiple angles to improve the accuracy of plaintext recognition.

[0098] Finally, a few points should be explained: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, which may refer to mechanical connection or electrical connection, or internal communication between two components, or direct connection. "upper", "lower", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may change;

[0099] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;

[0100] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for character string plaintext recognition based on feature information calculation, characterized in that: include: Obtain a certain amount of plaintext strings and ciphertext strings, parse the strings to be identified, and obtain the password fields contained in the URL respectively; The parsed password fields are trained to determine the recognition judgment items: information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio; the information entropy threshold, average distance threshold, conversion probability threshold, readability threshold and continuous segment threshold of the string are obtained respectively; Determine the priority of the recognition judgment items, calculate the information entropy, average distance between adjacent characters, character type conversion probability, string readability judgment and character type continuous segment ratio of the character string, and obtain the calculation score; Compare the first priority calculation score with the threshold corresponding to the first priority in order of priority to determine whether it is a plain text string, and if so, mark the string; The remaining character strings are compared with the corresponding threshold value by calculating the second priority score to determine whether they are plain text character strings, and the determined plain text character strings are marked, and this process is repeated until all five character string recognition determination items are completed; Count the plaintext character strings and ciphertext character strings identified according to the marking results; The method for obtaining the average distance calculation score of the adjacent character strings is: Determine the types of adjacent characters in the adjacent character string, set different reward and penalty coefficients for different adjacent character types, and determine the character length in the character string; The calculation score of the string is obtained by the following distance formula: In the formula, v i is the ASCII value of the i-th character, where v i+1 is the ASCII value of the i+1th character, n is the length of the string, α i，i+1 is the reward and penalty coefficient, j represents the group; The method for obtaining the string readability judgment calculation score is: Get a certain amount of English dictionaries and pinyin, use the hidden Markov model to count the probability of subsequence occurrence of characters in the string, and calculate the score of string readability judgment based on the subsequence occurrence probability: In the formula, p(sub n ) is the probability of subsequence occurrence, calculated using a simplified hidden Markov model, N is the maximum subsequence length, and L is the string length; When P is less than a certain value, the calculation score of the string readability judgment is:

2. The method for character string plaintext recognition based on feature information calculation as claimed in claim 1, characterized in that: The thresholds corresponding to the string recognition judgment items include a first threshold and a second threshold. The first threshold is used for plaintext string judgment, and the second threshold is used for ciphertext string judgment. After judging whether a string is a plaintext string, if not, the calculated score of the string is compared with the second threshold to judge whether the string is a ciphertext string. If so, it is also marked and the next round of judgment is not performed. The accuracy of the values ​​of the first threshold and the second threshold is judged as follows: the probability of being a plaintext string or a ciphertext string is 99%.

3. The method for character string plaintext recognition based on feature information calculation according to claim 1, characterized in that: The method for determining the threshold and priority of the identification judgment item is as follows: A clustering interval and a step length are set for each individual recognition judgment item. Within the clustering interval, the value that reaches the specified accuracy and has the highest recall is selected according to the set step length as the threshold of the recognition judgment item; Then, the recall rates corresponding to each recognition judgment item are sorted from large to small, and the recognition judgment item with the largest recall rate is given the first priority, and so on.

4. The method for character string plaintext recognition based on feature information calculation according to claim 1, characterized in that: The method for obtaining the character type conversion probability calculation score is: The switching frequency of character types in all strings is counted, and then the ratio of the switching frequency to the non-switching frequency of adjacent characters in the string is used as the calculation score ratio of the character type conversion probability. The calculation formula is: In the formula, m different is the switching frequency between adjacent characters in the string, m same It is the frequency of non-switching between adjacent characters in a string.

5. The method for character string plaintext recognition based on feature information calculation according to claim 1, characterized in that: The method for obtaining the character type continuous judgment ratio calculation score is: Count the number of all character segments and the number of continuous segments in the string. The criterion for determining a continuous character segment is: if three adjacent characters are of the same type, the calculation score for the continuous character type determination ratio is: Where, d 连续 is the number of consecutive segments in the string, and d is the number of all segments in the string.

6. The method for character string plaintext recognition based on feature information calculation according to claim 1, characterized in that: The method for obtaining the information entropy calculation score is: By counting the length of the corresponding string and the probability of any character appearing in the string, the calculation score corresponding to the information entropy is: Where n is the length of the string, f p The frequency of occurrence of the character in the string.

Images