Encrypted firmware identification method based on file slicing

Abstract

The present invention discloses a method for identifying encrypted firmware based on file slicing, comprising the following steps: dividing the firmware into file slices of equal size, and setting a reference value and a deviation value; reading binary data of fixed length for each file slice, and then converting it into hexadecimal data, if the hexadecimal data contains a plurality of consecutive identical characters, discarding the group of data and reading the next group, until all file slices are read; calculating the information entropy of the acquired hexadecimal data, and then calculating the variance of all entropy values; calculating the deviation between the variance and the reference value, if the deviation is greater than the deviation value, it can be directly determined whether it is encrypted firmware, otherwise searching for magic words in the file slice and then making a judgment. The present invention uses information entropy to determine the degree of confusion of firmware data, reads part of the data through file slicing, and realizes the identification of encrypted firmware, which has the advantages of simple implementation, high accuracy, and fast detection speed, and is suitable for large-scale firmware verification.

Description

Technical Field

[0001] The present invention belongs to the technical field of encrypted firmware identification, and in particular relates to an encrypted firmware identification method based on file slicing. Background Art

[0002] Firmware is a binary program running in the device, responsible for managing the hardware peripherals in the device and realizing the application functions of the device. According to whether the firmware contains a complete file system, the firmware is divided into two types: with file system and without file system. When analyzing firmware with file system, it is usually necessary to unpack the firmware, extract the complete file system, and then analyze the extracted files. For firmware without file system, since it does not contain a file system, reverse analysis software is usually used directly for analysis.

[0003] "Encrypted firmware" refers to a file that is obtained by fully or partially encrypting an unencrypted firmware file using encryption software and encryption algorithms. During the reverse analysis of IoT firmware, if the firmware is compressed using an unknown compression algorithm, such firmware files are also classified as encrypted firmware because existing tools cannot be used directly for reverse analysis.

[0004] There are generally two ways to identify encrypted firmware. The first is to search for magic words that appear in the file. Magic words are also called file signatures. They are a string of hexadecimal data used to identify file formats or protocols. By searching for magic words in the firmware, you can determine whether the firmware contains files of the corresponding format. If no matching magic words can be found after traversing the entire firmware, it is considered encrypted firmware. The second method is to judge based on the entropy value of the firmware file. Information entropy represents the uncertainty of the data. The greater the degree of data confusion, the greater the information entropy. Since the distribution of data information in unencrypted files is relatively orderly, while the encrypted file is the opposite, the degree of data confusion will be much greater than that of encrypted files. Therefore, the entropy value of the file can be used as a reference for determining encryption.

[0005] However, existing encrypted firmware identification methods have the problems of slow detection speed and low identification accuracy, making it difficult to implement large-scale firmware encryption verification. Summary of the invention

[0006] Technical problem solved: In response to the above technical problem, the present invention provides an encrypted firmware identification method based on file slicing, which can quickly and accurately detect encrypted firmware.

[0007] Technical solution: The encrypted firmware identification method based on file slicing includes the following steps:

[0008] (1) File slicing stage: Use file slicing technology to divide the firmware into file slices of equal size, and set the baseline value and deviation value;

[0009] (2) Data reading stage: Read binary data of fixed length for each file slice and convert the binary data into hexadecimal data. If the hexadecimal data contains multiple consecutive identical characters, the data set is discarded and the next set is read until all file slices are read;

[0010] (3) Calculation phase: Calculate the information entropy of the hexadecimal data obtained in step (2), and then calculate the variance of all entropy values;

[0011] (4) Encrypted firmware judgment stage: Calculate the deviation between the variance and the baseline value. If the deviation is greater than the deviation value, it can be directly judged whether it is encrypted firmware. Otherwise, search for magic words in the file slice and then make a judgment.

[0012] Preferably, the number of file slices in step (1) is 160.

[0013] Preferably, the reference value in step (1) is 15%.

[0014] Preferably, the length of the binary data read in step (2) is 215.

[0015] Preferably, the steps are specifically as follows:

[0016] (1) File slicing stage

[0017] Read the file as a binary data stream, calculate the file size, and logically divide the file into N equal-sized slices, each of which is size, and set the base value to base and the deviation value to deviation;

[0018] (2) Data reading stage

[0019] The file segments split in step (1) are piece[1], piece[2], piece[3]…piece[N];

[0020] The steps to read the data in the i-th file slice piece[i] are as follows:

[0021] Step S2.1, set the file pointer to cursor[i] = size * (i -1);

[0022] Step S2.2, read the binary data stream bytes[i] of length length from the position of cursor[i];

[0023] Step S2.3, convert binary bytes[i] into hexadecimal data hex[i];

[0024] Step S2.4, determine whether there are multiple consecutive identical characters in the data hex[i], if yes, go to step S2.5, otherwise go to step S2.6;

[0025] Step S2.5, discard the data hex[i] and set i to i+1;

[0026] Step S2.6, set i to i+1;

[0027] Repeat step (2) until i=N;

[0028] (3) Data calculation stage

[0029] Step S3.1, calculate the information entropy h[j] of hex[j], and add h[j] to the entropy value list;

[0030] Step S3.2: If , set j to j+1 and go to step S3.1, otherwise go to step S3.3;

[0031] Step S3.3, calculate the variance of all data in the entropy value list;

[0032] (4) Encrypted firmware judgment stage

[0033] Step S4.1, if |variance-base|>deviation, go to step S4.2, otherwise go to step S4.3;

[0034] Step S4.2, if variance>base, go to step S4.4, otherwise go to step S4.5;

[0035] Step S4.3, search for magic words from the position of file pointer cursor=0, if a magic word that meets the conditions is found, go to step S4.4, otherwise go to step S4.5;

[0036] Step S4.4, output the firmware as encrypted firmware, and end;

[0037] Step S4.5: Output the firmware as unencrypted firmware, and end.

[0038] Preferably, the information entropy h[j] in step S3.1 is calculated as follows: .

[0039] Preferably, the calculation method of the variance in step S3.3 is as follows:

[0040]

[0041]

[0042] Where h is the mean of all h[i] obtained in step S3.1.

[0043] Beneficial effect: The present invention provides an encrypted firmware detection method based on file slicing, which uses the variance of multiple information entropies as the basis for judging the encryption of the firmware, and can quickly and accurately determine whether the firmware is encrypted.

[0044] The present invention combines entropy value judgment and magic word judgment, and can fully detect firmware without omission on the basis of improving the recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of the encrypted firmware identification method of the present invention;

[0046] Figure 2 It is a specific algorithm flow chart of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0048] Example 1

[0049] like Figure 1 As shown, the encrypted firmware detection algorithm based on file slicing described in the present invention includes four stages, a file slicing stage, a data reading stage, a data calculation stage and an encrypted firmware judgment stage.

[0050] (1) File slicing stage: The file is read in binary format and logically divided into file slices of equal size, and the baseline value and deviation value are set;

[0051] (2) Data reading stage: Read binary data of fixed length from each file slice in turn, convert the binary data into hexadecimal data. If the hexadecimal data contains multiple consecutive identical characters, discard the data and read the next group until all file slices are read;

[0052] (3) Calculation phase: Calculate the information entropy of the hexadecimal data obtained in step (2), save all the obtained entropy values ​​in a list, and then calculate the variance of all entropy values;

[0053] (4) Encrypted firmware judgment stage: Calculate the deviation between the variance and the reference value. If the deviation is greater than the deviation value, it can be directly determined whether it is encrypted firmware. Otherwise, search for magic words in the file and then make a judgment.

[0054] The number of slices into which the file is cut and the length of each data segment to be read will affect the program's running time and the final result: too many slices will affect the program's running time, while too few slices will affect the randomness of the read data. Similarly, when the read data is too little, it cannot well reflect the degree of data obfuscation, and the result cannot be used as a basis for judging the encrypted firmware. Reading too much data will also cause the program to run too long. Therefore, the choice of the number of slices and the length of the data read will affect the judgment of the result.

[0055] In this embodiment, the number of slices pieces is set to 160, and the length of the read data length is set to 215.

[0056] Provide specific steps of the present invention below:

[0057] (1) File slicing stage

[0058] Read the file as a binary data stream, calculate the file size, and logically divide the file into N equal-sized slices, each of which is size, and set the base value to base and the deviation value to deviation;

[0059] (2) Data reading stage

[0060] The file segments split in step (1) are piece[1], piece[2], piece[3]…piece[N];

[0061] The steps to read the data in the i-th file slice piece[i] are as follows:

[0062] Step S2.1, set the file pointer to cursor[i] = size * (i -1);

[0063] Step S2.2, read the binary data stream bytes[i] of length length from the position of cursor[i];

[0064] Step S2.3, convert binary bytes[i] into hexadecimal data hex[i];

[0065] Step S2.4, determine whether there are multiple consecutive identical characters in the data hex[i], if yes, go to step S2.5, otherwise go to step S2.6;

[0066] Step S2.5, discard the data hex[i] and set i to i+1;

[0067] Step S2.6, set i to i+1;

[0068] Repeat step (2) until i=N;

[0069] (3) Data calculation stage

[0070] Step S3.1, use the entropy value calculation formula: , calculate the information entropy h[j] of hex[j], and add h[j] to the entropy value list;

[0071] Step S3.2: If , set j to j+1 and go to step S3.1, otherwise go to step S3.3;

[0072] Step S3.3, calculate the variance of all data in the entropy value list, the calculation method is as follows:

[0073]

[0074]

[0075] Where h is the mean of all h[i] obtained in step S3.1;

[0076] (4) Encrypted firmware judgment stage

[0077] Step S4.1, if |variance-base|>deviation, go to step S4.2, otherwise go to step S4.3;

[0078] Step S4.2, if variance>base, go to step S4.4, otherwise go to step S4.5;

[0079] Step S4.3, search for magic words from the position of file pointer cursor=0, if a magic word that meets the conditions is found, go to step S4.4, otherwise go to step S4.5;

[0080] Step S4.4, output the firmware as encrypted firmware, and end;

[0081] Step S4.5: Output the firmware as unencrypted firmware, and end.

[0082] Since the actual benchmark value needs to be set according to different firmware data sets, a reasonable benchmark value is one that accurately reflects the degree of difference in the data. In this embodiment, 15% is used as the benchmark value.

[0083] This embodiment is compared with the binwalk detection software, and 1262 router firmwares are analyzed respectively. The detection results are compared as follows:

[0084] Options Encrypted firmware number Unencrypted firmware Number of detection errors False Positive Rate Detection time / second This embodiment 108 1154 1 0.079% 78.265 binwalk 114 1148 8 0.634% 577.043

[0085] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. The encrypted firmware identification method based on file slicing is characterized by: The steps include: (1) File slicing stage: Use file slicing technology to divide the firmware into file slices of equal size, and set the baseline value and deviation value; Specifically, read the file as a binary data stream, calculate the file size, divide the file into N equal-sized slices in a logical manner, each slice is size, set the base value to base, and the deviation value to deviation; (2) Data reading stage: Read binary data of fixed length for each file slice and convert the binary data into hexadecimal data. If the hexadecimal data contains multiple consecutive identical characters, the data set is discarded and the next set is read until all file slices are read; Specifically, the file segments divided in step (1) are piece[1], piece[2], piece[3]...piece[N]; The steps to read the data in the i-th file slice piece[i] are as follows: Step S2.1, set the file pointer to cursor[i]=size*(i-1); Step S2.2, read the binary data stream bytes[i] of length length from the position of cursor[i]; Step S2.3, convert binary bytes[i] into hexadecimal data hex[i]; Step S2.4, determine whether there are multiple consecutive identical characters in the data hex[i], if yes, go to step S2.5, otherwise go to step S2.6; Step S2.5, discard the data hex[i] and set i to i+1; Step S2.6, set i to i+1; Repeat step (2) until i=N; (3) Calculation phase: Calculate the information entropy of the hexadecimal data obtained in step (2), and then calculate the variance of all entropy values; Specifically, step S3.1, calculate the information entropy h[j] of hex[j], and add h[j] to the entropy value list; the information entropy h[j] is calculated as follows: h(x) = -∑P(x j )·logP(x j ); Step S3.2, if j≠N, set j to j+1 and go to step S3.1, otherwise go to step S3.3; Step S3.3, calculate the variance of all data in the entropy value list; the calculation method of variance is as follows: Where h is the mean of all h[i] obtained in step S3.1; (4) Encrypted firmware judgment stage: Calculate the deviation between the variance and the reference value. If the deviation is greater than the deviation value, directly determine whether it is encrypted firmware. Otherwise, search for magic words in the file slice and make a judgment again. Specifically: Step S4.1, if |variance-base|>deviation, go to step S4.2, otherwise go to step S4.3; Step S4.2, if variance>base, go to step S4.4, otherwise go to step S4.5; Step S4.3, search for magic words from the position of file pointer cursor=0, if a magic word that meets the conditions is found, go to step S4.4, otherwise go to step S4.5; Step S4.4, output the firmware as encrypted firmware, and end; Step S4.5: Output the firmware as unencrypted firmware, and end.

2. The encrypted firmware identification method based on file slicing according to claim 1 is characterized in that: The number of file slices in step (1) is 160.

3. The encrypted firmware identification method based on file slicing according to claim 1 is characterized in that: The reference value in step (1) is 15%.

4. The encrypted firmware identification method based on file slicing according to claim 1, characterized in that: The length of the binary data read in step (2) is 215.

Images