Firmware targeted analysis method

Abstract

The application discloses a firmware target analysis method, comprising the following steps: identifying and decompressing a firmware binary file, and extracting root file system related feature field information; determining the CPU framework of the firmware according to the extracted related feature field information, and performing reverse assembly to obtain a firmware loading address, so that the firmware runs normally; classifying and storing the input and extracted files according to file suffixes and names; extracting target information from the files classified and parsed at different depths, and outputting the target information; for large-scale firmware analysis, an improved method based on adjacent version recursive search is proposed based on entropy analysis; for firmware that cannot be identified or analyzed abnormally, a Magic field feature library is constructed to perform length judgment, extraction analysis and feature comparison on the binary fields in the firmware header, so that target file system analysis is completed.

Description

Technical Field

[0001] This invention relates to the field of firmware security analysis technology, and in particular to a firmware targeted analysis method. Background Technology

[0002] Due to a lack of emphasis on security by device manufacturers, firmware often lacks sufficient security checks before release, containing numerous known vulnerabilities. Even for publicly disclosed insecurity issues, corresponding patches are not released in a timely manner. Furthermore, firmware integrity checks performed for security reasons hinder the development of third-party patches. Therefore, firmware security analysis is particularly important.

[0003] With the increasing variety and quantity of firmware, large-scale firmware security analysis helps to promptly locate and patch firmware vulnerabilities, improving device security. However, to protect the uniqueness of their technology, firmware vendors use various encrypted file systems and non-standard compression algorithms, making it difficult for mainstream firmware scanning tools such as Binwalk, BAT, and FRAK to effectively identify and extract important information such as the kernel and file system from firmware images, hindering in-depth firmware security analysis.

[0004] Meanwhile, due to the widespread use of firmware in various devices, attacks targeting firmware are increasing. Therefore, firmware security analysis is crucial. However, currently, apart from a few analyses targeting specific devices or software versions, there is no mature, large-scale firmware security analysis technology. Furthermore, firmware analysis mainly relies on manual extraction using mainstream static tools, resulting in slow speeds, limited single-item analysis capabilities, and low success rates in identifying encrypted firmware or firmware with obfuscated magic values. It is difficult to perform targeted extraction and analysis of firmware, and it is also impossible to comprehensively obtain key sensitive information. Summary of the Invention

[0005] To address the shortcomings of current firmware analysis methods, such as low efficiency in identifying vulnerabilities through single firmware analysis and manual target file retrieval, and the limited file system support and unsatisfactory parsing results of existing firmware parsing tools (e.g., Binwalk / BAT / FRAK) in locating and extracting critical firmware functional modules (e.g., binary code and configuration files), this paper proposes a targeted firmware analysis method. This method optimizes the compatibility of unpacking tools, expands file system support, and enables customized firmware parsing. By optimizing unpacking recognition formats and increasing compatibility with different firmware file types, the method aims to improve the accuracy of firmware parsing and its compatibility with various file systems. Furthermore, the provided API interface enables customized functions such as controllable parsing depth, intelligent file classification, and information classification output.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A firmware targeted analysis method, comprising:

[0008] Step 1: Identify and decompress the firmware binary file, and extract the root file system related feature fields;

[0009] Step 2: Based on the extracted relevant feature field information, determine the CPU framework of the firmware and disassemble it to obtain the firmware load address so that the firmware can run normally;

[0010] Step 3: Categorize and store the input and extracted files according to their file extensions and names;

[0011] Step 4: Extract target information from files parsed and classified at different depths, and output the target information by category.

[0012] Further, step 1 includes:

[0013] Perform initial analysis on the file, preset the number of parsing depths (depth), and determine whether it is a deep parsing or a shallow parsing;

[0014] For deep analysis, a recursive decompression method is used to read the compression mode feature value of the firmware and compare it with the value in the compression mode feature library to identify the compression mode of the target device firmware.

[0015] After a successful match and determination of the compression method, targeted decompression is performed. At the same time, all file types after decompression are scanned. After each complete decompression of the target firmware file, the depth layer number is decremented by one.

[0016] For any remaining compressed files, recursively decompress them until the final depth is 0 or there are no compressed files or unrecognizable compressed files.

[0017] For unrecognized compressed files, an error is reported. At the same time, the decompressed files are identified, and the files are segmented according to the feature values ​​in the library, and the unrecognized feature values ​​are segmented out.

[0018] Further, step 3 includes:

[0019] The system uses a pre-defined whitelist and blacklist to detect and determine the file type. Sensitive files whose names contain Password / Key / Secret / credential / auth / login or whose extensions are elf / py / java / bin / exe are stored and marked as sensitive information for later analysis.

[0020] Skip files that are on the blacklist and store them separately.

[0021] Further, step 4 includes:

[0022] The files excluding those on the blacklist are processed again after classification. The binary files are traversed, and the files are further parsed according to the file system structure. Information is extracted from the files according to their respective data structures, including Magic values ​​and CRC check codes.

[0023] A verification mechanism is used to check the files to ensure that the files are read completely. Files that cannot be identified or detected as abnormal are reported as abnormal. At the same time, the whitelist is judged. If there are many whitelisted files, they are marked as critical firmware and output to the target.

[0024] Furthermore, for large-scale firmware parsing, the method also includes:

[0025] The firmware decryption program is searched for adjacent firmware versions in the database. For encrypted high-entropy firmware, its version information is obtained by segmenting the firmware name and matched in the database. The decryption program is located for all versions by using preset keywords. After locating the decryption program, the corresponding QEMU is selected for simulated decryption by viewing the executable file's running architecture. The obtained decryption program is output, stored, and added to the decryption program library.

[0026] Furthermore, for firmware that cannot be identified or analyzed as abnormal, a Magic field feature library is constructed to perform length judgment, extraction analysis, and feature comparison on the binary fields of the firmware header, thereby completing targeted file system parsing. At the same time, different file system structures are studied to obtain Magic values ​​of different file systems and add them to the feature value library. During each firmware file system parsing, the Magic values ​​are classified and identified, and the feature value library is expanded.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This invention starts by acquiring firmware, analyzing its characteristic values, file system differences and commonalities, and making reasonable improvements to the current parser to address its shortcomings. This allows the parser to handle firmware analysis across different file systems. It also precisely categorizes different types of firmware for security analysis, enabling the analysis tool to classify different file systems during static firmware analysis, thus improving the efficiency of firmware parsing and achieving targeted firmware system analysis. For firmware that current mainstream parsers cannot identify or analyze abnormally, this invention studies its file system information, analyzes the header offset in the file system, collects its Magic value, and writes corresponding scripts through API interfaces to identify and extract specific encrypted file systems.

[0029] This research on firmware parser design is of significant research and practical importance for targeted analysis of firmware security and parsing. It significantly improves the scalability and usability of the firmware parser, and designs extended encrypted file system types to enhance the efficiency of large-scale firmware parsing and vulnerability discovery, providing a tool foundation for future firmware parsing work. Compared to traditional parsing methods, this invention improves parsing efficiency and success rate, optimizes the firmware analysis process, better classifies and outputs firmware information, and incorporates targeted extraction methods to achieve targeted identification and parsing of files with specific file systems and compression types, enabling targeted output of sensitive information. Recursive decompression is achieved by judging the parsing depth, resulting in more thorough and comprehensive firmware parsing. Attached Figure Description

[0030] Figure 1 A flowchart for device firmware identification provided in an embodiment of the present invention;

[0031] Figure 2 A flowchart of firmware decompression provided in an embodiment of the present invention;

[0032] Figure 3 A flowchart of a firmware targeted analysis method provided in an embodiment of the present invention;

[0033] Figure 4 A flowchart of the firmware decryption and parsing process based on recursive search of adjacent versions provided in this embodiment of the invention;

[0034] Figure 5 This is a flowchart of the Magic recognition and conversion process provided in an embodiment of the present invention;

[0035] Figure 6 Example diagram of the preliminary parsing results of the plug-in loop provided in the embodiment of the present invention;

[0036] Figure 7This is an example diagram showing the output results of the plugin's targeted extraction of sensitive files provided in an embodiment of the present invention;

[0037] Figure 8 This is one of the example images of Binwalk recognition results provided in an embodiment of the present invention;

[0038] Figure 9 The Magic value of the pre-encryption firmware provided in this embodiment of the invention;

[0039] Figure 10 This is a file system with modified Magic value provided in an embodiment of the present invention;

[0040] Figure 11 This is the second example of the Binwalk recognition result provided in this embodiment of the invention;

[0041] Figure 12 This is one of the example images of the E-Binwalk plugin recognition results provided in an embodiment of the present invention;

[0042] Figure 13 This is a schematic diagram of a tree-shaped directory obtained after parsing the firmware file system, provided in an embodiment of the present invention.

[0043] Figure 14 Example diagram of tar test file information provided in an embodiment of the present invention;

[0044] Figure 15 This is a schematic diagram illustrating the identification results of a tar file with obfuscated Magic values ​​using Binwalk before the improvement provided in this embodiment of the invention.

[0045] Figure 16 This is a schematic diagram illustrating the E-Binwalk recognition results for tar files with obfuscated Magic values, provided in an embodiment of the present invention.

[0046] Figure 17 This is a schematic diagram illustrating the search results for sensitive information in a tar file using E-Binwalk, as provided in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0048] A firmware targeting analysis method is proposed, which can be implemented from three aspects.

[0049] The first aspect is controllable parsing depth, intelligent file classification, and key information extraction. Customized functions are implemented using API interfaces. A recursive parsing method is used, setting an initial depth and recursively extracting until the preset depth is reached. The specific approach to firmware parsing is as follows: Firmware parsing involves two steps. First, the firmware binary file is identified (e.g., ... Figure 1 (as shown) and decompression (as shown) Figure 2 (As shown), and extract the root file system from it. For binary files, use tools to traverse all bytes and identify characteristic fields such as compression algorithms, header information, file systems, and operating system types based on the different offsets of each information to obtain the necessary basic characteristics and distribution information to extract the root file system.

[0050] After obtaining all relevant information, the second step, static analysis, is performed. Based on the information obtained in the first step, the CPU framework of the firmware is determined, and a suitable disassembler is selected to disassemble it to obtain its firmware load address, enabling the firmware to run normally. After that, disassembly analysis can be performed.

[0051] After parsing the firmware, the system intelligently categorizes and stores the input and extracted files based on information such as file extensions, formats, and names. For files parsed and categorized at different depths, a targeted information extraction method is used to specifically obtain key information, and this targeted information is then output in a categorized manner.

[0052] like Figure 3As shown, the specific algorithm implementation is as follows: Initial file analysis is performed. The user presets the parsing depth (depth layers) and determines whether it's deep or shallow parsing. For deep parsing, a recursive decompression method is used. The firmware's compression method feature values ​​are read and compared with values ​​in the compression method feature library to identify the target device's firmware compression method. After a successful match and confirmation of the compression method, targeted decompression is performed. Simultaneously, all file types after decompression are scanned. After each complete decompression of the target firmware file, the depth layer is decremented by one. For any remaining compressed files, recursive decompression is performed until the final depth is 0, or no compressed files or unrecognizable compressed files exist. Unrecognizable compressed files are reported as an anomaly. The decompressed files are then categorized, segmented based on feature values ​​in the library, and the unrecognizable feature values ​​are extracted and their filenames expanded. After the recursive decompression, all files are intelligently classified. The system uses a pre-defined whitelist and blacklist to detect and determine file types. Sensitive files with names containing "Password / Key / Secret / credential / auth / login" or with extensions like "elf / py / java / bin / exe" are stored and marked as sensitive information for later analysis. Blacklisted files, such as pdf / video / doc files, are skipped and stored separately. Then, the system processes files not on the blacklist again, traversing binary files and further parsing them based on file system structures. Since different file systems have different structures and information, information such as byte order, magic value, data blocks, version information, and CRC checksums are extracted from the files using parsers and plugins according to their respective data structures. Various verification mechanisms are used to comprehensively check files to ensure complete reading; files that cannot be identified or detected as abnormal are reported as exceptions. The whitelist is also reviewed; if many whitelisted files exist, they are marked as critical firmware and output accordingly.

[0053] The second aspect is the targeted decryption program location and extraction, such as... Figure 4 As shown, based on entropy analysis, an improved method based on recursive search of adjacent versions is proposed. Entropy analysis is used to determine high or low entropy for initial judgment. A large-scale search is then performed on all versions in the database, and this recursive operation is repeated until a firmware that can be parsed is found. Then, the decryption program is extracted from the firmware, and firmware simulation decryption is implemented using QEMU, completing the targeted decryption extraction. Based on this, a large number of encrypted firmware files in the firmware library can be targeted for decryption, improving firmware decryption efficiency.

[0054] The specific implementation plan involves using a large number of firmware files in the database to search for decryption programs for adjacent firmware versions. For encrypted high-entropy firmware, version information is obtained by segmenting the firmware name and matching it in the database. All versions are located through a large-scale search using preset keywords. The targeted decryption program `imgdecrypt` is located in the CGI that is called. After locating the decryption program, the corresponding QEMU is selected for simulated decryption by examining the executable file's runtime architecture. The obtained decryption programs are output, stored, and added to the decryption program library. Through large-scale analysis, a variety of decryption programs are obtained for decryption and reverse analysis.

[0055] The third aspect is targeted file system parsing. By analyzing the file system structure in firmware that cannot be recognized or shows abnormalities, parsers and plugins are used for extraction. A Magic field feature library is constructed using the feature field matching principle to perform length judgment, extraction analysis, and feature comparison on the firmware header binary fields, thus completing targeted file system parsing. Simultaneously, different file system structures are studied, and Magic values ​​for different file systems are obtained and added to the feature value library. During each firmware file system parsing, Magic values ​​are categorized and identified, expanding the feature value library. Structural analysis of different file systems and compressed files increases the types of recognizable file systems, and the Magic library is expanded to support the recognition of some simple encrypted Magic values. The Magic library can also be expanded by judging file systems with unknown Magic values, preventing abnormal recognition when identifying the same file system subsequently. The binary information of the targeted file system is analyzed to obtain feature values ​​for different files for feature recognition, such as the header offset of squashfs:

[0056]

[0057]

[0058] Tar files have a Magic value of 'ustar' and their file descriptions begin with 'posix tar archive'. Therefore, the research approach focuses on increasing the recognition method by expanding the Magic value of tar files and analyzing both the file description information and the Magic value during recognition (the Magic recognition and conversion process is as follows). Figure 5 (As shown), thus enabling complete and comprehensive identification of the obfuscated firmware information.

[0059] For unknown file systems or compressed files that do not exist in the Magic value, anomalies are recorded and output for later manual analysis of their header information to obtain feature values; for files that can match feature values ​​but whose string addresses do not match the information stored in the feature library, they are identified as special file systems or compressed files and the feature value library is expanded; for firmware files that cannot match feature values ​​but have obvious string identifiers at specific addresses in the existing library, their strings are recorded and output and added to the feature library, and saved as newly identified file system types.

[0060] To verify the effectiveness of the present invention, the following tests were conducted:

[0061] As a specific implementation method, the method of this invention is programmed into a firmware targeted analysis enhancement plugin (hereinafter referred to as the enhancement plugin). The enhancement plugin performs recursive parsing on batches of firmware files, and based on an established sensitive information database, initially extracts all files in the target folder and compares them with the database. Files with sensitive information in their names are output to a specified folder, and the initial parsing results are displayed on the output interface. Subsequently, targeted extraction can be performed, where the user specifies the names and types of sensitive information files to be searched, and the plugin intelligently parses and outputs the results. Test results are as follows... Figure 6 As shown.

[0062] The test database is set to sensitive keywords such as key, keyword, and passwd, and the output directory is as follows. Figure 7 As shown.

[0063] As can be seen, the enhanced plugin successfully completed the targeted extraction of sensitive files and preliminary firmware parsing. It also used a loop-based approach to directly parse and identify all files in the target folder, significantly reducing the extremely low efficiency of traditional single-file parsing methods in large-scale firmware analysis.

[0064] By extracting and testing the file systems, starting from several file systems under study, files with the suffixes cramfs / jffs2 / squashfs were encrypted by rewriting their Magic values, and then identified using the method of this invention. It can be seen that by splitting the header bytes, the corresponding file system offset string can be correctly identified, the representative file system of the Magic value can be found, and the extended values ​​in the Magic library can also be identified.

[0065] ① Test on identification and extraction of the squashfs file system after obfuscating the Magic signature value

[0066] First, the firmware image `openwrt-rockchip-R23.4.20-hinlink_opc-h66k-squashfs-sysupgrade.img.gz` was parsed and extracted for comparison testing. Based on the previously extracted name information, it was known that this firmware image contains a squashfs file system. An unoptimized Binwalk tool was used for identification, and the results are as follows... Figure 8 As shown.

[0067] As can be seen, Binwalk has a certain ability to recognize unencrypted squashfs file systems. It located the beginning segment of the squashfs file system at 134217728 (0x8000000) and can output its version number, endianness, and other information. Next, the 010-Editor binary tool is used to encrypt and modify its Magic value. The following image shows the tool locating its Magic value. Since squashfs header features have seven types: sqsh, hsqs, qshs, shsq, hsqt, tqsh, and sqlz, the tool searches for and locates each Magic value one by one. The results are as follows. Figure 9 As shown.

[0068] As you can see, its Magic value is hsqs, which is 58 73 71 73. After modifying its Magic value to abcd using 010Editor and performing a recognition test, the modified file system looks like this. Figure 10 As shown.

[0069] The recognition results after testing with Binwalk are as follows: Figure 11 As shown.

[0070] As can be seen, after changing the Magic value hsqs to abcd at 134217728 (0x8000000), Binwalk cannot recognize the encrypted Magic value abcd starting from 134217838 (0x800006E). Therefore, it starts to make a recognition error at offset 6E and does not output the correct file system information of the firmware.

[0071] Therefore, the plugin design approach can be summarized from the errors identified by Binwalk: First, a Magic library is established to facilitate file system recognition and the expansion of Magic values. Then, the byte information of the file is read and compared with the values ​​in the Magic library. If a match is found, the result is output. The E-Binwalk intelligent parsing tool calls the Pysquashfs tool for the squashfs file system encryption parsing plugin to read the binary information of the firmware, perform matching, and generate a tree-like information directory.

[0072] Previous binary analysis of the bin file revealed that changing the Magic value at 0x8000000 alters the information read. Further research led to a solution where the ASCII code is first converted to hexadecimal in the plugin, and then converted to decimal for matching after reading. This approach effectively accommodates the impact of uppercase and lowercase characters on the Magic value.

[0073] The following are the results obtained after recognition using the enhancement plugin:

[0074] First, it will output information about the file system: such as Figure 12 As shown, the encrypted bin firmware can still be recognized as a squashfs file system. It also outputs the decimal information of its Magic value, which, after conversion to hexadecimal, is 0x64636261, corresponding to the ASCII code dcba (different from abcd modified during Magic obfuscation due to the endianness of the file system's storage).

[0075] It will also generate a tree-like directory structure obtained after parsing the firmware file system, such as Figure 13 As shown, sensitive files can be clearly identified.

[0076] ② Test on the identification and extraction of tar files after obfuscating the Magic signature value

[0077] The test uses a tar-type test file from Binwalk, similar to squashfs, whose Magic value and binary information are as follows: Figure 14 As shown.

[0078] Similar to the squashfs file system recognition test, the binary information of its Magic value is modified and changed to a test encrypted value such as abcde that Binwalk does not support.

[0079] First, we tested the ability of the unenhanced raw tool Binwalk to recognize tar files with obfuscated Magic signature values. The results are as follows: Figure 15 As shown, it cannot be successfully recognized.

[0080] The enhanced E-Binwalk's ability to recognize obfuscated tar files is as follows: Figure 16 As shown.

[0081] Meanwhile, the optimized E-Binwalk supports searching for sensitive characters in tar files and outputting the results. For example, the search results for sensitive characters in a test tar file are shown below. Figure 17 As shown.

[0082] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A firmware targeted analysis method, characterized in that, include: Step 1: Identify and decompress the firmware binary file, and extract the root file system related feature fields; Step 2: Based on the extracted relevant feature field information, determine the CPU framework of the firmware and disassemble it to obtain the firmware load address so that the firmware can run normally; Step 3: Categorize and store the input and extracted files according to their file extensions and names; Step 4: Extract targeted information from files parsed and classified at different depths, and output the targeted information by category; including: performing operations on files other than those on the blacklist after classification, traversing binary files, further parsing files according to the file system structure, and extracting information from files according to their respective data structures, including Magic value and CRC check code; The method also includes: for unidentified or parsed firmware, analyzing its file system structure, extracting data using a parser and parser plugins, constructing a Magic field feature library using feature field matching, performing length judgment, extraction analysis, and feature comparison on the binary fields in the firmware header to complete targeted file system parsing; analyzing different file system structures, obtaining Magic values ​​for different file systems and adding them to the Magic field feature library, classifying and identifying Magic values ​​and expanding the Magic field feature library during each firmware file system parsing; recording unknown file systems or compressed files not stored in the Magic field feature library as abnormal and outputting abnormal information for subsequent analysis of their header information to obtain Magic values; identifying Magic values ​​that can be matched but whose corresponding feature string offset address is inconsistent with the information stored in the Magic field feature library as special file systems or compressed files and expanding the Magic field feature library; recording and outputting the string for firmware files that cannot be matched with Magic values ​​but have a preset feature string identifier at a specific offset address recorded in the Magic field feature library, adding it to the Magic field feature library, and saving it as a newly identified file system type.

2. The firmware targeted analysis method according to claim 1, characterized in that, Step 1 includes: Perform initial analysis on the file, preset the number of parsing depths (depth), and determine whether it is a deep parsing or a shallow parsing; For deep analysis, a recursive decompression method is used to read the compression mode feature value of the firmware and compare it with the value in the compression mode feature library to identify the compression mode of the target device firmware. After a successful match and determination of the compression method, targeted decompression is performed. At the same time, all file types after decompression are scanned. After each complete decompression of the target firmware file, the depth layer number is decremented by one. For any remaining compressed files, recursively decompress them until the final depth is 0 or there are no compressed files or unrecognizable compressed files. For unrecognized compressed files, an error is reported. At the same time, the decompressed files are identified, and the files are segmented according to the feature values ​​in the library, and the unrecognized feature values ​​are segmented out.

3. The firmware targeted analysis method according to claim 1, characterized in that, Step 3 includes: The system uses a pre-defined whitelist and blacklist to detect and determine the file type. Sensitive files whose names contain Password / Key / Secret / credential / auth / login or whose extensions are elf / py / java / bin / exe are stored and marked as sensitive information for later analysis. Skip files that are on the blacklist and store them separately.

4. The firmware targeted analysis method according to claim 1, characterized in that, Step 4 also includes: A verification mechanism is used to check the files to ensure that the files are read completely. Files that cannot be identified or detected as abnormal are reported as abnormal. At the same time, the whitelist is judged. If there are many whitelisted files, they are marked as critical firmware and output to the target.

5. The firmware targeted analysis method according to claim 1, characterized in that, For large-scale firmware parsing, the method also includes: The firmware decryption program is searched for adjacent firmware versions in the database. For encrypted high-entropy firmware, its version information is obtained by segmenting the firmware name and matched in the database. The decryption program is located for all versions by using preset keywords. After locating the decryption program, the corresponding QEMU is selected for simulated decryption by viewing the executable file's running architecture. The obtained decryption program is output, stored, and added to the decryption program library.

Images