An automated intelligent embedded firmware analysis and vulnerability mining method

Abstract

The present invention discloses an automated intelligent embedded firmware analysis and vulnerability mining method, comprising the following steps: 1) obtaining firmware; 2) unpacking the firmware; 3) extracting executable files; 4) parsing file structures; 5) disassembly and control flow analysis; 6) function partitioning and cross-reference; 7) decompilation and in-function variable tracking; 8) vulnerability mining; this method supports the combined operation function of side-channel and error injection, and the scripted firmware analysis function, ensuring that known software vulnerabilities are detected without omission, or more careful analysis is carried out for a certain vulnerability, providing post-processing for firmware unpacking and analysis, and realizing the automation of the main process from firmware files to vulnerability reports.

Description

Technical Field

[0001] The present invention relates to an automated intelligent embedded firmware analysis and vulnerability mining method. Background Art

[0002] In recent years, the application scenarios of embedded systems in the Internet of Things have developed rapidly. However, its basic characteristics of having no display terminal and limited hardware resources also make users and manufacturers have different views on it from standard computers. From the perspective of users, an embedded system is a standard hardware, often ignoring the dominant software firmware inside it. Its concealment also makes users not care about firmware upgrades to repair software defects or vulnerabilities, or the backdoors deliberately attached by a malicious manufacturer or any endpoint in the supply chain. On the other hand, from the perspective of manufacturers, under the condition of limited hardware resources, functional development often takes precedence over security considerations. Therefore, its security is generally much lower than that of standard computer systems. However, embedded systems are often widely used in occasions that require special attention to their security, such as gateways, routers, etc. Their vulnerability creates the most concealed gap in the overall system security, which is a worrying situation.

[0003] Therefore, a system that can perform planned security testing and evaluation on embedded systems plays an extremely important role in the scenario of the explosion of Internet of Things applications. Different from standard computers, embedded systems have great technical differences, and this difference involves both software and hardware conditions. Therefore, it is very challenging to perform planned security testing and evaluation on embedded systems considering the needs of relevant technical personnel and equipment. In addition, due to the differences of embedded systems, the testing and evaluation system has higher requirements for automation and quantification. The technical gap caused by the above-mentioned layers of difficulties in technical personnel and equipment is the root cause of the problem in realizing automation and quantification.

[0004] The existing technologies have the following problems: the existing technologies lack comprehensive analysis functions; for example, mainly aiming at software-type vulnerability analysis, they cannot support hardware-type vulnerabilities (side channels and fault injection), mainly rely on manual operations to provide the firmware to be analyzed, lacking automation and quantification functions, and lacking perfect processing after firmware unpacking. For example, the existing technologies lack related operations at the toolchain level, limiting the maximum degree of automation and quantification analysis functions.

[0005] Therefore, it is necessary to seek an automated method for the main process from firmware files to vulnerability reports. Summary of the Invention

[0006] In view of this, the object of the present invention is to provide an embedded firmware analysis and vulnerability mining method that supports the combined operation function of side-channel and error injection, scripted firmware analysis function, ensures that known software vulnerabilities are not missed in detection, or conducts more careful analysis for a certain vulnerability, and provides firmware unpacking and post-analysis processing.

[0007] To solve the above technical problems, the technical solution of the present invention is as follows:

[0008] An automated intelligent embedded firmware analysis and vulnerability mining method, comprising the following steps:

[0009] 1) Obtain the firmware. Before starting the firmware analysis, first, it is necessary to be able to obtain the firmware of the target embedded device. If the target firmware is distributed to users through the public domain, then researchers can directly use these firmware packages for subsequent automatic analysis processing. If the firmware cannot be obtained in the public domain, then it is necessary to directly extract the firmware from the hardware;

[0010] 2) Unpack the firmware; if the firmware package is directly extracted from the target device, then the dumped binary file can be directly used for subsequent analysis. When some firmware is released, for various considerations such as firmware package size and file integrity verification, it may be packed multiple times. For these firmware files, the first step is to decompress these files layer by layer until a completely decompressed binary file is obtained;

[0011] 3) Extract the executable files; after obtaining the binary files, more abstract analysis needs to be carried out. Feature scanning is performed on these file system formats, and all files are completely extracted from the file system;

[0012] 4) Analyze the file structure; after obtaining the executable files, the analysis of the file format needs to be carried out. Taking the Linux kernel as an example, in the Linux system, the executable files are generally in the ELF format. The ELF data required for the executable files mainly include instruction set information (platform / instruction set version / endianness), symbol table / import table (referenced external libraries, especially system libraries), etc.;

[0013] 5) Disassemble and perform control flow analysis; after loading the data in the file into memory according to the information contained in the ELF, the code segment can be disassembled. The role of disassembly is to convert the data flow in the code segment into an instruction control flow. As long as the mainstream instruction set is supported, most executable files can be successfully disassembled. Only by obtaining the instruction-level control flow and knowing how the program operates can subsequent higher-level analysis be possible;

[0014] 6) Function partitioning and cross-reference; The control flow obtained from the data flow conversion is flat in memory. Only by sorting out the function list can we analyze the parameter passing and data flow in the program. To reconstruct the functions that may exist in the source code from the flat control flow through methods such as simulation execution, the program entry and export table items obtained from the previous analysis of ELF metadata can both serve as the entry points for simulation execution. Through recursive simulation execution, when a function call is encountered, it recursively enters the simulation execution of another function. Eventually, the entry points of a considerable part of the functions in this executable file can be obtained. On the basis of recursive simulation execution, a scan of the function entry points in the data segment is also added to complete the function list. Then, by analyzing the instructions, a basic cross-reference database is constructed while simulating the execution of each function;

[0015] 7) Decompilation and variable tracking within functions; Decompilation mainly includes three processes: The first process is from the original instructions to intermediate expressions. Regardless of the CPU instruction architecture, the most basic and indispensable operations belong to the category of the arithmetic logic unit. After this translation, regardless of the original architecture of the executable file, it is converted into a unified intermediate language, making the subsequent processing logic no longer need to consider the differences of each original architecture; The second process is to construct a structure similar to an abstract syntax tree (AST). The flat control flow analysis was sorted out into individual functions before. Now, this construction similar to an AST combines the flat control blocks in the function with each other and nests them into the nodes on the syntax tree. After this step of processing, various syntax blocks can be generated, and the prototype of the decompilation result emerges; The third process is optimization, which involves variable sorting and expression contraction, etc. Intermediate variables that are meaningless for analysis will be hidden, leaving only meaningful expressions;

[0016] 8) Vulnerability mining; First is the vulnerability mining plugin, which applies complete data tracking. Through this plugin system, users can define the inputs and outputs that may have vulnerabilities and run the plugin to automatically search for vulnerabilities;

[0017] Preferably, in the firmware obtained in step 1), the main hardware-level debugging protocols in the market are UART, JTAG, etc. Although the details of these protocols are public, embedded device manufacturers often do not mark the functions of each pin on the circuit board. If the target device has a large number of pins, it is quite difficult to discover the key pins from this exponentially expanding permutation and combination. The tool board of this solution has multiple pins and the ability to monitor and write to these pins. Only need to connect all the pins on the target device to this tool board and start the target device, then this tool board can automatically sense various debugging protocols provided by the target device. Through these protocols, functions such as hardware information reading, non-volatile storage transfer, memory operation, and instruction-level debugging can be realized.

[0018] Preferably, in step 6), the cross-reference database is obtained by analyzing instructions. While simulating the execution of each function, it can be known what memory address a certain line of instruction operates on. These operations may be reads or writes. Organize these operations on memory addresses, and then add the call relationships between functions to form a basic cross-reference database. With this cross-reference data, given a memory address, it is possible to immediately find out which functions / instructions read and write to this address. Given a function entry, it is possible to find out which other functions call this function.

[0019] Preferably, in step 8), as an aid to vulnerability discovery, two sets of simulation debugging systems are also provided. The user-mode simulation debugging system can conveniently perform various tests or debugging on a single executable file, while the hardware-level simulation debugging system can simulate the boot process of real hardware according to the kernel, file system, etc. provided by the user and allow the user to perform kernel debugging. In addition, a fuzzing framework is provided for black-box analysis. By defining the format of user input, the fuzzing framework can automatically generate test input data, use the above-mentioned simulation systems to test a specified executable program, and capture various abnormal behaviors to detect whether there are potential vulnerabilities such as denial-of-service attacks.

[0020] The technical effects of the present invention are mainly reflected in the following aspects:

[0021] 1. Side-channel and error injection; through external physical intervention, the operation of the hardware is brought into an abnormal state, which has the effect of promoting the researcher's privileges.

[0022] 2. Automatic detection of debugging interfaces; by reading the information of each pin of the target device, automatically detect the pins corresponding to various debugging interfaces.

[0023] 3. Decompilation; decompilation is the core of the static analysis of executable files and is also an important reference for researchers to develop PoC.

[0024] 4. Vulnerability Mining: The self-developed vulnerability mining plug-in SDK allows researchers to write corresponding automated vulnerability mining plug-ins for different types of vulnerabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of an automated intelligent embedded firmware analysis and vulnerability mining method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] The following will further elaborate on the specific implementation manners of the present invention in conjunction with the Figure 1 drawings, so that the technical solutions of the present invention are easier to understand and master. Embodiment

[0027] Existing technical solutions have two problems. First, there is a gap between software and hardware. Second, the scope of cross-domain coverage. The problem of the gap between software and hardware mainly refers to the lack of relevance between the knowledge fields of relevant technical personnel and related devices. For example, mastering hardware knowledge requires an electronics major, and mastering software knowledge requires an information major. These two majors often lack commonality for technical personnel. For example, it is quite difficult to talk about field bus technology to software personnel or the differences between API and ABI to hardware personnel. If this problem is brought into the debugging level of embedded systems, hardware personnel can use a circuit board in combination with an oscilloscope or logic tester to find the debugging interface and connect a specific debugger. Then the software debugging is handed over to software personnel, who generally do not have or master the operation knowledge of an oscilloscope or logic tester. In the field of embedded security testing and evaluation, finding hidden test ports is one of the most important steps. A circuit board has countless connection endpoints. It is quite difficult for hardware personnel to find a few debugging lines among these numerous endpoints, let alone software personnel without a hardware knowledge background. For the found debugging lines, to connect them to a computer and successfully use general-purpose open-source debugging software to debug the board, from the recognition of connection definitions, port connections, device configurations, to communication methods, etc., this process poses a high challenge to software and hardware operators, and the technical gap also limits the degree of automation and quantification.

[0028] Regarding the cross - domain scope, the security of embedded systems is very broad in the fields from hardware to software, including pure hardware security, software - hardware security, pure software security, and communication security. Pure hardware security includes side - channel and fault injection; software - hardware security includes debug port access, field - bus intrusion, hardware - level access control, etc.; pure software security undertakes all considerations of standard computer security such as stack and heap overflows, integer overflows, string format defects, etc.; communication security covers signal security from the physical layer to API and encryption / decryption security at the application layer. To cover all these fields requires long - term investment in funds, technology, and knowledge for relevant technicians and equipment.

[0029] An automated intelligent embedded firmware analysis and vulnerability mining method of the present application, as Figure 1 shown, includes the following steps:

[0030] 1) Obtain the firmware

[0031] Before starting the firmware analysis, it is first necessary to obtain the firmware of the target embedded device. If the target firmware is distributed to users through the public domain, then researchers can directly use these firmware packages for subsequent automatic analysis and processing.

[0032] However, if the firmware cannot be obtained in the public domain, then it is necessary to directly extract the firmware from the hardware. The main hardware - level debugging protocols on the market are UART, JTAG, etc. Although the details of these protocols are public, embedded device manufacturers often do not label the functions of each pin on the circuit board. To know the function of a certain pin, the traditional method can only rely on the experience of researchers to guess and test the specific function of a certain pin. Generally speaking, a debugging protocol requires the cooperation of multiple pins to be realized. If the target device has a large number of pins, it can be imagined that it is quite difficult to discover the key pins from this exponentially expanding permutation and combination.

[0033] This technical solution includes a self - developed toolboard. It has multiple pins and the ability to monitor and write to these pins. As mentioned above, various debugging protocols are public. That is to say, the characteristics of these debugging protocols, their handshakes. Even for entry - level researchers, as long as all the pins on the target device are connected to this development board and the target device is started, this development board can automatically sense various debugging protocols provided by the target device. Through these protocols, functions such as hardware information reading, non - volatile storage transfer, memory operation, instruction - level debugging, etc. can be realized. The firmware exists in non - volatile storage, so as long as the content of the entire non - volatile storage is extracted, then the firmware must be included in it for subsequent analysis and use.

[0034] 2) Unpack the firmware

[0035] If the firmware package is directly extracted from the target device, the dumped binary file can be directly used for subsequent analysis. If the firmware package is public, generally speaking, these firmware files distributed to end-users have been file-packaged, and common packaging formats include zip, tar, etc. When some firmware is released, for various considerations such as the size of the firmware package and file integrity verification, it may be packaged multiple times. For these firmware files, the first step is to decompress these files layer by layer until the completely decompressed binary file is obtained.

[0036] 3) Extract executable files

[0037] After obtaining the binary file, more abstract analysis needs to be carried out. Although embedded manufacturers do not disclose the details of the firmware to the public, the source code of the systems used by mainstream firmware often is not developed independently from scratch. Since they are based on existing projects with additions and deletions, it means that these firmware can be roughly adapted according to the existing projects.

[0038] For example, in the router field, many manufacturers will choose OpenWRT and conduct secondary development. The Linux kernel used by OpenWRT supports multiple file systems. Depending on the hardware configuration of the router, whether it is the most space-saving SquashFS or the most feature-rich EXT file system, the formats of these file systems are public. Even if the manufacturer makes changes, the core structure will not change. Among these file systems that work with the kernel, there are various library files and executable files, and these files may be the locations of vulnerabilities and the focus of subsequent analysis. Therefore, this technical solution will perform feature scanning on these file system formats and completely extract all files from the file system.

[0039] 4) Parse file structure

[0040] After obtaining the executable file, the first thing to do is to analyze the file format. Continuing with the Linux kernel as an example, in the Linux system, executable files are generally in the ELF format. The ELF header describes the environment required by the ELF, including the hardware type (such as MIPS, ARM), file type (such as executable file, shared library file), etc. Following the ELF header are the program header and session header. The former determines how the file content should be loaded into memory, while the latter describes the nature and role of each section in memory. For actual file analysis, the main ELF metadata required includes instruction set information (platform / instruction set version / endianness), symbol table / import table (referenced external libraries, especially system libraries), etc. This is the first step in file analysis, aiming to know how the file is loaded and run in the actual environment.

[0041] 5) Disassembly and Control Flow Analysis

[0042] After loading the data in the file into memory according to the information contained in the ELF, the code segment can be disassembled. The role of disassembly is to convert the data stream in the code segment into an instruction control flow. In other words, what is obtained through disassembly is exactly the operations that the CPU will perform when interpreting and executing this data stream. Although the instruction set itself is very likely protected by copyright and other laws and regulations, the encoding method of instructions is public. Except for extremely confidential hardware, most embedded devices use the original factory instructions without modification. Therefore, as long as the mainstream instruction set is supported, most executable files can be successfully disassembled. Only by obtaining the control flow at the instruction level and knowing how the program performs operations is it possible to perform subsequent higher-level analyses.

[0043] 6) Function Partitioning and Cross-References

[0044] The control flow obtained from the conversion of the data stream is flat in memory. However, in actual program development, there will definitely be concepts such as functions. Only by sorting out the function list can the passing of parameters and the flow of data in the program be analyzed. Therefore, the analysis to be done here is to reconstruct the functions that may exist in the source code from the flat control flow through methods such as simulated execution. The program entry and export table items obtained from the analysis of the ELF metadata earlier can all serve as the entry points for simulated execution. Through recursive simulated execution, when a function call is encountered, recursive entry into the simulated execution of another function is made, and finally, the function entry points of a considerable part of this executable file can be obtained. However, although this recursive analysis is the most reliable, it may have omissions. When a function is not called directly (such as through a function dispatch table call), the simulated execution may not be able to reach it. Therefore, on the basis of recursive simulated execution, a scan of the function entry points in the data segment is also added to complete the function list.

[0045] The triggering of vulnerabilities often involves the input of malicious data. Therefore, the tracking of data flow is quite important. When obtaining the data at the function level, the cross-referenced data also emerges. By analyzing the instructions, while simulating the execution of each function, it can also be known which memory addresses a certain line of instruction operates on. These operations may be reads or writes. Sorting out these operations on memory addresses, together with the call relationships between functions, constitutes a basic cross-reference database. With this cross-reference data, given a memory address, it is possible to immediately find out which functions / instructions have read and written to this address; given a function entry point, it is possible to find out which other functions call here.

[0046] 7) Decompilation and Tracking of Variables within Functions

[0047] Although theoretically the analysis at the disassembly level is sufficient, there are still deficiencies in practical applications. For example, it is difficult to represent constructs such as the switch block in C grammar at the assembly level, and the assembly representation will be quite laborious when showing the control flow. Even if malicious input is fully observed during data tracking based on assembly analysis, the subsequent vulnerability verification will increase the labor cost because the assembly is too obscure. Thus, decompilation is actually an essential and important step.

[0048] Decompilation mainly consists of three processes. The first process is the translation from the original instructions to the intermediate representation (IR). Regardless of the CPU instruction architecture, the most fundamental and indispensable operations at the lowest level belong to the category of the arithmetic logic unit (ALU). Generally speaking, the ALU implements operations such as addition, subtraction, multiplication, division, and various bitwise operations. What the intermediate expression represents are precisely these most basic operations. After this translation, regardless of the original architecture of the executable file, it is transformed into a unified intermediate language, making the subsequent processing logic no longer need to consider the differences of each original architecture. The second process is to construct a structure similar to the abstract syntax tree (AST). If the function partitioning described above organizes the flat control flow analysis into individual functions, then the construction of a structure similar to the AST now combines the flat control blocks in the function and nests them with each other to form the nodes on the syntax tree. For example, the if block in the source code corresponds to a comparison jump in the instructions. At the instruction level, the code in the if block and the code block where the if header is located do not show a hierarchical relationship. However, in the syntax tree, the code in the if statement is a sub-block of the if block, and there is an obvious hierarchical relationship. After this step of processing, various syntax blocks can be generated, and the prototype of the decompilation result emerges. The third process is optimization. Optimization involves variable arrangement and expression contraction, etc. For example, the chained calls in the source code are represented by multiple intermediate expressions before optimization. The function pointer returned by the previous expression is called by the next expression. These scattered multiple expressions can be contracted by replacing the previous expression into the current expression to restore the chained call representation in the source code. After this step, the intermediate variables that are meaningless for analysis will be hidden, leaving only the meaningful expressions.

[0049] Obviously, to know where an expression should be shrunk, it is necessary to have data (variable) flow information within the function. Additionally, the cross-references mentioned earlier show the mutual references at the granularity of functions and memory addresses, but it still lacks the data flow information within the function. Considering these two factors, it can be known that the data flow within the function is also quite important for the entire analysis. By performing static single assignment (SSA) on each variable in a function, the lifecycle of a variable within the function and its read / write situations can be determined. With this information, the data flow within the function can be traced. Combining with the cross-reference information obtained earlier, the flow of a certain data can be fully displayed across functions and variables.

[0050] 8) Vulnerability Mining

[0051] With the above various basic components, various vulnerability mining methods have sufficient technical support.

[0052] First is the vulnerability mining plugin. This plugin applies the complete data tracing described earlier. Through this plugin system, users can define the inputs and outputs that may have vulnerabilities and run the plugin to automatically search for vulnerabilities. Here, taking the arbitrary code execution vulnerability as an example, for a common gateway interface, the input is defined as the environment variable QUERY_STRING, and the output is defined as the system function system(). If it is found in the data tracing that there is a connection between the input and output, that is, the uncontrollable user data passed in from QUERY_STRING may eventually be passed into system(), then the common gateway interface may have a vulnerability.

[0053] As an auxiliary for vulnerability discovery, this technical solution also comes with two sets of simulation debugging systems. The user-mode simulation debugging system can conveniently perform various tests or debugging on a single executable file, while the hardware-level simulation debugging system can simulate the startup process of real hardware according to the kernel, file system, etc. provided by the user and allow the user to perform kernel debugging.

[0054] In addition, this technical solution also provides a fuzzing framework for black-box analysis. By defining the format of user inputs, the fuzzing framework can automatically generate test input data, use the simulation systems mentioned above to test the specified executable program, and capture various abnormal behaviors to detect whether there are potential vulnerabilities such as denial-of-service attacks.

[0055] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in any regard, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention.

[0056] In addition, it should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An automated intelligent embedded firmware analysis and vulnerability mining method, characterized in that, It includes the following steps: 1) Obtain the firmware. Before starting the firmware analysis, first obtain the firmware of the target embedded device. If the firmware is distributed through public channels, directly use these firmwares for subsequent automatic analysis and processing; If it cannot be obtained from public channels, extract the firmware from the hardware; 2) Firmware unpacking; For the firmware directly extracted from the target embedded device, convert it into a binary file for subsequent analysis. For the firmware that has been packaged multiple times during release, unpack these firmwares layer by layer until the unpacked binary file is obtained; 3) Extract the executable file; After obtaining the binary file, perform a feature scan for the file system format and completely extract all files from the file system; 4) Parse the file structure; After obtaining the executable file, analyze the file format. The format of the executable file is the ELF format. The data in the executable file includes the platform, instruction set version, endian instruction set information, symbol table, and import table; 5) Disassembly and control flow analysis; After loading the data in the executable file into memory, disassemble the code segment, convert the data flow in the code segment into an instruction control flow, and obtain the instruction-level control flow; 6) Function partitioning and cross-reference; By means of simulated execution, reconstruct the control flow into the functions existing in the source code. Use the program entry and export table items obtained from the analysis in step 5) as the entry of the simulated execution. During the simulated execution, when a function call is encountered, enter the simulated execution of the corresponding function. Finally, obtain the function entry list of the executable file, scan the function entry list, complete the function entry list, and construct a basic cross-reference database while analyzing each function by analyzing the instructions; 7) Decompilation and variable tracking within functions; Decompilation includes three processes: First, the conversion from the original instructions to intermediate expressions: Translate the operations of the CPU instruction architecture into a unified intermediate language so that the subsequent processing logic does not need to consider the differences in the original architectures; Second, construct the abstract syntax tree structure: Combine and nest each control block in the function into each node on the syntax tree to generate the preliminary structure of the decompilation result; Third, the optimization process: Hide the intermediate variables that are useless for analysis and only retain the necessary expressions; 8) Vulnerability mining; Use the vulnerability mining plugin and apply data tracking. Through this vulnerability mining plugin, the user defines the possible input and output with vulnerabilities, and runs the vulnerability mining plugin to automatically find vulnerabilities.

2. The automated intelligent embedded firmware analysis and vulnerability mining method according to claim 1, characterized in that: In step 6), the cross-reference database is determined by analyzing the instructions and determining the memory addresses operated by each line of instructions while simulating the execution of each function. These operations include reading and writing. Organize these operations on the memory addresses and combine the call relationships between functions to form a basic cross-reference database. Through this cross-reference database, given a memory address, it is possible to find out which functions or instructions have read and written to this address; given a function entry, it is possible to find out which other functions have called this function.

3. The automated intelligent embedded firmware analysis and vulnerability mining method according to claim 1, characterized in that: In step 8), as an aid to vulnerability mining, two sets of simulation debugging systems are provided: the first is a user-mode simulation debugging system for testing or debugging a single executable file; the second is a hardware-level simulation debugging system that, based on the kernel and file system provided by the user, simulates the boot process of real hardware and allows the user to perform kernel debugging.

Images