A firmware vulnerability judgment and positioning method based on syntax similarity analysis
By employing a syntax similarity analysis-based approach, utilizing deep neural network models and improved tools, we can quickly and automatically detect and locate n-day vulnerabilities in firmware. This solves the problems of long detection times and false positives in existing technologies, achieving efficient vulnerability location and determination.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-05-11
- Publication Date
- 2026-07-03
Smart Images
Figure CN116644430B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of firmware vulnerability search, specifically, a method for determining and locating firmware vulnerabilities based on syntax similarity analysis. Background Technology
[0002] The rapid development of the Internet of Things (IoT) offers immense convenience but also faces increasingly serious security threats. As IoT devices form a crucial foundation of the IoT, their security is paramount. With IoT devices becoming increasingly powerful and their components more complex, no single company can develop them independently. To improve development efficiency and reduce costs, more and more firmware is incorporating third-party software. However, this practice also allows vulnerabilities in third-party software to propagate into the firmware itself, compromising its security.
[0003] N-day vulnerabilities introduced by third-party software are often more threatening than undisclosed zero-day vulnerabilities because they are publicly known. This means that any malicious actor can easily understand the principles and exploitation methods of these vulnerabilities through public channels. However, due to the lack of attention from firmware vendors, these n-day vulnerabilities are often difficult to properly patch in firmware, resulting in their widespread presence.
[0004] Existing firmware vulnerability detection and search technologies mainly suffer from the following problems:
[0005] (1) Existing vulnerability mining techniques mainly target 0-day vulnerabilities for analysis, and very few studies have analyzed the existence of n-day vulnerabilities introduced by third-party software in firmware.
[0006] (2) Existing vulnerability discovery techniques, such as dynamic analysis based on fuzz and static search analysis based on pattern, often require a lot of manual operation and consume a lot of time, making them difficult to apply to large-scale analysis of similar firmware file systems.
[0007] (3) Due to issues such as compilation and configuration, firmware may not necessarily introduce corresponding n-day vulnerabilities when third-party software is introduced. Therefore, directly using third-party software to analyze firmware to determine its possible n-day vulnerabilities can easily lead to a large number of false positives.
[0008] Therefore, it is necessary to propose a highly usable, fast and efficient method for identifying and locating vulnerabilities in firmware to address the problem of firmware security being threatened by n-day vulnerabilities introduced by third-party software packages. Summary of the Invention
[0009] The purpose of this invention is to propose a firmware vulnerability identification and location method based on syntax similarity analysis to address the threat posed by n-day vulnerabilities in firmware.
[0010] This invention is achieved through the following technical solution:
[0011] This invention discloses a firmware vulnerability determination and location method based on syntax similarity analysis. Based on collected third-party software vulnerability patch information and firmware database, the method extracts the syntax features of firmware and third-party software vulnerabilities, and compares the extracted feature information through similarity analysis technology to determine whether the vulnerability exists in the firmware and locate the location of the vulnerable function, thus completing the determination of the existence of the vulnerability and the location of the vulnerability.
[0012] As a further improvement, the present invention includes the following steps:
[0013] 1) Collection of vulnerability information and firmware; Collect publicly available vulnerability information from the network and extract vulnerability patch information. After preprocessing, store the vulnerability information in the database. Crawl firmware from firmware vendor websites and firmware collection websites and store the firmware information in the firmware database. Finally, obtain vulnerability patch information and firmware image files.
[0014] 2) Model training based on deep neural networks; Based on the patch source code obtained in step 1), a graph embedding model based on deep learning is trained to convert the ACFG (attribute control flow graph) of the vulnerability function into a one-dimensional feature vector, so that the model can fully learn the feature information of the vulnerability function and the patch code, which facilitates subsequent similarity calculation.
[0015] 3) Vulnerability Patch Feature Extraction; Based on the vulnerability patch information obtained in step 1), obtain the patch source code, and use a source code parsing tool to parse the source code, extract the syntax features and ACFG of the patch code, and use the deep neural network graph embedding model trained in step 2) to convert the ACFG into a one-dimensional feature vector for subsequent analysis.
[0016] 4) Firmware unpacking and analysis; The improved binwalk tool is used to extract the file system from the firmware image file collected in step 1), and the files in the file system are analyzed to extract the binary executable files for subsequent analysis;
[0017] 5) Syntax feature comparison; For the binary executable file obtained in step 4), the syntax features of the binary executable file are extracted using a self-developed binary executable file parsing tool, and the similarity analysis of the syntax features of the vulnerability patch extracted in step 3) is performed. If the similarity does not exceed the set threshold, it proves that the vulnerability does not exist and the analysis stops. If the similarity exceeds the threshold, a potentially vulnerable binary executable file is obtained.
[0018] 6) ACFG comparison: Use a disassembler to extract the function ACFGs of the potentially vulnerable binary executable file obtained in step 5), and further use the model obtained in step 2) in combination with the vulnerable function ACFGs extracted in step 3) to perform similarity analysis between ACFGs, and finally determine the existence of the vulnerability and the location of the vulnerable function.
[0019] As a further improvement, the vulnerability information in step 1) of this invention includes: the release time of the vulnerability, the threat level of the vulnerability, the patch information of the vulnerability, and the source code information of the software affected by the vulnerability.
[0020] As a further improvement, the preprocessing operation of patch information in step 1) of the present invention includes deleting useless code in the patch, locating the patch source code, and locating the patch function, thereby determining the location of the vulnerable function in the source code.
[0021] As a further improvement, the training method for training the deep learning-based graph embedding model in step 2) of the present invention is to use a cross-compilation toolchain to compile the code before and after patching into binary executable files of different architectures such as x86, ARM, and MIPS, extract the ACFG of the binary files and source code, and train the model and save the model parameters based on these known ACFG results.
[0022] As a further improvement, the improved binwalk tool in step 4) of this invention is designed to write corresponding firmware unpacking code and binwalk plugins for firmware image files with different packaging formats and file systems of different formats contained in the image files, thereby improving the firmware unpacking rate and the success rate of extracting the file system.
[0023] As a further improvement, the syntactic features in steps 3) and 5) of this invention include all visible strings in the source file where the vulnerability is located and the binary executable file, as well as the function names of referenced external functions.
[0024] As a further improvement, the similarity analysis of syntactic features in step 5) of this invention is based on an improved Jaccard set similarity algorithm, which compares the similarity between the set of syntactic features of the binary executable file in the firmware file system and the set of syntactic features in the vulnerability source code file, and finally determines whether the source code in the binary executable file has been compiled.
[0025] As a further improvement, the similarity analysis between ACFGs in step 6) of this invention is based on a graph embedding model of a deep neural network. The ACFG of the function in the binary executable file is converted into a one-dimensional feature vector, and the cosine distance between it and the vulnerability feature vector is calculated as a similarity score. By comparing it with a set threshold, the similarity between the function in the binary executable file and the vulnerability function is finally determined, and it is finally determined whether it contains a vulnerability and the location of the vulnerability function is located.
[0026] The beneficial effects of this invention are as follows:
[0027] This invention addresses the security threats posed by n-day vulnerabilities introduced by third-party software in firmware by proposing a method for firmware vulnerability identification and localization based on syntactic similarity analysis. This method extracts syntactic features from both a collected vulnerability database and a firmware database, and then performs coarse-grained and fine-grained similarity analysis to determine the existence of n-day vulnerabilities introduced by third-party software in the firmware and locate the vulnerable functions, thus completing the search and analysis of n-day vulnerabilities introduced by third-party software in firmware.
[0028] Compared to existing methods, this method boasts a high degree of automation and portability across different firmware file systems, significantly reducing the skill requirements for analysts. Secondly, before using precise algorithms to locate vulnerable functions, a coarse-grained screening is performed by comparing the entire syntactic features of the binary executable file, which greatly shortens the time required for vulnerability search and analysis, improving the efficiency of n-day vulnerability searches. Furthermore, by utilizing graph embedding algorithms to analyze the ACFG of vulnerable functions, n-day vulnerabilities introduced by third-party software in the firmware can be accurately identified, and the location of vulnerable functions can be determined, significantly improving the efficiency of security personnel in analyzing n-day vulnerabilities in firmware. Attached Figure Description
[0029] Figure 1 This is a flowchart of a firmware vulnerability identification and location method based on syntax similarity analysis; Detailed Implementation
[0030] This invention discloses a firmware vulnerability identification and localization method based on syntax similarity analysis. To make the objectives and effects of this invention clearer, the implementation process of the vulnerability identification and localization scheme in firmware is described in detail below as an example, with reference to the accompanying drawings and preferred embodiments. Specifically, this embodiment uses LLVM to analyze the vulnerable source code, utilizes IDA Pro to analyze the binary files in the firmware, and employs a graph embedding model based on an improved graph convolution scheme as the feature extraction model for ACFG. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.
[0031] The specific technical solution of the present invention will be further described below with reference to the accompanying drawings:
[0032] Figure 1 This is a flowchart of a firmware vulnerability identification and location method based on syntax similarity analysis, which specifically includes the following steps:
[0033] 1) Collection of vulnerability information and firmware; Collect publicly available vulnerability information from the network and extract vulnerability patch information. After preprocessing, store the vulnerability information in the database. Crawl firmware from firmware vendor websites and firmware collection websites and store the firmware information in the firmware database. Finally, obtain vulnerability patch information and firmware image files.
[0034] Specifically, using the Scrapy web scraping framework in Python, firmware image files and vulnerability information, including the release time of vulnerabilities, the threat level of vulnerabilities, vulnerability patch information, and the source code information of the software affected by the vulnerabilities, are crawled from the network and finally stored in the vulnerability information database.
[0035] 2) Deep Neural Network-Based Model Training: Based on the patch source code obtained in step 1), a deep learning-based graph embedding model is trained to convert the ACFG (Attribute Control Flow Graph) of the vulnerability function into a one-dimensional feature vector. This allows the model to fully learn the feature information of the vulnerability function and the patch code, facilitating subsequent similarity calculations. Specifically, this includes:
[0036] (a) Use cross-compilation toolchains to compile the source code of third-party software before and after patching into binary executable files for different architectures such as x86, ARM, and MIPS.
[0037] (b) The binary executable file obtained in (a) was disassembled using IDA Pro and the ACFG of the functions in the binary executable file was extracted using the IDAPython script. Here, the visible strings, constants, number of instructions, number of calling instructions, number of successor nodes and node betweenness centrality in the basic block of the function CFG (control flow graph) were used as the attribute values of the nodes to form the ACFG. The ACFG of the vulnerable functions in the source code of the third-party software before and after patching was extracted using LLVM.
[0038] (c) Pair the ACFG in the source code with the ACFG in the binary code. If they are both before or after the patch, they are labeled as similar; otherwise, they are labeled as dissimilar. At the same time, some unrelated functions' binary code ACFG are added during the pairing process, and all of them are labeled as dissimilar.
[0039] (d) Using the ACFG pairs and corresponding labels obtained above, a graph embedding model based on graph convolutional networks is trained based on the Siamese network model, so that the model can fully learn the feature information of the vulnerability function and the patch code.
[0040] 3) Vulnerability Patch Feature Extraction; Based on the vulnerability patch information obtained in step 1), obtain the patch source code, and use a source code parsing tool to parse the source code, extract the syntax features and ACFG of the patch code, and use the deep neural network graph embedding model trained in step 2) to convert the ACFG into a one-dimensional feature vector for subsequent analysis.
[0041] Specifically, LLVM is used to parse the source code file containing the vulnerability, extract visible strings and external imported functions (such as C standard library functions) in the source file as syntactic feature information, extract the ACFG of the vulnerable function before and after patching as syntactic structure features, and use the neural network model trained in step 2) to convert the ACFG from high-dimensional feature information into a one-dimensional feature vector.
[0042] 4) Firmware unpacking and analysis; The improved binwalk tool is used to extract the file system from the firmware image file collected in step 1), and the files in the file system are analyzed to extract the binary executable files for subsequent analysis;
[0043] Specifically, the improved binwalk tool is designed to develop corresponding firmware unpacking code and binwalk plugins for firmware image files with different packaging formats and file systems contained within the image files, thereby improving the firmware unpacking rate and the success rate of extracting file systems.
[0044] 5) Syntax Feature Comparison: For the binary executable file obtained in step 4), the syntax features of the binary executable file are extracted using a self-developed binary executable file parsing tool. This is then combined with the vulnerability patch syntax features extracted in step 3) to perform a syntax feature similarity analysis. If the similarity does not exceed a set threshold, it proves that the vulnerability does not exist, and the analysis stops. If it exceeds the threshold, a potentially vulnerable binary executable file is obtained, specifically including:
[0045] (a) Using a self-developed binary executable file analysis tool, extract all visible strings and external imported functions in the binary file as the syntax feature information of the binary file.
[0046] (b) Based on the improved Jaccard set similarity algorithm, calculate the similarity value between the syntax features of the binary executable file and the syntax features of the vulnerability, and compare it with the set similarity threshold. If the calculated similarity value exceeds the threshold, the source code containing the vulnerability is compiled into the binary executable file; otherwise, the binary executable file does not contain the corresponding vulnerability because the corresponding source code has not been compiled, and subsequent analysis is stopped.
[0047] 6) ACFG Comparison: Using a disassembler, extract the function ACFGs of the potentially vulnerable binary executable files obtained in step 5). Further utilize the model obtained in step 2), combined with the vulnerable function ACFGs extracted in step 3), to perform similarity analysis between ACFGs, ultimately determining the existence of the vulnerability and identifying the location of the vulnerable function. Specifically, this includes:
[0048] (a) Disassemble the potentially vulnerable binary executable obtained in step 5) using IDA Pro and extract the ACFG of all functions.
[0049] (b) Using the deep neural network-based model trained in step 2), convert the ACFG of all functions in the binary executable file into one-dimensional feature vectors;
[0050] (c) Calculate the cosine distance between the one-dimensional feature vector obtained in (b) and the one-dimensional feature vector obtained in step 3) from the feature transformation of the vulnerability function structure before and after patching, and compare it with a set threshold. If it exceeds a certain threshold, the function can be considered similar to the vulnerability function, and it can be determined whether it has been patched.
[0051] It will be understood by those skilled in the art that the above description is merely a single example of the invention and is not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.
Claims
1. A firmware vulnerability identification and location method based on syntax similarity analysis, characterized in that, Based on the collected third-party software vulnerability patch information and firmware database, the syntactic features of firmware and third-party software vulnerabilities are extracted. By using similarity analysis technology, the extracted feature information is compared to determine whether the vulnerability exists in the firmware and to locate the vulnerable function. This completes the determination of the existence of the vulnerability and the location of the vulnerability. Includes the following steps: 1) Collection of vulnerability information and firmware; Collect publicly available vulnerability information from the network and extract vulnerability patch information. After preprocessing, store the vulnerability information in the database. Crawl firmware from firmware vendor websites and firmware collection websites and store the firmware information in the firmware database. Finally, obtain vulnerability patch information and firmware image files. 2) Model training based on deep neural networks; Based on the patch source code obtained in step 1), a graph embedding model based on deep learning is trained to convert the ACFG attribute control flow graph of the vulnerability function into a one-dimensional feature vector, so that the model can fully learn the feature information of the vulnerability function and the patch code, which facilitates subsequent similarity calculation. 3) Vulnerability Patch Feature Extraction; Based on the vulnerability patch information obtained in step 1), obtain the patch source code, and use a source code parsing tool to parse the source code, extract the syntax features and ACFG of the patch code, and use the deep neural network graph embedding model trained in step 2) to convert the ACFG into a one-dimensional feature vector for subsequent analysis. 4) Firmware unpacking and analysis; The improved binwalk tool is used to extract the file system from the firmware image file collected in step 1), and the files in the file system are analyzed to extract the binary executable files for subsequent analysis; 5) Syntax feature comparison; For the binary executable file obtained in step 4), the syntax features of the binary executable file are extracted using a self-developed binary executable file parsing tool, and the similarity analysis of the syntax features of the vulnerability patch extracted in step 3) is performed. If the similarity does not exceed the set threshold, it proves that the vulnerability does not exist and the analysis stops. If the similarity exceeds the threshold, a potentially vulnerable binary executable file is obtained. 6) ACFG comparison: Use a disassembler to extract the function ACFGs of the potentially vulnerable binary executable file obtained in step 5), and use the model obtained in step 2) in combination with the vulnerable function ACFGs extracted in step 3) to perform similarity analysis between ACFGs, and finally determine the existence of the vulnerability and the location of the vulnerable function. The vulnerability information in step 1) includes: the release time of the vulnerability, the threat level of the vulnerability, the patch information of the vulnerability, and the source code information of the software affected by the vulnerability. The preprocessing operation of the patch information in step 1) includes deleting useless code in the patch, locating the patch source code, and locating the patch function, thereby determining the location of the vulnerable function in the source code; The training method for the deep learning-based graph embedding model in step 2) is to use a cross-compilation toolchain to compile the code before and after patching into binary executable files for different architectures such as x86, ARM, and MIPS. The ACFG of the binary files and source code is extracted, and the model is trained and the model parameters are saved based on these known ACFG results.
2. The firmware vulnerability determination and location method based on syntax similarity analysis according to claim 1, characterized in that, The improved binwalk tool in step 4) is designed to write corresponding firmware unpacking code and binwalk plugins for firmware image files with different packaging formats and file systems of different formats contained in the image files, thereby improving the firmware unpacking rate and the success rate of extracting the file system.
3. The firmware vulnerability determination and location method based on syntax similarity analysis according to claim 1, characterized in that, The syntactic features in steps 3) and 5) include all visible strings in the source file and binary executable file where the vulnerability is located, as well as the function names of referenced external functions.
4. The firmware vulnerability determination and location method based on syntax similarity analysis according to claim 1, 2, or 3, characterized in that, The similarity analysis of syntactic features in step 5) is based on the improved Jaccard set similarity algorithm. It compares the similarity between the set of syntactic features of the binary executable file in the firmware file system and the set of syntactic features in the vulnerability source code file, and finally determines whether the source code in the binary executable file has been compiled.
5. The firmware vulnerability determination and location method based on syntax similarity analysis according to claim 4, characterized in that, The similarity analysis between ACFGs in step 6) is based on the graph embedding model of a deep neural network. The ACFG of the function in the binary executable file is converted into a one-dimensional feature vector, and the cosine distance between it and the vulnerability feature vector is calculated as the similarity score. By comparing it with a set threshold, the similarity between the function in the binary executable file and the vulnerability function is finally determined, and it is finally determined whether it contains a vulnerability and the location of the vulnerability function is located.