Non-standardized firmware unpacking method and unpacking tool chain based on static arrangement
By using a static orchestration method to automatically parse the firmware structure, dynamically select tools, and handle fault tolerance, the compatibility and reliability issues of non-standard firmware unpacking are solved, achieving an efficient and reliable firmware unpacking process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 安徽明生恒卓科技有限公司
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-17
AI Technical Summary
Existing firmware unpacking tools suffer from poor compatibility, weak ability to handle nested structures, lack of fault tolerance mechanisms, and insufficient adaptation to special formats when dealing with non-standard, complex, or partially damaged firmware. This results in a high unpacking failure rate and reliance on manual experience.
A static orchestration-based approach is adopted to obtain firmware structural features through automated preliminary analysis, identify logical partition structures, generate adjustable partition description information, dynamically select and call underlying tools, and switch to backup tools when execution failures occur, and integrate to generate standardized output.
It significantly improves the success rate and automation level of unpacking non-standard firmware, reduces reliance on human experience, ensures the reliability and reproducibility of the unpacking process, and generates standardized unpacking reports.
Smart Images
Figure CN121879818A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embedded system firmware analysis technology, specifically a non-standard firmware unpacking method and unpacking toolchain based on static orchestration. Background Technology
[0002] Firmware, as a core component of embedded devices, is widely found in various products such as IoT terminals, industrial control equipment, and smart hardware. It not only carries the operating system kernel and drivers necessary for device operation but also contains application-layer functional modules and hardware configuration parameters. Its integrity and security directly determine the device's operational stability and data security. In the field of embedded system security, firmware unpacking and analysis is a crucial prerequisite for vulnerability discovery, malware detection, and compliance verification. Only by unpacking and obtaining the internal file structure of the firmware can potential security risks such as permission vulnerabilities and hard-coded keys be further investigated, providing a technical basis for device security protection.
[0003] Currently, mainstream firmware unpacking tools are typically represented by binwalk. Its core principle is to automatically unpack standard format firmware by matching file system characteristics using a pre-defined "Magic Number" database. However, when dealing with non-standard firmware in scenarios such as industrial equipment and niche brand hardware, these tools exhibit significant limitations: First, they have poor compatibility with vendor-defined formats. For example, binwalk cannot recognize proprietary file systems or non-standard partition tables due to a lack of corresponding feature libraries. Second, they have weak nested structure processing capabilities. When firmware has multiple nested layers such as "main partition - sub-compressed image - deep file system," binwalk can only parse the surface structure and cannot recursively identify deeper content. Third, they lack fault tolerance mechanisms. If the firmware experiences packet loss or storage corruption resulting in abnormal data blocks, binwalk often terminates the unpacking process directly, failing to retain valid data. Fourth, they lack adaptation to special formats. For non-standard compression algorithms or lightweight encryption protection used by vendors, binwalk lacks corresponding decompression and decryption capabilities, leading to unpacking interruptions.
[0004] In existing technical solutions, engineers typically need to manually call tools such as xxd, dd, and sasquatch for step-by-step processing. The entire process requires manual parameter calculation and result judgment, which is not only inefficient but also highly dependent on the engineer's experience. Furthermore, the process lacks standardized documentation, making it difficult to reproduce and audit. Therefore, a firmware unpacking solution is needed that can automatically orchestrate underlying tools, adapt to non-standardized scenarios, and possess fault tolerance capabilities to overcome traditional technical bottlenecks and improve unpacking efficiency and reliability. Summary of the Invention
[0005] To address the inherent contradiction between the rigid toolchain and the diversity and uncertainty of firmware when processing non-standard, complex, or partially damaged firmware, which leads to a high failure rate in the unpacking process, this invention provides a non-standard firmware unpacking method and unpacking toolchain based on static orchestration.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A non-standard firmware unpacking method based on static orchestration includes: A preliminary analysis is performed on the firmware to be analyzed to obtain its structural feature information; Based on structural feature information, the logical partition structure of the firmware is identified, and adjustable partition description information is generated. Based on the partition description information and the predefined tool mapping strategy, select and call the corresponding underlying tools to extract data from each partition; If the current reading tool fails during the data reading process, the system will automatically switch to the backup reading tool according to the predetermined fault tolerance strategy. Integrate the reading process and result data to generate standardized unpacking output.
[0007] As a further aspect of the present invention, the preliminary analysis includes at least one of file type identification, feature string extraction, and binary visualization.
[0008] As a further aspect of the present invention, the logical partition structure includes searching for file system feature signatures in the firmware binary data and determining the boundaries of each partition based on the location of the found signatures.
[0009] As a further aspect of the present invention: when determining the partition boundary, the starting position of the next file system feature signature is used as the end boundary of the current partition, or the starting position of consecutive empty data blocks is used as the end boundary.
[0010] As a further aspect of the present invention, the tool mapping strategy defines the correspondence between file system types, data characteristics and underlying extraction tools, including mapping specific file system types to dedicated unpacking tools and mapping unknown or unrecognizable types to raw data copying tools.
[0011] As a further aspect of the present invention, the fault-tolerance strategy includes automatically switching to a second extraction tool with data recovery capabilities to continue execution when the first reading tool fails due to a reading error.
[0012] As a further aspect of the present invention, generating standardized unpacking output includes generating a structured unpacking report, which contains firmware metadata, extraction status logs for each partition, and the final file directory tree information.
[0013] As a further aspect of the present invention, the unpacking method also includes a fully automatic mode and an interactive mode; in the interactive mode, users are allowed to verify and modify the adjustable partition description information.
[0014] This invention also provides a statically orchestrated non-standard firmware unpacking toolchain for implementing a statically orchestrated non-standard firmware unpacking method, comprising: The firmware preliminary analysis module is used to perform automated preliminary analysis of the firmware to be analyzed and obtain its structural feature information. The structure recognition and partitioning module is used to identify the logical partition structure of the firmware based on structural feature information and generate adjustable partition description information. The tool selection and execution module is used to automatically select and call the corresponding underlying tools to extract data from each partition based on the partition description information and the predefined tool mapping strategy. The exception handling and recovery module is used to monitor the execution status of the tool during the data extraction process, and to control the switch to the backup extraction tool according to the predetermined fault tolerance strategy when the current extraction tool fails. The automated integration module is used to integrate the extraction process and result data to generate standardized unpacking output.
[0015] As a further aspect of the present invention: the preliminary analysis module performs analysis by calling at least one of a file type identification tool, a string extraction tool, and a binary viewing tool.
[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention first proactively senses the internal structure of firmware through automated preliminary analysis, replacing passive matching that relies on static feature libraries. It then generates adjustable partition description information, providing a flexible and reliable basis for subsequent processing. Next, it dynamically selects and calls appropriate tools through predefined strategies, and automatically switches to backup solutions based on fault tolerance strategies when execution fails, ensuring the unpacking process is not interrupted by localized problems. Finally, it integrates and generates standardized output, achieving full auditability and reproducibility. This transforms firmware unpacking from a fragile manual process dependent on specific tools and human intervention into an intelligent pipeline based on rules and strategies, capable of self-adaptation and recovery, significantly improving the unpacking success rate, automation level, and overall reliability in complex scenarios. Therefore, this invention, by constructing an adaptive unpacking method with a closed-loop capability of "perception-decision-execution-fault tolerance," fundamentally solves the inherent contradiction between the rigid toolchain and the diversity and uncertainty of firmware in traditional firmware unpacking technology when dealing with non-standardized, structurally complex, or partially damaged firmware, resulting in high failure rates and heavy reliance on human experience.
[0017] 2. This invention replaces manual visual analysis with automated preliminary analysis, automatically matches and invokes underlying tools based on strategies to replace manual command assembly, and automatically handles anomalies through fault-tolerant logic to replace manual intervention and debugging. Ultimately, it transforms the entire unpacking process into a standardized operation that can be executed with a single click, yields consistent results, and is fully traceable. This not only reduces the depth of professional background required of operators, enabling ordinary technicians to complete complex unpacking tasks, but also significantly improves processing efficiency and process reliability by eliminating human decision-making and operational delays.
[0018] 3. During the data extraction phase, the system continuously monitors the tool's execution status (e.g., return codes, error output). If the current tool fails to read bad blocks or experiences data anomalies, a predefined fault-tolerance strategy is triggered, automatically and seamlessly switching to a backup tool with data recovery capabilities to continue extraction, rather than terminating the entire process. Simultaneously, the system accurately records the location and context of the error and extracts the remaining intact data partitions first, achieving "partial isolation" of the damaged area and "maximum preservation" of valid data. This process is fully automated, requiring no manual intervention, ensuring the unpacking process can continue even with partial corruption. Ultimately, it generates standardized output containing successfully extracted content and clear error logs, significantly improving data acquisition rate and process robustness under non-ideal conditions. Attached Figure Description
[0019] Figure 1 This is a flowchart of the unpacking method in this invention.
[0020] Figure 2 This is a schematic diagram of the unpacking toolchain in this invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Please see Figure 1 In this embodiment of the invention, the unpacking method includes the following:
[0023] I. Environmental Preparation Set up the necessary hardware and software environment for the toolchain to run, and automatically install underlying tools such as xxd / hexdump, strings, dd, ddrescue, and sasquatch as shown in Table 1 through the system package manager, and establish a hierarchical directory structure to achieve data isolation and process traceability.
[0024] Table 1. Underlying Tools and Functions Tool Name Function xxd / hexdump Hexadecimal visualization and offset positioning for binary files strings Extract printable strings from firmware dd Used to precisely extract raw data blocks by offset. ddrescue Data recovery for corrupted firmware sasquatch Processing vendor-modified SquashFS file system file Auxiliary tools for identifying basic file types II. Preliminary Analysis By using multi-dimensional tools for analysis, the basic attributes and potential features of the firmware are extracted, mainly including file type identification, feature string extraction, binary visualization, etc., and integrated to form an analysis report containing basic firmware information, identified file system signatures, and suspicious offsets.
[0025] The specific methods for tool invocation and data extraction are as follows: (1) File type identification: Call file firmware.bin to obtain the basic type of the firmware and determine whether it is a known format; (2) Feature string extraction: Execute strings -n 8 firmware.bin (extract strings with a length of ≥8) to filter out keywords related to file system, architecture, compression algorithm, etc.
[0026] (3) Binary visualization: Use xxd -a firmware.bin | head -n 200>analysis / hex_preview.txt and hexdump -C firmware.bin | head -n 200>analysis / hexdump_preview.txt to generate a hexadecimal-ASCII code comparison view of the firmware to help identify duplicate data blocks and partition table markers.
[0027] III. Zoning Based on the preliminary analysis results, the firmware's partition table format is identified. Then, the partition's starting offset is determined based on the file system signature, and the partition size is calculated. The final output is a CSV format information table containing the partition name, inferred type, hexadecimal starting offset, and size. This table supports manual editing and correction to adapt to non-standardized partition scenarios.
[0028] The method for identifying partitioned tables is as follows: (1) For the standard partition table, locate the partition by searching for feature markers (such as the 0x55AA marker at offset 0x1FE in MBR, and the “EFIPART” signature in GPT), and parse the partition table entries to obtain the start / end address of the partition.
[0029] (2) For vendor-customized partition tables: Combine the strings (such as “partition_table”, “offset”) from the preliminary analysis with the binary view, manually locate the starting position of the partition table and parse the partition description in the private format.
[0030] The specific methods for file system signature matching and boundary determination are as follows: (1) Starting offset: Search for the file system signature (such as b'hsqs' of SquashFS) identified in step two in the firmware binary data, and record the offset of the signature (such as 0x100000) as the starting point of the partition.
[0031] (2) End offset: Determined in the following way: If a next partition signature exists, its starting offset is used as the end point of the current partition; If it is the last partition, the partition size is the total firmware size minus the starting offset; If an empty data block exists (a continuous sequence of 0x00 or 0xFF), the starting position of the empty block is used as the ending point.
[0032] IV. Data Extraction Based on the partition information table, the appropriate tool is called to extract data from each partition. The partition information table is read, and the tool is matched according to the partition type. Simultaneously, the tool's execution status is monitored in real time. If a read failure occurs, the tool is automatically switched to another tool. The error location is logged, and intact data is extracted first. After extraction, the partition file type is verified a second time using the `file` command to ensure consistency with expectations and avoid subsequent parsing deviations.
[0033] The tool matching logic is as follows: For SquashFS (including its modified version), sasquatch is used for decompression first. If sasquatch fails to decompress (e.g., due to an unsupported compression algorithm), it is downgraded to dd to extract the original image for subsequent manual analysis. For standard file systems, binwalk is used for extraction. When encountering unknown data types or extraction failures, dd is used to copy the original data by offset. If dd encounters a bad block read failure, it automatically switches to ddrescue.
[0034] V. Results Integration Summarize the execution data from all the above steps and generate a standardized report. First, generate an unpacking report in JSON format, including basic firmware information, execution status of each step (success / failure and reason), and partition extraction statistics. Then, organize the final unpacked file system contents by partition name to a specified directory and archive all intermediate files (analysis report, partition table, tool log). Finally, generate a README file explaining the directory structure, key findings, and subsequent processing suggestions.
[0035] II. Firmware Unpacking Toolchain like Figure 2 As shown, the unpacking toolchain includes the following modules: Firmware Preliminary Analysis Module: Extracts basic firmware information, characteristic strings, and binary views using tools such as file, strings, xxd, and hexdump, and identifies potential file system signatures and architecture information to provide a basis for subsequent processing.
[0036] Structure recognition and partitioning module: Based on the preliminary analysis results, it identifies the partition table format (MBR, GPT or custom), determines the starting offset of the partition by searching the file system signature, determines the ending offset by combining binary analysis, and generates a partition information table containing the partition name, type, offset and size.
[0037] Tool selection and execution module: Establishes a mapping relationship between file system types and extraction tools (e.g., SquashFS corresponds to sasquatch, and unknown types correspond to dd), automatically generates tool calling commands based on the partition information table (e.g., calculates the skip and count parameters of dd), and executes commands according to priority.
[0038] Error handling and recovery module: Monitors the execution status of the tool, automatically switches to the backup tool (e.g., switches to ddrescue when dd fails) when extraction fails, records the error location and logs, and ensures that valid data is extracted to the maximum extent.
[0039] Automated integration module: Connects the above modules to form a complete workflow, supports fully automatic mode (no manual intervention required) and manual mode (allows users to edit partition information), and generates standardized unpacking reports.
[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for unpacking non-standardized firmware based on static orchestration, characterized in that, include: A preliminary analysis is performed on the firmware to be analyzed to obtain its structural feature information; Based on structural feature information, the logical partition structure of the firmware is identified, and adjustable partition description information is generated. Based on the partition description information and the predefined tool mapping strategy, select and call the corresponding underlying tools to extract data from each partition; If the current reading tool fails during the data reading process, the system will automatically switch to the backup reading tool according to the predetermined fault tolerance strategy. Integrate the reading process and result data to generate standardized unpacking output.
2. The non-standardized firmware unpacking method based on static arrangement according to claim 1, characterized in that, The preliminary analysis includes at least one of file type identification, feature string extraction, and binary visualization.
3. The non-standardized firmware unpacking method based on static arrangement according to claim 2, characterized in that, The logical partitioning structure involves searching for file system signature features in the firmware binary data and determining the boundaries of each partition based on the location of the found signature.
4. The non-standardized firmware unpacking method based on static arrangement according to claim 3, characterized in that, When determining partition boundaries, the starting position of the next file system signature is used as the end boundary of the current partition, or the starting position of consecutive empty data blocks is used as the end boundary.
5. The non-standard firmware unpacking method based on static orchestration according to claim 1, characterized in that, The tool mapping strategy defines the correspondence between file system types, data characteristics and underlying extraction tools, including mapping specific file system types to dedicated unpacking tools and mapping unknown or unrecognizable types to raw data copying tools.
6. The non-standard firmware unpacking method based on static orchestration according to claim 5, characterized in that, The fault-tolerance strategy includes automatically switching to a second extraction tool with data recovery capabilities to continue execution when the first reading tool fails due to a reading error.
7. The non-standard firmware unpacking method based on static orchestration according to claim 1, characterized in that, Generating standardized unpacking output includes generating a structured unpacking report, which contains firmware metadata, extraction status logs for each partition, and the final file directory tree information.
8. The non-standard firmware unpacking method based on static orchestration according to claim 1, characterized in that, The unpacking method also includes a fully automatic mode and an interactive mode; in interactive mode, users are allowed to verify and modify the adjustable partition description information.
9. A non-standard firmware unpacking toolchain based on static orchestration, characterized in that, A method for implementing a non-standard firmware unpacking method based on static orchestration as described in any one of claims 1-8 includes: The firmware preliminary analysis module is used to perform automated preliminary analysis of the firmware to be analyzed and obtain its structural feature information. The structure recognition and partitioning module is used to identify the logical partition structure of the firmware based on structural feature information and generate adjustable partition description information. The tool selection and execution module is used to automatically select and call the corresponding underlying tools to extract data from each partition based on the partition description information and the predefined tool mapping strategy. The exception handling and recovery module is used to monitor the execution status of the tool during the data extraction process, and to control the switch to the backup extraction tool according to the predetermined fault tolerance strategy when the current extraction tool fails. The automated integration module is used to integrate the extraction process and result data to generate standardized unpacking output.
10. A non-standard firmware unpacking toolchain based on static orchestration according to claim 9, characterized in that, The preliminary analysis module performs analysis by calling at least one of the following: a file type identification tool, a string extraction tool, and a binary viewer.