A binary program dynamic analysis method based on process hollowing technology

Abstract

The application discloses a binary program dynamic analysis method based on a process hollowing technology, utilizes a system kernel module to further expand the ability of a traditional binary dynamic plug-in tool when performing fine-grained analysis, constructs a brand-new fine-grained analysis environment in a new hollowing process by combining a code decoupling analysis thought, can reduce the influence on the execution of a target program, improves the application range and analysis reliability of binary analysis, and can reuse the function of an application layer dynamic plug-in tool. The application utilizes the operating system characteristics to realize more flexible and efficient analysis function, gives a hollowing process environment and an analysis task construction method, and can realize fine-grained data flow analysis on special binary programs such as a vulnerability exploitation program and a countermeasure malicious program. The application method can effectively improve the application range of binary program automatic analysis, simplifies the complexity of analysis memory management, and improves the performance and stability of dynamic analysis on binary codes.

Description

Technical Field

[0001] This invention belongs to the field of software security analysis technology, and specifically relates to a method for dynamic analysis of binary programs based on process cutout technology. Background Technology

[0002] With the increasing prevalence of cyberattacks, vulnerable applications and malicious programs delivered to target systems play a major role. As software security strategies become more sophisticated, simple program analysis is no longer sufficient to uncover potential security vulnerabilities or threats; more granular and logically rich analysis methods have become essential in various scenarios. Similarly, attackers are employing more covert obfuscation techniques to hide their actual functions. Security personnel and automated analysis platforms need to conduct more detailed analysis of malicious code, going beyond simply intercepting basic program behavior. For example, some malicious code on Windows systems uses undocumented system interfaces to call system functions, circumventing traditional function hooking-based detection methods. Furthermore, security researchers have been exploring dynamic, fine-grained analysis methods for many years, represented by dynamic data flow analysis or dynamic taint analysis, achieving significant results while also facing challenges such as improving analysis performance, adapting to new analysis environments, addressing specific customized analysis needs, and relying on virtualization technology.

[0003] Existing technical solutions focus on building new analysis schemes, but are mainly based on or geared towards Linux system environments. However, the demand for program and malware analysis is greater on Windows platforms, and the supporting technical methods are insufficient. These solutions primarily rely on virtualization methods, lacking deployment flexibility and making it difficult to deploy and implement analysis directly on target systems. Other research focuses on application-layer dynamic instrumentation methods, building more efficient and in-depth analysis methods on this basis, but fails to fully exploit operating system characteristics, resulting in shortcomings in practicality. Furthermore, the widespread adoption of 64-bit system environments, increased program code size, and higher memory consumption during runtime bring new challenges to the design and implementation of analysis frameworks.

[0004] Fine-grained dynamic program analysis requires instrumentation of the target program, which may affect its normal operation, especially the large allocation of shadow memory, which can impact the virtual memory space layout of the target process. These large amounts of placeholders can alter the memory address allocation locations normally used by the target program, potentially changing the application's execution process. For example, some exploits use heap spraying techniques to optimize memory layout for code execution, and the range of heap spraying can vary between different samples. Therefore, when performing dynamic analysis on memory-intensive software, exploits, or other specialized applications, introducing too many interfering factors into the target process space can easily lead to memory address allocation conflicts, resulting in inconsistent execution outcomes. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by providing a dynamic program analysis method based on process hollowing-out technology. This method further expands the concept of decoupled analysis by introducing kernel module support and constructing analysis code within an independent hollowed-out process, thereby improving analysis performance and flexibility while reducing the impact on the normal execution of the target program. A novel memory allocation and management scheme for dynamic fine-grained binary analysis is proposed. By constructing the memory layout required for dynamic data flow analysis within the hollowed-out process, the difficulty of shadow memory allocation and storage is simplified, further improving the capabilities of dynamic fine-grained binary analysis. In the analysis of binary programs, the application of this invention can particularly expand the applicability of dynamic analysis and improve analysis performance.

[0006] The technical solution to achieve the purpose of this invention is: a dynamic program analysis method based on process cutout technology, the method comprising the following steps:

[0007] Step 1: Based on the actual analysis scenario and objectives, build and configure the basic analysis host program, in which the analysis task inspection function code and user interaction interface are built. After completion, compile and generate a new analysis host program, which will be mapped to the hollow process.

[0008] Step 2: Based on the characteristics of the target program to be analyzed and the specific analysis requirements, construct the dynamic link library modules that the instrumentation tool needs to load. During construction, build the corresponding analysis functions based on the interface provided by the general instrumentation tool, including execution code tracing, runtime information recording, and analysis code construction.

[0009] Step 3: Start the target program to be analyzed using a general instrumentation tool, and create the instrumentation process where the target program actually runs. When the process starts, it will load the dynamic link library module built in Step 2 to complete the memory allocation and instrumentation of the target program during runtime.

[0010] Step 4: Based on Step 3, the functional modules in the instrumentation tool will create a hollow process and load the new analysis host program built in Step 1. The analysis code will run in this process environment.

[0011] Step 5: Based on step 4, the hollow process continues to load the pre-compiled kernel driver module after it runs, which assists in the subsequent analysis environment initialization and program analysis. Then, the analysis environment initialization process will begin, including creating analysis threads, saving the running state, and adjusting the memory layout of the hollow process.

[0012] Step 6: Execute the analysis code corresponding to the target program in the hollow process. Then, the kernel module is responsible for notifying the analysis thread to start executing the analysis code. The analysis thread executes the analysis code pre-built in step 2 and cyclically obtains runtime information and subsequent analysis code to complete the entire program analysis process.

[0013] Step 7: During the execution and analysis of the target program, when the target program executes system calls related to synchronization events, the kernel module will switch the buffer of the current thread to speed up the analysis process and alleviate the synchronization problem in multi-threaded scenarios.

[0014] Step 8: After the analysis code corresponding to the target program is executed, the analysis task is triggered. This analysis task is set in step 2. For the analysis task that needs to be checked, the target program will enter the kernel module after completing the analysis task. The kernel module checks and outputs the analysis results.

[0015] Compared with the prior art, the significant advantages of this invention are:

[0016] (1) This invention utilizes process cutout technology and kernel technology to realize dynamic binary program analysis, which expands the capabilities of dynamic binary analysis and enables automated deep analysis of applications that are sensitive to memory allocation and memory layout.

[0017] (2) This invention proposes a memory management scheme for dynamic fine-grained analysis, which simplifies the complexity of shadow memory and analysis code management and further improves the analysis performance of binary programs.

[0018] (3) The present invention can directly analyze binary programs without preprocessing or other optimization operations on the program code, and has good versatility. It can also be combined with other program analysis and optimization methods.

[0019] (4) This invention can integrate the functions of existing instrumentation tools without changing the original tools used. It can reuse the rich interfaces provided by various mature analysis platforms, simplify the complexity of the framework, and can be quickly applied in practice.

[0020] (5) The present invention can be deployed directly in a real target environment, and has better flexibility and lower performance overhead, while avoiding the introduction of anti-virtualization detection overhead. Attached Figure Description

[0021] Figure 1 This is a framework diagram of the binary program dynamic analysis method based on process cutout technology of the present invention.

[0022] Figure 2 This is a flowchart of the analysis environment initialization process built in one embodiment.

[0023] Figure 3 This is a memory space layout diagram of the various module processes constructed in one embodiment.

[0024] Figure 4 This figure shows the performance evaluation results of a binary dynamic program analysis framework constructed in one embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] In one embodiment, combined Figure 1 This paper provides a method for dynamic analysis of binary programs based on process cutout technology, which includes the following steps:

[0027] Step 1: Based on the actual analysis scenario and objectives, construct the analysis task inspection function code and define the interaction interface in the basic host program corresponding to the hollowing process. After completion, compile and generate a new analysis host program. This step specifically includes:

[0028] Step 1.1: Construct a basic analysis host program, including the program startup main function, thread initialization function, and analysis task distribution and processing function. When compiling the program, static linking is used, so there is no need to link external dynamic link library modules. In the analysis task processing function, the interfaces exported by the kernel module can be called to complete other operations.

[0029] Step 1.2: In the basic analysis host program's thread initialization function, save the task dispatch function address to the specified register. For example, on a 64-bit system, register R15 can be specified, and on a 32-bit system, register ESI can be specified.

[0030] Step 1.3: In the basic analysis host program's analysis task dispatch function, add different analysis task processing functions as needed, and specify special registers to pass the task type. For example, on a 64-bit system, register R10 can be specified, and on a 32-bit system, register EDI can be specified.

[0031] Step 1.4: Compile and generate a new analysis host program. During compilation, other default libraries are ignored and no other library functions are introduced to avoid interference with the memory address space. Calls to other system library functions are mainly completed by simple encapsulation functions in the program.

[0032] Step 2: Based on the characteristics and specific analysis requirements of the target program, construct corresponding analysis functions using interfaces provided by general instrumentation tools, such as DynamoRIO and Pin. Then, compile these functions to generate a dynamic link library module containing the analysis function code. Specific steps include:

[0033] Step 2.1: Construct functional code to trace the execution process of the target program, build instrumentation code for instructions related to the analysis task, and use the extended thread TLS local storage buffer to save pointers to the thread runtime context register and record buffer. In this process, optimization is combined with kernel modules to further improve the efficiency of runtime information recording.

[0034] Step 2.2: Construct runtime information recording code for the target program, recording memory address information associated with the analysis instructions, including the flag register and register values ​​that are difficult to determine statically, such as the value of the cl register in the shift instruction shlrbx,cl. The recording code uses simple instructions such as LEA and MOV to retrieve values, inserting relevant code before the original executed instructions;

[0035] Step 2.3: While instrumenting and constructing code to record runtime information, corresponding analysis code is simultaneously constructed on a per-program-basic-block basis. Recording code is inserted into the target program code on a per-program-basic-block basis, and analysis code corresponding to the basic block is generated. For each instruction being analyzed, if runtime recording information exists, recording code will be added to it during execution, and the operation of retrieving data from the buffer will be constructed in the corresponding analysis code, maintaining the corresponding relationship between the data access of the two parts.

[0036] Step 2.4: For each basic block of program execution, construct functional code. When the program runs, insert code at the beginning of its instructions to record the address of the constructed analysis code, so that when the analysis thread reads the record buffer, it can directly obtain the analysis code corresponding to the next basic block.

[0037] Step 2.5: Construct the linker code for the analysis code execution. In the analysis code, specify a register to point to the next location in the buffer to be read. For example, in a 64-bit system, the RBX register can be specified. In the analysis code, after retrieving the data, it will be read again to determine whether the retrieved value is a valid analysis code address. In this way, if an exception occurs during the execution of the analysis thread, the kernel module will handle it and switch the record buffer, so that the analysis code can continue to execute.

[0038] Step 2.6: Insert task setup and check code at the function locations related to the analysis task. Use instrumentation tools to construct instruction-level task function call code to trigger the execution of the actual analysis task. For each analysis task, a simple function call instruction is generated during instrumentation, and the generated code address is saved to the record buffer. The code saves the task type, passed parameters, etc., to registers or pushes them onto the thread stack, and then calls the location register of the saved function. For example, the task type is passed using register R10, and the target address is used using register R15, consistent with the settings in Step 1.2.

[0039] Step 3: Start the target program to be analyzed using a general instrumentation tool to create the process that the target program actually executes, i.e., the instrumentation process. When the process starts, it will load the dynamic link library module built in Step 2 to complete functions such as memory allocation and target program execution tracing.

[0040] Step 4: Based on Step 3, the functional modules in the instrumentation tool will create a hollow process and load the new executable analysis host program built in Step 1. The analysis code will run in this process environment.

[0041] Step 5: Building on Step 4, the hollow process will continue to load the pre-compiled kernel module after it runs, to assist in the subsequent analysis environment initialization and program analysis. Then, the analysis environment initialization process will begin.

[0042] In one embodiment, combined Figure 2 The specific initialization steps include:

[0043] Step 5.1 involves constructing a kernel module. This module registers system callbacks and creates synchronization objects to monitor and manage user-mode process and thread activity. The kernel module is implemented as a driver, loaded and run by the application layer program. It includes monitoring functions for the target process and the analysis process, and handles synchronization of analysis events between them. Specifically, during process tracing of the instrumented process, function call characteristics are directly used to obtain the memory allocation status of the instrumented process without modifying the instrumentation tool.

[0044] Step 5.2: Create several analysis threads in the hollow process to wait for the execution of analysis tasks, manage them using a thread pool, and put the analysis threads in a suspended state. The analysis threads will wait for subsequent notifications from the kernel module to execute specific analysis tasks.

[0045] Step 5.3: After all analysis threads are created, the kernel module saves the state of the process control block (PEB) structure of the hollow process and the control block (TEB) structure of all analysis threads in the kernel space buffer.

[0046] Step 5.4: The kernel module starts a new initialization thread, traverses the address space of the hollow process through the memory management functions exported by the system, and releases the virtual memory address segments that have been allocated in the process space, but retains the code area loaded when the program starts.

[0047] Step 5.5: Allocate a shared memory space with the instrumentation process in the instrumentation process. This space is mainly used to record data during the runtime of the target program. This buffer is transparent to the instrumentation tool and does not require the target program or the instrumentation tool to participate in its management.

[0048] Step 5.6: After the instrumentation tool itself has completed the memory allocation required during runtime, the new initialization thread begins to readjust the memory layout in the instrumentation process. The program code portion corresponds to the memory area occupied by the instrumentation tool code within the instrumentation process, while some space is reserved for analysis tasks and can be freely allocated during analysis; this corresponds to the memory space reserved by the instrumentation process for the instrumentation tool. The analysis buffer is allocated by the kernel module during the initialization process. In one embodiment, combined with... Figure 3 The memory layout design specifically includes:

[0049] (1) The memory address space occupied by the instrumentation tool module is used to map the new analysis host program code developed by the user during the hollowing process, including thread scheduling code and analysis status check code.

[0050] (2) The instrumentation process allocates and manages the memory address space for caching instrumentation code. In the container process, all memory related to analysis is allocated from this space. The instrumentation tool allocates a portion of it to store shared analysis code, which facilitates the generation and saving of analysis code during instrumentation.

[0051] (3) The record buffer allocated by the kernel module. The physical pages corresponding to this part of virtual memory are mapped to the same virtual address in the hollow process. This part of virtual memory address is allocated in advance when the target process starts, and a fixed space is allocated from it when the target thread starts.

[0052] (4) Shadow memory required for dynamic data flow analysis: The kernel module tracks the memory allocation and, in response to the target program's allocation, synchronously submits the corresponding address space allocation in the hollow process, keeping in sync with the target program in terms of page allocation.

[0053] Step 5.7: Reallocate TEB and Stack space for all analysis threads. For pre-created task threads, use a small global array as a temporary Stack space. This memory is allocated in the program code segment and is not affected when the process is interrupted. In this way, when the analysis thread is rescheduled and returned to user code, the original user-mode Stack space still exists.

[0054] Step 5.8: After the analysis thread starts executing the actual analysis code, a stack space switch will be performed. This way, the thread will use the newly allocated, larger Stack buffer when executing analysis tasks, ensuring that subsequent analysis tasks can be completed normally.

[0055] Step 6: Execute the analysis code corresponding to the target program in the hollow process. After the target program generates a full buffer or before the target thread exits, the kernel module is responsible for notifying the analysis thread to start executing the analysis code. The analysis thread executes the analysis code pre-built in step 2 and obtains runtime information and the next analysis code from the record buffer, thereby completing the analysis of the entire program.

[0056] Step 7: During the execution and analysis of the target program, the kernel module continuously filters the system call process. When the target program executes a system call related to a synchronization event, a new buffer will be switched for the current thread of the target program to record runtime information, thereby speeding up the analysis process and alleviating the synchronization problem in multi-threaded scenarios.

[0057] Step 8: The analysis code corresponding to the target program is executed and the analysis task is triggered. This analysis task is set in step 2. During the execution process, multiple analysis tasks may be set. For each analysis task that needs to be checked, after the target program completes the analysis task, its execution will enter the kernel module, and the kernel module will check and output the analysis results.

[0058] The method of this invention can solve problems such as process space interference encountered in real-world application environments. It can be directly deployed and run on the target operating system without relying on virtualization features. By introducing kernel module support, it optimizes the analysis memory management strategy, improves the performance of dynamic data flow analysis, and utilizes a hollowed-out process to execute analysis in a new context, reducing the impact on the target program and ensuring that the analysis of memory-sensitive programs can be performed. Compared with other methods, the analysis method of this invention has better flexibility, lower performance overhead, and a wider range of applicability. Figure 4 The results of the performance test program based on the method of this invention show that, compared with the classical method, this method can reduce the impact on the target program and greatly improve the performance of dynamic fine-grained analysis.

[0059] In summary, the binary program dynamic analysis method based on process tunneling technology proposed in this invention can achieve efficient dynamic analysis of binary programs such as large applications, exploit programs, and malicious code. The main idea is to transparently extend the functionality of existing program instrumentation frameworks using kernel modules, and based on the concept of decoupled analysis, to construct a fine-grained analysis environment in independent tunneled processes. Furthermore, a new shadow memory allocation scheme is designed, improving the management and utilization efficiency of memory state analysis, and further enhancing the analysis capabilities of dynamic binary programs. The method of this invention helps to enhance the effectiveness and adaptability of binary code analysis.

[0060] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for dynamic analysis of binary programs based on process cutout technology, characterized in that, Includes the following steps: Step 1: Based on the actual analysis scenario and objectives, build and configure the basic analysis host program, in which the analysis task inspection function code and user interaction interface are built. After completion, compile and generate a new analysis host program. Step 2: Based on the characteristics of the target program to be analyzed and the specific analysis requirements, construct the dynamic link library modules that the instrumentation tool needs to load. During construction, build the corresponding analysis functions based on the interface provided by the general instrumentation tool, including execution code tracing, runtime information recording, and analysis code construction. Step 3: Start the target program to be analyzed using a general instrumentation tool, create the instrumentation process where the target program actually runs, and load the dynamic link library module built in Step 2 when the instrumentation process starts to complete the memory allocation and target program instrumentation functions during program runtime. Step 4: The instrumentation tool's functional modules create a hollow process and simultaneously load the new analysis host program built in Step 1. The analysis code runs in the hollow process environment. Step 5: After the hollow process runs, it continues to load the pre-compiled and built kernel module to assist in the subsequent analysis environment initialization and program analysis. Then, the analysis environment initialization process begins, including creating analysis threads, saving the running state, and adjusting the memory layout of the hollow process. Step 6: Execute the analysis code corresponding to the target program in the hollow process. Then, the kernel module is responsible for notifying the analysis thread to start executing the analysis code. The analysis thread executes the analysis code pre-built in step 2 and cyclically obtains runtime information and subsequent analysis code to complete the entire program analysis process. Step 7: During the execution and analysis of the target program, when the target program executes system calls related to synchronization events, the kernel module switches the buffer of the current thread to speed up the analysis process and alleviate the synchronization problem in multi-threaded scenarios. Step 8: After the analysis code corresponding to the target program is executed, the analysis task is triggered. The analysis task is set in step 2. For the analysis task that needs to be checked, the target program will enter the kernel module after completing the analysis task. The kernel module checks and outputs the analysis results.

2. The binary program dynamic analysis method based on process cutout technology according to claim 1, characterized in that, In step 1, the new analysis host program includes a main function, a thread initialization function, and analysis task dispatch and processing functions. The specific construction method includes: Step 1.1: Build a basic analysis host program. When compiling the program, use static linking, so there is no need to link external dynamic link library modules. In the analysis task processing function, call the interface exported by the kernel module to complete other operations. Step 1.2: In the basic analysis of the host program's thread initialization function, save the address of the task dispatch function to the specified register; Step 1.3: In the basic analysis host program's analysis task dispatch function, add different analysis task processing functions as needed, and specify a special register to pass the task type; Step 1.4: Compile and generate a new analysis host program. During compilation, other default libraries are ignored and no other library functions are introduced. Calls to other system library functions are completed by simple encapsulation functions in the program.

3. The binary program dynamic analysis method based on process cutout technology according to claim 1, characterized in that, In step 2, corresponding analysis functions are built based on the interface provided by the general dynamic instrumentation tool, specifically including: Step 2.1: Construct functional code to trace the execution process of the target program, build instrumentation code for instructions related to the analysis task, and combine the kernel module to extend the thread TLS storage buffer to save runtime data; Step 2.2: Construct runtime information recording code for the target program, record memory address information associated with the analysis instructions, including flag registers and register values ​​that are difficult to determine statically, and insert relevant code before the original execution instructions; Step 2.3: While instrumenting and building code that records runtime information, corresponding analysis code is also built at the same time, based on the basic blocks of the program, to maintain the correspondence between the execution code and the analysis code in terms of data access. Step 2.4: For each basic block of program execution, construct functional code, and when running, insert code at the beginning of its instructions to record the address of the constructed analysis code; Step 2.5: Construct the linker code for the analysis code execution, specifying in the analysis code that the register points to the next location in the buffer to be read, and have the kernel module handle execution exceptions; Step 2.6: Insert task setup and check code at the function locations related to the analysis task, and use instrumentation tools to build instruction-level task function call code to trigger the execution of the actual analysis task.

4. The binary program dynamic analysis method based on process cutout technology according to claim 1, characterized in that, Step 5, the analysis environment initialization process specifically includes: Step 5.1: Build a kernel module, register system callbacks using the kernel module, and create synchronization objects to monitor and manage user-mode process and thread activities. During the process of instrumenting process tracing, the memory allocation status of the instrumenting process is obtained directly using function call characteristics without modifying the instrumentation tool. Step 5.2: Create several analysis threads in the hollow process and put them in a suspended state, waiting for subsequent notification from the kernel module to execute specific analysis tasks; Step 5.3: After all analysis threads are created, the kernel module saves the state of the process control block (PEB) of the hollow process and the control block (TEB) structure of all analysis threads in the kernel space buffer. Step 5.4: The kernel module starts a new initialization thread, traverses the address space of the hollow process through the memory management functions exported by the system, and releases the virtual memory address segments that have been allocated in the process space, while retaining the code area loaded when the program starts. Step 5.5: Allocate a shared memory space with the instrumentation process in the instrumentation process to record the data of the target program during runtime. The buffer does not require the target program or the instrumentation tool to participate in its management. Step 5.6: Once the instrumentation tool itself has completed the memory allocation required during runtime, a new initialization thread begins to readjust the memory layout in the cutout process.

5. The binary program dynamic analysis method based on process cutout technology according to claim 4, characterized in that, During the analysis environment initialization process, the kernel thread resets the memory layout in the hollow process. The specific layout includes: (1) The memory address space occupied by the instrumentation tool is used to map the new analysis host program code developed by the user during the hollowing process, including thread scheduling code and analysis status check code. (2) The memory address space allocated by the instrumentation process for managing and caching instrumentation code is allocated from this space in the container process. The instrumentation tool allocates space in this space to store shared analysis code, so as to generate and save analysis code during instrumentation. (3) The record buffer allocated by the kernel module has its virtual memory corresponding physical page mapped to the same virtual address in the hollow process. It is allocated in advance when the target process starts and a fixed space is allocated from it when the target thread starts. (4) The shadow memory used in dynamic data flow analysis is tracked by the kernel module. When the target program submits an allocation, the corresponding address space is submitted synchronously in the hollow process to keep in sync with the target program in terms of page allocation.

Images