System call interception method based on dynamic binary code rewriting

Abstract

A system call interception method based on dynamic binary code rewriting comprises the steps that a corresponding rewriting strategy is configured according to the characteristics of an application to be operated, and a springboard area and a hook function are set according to the rewriting strategy when an operating system is initialized; the method comprises the following steps: after intercepting a system call created or loaded during application running, identifying a precise code byte representing a system call instruction when the system call instruction is executed for the first time, and performing binary code rewriting on the intercepted precise code byte of the system call instruction according to a rewriting strategy pair; therefore, program monitoring is realized through the hook function during operation of the operating system. According to the characteristics of the to-be-operated application, the corresponding rewriting strategy is configured, and efficient system call interception is achieved; the binary code of the dynamic system calling instruction is rewritten, namely, the binary code is rewritten when the application runs or is loaded.

Description

Technical Field

[0001] The present invention relates to a technology in the field of information security, in particular to a system call interception method based on dynamic binary code rewriting. Background Art

[0002] As the complexity of applications continues to increase, the demand for adding application monitoring capabilities to operating systems is also increasing. System calls (syscalls) are programming interfaces provided by the operating system to applications, allowing them to interact with the operating system kernel. Through system calls, programs can access services and resources provided by the operating system, or perform some privileged operations, such as file access, process management, network communication, etc. System call interception (syscall hook) intercepts system calls and inserts and runs a custom hook function before and after the system call execution to implement operations such as checking, filtering, intercepting, modifying, and releasing related system calls, thereby achieving the goal of monitoring program execution. System call interception is usually implemented through a trap-simulation mechanism, but this method introduces context switching, resulting in significant performance degradation. How to provide an efficient system call interception method is a long-standing technical problem. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a system call interception method based on dynamic binary code rewriting.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a system call interception method based on dynamic binary code rewriting. According to the characteristics of an application to be run, a corresponding rewriting strategy is configured. When the operating system is initialized, a springboard area and a hook function are set according to the rewriting strategy. After intercepting the system call created or loaded when the application is running, the precise code bytes representing the instruction when the system call instruction is first executed are identified, and according to the rewriting strategy, the precise code bytes of the intercepted system call instruction are rewritten into binary code, thereby realizing program monitoring through the hook function during the operation of the operating system.

[0006] The rewrite strategy includes: basic strategy, non-zero page execution strategy and red zone protection strategy;

[0007] The setting of the springboard area and the hook function refers to: setting the memory page of the springboard area to be executable only, placing the hook function at the end of the corresponding springboard area, and catching illegal or unintentional NULL execution when the springboard area is set to zero page;

[0008] The binary code rewriting includes: under the basic strategy, dynamically rewriting the system call instruction (syscall / sysenter) into callq*%rax or call*%eax instruction; under the non-zero page execution strategy or the red zone protection strategy, dynamically rewriting in progressive steps.

[0009] The progressive steps include: a heuristic step and a full capture step.

[0010] The heuristic step is as follows: first determine whether the heuristic exists; if so, rewrite movq#syscall,%rax to movq f(#syscall),%rax, where: f(#syscall) is a specific encoding method based on the system call number (#syscall) to avoid the problems of zero address execution and red zone destruction; rewrite the system call instruction to a call instruction under the non-zero page execution strategy, and rewrite the system call instruction to a jmp instruction under the red zone protection strategy.

[0011] The full capture step refers to: handling special situations that cannot be satisfied by the heuristic step, and dynamically rewriting using the overwrite-recovery mechanism and the trap-simulation mechanism.

[0012] The present invention relates to a system for implementing the above method, comprising: a configuration unit, an initialization unit, an interception unit and a rewriting unit, wherein: the configuration unit configures a corresponding rewriting strategy according to the characteristics of an application to be run, the initialization unit sets a springboard area and a hook function according to the rewriting strategy when the operating system is initialized, the interception unit intercepts the system call created or loaded when the application is running, and the rewriting unit identifies the precise code bytes representing the system call instruction when the system call instruction is first executed, and rewrites the precise code bytes of the intercepted system call instruction into binary code according to the rewriting strategy. Technical Effects

[0013] Compared with the prior art, the present invention configures corresponding rewriting strategies according to the characteristics of the application to be run, thereby realizing efficient system call interception; binary code rewriting of dynamic system call instructions, that is, binary code rewriting is performed when the application is running or loaded. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention;

[0015] Figure 2 It is a schematic diagram of the rewriting strategy of the present invention;

[0016] Figure 3 A schematic diagram of the dynamic rewriting method under the basic strategy;

[0017] Figure 4Set up and rewrite the schematic diagram for the springboard area under the basic strategy;

[0018] Figure 5 It is a schematic diagram of the principle of heuristic steps;

[0019] Figure 6 Set up and rewrite schematics for heuristic step springboard areas;

[0020] Figure 7 A flowchart of the full capture steps. DETAILED DESCRIPTION

[0021] like Figure 1 As shown, this embodiment relates to a system call interception method based on dynamic binary code rewriting, including:

[0022] Step 1: Configure the rewrite strategy according to the characteristics of the application to be run, including: a basic strategy for applications that do not consider the special characteristics of zero pages or do not use red areas, a non-zero page execution strategy for applications that pay attention to zero page security issues, and a red area protection strategy for applications that do not use red areas.

[0023] The zero page refers to: the first memory block in the computer address space, that is, the page with a starting address of zero; usually the operating system reserves this page and sets it to prohibit access;

[0024] The red zone is the 128-byte area below the stack pointer that can be used for temporary storage of program calls. Applications can use the red zone to avoid the operational overhead caused by modifying the stack pointer, thereby improving performance. The operating system usually ensures the integrity and security of the red zone.

[0025] Step 2: According to the rewrite strategy, set the springboard area and hook function when the operating system is initialized, including:

[0026] Step 2.1: Under the basic strategy, set the springboard area to zero page, fill the memory at the address corresponding to the system call number with nop instructions, and place the hook function at the end of all nop instructions;

[0027] In actual scenarios, applications have illegal or unintentional memory access to zero pages, i.e., NULL pointer access. Due to the lack of physical memory mapping at the zero page, a page fault will occur and the user space program will be terminated. In order to prevent the above problems from occurring, the embodiment of this specification sets the memory page of the springboard area to be executable only, and prepares a bitmap for maintaining the call point virtual address of the replaced system call to capture illegal or unintentional NULL execution of the application. If the call point address is not in the bitmap, this solution considers it to be an illegal NULL execution and terminates the user space program. Otherwise, it is considered to be normal behavior.

[0028] Step 2.2, under the non-zero page execution strategy, set the springboard area to a non-zero specific area, and the address range of the specific area is based on Figure 5 The address range after the system call number is encoded as shown and the hook function is placed at the end of the springboard area corresponding to each system call;

[0029] Step 2.3, under the red zone protection strategy, set the springboard area to a specific area, and the address range of the specific area is based on the following Figure 5 The address range after the system call number is encoded as shown and the hook function is placed at the end of the springboard area corresponding to each system call;

[0030] Step 3: intercept the system call created or loaded when the application is running and use the operating system characteristics or hardware characteristics to identify the precise code bytes representing the system call instruction when the instruction is first executed.

[0031] The operating system or hardware characteristics include but are not limited to system call interception characteristics provided by Linux such as Syscall User Dispatch or hardware characteristics such as SIGILL exception.

[0032] Step 4: Figure 2 As shown, the exact code bytes of the intercepted system call instructions are dynamically binary rewritten according to the base policy, the non-zero page execution policy, or the red zone protection policy; Figure 3 As shown, taking the basic strategy as an example, before rewriting, when the system call instruction is executed for the first time, the rewriting logic is triggered and the hook function is called immediately; after rewriting, the application will directly execute the rewritten instruction and the hook function will be called immediately; based on the dynamic rewriting method, this solution only rewrites the system call instructions that are actually used, and does not rely on conventional assumptions or system call instructions identified by disassembler analysis during loading, so as to improve the compatibility and reliability of the rewriting method.

[0033] The step 4 specifically includes:

[0034] Step 4.1: Under the basic strategy, dynamically rewrite the system call instruction (syscall / sysenter) to callq*%rax or call*%eax, such as Figure 4 Shown is a schematic diagram of the springboard area setting and overriding method under the basic strategy;

[0035] Although the base policy takes zero page access into account during OS initialization, it still allows code execution on zero page, which makes it possible for an attacker to manipulate the rax or eax registers and the return address on the stack, bypassing the NULL execution check and leading to arbitrary system call execution. At the same time, the rewrite policy of replacing syscall / sysenter with callq / call may lead to ABI violations. Specifically, if the application uses the red zone, callq / call will implicitly save the return address on the stack, resulting in the integrity of the red zone being compromised and potential program crashes.

[0036] Following the ABI calling convention, before executing the syscall instruction, the application sets the system call number to the rax or eax register;

[0037] All three strategies utilize the dynamic system call interception in the previous step and perform rewriting at the first interception.

[0038] Step 4.2: Under the non-zero page execution strategy or the red zone protection strategy, a progressive step including a heuristic step and a full capture step is used for dynamic rewriting, including:

[0039] ① Heuristic step: Identify the heuristic based on whether there is an operation in the code segment before the system call instruction that transfers the system call number to the rax or eax register through the mov instruction;

[0040] If the heuristic exists, rewrite movq#syscall,%rax to movqf(#syscall),%rax, where: f(#syscall) is a specific encoding based on the system call number (#syscall) to avoid zero-address execution and red zone corruption issues;

[0041] like Figure 5 As shown, bits [0,8) are used to avoid the red area corruption problem; bits [8,16) are used to save the original system call number; bits [16,24) are used to avoid the zero address execution problem. The specific steps are as follows:

[0042] Under the non-zero page execution policy, rewrite movq #syscall, %rax to movq f(pf×, #syscall), %rax, and rewrite syscall / sysenter to callq *%rax; where pf× is a one-byte prefix offset that controls the target (non-zero) springboard area to which callq *%rax jumps.

[0043] Under the red zone protection strategy, rewrite movq#syscall,%rax to movq f(#syscall,id×),%rax, and rewrite syscall / sysenter to jmpq*%rax; id× is a one-byte index used to identify the call point of the replaced system call; the jmp instruction is used to prevent the call instruction from pushing the return address into the stack to destroy the red zone. During the rewriting process, the return address needs to be recorded. Since there are multiple call points in actual applications that call the same system call, in order to correctly identify the return address of each call point, this solution records and maintains the index of different call points for each system call, and embeds the index id× into the bit [0,8) of the rax or eax register. Therefore, this solution allows each system call to come from up to 256 different call points; system calls are usually initiated through public libraries, so in actual situations, this limit is rarely exceeded; if the limit is exceeded, the full capture step will be triggered.

[0044] For example, under the non-zero page execution strategy and the red area protection strategy, the above two methods are combined to rewrite movq#syscall,%rax to movq f(pf×,#syscall,id×),%rax; rewrite syscall / sysenter to jmpq*%rax; the springboard area setting and rewriting method are as follows Figure 6 shown.

[0045] ② Full capture step: Since the heuristic does not always exist in the program, a full capture step needs to be introduced to handle all situations. Figure 7 Schematic diagram of the whole capture process.

[0046] Under the red zone protection strategy, an overwrite-restore mechanism is used in the dynamic rewrite process. Specifically, 0×590×ff0×d00×600×cc 0×cc (i.e., pop%rc×; callq*%rax; 0×60; int3; int3) is used to replace syscall and subsequent instructions. In this scheme, pop is used to adjust rsp so that the red zone remains intact when the call instruction is called. The above code sequence is triggered by exceptions of +2 bytes (0×d00×600×cc=shlb 0×cc(%rax)), +3 bytes until the last byte (int3), identifying and processing direct jumps in the program whose target is the replaced byte. Afterwards, the control flow is redirected back to the saved code block before overwriting to achieve repair.

[0047] In special cases where the non-zero page execution strategy and the overwrite-restore mechanism cannot meet the requirements, the trap-simulation mechanism is used.

[0048] After specific actual experiments, the experimental equipment is a 2-socket 36-core Intel Xeon Platinum 8360YCPU@2.40GHz, 256GB memory server, based on the operating system Ubuntu 20.04LTS (kernel version 6.2.0), the test applications include website applications, database applications and artificial intelligence applications. The experimental data that can be obtained shows that this method can achieve up to twice the improvement compared with the existing technology.

[0049] Compared with the existing technology, this method configures the rewriting strategy according to the characteristics of the application to be run, and adapts to different application scenarios; it adopts the method of rewriting binary code when the application is running or loading, saving the work and time of offline static rewriting; it adopts a progressive rewriting step, and the heuristic step can cover 97% of the actual application scenarios.

[0050] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A system call interception method based on dynamic binary code rewriting, characterized in that: According to the characteristics of the application to be run, a corresponding rewrite strategy is configured, and a springboard area and a hook function are set according to the rewrite strategy when the operating system is initialized; after intercepting the system call created or loaded when the application is running, the precise code bytes representing the instruction when the system call instruction is first executed are identified, and according to the rewrite strategy, the precise code bytes of the intercepted system call instruction are rewritten into binary code, thereby realizing program monitoring through the hook function during the operation of the operating system; The rewrite strategy includes: a basic strategy, a non-zero page execution strategy and a red zone protection strategy.

2. The system call interception method based on dynamic binary code rewriting according to claim 1, characterized in that: The setting of the springboard area and the hook function refers to: setting the memory page of the springboard area to be executable only, placing the hook function at the end of the corresponding springboard area, and catching illegal or unintentional NULL execution when the springboard area is set to a zero page.

3. The system call interception method based on dynamic binary code rewriting according to claim 1, characterized in that: The binary code rewriting includes: under the basic strategy, dynamically rewriting the system call instruction (syscall / sysenter) into a callq*%rax or call*%eax instruction; under the non-zero page execution strategy or the red zone protection strategy, dynamically rewriting in progressive steps; The progressive steps include: a heuristic step and a full capture step.

4. The system call interception method based on dynamic binary code rewriting according to claim 3 is characterized in that: The heuristic step is as follows: first determine whether the heuristic exists; if so, rewrite movq#syscall,%rax to movqf(#syscall),%rax, where: f(#syscall) is a specific encoding method based on the system call number (#syscall) to avoid the problems of zero address execution and red area destruction; rewrite the system call instruction to a call instruction under the non-zero page execution strategy, and rewrite the system call instruction to a jmp instruction under the red area protection strategy.

5. The system call interception method based on dynamic binary code rewriting according to claim 3 is characterized in that: The full capture step refers to: handling special situations that cannot be satisfied by the heuristic step, and dynamically rewriting using the overwrite-recovery mechanism and the trap-simulation mechanism.

6. A system call interception system based on dynamic binary code rewriting for implementing any of the methods described in claims 1-5, characterized in that: include: A configuration unit, an initialization unit, an interception unit and a rewrite unit, wherein: the configuration unit configures a corresponding rewrite strategy according to the characteristics of the application to be run; the initialization unit sets a springboard area and a hook function when the operating system is initialized according to the rewrite strategy; the interception unit intercepts the system call created or loaded when the application is running; the rewrite unit identifies the precise code bytes representing the system call instruction when the system call instruction is first executed, and rewrites the precise code bytes of the intercepted system call instruction into binary code according to the rewrite strategy.

Images