A system call takeover method suitable for Linux system compatibility mode and switch distribution scenarios

Abstract

The application provides a system call takeover method suitable for Linux system compatibility mode and a switch distribution scene, comprising the following steps: when it is detected that an IA32 compatible system call table is unavailable or does not serve as a valid distribution path, locating a direct call point of an original dispatch function in an IA-32 compatible entry function, and by modifying a relative call instruction at the call point, the call point is redirected to a self-defined dispatch function; the self-defined dispatch function implements hierarchical distribution according to a target system call number, executes self-defined logic on a target system call of interest, and transparently returns to the original dispatch function to maintain kernel original semantics for a target system call not of interest; and by means of a multi-entry parallel patching and a runtime self-repairing mechanism, completeness and cross-version stability of the takeover path are ensured. The application does not depend on export of a system call table, can be suitable for new kernels such as Linux 6.x, and is applied to fields such as file transparent encryption and decryption, security auditing and the like.

Description

Technical Field

[0001] This invention relates to the field of operating system kernel security technology, and in particular to a system call takeover method adapted to Linux system compatibility mode and switch distribution scenarios. Background Technology

[0002] The Linux operating system supports running 32-bit user-space programs within a 64-bit kernel (x86_64). This mechanism is known as IA-32 compatible mode or compat mode. When a 32-bit program requests kernel services via a system call, it can enter the kernel through various entry points, such as the int 0x80 interrupt, the SYSENTER instruction, or the SYSCALL instruction. The kernel then dispatches the request to the appropriate compatible system call handling logic within the corresponding entry point processing function.

[0003] In traditional Linux kernel versions, system call dispatch relies on the system call table. For 64-bit applications, the kernel uses `sys_call_table`; for 32-bit compatible applications, it uses `ia32_sys_call_table`. Both tables are arrays of function pointers, indexed by system call numbers, storing the addresses of the kernel handler functions corresponding to each system call. Based on this, kernel modules for security monitoring, auditing, transparent file encryption / decryption, and access control often employ system call hijacking: when a kernel module is loaded, the address of `ia32_sys_call_table` is located, the original entries are saved, and the entry corresponding to the target system call number is replaced with the address of a custom handler function, thereby taking over specific system calls.

[0004] However, in newer Linux kernels (especially Linux 6.x series), the dispatch mechanism for IA-32 compatible system calls has undergone fundamental changes, rendering the aforementioned traditional hijacking schemes ineffective, mainly due to the following problems: In the Linux 6.x kernel, the dispatch of IA-32 compatible system calls no longer relies entirely on the ia32_sys_call_table function pointer table. Instead, within dispatch functions like ia32_sys_call, a switch statement is used to directly jump to the corresponding handler function based on the system call number, employing a compile-time fixed direct dispatch method. Because the jump table or conditional jump instructions generated after the switch statement is compiled are embedded in the function code segment, rather than stored in a dynamically modifiable function pointer table, the traditional method of replacing table entries cannot effectively take over the target system call, leading to the failure of security and monitoring functions.

[0005] Even if some kernel versions still retain ia32_sys_call_table, for security hardening purposes, this symbol may no longer be exported through EXPORT_SYMBOL, or it may not be located through kernel symbol lookup functions such as kallsyms_lookup_name, making it impossible for kernel modules to obtain the base address of the table, let alone implement table entry replacement.

[0006] Furthermore, differences in configuration and patches across different kernel versions and distributions can cause changes in the entry function name, dispatch path, and symbol visibility of IA-32 compatible system calls. Traditional strategies lack runtime self-checking and adaptation capabilities, potentially failing to detect monitoring failures for extended periods, thus creating security vulnerabilities.

[0007] In summary, there is an urgent need for a solution that can still accurately take over the target 32-bit compatible system calls when the IA-32 compatible system call table is unavailable, unreliable, or no longer serves as the primary dispatch path. Summary of the Invention

[0008] To address the aforementioned technical issues, this application proposes a system call takeover method adapted to Linux system compatibility mode and switch distribution scenarios, comprising the following steps: S1. When a Linux system kernel module is loaded or initialized, it checks whether the current kernel environment of the Linux system can obtain a valid IA32-compatible system call table.

[0009] Furthermore, the detection conditions for determining whether the current kernel environment of the Linux system can obtain a valid IA32-compatible system call table include the following conditions: (1) The current kernel module version of the Linux system is not lower than the preset version threshold; (2) The current kernel has enabled syscall wrapper configuration; (3) The symbol ia32_sys_call exists in Linux IA32 compatible systems, but ia32_sys_call_table cannot be obtained by scanning, symbol lookup or other methods; (4) At least one entry in the IA32 compatible system call table has been modified, but no custom processing function has been detected to be called within a preset time.

[0010] If any one of the conditions (1)-(4) above is met, it is determined that the current kernel environment of the Linux system cannot obtain a valid IA32 compatible system call table.

[0011] S2. If a valid IA32-compatible system call table can be obtained, the original table entry is saved, and the table entry corresponding to the target system call number is replaced with a custom processing function, and then the step ends; if a valid IA32-compatible system call table cannot be obtained, the address of at least one IA-32-compatible entry function and the address of the original system call dispatch function are obtained, and then the process proceeds to step S3.

[0012] The target system call number includes a 32-bit numerical identifier for target system calls such as file access, file status, memory mapping, read / write, truncation, file sending, and file range copying.

[0013] Furthermore, when obtaining the address of at least one IA-32 compatible entry function, the search is performed in the following priority order: First, search for do_int80_syscall_32; if not found, search for do_int80_emulation; if still not found, then search for do_fast_syscall_32, __do_fast_syscall_32, do_SYSENTER_32, or their equivalent entry functions in sequence.

[0014] S3. Using the address of the IA-32 compatible entry function as the starting base address, search for relative call instructions within a preset scanning range, parse the displacement of the relative call instructions and calculate the actual target address, compare the actual target address with the address of the original system call dispatch function, if they match, determine that the relative call instruction is the direct call point to the original system call dispatch function; if they do not match, continue searching for the next relative call instruction within the preset scanning range of the IA-32 compatible entry function and repeat the comparison.

[0015] Furthermore, the address of the original system call dispatch function is saved as the original rollback path for subsequent operations.

[0016] S4. Modify the relative call instruction at the direct call point so that the relative call instruction points to a preset custom dispatch function; S5. In the custom dispatch function, if the current target system call number belongs to the preset target system call number set, the custom switch dispatch is executed according to the target system call number to dispatch the target system call to the corresponding custom processing function; otherwise, the address of the saved original system call dispatch function is called to maintain the original kernel behavior.

[0017] The custom processing function is a function written for a specific target system call and contains specific business logic.

[0018] Furthermore, the custom dispatch function has the same function signature as the original system call dispatch function, and both receive parameters including at least a pointer to the processor register state structure and the target system call number to ensure consistent calling conventions.

[0019] Furthermore, to avoid disrupting non-target system calls and special system calls, the custom processing function internally includes the following dual original fallback mechanism: If the custom processing function contains a raw 32-bit wrapper function corresponding to the current target system call, then the raw 32-bit wrapper function will be called first. Otherwise, the saved original system call dispatch function is invoked back; If neither of the above is available, return the error code agreed upon by the kernel module.

[0020] Furthermore, since different kernel module versions and distributions may use different IA32 entry paths, the method also sets up the following multi-entry self-healing mechanism: During the execution of the kernel module, the call status of the custom processing function is periodically monitored; If the custom processing function is not triggered after a preset time, a preset number of calls, or a preset system event, or if a certain type of 32-bit compatible program is detected not to have been hooked, a re-check of the IA-32 compatible entry function will be initiated. The re-examination process includes: (A) Reacquire the current addresses of multiple IA-32 compatible entry functions; The plurality of IA-32 compatible entry functions include at least two of do_int80_syscall_32, do_int80_emulation, do_fast_syscall_32, __do_fast_syscall_32, and do_SYSENTER_32; (B) For each reacquired IA-32 compatible entry function, scan and search for relative call instructions within their respective preset scan range, parse the displacement of the relative call instructions and calculate the actual target address, and compare the actual target address with the address of the original system call dispatch function that has been saved to confirm whether there is an unmodified direct call point to the original system call dispatch function. (C) For confirmed unmodified direct call points, modify the relative call instruction at the direct call point so that the relative call instruction points to the custom dispatch function; (D) For direct call points that have been modified, record the original target address and modification status to avoid applying patches repeatedly or causing error rollbacks; If the direct call point of at least one IA-32 compatible entry function is successfully modified, the repair success status is recorded, and operation continues to maintain continuous control over IA-32 compatible system calls.

[0021] Through multi-entry self-healing, this solution can still compensate for the takeover path at runtime, even if some kernel paths do not go through the originally selected IA-32 compatible entry function.

[0022] Self-built switch distribution can cover critical target system calls required for transparent file encryption / decryption, access control, auditing, or policy judgment. For target system calls that do not need to be taken over, their execution path and return value are not changed.

[0023] The beneficial effects of the system call takeover method adapted to Linux system compatibility mode and switch dispatch scenario are as follows: (1) The present invention does not rely on the function pointer table of the IA32 compatible system call table, but takes over by modifying the direct call point of the original dispatch function in the IA-32 compatible entry function, which directly acts on the actual dispatch path, fundamentally solving the core problem that the traditional table replacement scheme fails due to the use of switch direct dispatch in new kernels such as Linux 6.x; (2) Even if the ia32_sys_call_table symbol is hidden by the kernel or the table has been removed, the takeover deployment can still be completed, which greatly improves the adaptability of the scheme to different kernel compilation options and security hardening strategies; (3) Through parallel patching and other IA-32 compatible entry paths, combined with the runtime multi-entry self-repair mechanism, it ensures that no matter what instruction the 32-bit program uses to initiate the system call, it can be effectively taken over, avoiding the monitoring blind spot caused by different entry paths; (4) Through the layered dispatch in the custom dispatch function. The dual fallback mechanism within the logic and custom processing functions ensures that non-target system calls are returned to the original kernel dispatch logic without any abnormal return, thus ensuring full compatibility with existing 32-bit applications and not affecting the stability and normal function of the system; (5) During the operation of the kernel module, the takeover status is continuously monitored. Once the takeover path is found to be invalid, the multi-entry rescanning and patching process is automatically triggered. The takeover capability can be restored without reloading the kernel module or restarting the Linux system, effectively addressing the problem of takeover point failure caused by kernel minor version upgrades, online patches or runtime state changes, and ensuring the reliability and maintainability of long-term operation. Attached Figure Description

[0024] Figure 1This is a schematic diagram of the overall workflow of a system call takeover method adapted to Linux system compatibility mode and switch distribution scenario according to the present invention; Figure 2 This is a schematic diagram of the dual original fallback mechanism of the system call takeover method adapted to Linux system compatibility mode and switch distribution scenario in this invention; Figure 3 This is a schematic diagram of a multi-entry self-repair mechanism for a system call takeover method adapted to Linux system compatibility mode and switch distribution scenario according to the present invention. Detailed Implementation

[0025] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0026] like Figure 1 As shown, the system call takeover method adapted to Linux system compatibility mode and switch distribution scenarios of the present invention includes the following steps: S1. When a Linux system kernel module is loaded or initialized, it checks whether the current kernel environment of the Linux system can obtain a valid IA32-compatible system call table. S2. If a valid IA32-compatible system call table can be obtained, the original table entry is saved, and the table entry corresponding to the target system call number is replaced with a custom processing function, and then the step ends; if a valid IA32-compatible system call table cannot be obtained, the address of at least one IA-32-compatible entry function and the address of the original system call dispatch function are obtained, and then the process proceeds to step S3. S3. Using the address of the IA-32 compatible entry function as the starting base address, search for relative call instructions within a preset scanning range, parse the displacement of the relative call instructions and calculate the actual target address, compare the actual target address with the address of the original system call dispatch function, if they match, determine that the relative call instruction is the direct call point to the original system call dispatch function; if they do not match, continue searching for the next relative call instruction within the preset scanning range of the IA-32 compatible entry function and repeat the comparison. The address of the original system call dispatch function is also saved as the original rollback path for later use; S4. Modify the relative call instruction at the direct call point so that the relative call instruction points to a preset custom dispatch function; S5. In the custom dispatch function, if the current target system call number belongs to the preset target system call number set, the custom switch dispatch is executed according to the target system call number to dispatch the target system call to the corresponding custom processing function; otherwise, the address of the saved original system call dispatch function is called to maintain the original kernel behavior.

[0027] Example 1: This example describes how a kernel module adaptively selects a takeover path based on the current kernel environment during loading and initialization.

[0028] After a kernel module is loaded, an environment check is first performed to determine if the current kernel is a newer version that uses the syscall wrapper mechanism. Specifically, this is done by checking the kernel symbol table for wrapper-style naming symbols such as __x64_sys_open and __ia32_sys_open. If such wrapper-style naming symbols are detected, it indicates that the current kernel has introduced the syscall wrapper mechanism and a new takeover path is required.

[0029] The module then attempts to obtain the address of the IA32-compatible system call table (ia32_sys_call_table). Methods for obtaining this address include, but are not limited to: searching using the kallsyms_lookup_name function, parsing the / proc / kallsyms file, and scanning the symbol table of the kernel image.

[0030] If the address is successfully obtained, it indicates that the current kernel still retains the ia32_sys_call_table and its symbols are visible. The kernel module saves the pointer to the original handler function corresponding to the target system call number in the ia32_sys_call_table and replaces this entry with the address of the custom handler function, thereby completing the takeover of the target system call. This path is used for compatibility with kernel versions or configurations that still distribute data through ia32_sys_call_table.

[0031] If, after the above attempts, the address of ia32_sys_call_table cannot be obtained, or if at least one entry in ia32_sys_call_table is modified after testing, but no custom processing function is detected to be called within a preset time, then ia32_sys_call_table is determined to be unavailable or no longer used as an actual distribution path. In this case, the kernel module records this state and prepares to take over using the IA-32 compatible entry point call point modification method described in subsequent embodiments.

[0032] Example 2: This example describes how to take over the IA-32 compatible entry function by modifying the relative call instructions when ia32_sys_call_table is unavailable.

[0033] First, the kernel module searches for the address of the IA-32 compatible entry function, prioritizing the symbol address of `do_int80_syscall_32`. If this symbol does not exist or is not visible, it then searches for the symbol address of `do_int80_emulation`. Simultaneously, the kernel module searches for the symbol address of the original system call dispatch function `ia32_sys_call`.

[0034] Once the address of the IA-32 compatible entry function and the symbolic address of ia32_sys_call have been successfully obtained, the kernel module scans the code segment of the IA-32 compatible entry function. The scan range is a preset length (the first 256 bytes) starting from the start address of the IA-32 compatible entry function, within which a relative call instruction with opcode 0xE8 is searched.

[0035] For each relative call instruction found, the kernel module reads the displacement field in the relative call instruction and calculates the actual target address by combining it with the address of the next instruction of the relative call instruction.

[0036] The calculated actual target address is compared with the obtained ia32_sys_call address. If they match, the current relative call instruction is determined to be the direct call point of the original system call dispatch function in the IA-32 compatible entry function.

[0037] After determining the direct call point, the module calculates the displacement of the custom dispatch function `my_ia32_sys_call` relative to the current relative call instruction, and modifies the displacement field of the current relative call instruction to the new displacement, thus changing the execution target of the current relative call instruction from `ia32_sys_call` to `my_ia32_sys_call`. During the modification process, concurrent execution on other processors should be stopped or text segment write protection should be temporarily disabled. After the modification is completed, the instruction cache should be flushed to ensure immediate effect.

[0038] After the modification is completed, when a 32-bit compatible process enters the kernel module through the entry point corresponding to the successfully modified IA-32 compatible entry function, it will no longer call the original ia32_sys_call, but will instead enter the custom dispatch function my_ia32_sys_call.

[0039] Example 3: This example describes the processing logic for self-built switch distribution.

[0040] The custom dispatch function receives the same or compatible parameters as the original system call dispatch function, including at least a structure pointer regs pointing to the processor register state and the target system call number nr.

[0041] First layer: Enter the self-built switch dispatch branch. The switch statement uses the target system call number nr as the judgment condition, matches one by one with the preset target system call number set, and dispatches the target system call to the corresponding custom processing function.

[0042] When nr equals __NR_ia32_open, the custom open handler function is called; when nr equals __NR_ia32_read, the custom read handler function is called; when nr equals __NR_ia32_pread64, the custom compat pread64 handler function is called.

[0043] The second layer: For target system call numbers not covered in the switch statement, i.e. target system calls that do not belong to the preset set of target system call numbers, the custom dispatch function calls the saved original system call dispatch function ia32_sys_call(regs, nr), completely returning the processing power to the kernel module's native logic, so that non-target system calls continue to execute according to their original semantics, which is completely transparent to the caller.

[0044] Example 4: This example describes how a custom processing function can roll back to the original kernel implementation after completing its custom logic.

[0045] Taking the 32-bit pread64 system call as an example, the corresponding custom processing function first performs file access policy judgment, file path identification, or transparent encryption / decryption related operations. Then, when it needs to call the original kernel implementation to complete the actual system call function, such as... Figure 2 As shown, the custom processing function performs a double backoff check: The first step is to check if a raw 32-bit wrapper function exists that corresponds to the current target system call. If it does, the raw 32-bit wrapper function is called first, and the regs parameter is passed to it to complete the execution.

[0046] The second step is to roll back if the original 32-bit wrapper function does not exist, but the original system call dispatch function ia32_sys_call has been saved. Then, the rollback is performed by ia32_sys_call, which redistributes the system call to the corresponding kernel module implementation based on the target system call number.

[0047] If neither of the above two fallback paths is available, the kernel-defined error code -ENOSYS will be returned.

[0048] The double fallback mechanism ensures that even in complex situations such as missing entries in the ia32_sys_call_table or when the symbols of the original 32-bit wrapper function are not visible, the target system call can still invoke the original kernel module implementation, thus guaranteeing the integrity and compatibility of system functions.

[0049] Example 5: This example describes how to automatically repair and cover multiple entry points when a potential takeover path failure is detected during kernel module operation.

[0050] like Figure 3 As shown, during operation, the kernel module maintains a trigger counter or flag for each target system call to indicate whether the custom processing function has been actually called.

[0051] The kernel module checks the aforementioned counters or flags periodically or when a preset event is triggered. If it detects that a custom handler function for a target system call has not been invoked within a preset time, but there is an active 32-bit user-space process in the system, it determines that the current takeover path may have a blind spot or has failed, and triggers a self-repair process.

[0052] The self-healing process includes the following steps: First, retrieve the address of the following IA-32 compatible entry function: 1. do_int80_syscall_32, if it does not exist, then get do_int80_emulation; 2. do_fast_syscall_32; 3. __do_fast_syscall_32; 4. do_SYSENTER_32.

[0053] Then, for each IA-32 compatible entry function that successfully obtains an address, the scanning and matching process as in Example 2 is performed, that is, the relative call instruction that calls ia32_sys_call is searched in the code segment of the current IA-31 compatible entry function.

[0054] If a matching direct call point is found in an IA-32 compatible entry function, and this direct call point has not yet been patched (as determined by the maintained patch status record), then the displacement of the corresponding relative call instruction is modified so that displacement points to the custom dispatch function `my_ia32_sys_call`. After successful patching, the patch status record of the corresponding entry point for the current IA-32 compatible entry function is updated.

[0055] Through the aforementioned multi-entry self-repair mechanism, changes in the kernel module entry path can be automatically detected at runtime, and the uncovered entry paths can be compensated and repaired, thereby eliminating monitoring blind spots caused by 32-bit programs using different entry instructions.

[0056] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A system call takeover method adapted to Linux system compatibility mode and switch dispatch scenarios, characterized in that, Includes the following steps: S1. When a Linux system kernel module is loaded or initialized, it checks whether the current kernel environment of the Linux system can obtain a valid IA32-compatible system call table. S2. If a valid IA32-compatible system call table can be obtained, the original table entry is saved, and the table entry corresponding to the target system call number is replaced with a custom processing function, and then the step ends; if a valid IA32-compatible system call table cannot be obtained, the address of at least one IA-32-compatible entry function and the address of the original system call dispatch function are obtained, and then the process proceeds to step S3. The target system call number includes a 32-bit numerical identifier for target system calls, including file access, file status, memory mapping, read / write, truncation, sending files, and copying file ranges. S3. Using the address of the IA-32 compatible entry function as the starting base address, search for relative call instructions within a preset scanning range, parse the displacement of the relative call instructions and calculate the actual target address, compare the actual target address with the address of the original system call dispatch function, if they match, determine that the relative call instruction is the direct call point to the original system call dispatch function; if they do not match, continue searching for the next relative call instruction within the preset scanning range of the IA-32 compatible entry function and repeat the comparison. The address of the original system call dispatch function is also saved as the original rollback path for later use; S4. Modify the relative call instruction at the direct call point so that the relative call instruction points to a preset custom dispatch function; S5. In the custom dispatch function, if the current target system call number belongs to the preset target system call number set, then the custom switch dispatch is executed according to the target system call number to dispatch the target system call to the corresponding custom processing function; otherwise, the address of the saved original system call dispatch function is called to maintain the original kernel behavior. The custom processing function is a function written for a specific target system call and contains specific business logic.

2. The method according to claim 1, characterized in that, In step S1, the detection conditions for determining whether the current kernel environment of the Linux system can obtain a valid IA32-compatible system call table include the following conditions: (1) The current kernel module version of the Linux system is not lower than the preset version threshold; (2) The current kernel has enabled syscall wrapper configuration; (3) The symbol ia32_sys_call exists in Linux IA32 compatible systems, but ia32_sys_call_table cannot be obtained by scanning, symbol lookup or other methods; (4) At least one entry in the IA32 compatible system call table has been modified, but no custom processing function has been detected to be called within a preset time. If any one of the conditions (1)-(4) above is met, it is determined that the current kernel environment of the Linux system cannot obtain a valid IA32 compatible system call table.

3. The method according to claim 1, characterized in that, In step S2, when obtaining the address of at least one IA-32 compatible entry function, the search is performed in the following priority order: First, search for do_int80_syscall_32; if not found, search for do_int80_emulation; if still not found, then search for do_fast_syscall_32, __do_fast_syscall_32, do_SYSENTER_32, or their equivalent entry functions in sequence.

4. The method according to claim 1, characterized in that, The custom dispatch function has the same function signature as the original system call dispatch function, and both receive parameters including at least a pointer to the processor register status structure and the target system call number to ensure consistent calling conventions.

5. The method according to claim 1, characterized in that, The custom processing function has the following dual initial rollback mechanism: If the custom processing function contains a raw 32-bit wrapper function corresponding to the current target system call, then the raw 32-bit wrapper function will be called first. Otherwise, the saved original system call dispatch function is invoked back; If neither of the above is available, return the error code agreed upon by the kernel module.

6. The method according to claim 1, characterized in that, The following multi-entry self-healing mechanism is also included: During the execution of the kernel module, the call status of the custom processing function is periodically monitored; If the custom processing function is not triggered after a preset time, a preset number of calls, or a preset system event, or if a certain type of 32-bit compatible program is detected not to have been hooked, a re-check of the IA-32 compatible entry function will be initiated. If the direct call point of at least one IA-32 compatible entry function is successfully modified, the repair success status is recorded, and operation continues to maintain continuous control over IA-32 compatible system calls.

7. The method according to claim 6, characterized in that, The re-checking process in the multi-entry self-healing mechanism includes: (A) Reacquire the current addresses of multiple IA-32 compatible entry functions; The plurality of IA-32 compatible entry functions include at least two of do_int80_syscall_32, do_int80_emulation, do_fast_syscall_32, __do_fast_syscall_32, and do_SYSENTER_32; (B) For each reacquired IA-32 compatible entry function, scan and search for relative call instructions within their respective preset scan range, parse the displacement of the relative call instructions and calculate the actual target address, and compare the actual target address with the address of the original system call dispatch function that has been saved to confirm whether there is an unmodified direct call point to the original system call dispatch function. (C) For confirmed unmodified direct call points, modify the relative call instruction at the direct call point so that the relative call instruction points to the custom dispatch function; (D) For direct call points that have been modified, record the original target address and modification status to avoid applying patches repeatedly or causing error rollbacks.

Images