Execute system calls in the isolated address space of the operating system kernel

Abstract

Embodiments of the present systems and methods may provide additional security mechanisms within the operating system kernel itself by executing system calls in a dedicated address space to reduce the amount of shared resources visible to and therefore exploitable by malicious applications. For example, in an embodiment, a method implemented in a computer may include a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method may include: when a user process makes a system call, switching to kernel mode and using a system call page table for the user process to execute a system call handler, when the system call handler attempts to access unmapped kernel space memory, generating a page fault, and handling the page fault by determining whether the attempted access to the unmapped kernel space memory is allowed.

Description

Background Art

[0001] The present invention relates to techniques for providing additional security mechanisms within the operating system kernel itself by executing system calls in a private address space to reduce the amount of shared resources that are visible to and therefore exploitable by malicious applications.

[0002] System calls are an important part of the kernel application binary interface (ABI) of operating systems (such as Linux) that are exposed to user space applications. When a system call is executed, the CPU transitions from unprivileged mode (e.g., ring-3 on x86) to supervisor mode (e.g., ring-0 on x86) and the routine called during the execution of the system call has the highest privilege level and therefore has the ability to access and modify system resources. Malicious user space applications can exploit vulnerabilities in the system call implementation to gain access to these system resources and cause the system to crash, read sensitive information, or obtain superuser privileges.

[0003] Microkernels avoid extensive exposure of privileged state by minimizing the amount of code running in supervisor mode and executing most operating system services with user privileges in isolated address spaces. The main disadvantages of the microkernel approach are the performance costs associated with inter-process communication and the complexity of the interfaces between different components of the system.

[0004] As a mitigation for the Meltdown vulnerability, the Page Table Isolation (PTI) mechanism, which completely isolates the user and kernel address spaces, was added to the Linux kernel. However, even with PTI, the entire kernel code shares the same address space, and an exploit in any system call handler makes the entire system vulnerable.

[0005] Thus, a need has arisen for techniques to provide additional security mechanisms within the operating system kernel itself. Summary of the invention

[0006] Embodiments of the present systems and methods may provide additional security mechanisms within the operating system kernel itself by executing system calls in a private address space to reduce the amount of shared resources that are visible to, and therefore exploitable by, malicious applications.

[0007] In an embodiment, when a user space application executes a system call, the kernel entry code may select an address space appropriate for the execution of that particular system call. If the execution of the system call attempts to access memory outside of that address space, a page fault occurs. The page fault handler may verify whether the requested memory range is allowed to be accessed by the system call handler routine, and if so, this range may be added to the private address space. Upon completion of the system call, the address space definition may be restored to the original.

[0008] For example, in an embodiment, a method implemented in a computer, which may include a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, may include: when a user process makes a system call, switching to kernel mode and using a system call page table for the user process to execute a system call handler, when the system call handler attempts to access unmapped kernel space memory, generating a page fault, and handling the page fault by determining whether the attempted access to the unmapped kernel space memory is allowed.

[0009] In an embodiment, the computer system operating system may be a Linux operating system. The system call page table may include multiple kernel space address ranges that allow the execution of kernel code and access to kernel data. The system call page table may be created in the following manner: creating a template system call page table, the template system call page table includes a mapping of kernel code and data required to execute an entry to kernel mode when executing a system call, a table with a pointer to a system call handler, and an entry code for each system call handler, when a user process is created, creating a copy of the template system call page table for the user process in the process state representation, and filling the per-process entry page table with the user space mapping together with the original template, and when it is determined that a page fault is allowed, adding a mapping of the kernel space address range that allows the execution of the system call handler that attempts to access the kernel space memory. The method may further include: when it is determined that a page fault is not allowed, terminating the execution of the user process that makes a system call that attempts to access the kernel space memory. The method may further include: when the execution of the system call handler ends, removing the mapping of the kernel space address range added during the execution of the system call handler.

[0010] In an embodiment, a system may include a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform the following operations: when a user process makes a system call, switch to kernel mode and execute a system call handler using a system call page table for the user process, generate a page fault when the system call handler attempts to access unmapped kernel space memory, and handle the page fault by determining whether the attempted access to the unmapped kernel space memory is allowed.

[0011] In an embodiment, a computer program product may include a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions being executable by a computer to cause the computer to perform a method, the method may include: when a user process makes a system call, switching to kernel mode and executing a system call handler using a system call page table for the user process, generating a page fault when the system call handler attempts to access unmapped kernel space memory, and handling the page fault by determining whether the attempted access to the unmapped kernel space memory is allowed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The details of the present invention, both as to its structure and operation, may be best understood by reference to the accompanying drawings, in which like reference numerals and designations refer to like elements.

[0013] Figure 1 is an exemplary schematic diagram of a system memory space according to an embodiment of the present system and method.

[0014] Figure 2 is an exemplary flow chart of the process of operation of an embodiment of the present system and method.

[0015] Figure 3 is an exemplary block diagram of a computer system in which the processes involved in the embodiments described herein may be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present system and method can provide additional security mechanisms within the operating system kernel itself by performing system calls in a dedicated address space to reduce the amount of shared resources visible to malicious applications and therefore available for use by malicious applications. Conventional monolithic operating system kernels (such as the Linux kernel) that only run code in a single address space can be modified to run code in multiple address spaces to increase security. Additional attributes can be used to enhance security. For example, in a system call environment, code in multiple address spaces can be run in kernel (privileged) mode, which can be used to ensure that if the entity executing the system call attempts access outside its allowed address space, the attempted access can be safely detected and processed. Address spaces can be created per user process and can be used to implement security attributes on user processes.

[0017] In an embodiment, when a user space application executes a system call, the kernel entry code may select an address space appropriate for the execution of that particular system call. If the execution of the system call attempts to access memory outside of that address space, a page fault occurs. The page fault handler may verify whether the requested memory range is allowed to be accessed by the system call handler routine, and if so, this range may be added to the private address space. Upon completion of the system call, the address space definition may be restored to the original.

[0018] Figure 1 An exemplary schematic diagram of system memory space 100 is shown in FIG. This example shows kernel page table 102, system call page table 104, and user page table 106. In addition, there may be a template system call page table 126 described below. In this example, kernel page table 102 may be associated with a specific user process and may include user space entry 108, kernel entry page table 110, and kernel space entry 112. User space entry 108 may include user space address mappings used when the system is running in kernel (privileged) mode. Kernel space entry 112 may include kernel space address mappings used when the system is running in kernel (privileged) mode. Kernel entry page table 110 may include a CPU entry area that maps the structures necessary to perform a switch from unprivileged to kernel (privileged) mode. User page table 106 may be associated with a specific user space process, and user space entry 114 may only include user space mappings used by the user space process. Kernel entry page table 116 may include a CPU entry area that maps the structures necessary to perform a switch from unprivileged to kernel (privileged) mode. This separation of page tables, including kernel entry page tables 110, 116, and 120, may mitigate security vulnerabilities.

[0019] User processes typically run in unprivileged mode and have access to user page tables 106 that include only user space entries 114 and kernel entry page tables 116. Previously, when a user process makes a system call, the system would switch from unprivileged mode to kernel (privileged) mode, and the process would have access to the entire kernel page table including user space entries 108, kernel entry page tables 110, and kernel space entries 112. However, in embodiments of the present systems and methods, when a user process makes a system call, the system may switch from unprivileged mode to kernel (privileged) mode, and the process may access system call page tables 104. System call page tables 104 may extend user process visible mappings with additional system call (syscall) entry page tables 122. Syscall entry page tables 122 may allow a user process to access a limited kernel space address range to execute limited additional portions of code or access to data after switching from unprivileged mode to kernel (privileged) mode. Syscall entry page table 122 may provide access to a portion of code that is restricted enough to prevent malicious access to kernel code and data, but provides access to more code than is accessible from user space entry 118. When code is accessed through syscall entry page table 122, secure code block 124 may verify that any access made is safe. In an embodiment, any of a variety of verifications may be performed by secure code block 124. For example, in an embodiment, secure code block 124 may verify that access is made only to known symbols.

[0020] Figure 2 An exemplary flow chart of a process 200 of operation of an embodiment of the present system and method is shown in FIG. Figure 1 Process 200 begins at 202, where during Linux kernel initialization, a Figure 3 The additional template system call page table 126 is shown. The template system call page table 126 can be used to create a functional system call page table 104 for executing system call handlers in an isolated address space. The template system call page table 126 can have a mapping 128 of kernel code and data required to execute an entry to kernel (privileged) mode when executing a system call, a table with pointers to system call handlers, and an entry code for each system call handler in the syscall entry page table 122. In addition, the template system call page table 126 can have a populated kernel entry page table 120 and an empty user space page table 118.

[0021] At 204, when a process is created, the template system call page table 126 may be copied to the process state representation to form the operating system call page table 104. In addition to the original template including the populated kernel entry page table 120 and syscall entry page table 122, the empty user space page table 118 of the per-process system call page table 104 may be populated with user space mappings in the same manner as a normal process page table.

[0022] At 206, the user space process may execute a system call. The CPU may transition to kernel (privileged) mode and may call a generic system call entry routine. The routing may switch the CPU page table pointer register to use the system call page table 104 for the process and may jump to a specific system call handler.

[0023] At 208, during execution of the system call handler, any access to kernel space kernel code and / or data may cause a page fault. The page fault handler of the security block 124 may verify whether the requested memory access is safe or allowed. If the access is found to be safe or allowed, a mapping of the requested memory range may be added to the per-process system call page table 104. If the access is deemed unsafe or not allowed, the user space process may be terminated and / or other security actions may be taken.

[0024] When execution of the system call handler ends, control may be transferred back to the general system call entry routine at 210. This routine may clean up the per-process system call page table 104 and may remove mappings added during execution of the system call handler.

[0025] Figure 3 , an exemplary block diagram of a computer system 300 in which the processes involved in the embodiments described herein may be implemented. The computer system 300 may be implemented using one or more programmed general-purpose computer systems (e.g., embedded processors, systems on a chip, personal computers, workstations, server systems, and minicomputers or mainframe computers) or in a distributed networked computing environment. The computer system 300 may include one or more processors (CPUs) 302A-302N, input / output circuits 304, a network adapter 306, and a memory 308. The CPUs 302A-302N execute program instructions to perform the functions of the present communication systems and methods. Typically, the CPUs 302A-302N are one or more microprocessors, such as INTEL processor. Figure 3An embodiment in which the computer system 300 is implemented as a single multi-processor computer system is shown, in which multiple processors 302A-302N share system resources, such as memory 308, input / output circuitry 304, and network adapter 306. However, the present communication systems and methods also include embodiments in which the computer system 300 is implemented as multiple networked computer systems, which may be single-processor computer systems, multi-processor computer systems, or a mixture thereof.

[0026] Input / output circuitry 304 provides the ability to input data to or output data from computer system 300. For example, input / output circuitry may include input devices such as keyboards, mice, touch pads, trackballs, scanners, analog-to-digital converters, etc., output devices such as video adapters, monitors, printers, etc., and input / output devices such as modems, etc. Network adapter 306 interfaces device 300 to network 310. Network 310 may be any public or private LAN or WAN, including but not limited to the Internet.

[0027] The memory 308 stores program instructions executed by the CPU 302 and data used and processed by the CPU 302 to perform the functions of the computer system 300. The memory 308 may include, for example, electronic memory devices such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc., as well as electromechanical storage such as a disk drive, a tape drive, an optical drive, which may use an integrated drive electronics (IDE) interface or a variation or enhancement thereof, such as enhanced IDE (EIDE) or ultra direct memory access (UDMA), or an interface based on a small computer system interface (SCSI) or a variation or enhancement thereof, such as fast SCSI, wide SCSI, fast and wide SCSI, serial advanced technology attachment (SATA) or a variation or enhancement thereof, or a fiber channel arbitrated loop (FC-AL) interface.

[0028] The contents of memory 308 may vary depending on the functions that computer system 300 is programmed to perform. Figure 3 In the illustrated example, exemplary memory contents representing routines and data for embodiments of the above-described processes are shown. However, those skilled in the art will recognize that, based on well-known engineering considerations, these routines and memory contents associated with these routines may not be included in one system or device, but may be distributed among multiple systems or devices. The present communication systems and methods may include any and all such arrangements.

[0029] exist Figure 3In the example shown, memory 308 may include kernel space 312, user space 314, security block routines 324, and other operating system routines 322. Kernel space 312 may include code and data that resides in an elevated system state, and it may include protected memory space and full access to hardware. Kernel space 312 may include kernel page tables 316, system call (syscall) page tables 318, and template system call page tables 320. Kernel page tables 316 may be associated with a particular user process and may include user space page tables 108, kernel entry page tables 110, and kernel space page tables 112, as shown in FIG. Figure 1 The system call page table 318 may be extended with additional syscall entry page tables 122 to extend the user process visible mappings, such as Figure 1 , and as described above. The template system call page table 320 can be used to create a functional system call page table 104 for executing a system call handler in an isolated address space, as described above. The user space 314 may contain user applications, programs, tasks, processes, etc., which can be executed with reduced privileges, where they can only access a subset of the available resources of the hardware and software. The user space 314 may include a user page table 322 that can be mapped to the user space unprivileged access. The security block routine 324 can verify that any access made using the system call page table 318 is safe or allowed. Other operating system routines 322 can provide additional system functions.

[0030] like Figure 3 As shown, the communication system and method may include implementations on one or more systems that provide multi-processor, multi-tasking, multi-process and / or multi-threaded computing, as well as implementations on systems that only provide single-processor, single-threaded computing. Multi-processor computing involves using more than one processor to perform computing. Multi-tasking computing involves using more than one operating system task to perform computing. A task is an operating system concept that refers to a combination of an executing program and bookkeeping information used by the operating system. Whenever a program is executed, the operating system creates a new task for it. A task is like an envelope for a program because it identifies the program with a task number and attaches other bookkeeping information to the program. Many operating systems (including Linux, and ) are capable of running many tasks simultaneously and are known as multitasking operating systems. Multitasking is the ability of an operating system to execute more than one executable file at the same time. Each executable file runs in its own address space, which means that the executable files cannot share any of their memory. This has advantages because it is impossible for any program to damage the execution of any other program running on the system. However, there is no way for programs to exchange any information except through the operating system (or by reading files stored on the file system). Multiprocess computing is similar to multitasking computing in that the terms task and process are often used interchangeably, although some operating systems distinguish between the two.

[0031] The present invention may be a system, method and / or computer program product of any possible integrated technical detail level. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform aspects of the present invention. A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.

[0032] The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, a computer readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0033] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards these computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0034] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA)) can execute a computer-readable program instruction to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present invention.

[0035] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0036] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that is executed by a processor of a computer or other programmable data processing device to create a device for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that may direct a computer, a programmable data processing device, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored therein includes an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0037] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operating steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0038] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to different embodiments of the present invention.To this end, each box in the flow chart or block diagram can represent a part of a module, segment or instruction, which includes one or more executable instructions for realizing the logical function of the specification.In some alternative embodiments, the function marked in the box may not occur in the order marked in the figure.For example, depending on the function involved, the two boxes shown in succession can actually be performed substantially at the same time, or these boxes can sometimes be performed in reverse order.It will also be noted that each box in the block diagram and / or flow chart and the combination of the boxes in the block diagram and / or flow chart can be realized by a system based on special-purpose hardware, and the system based on special-purpose hardware performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0039] Although specific embodiments of the present invention have been described, those skilled in the art will appreciate that there are other embodiments that are equivalent to the described embodiments. Therefore, it should be understood that the present invention is not limited to the specific embodiments described, but only to the scope of the appended claims.

Claims

1. A method implemented in a computer, the computer comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising: When a user process makes a system call, switching to kernel mode, using a system call page table for the user process to execute a system call handler; When the system call handler attempts to access unmapped kernel space memory, a page fault is generated; and handling the page fault by determining whether the attempted access to unmapped kernel space memory is allowed; Wherein, the system call page table includes: a plurality of kernel space address ranges that allow execution of kernel code and access to kernel data; and wherein the system call page table is created by the following steps: creating a template system call page table including a mapping of kernel code and data required to execute an entry to kernel mode when a system call is executed, a table with pointers to system call handlers, and an entry code for each system call handler; When the user process is created, creating a copy of the template system call page table for the user process in a process state representation to form a system call page table, and populating the system call page table with a user space mapping; and When it is determined that page faults are allowed, a mapping of a kernel space address range is added that allows execution of the system call handler that attempts to access kernel space memory.

2. The method of claim 1, wherein: The computer system operating system is the Linux operating system.

3. The method of claim 1, further comprising: When it is determined that the page fault is not allowed, execution of the user process that made the system call attempting to access kernel space memory is terminated.

4. The method of claim 3, further comprising: When execution of the system call handler ends, the mapping of the kernel space address range added during execution of the system call handler is removed.

5. A system comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to: When a user process makes a system call, switching to kernel mode, executing a system call handler using a system call page table for the user process; When the system call handler attempts to access unmapped kernel space memory, a page fault is generated; and handling the page fault by determining whether the attempted access to unmapped kernel space memory is allowed; in, The system call page table includes: a plurality of kernel space address ranges that allow execution of kernel code and access to kernel data; and wherein the system call page table is created by the following steps: creating a template system call page table including a mapping of kernel code and data required to execute an entry to kernel mode when a system call is executed, a table with pointers to system call handlers, and an entry code for each system call handler; When the user process is created, creating a copy of the template system call page table for the user process in a process state representation to form a system call page table, and populating the system call page table with a user space mapping; and When it is determined that page faults are allowed, a mapping of a kernel space address range is added that allows execution of the system call handler that attempts to access kernel space memory.

6. The system of claim 5, wherein: The computer system operating system is the Linux operating system.

7. The system of claim 5, further comprising: When it is determined that the page fault is not allowed, execution of the user process that made the system call attempting to access kernel space memory is terminated.

8. The system of claim 7, further comprising: When execution of the system call handler ends, the mapping of the kernel space address range added during execution of the system call handler is removed.

9. A computer program product comprising program instructions, wherein the program instructions are executable by a computer to cause the computer to perform a method comprising: When a user process makes a system call, switching to kernel mode, executing a system call handler using a system call page table for the user process; When the system call handler attempts to access unmapped kernel space memory, a page fault is generated; and handling the page fault by determining whether the attempted access to unmapped kernel space memory is allowed; Wherein, the system call page table includes: a plurality of kernel space address ranges that allow execution of kernel code and access to kernel data; and wherein the system call page table is created by the following steps: creating a template system call page table including a mapping of kernel code and data required to execute an entry to kernel mode when a system call is executed, a table with pointers to system call handlers, and an entry code for each system call handler; When the user process is created, creating a copy of the template system call page table for the user process in a process state representation to form a system call page table, and populating the system call page table with a user space mapping; and When it is determined that page faults are allowed, a mapping of a kernel space address range is added that allows execution of the system call handler that attempts to access kernel space memory.

10. The computer program product of claim 9, wherein: The computer system operating system is the Linux operating system.

11. The computer program product of claim 9, further comprising: When it is determined that the page fault is not allowed, execution of the user process that made the system call attempting to access kernel space memory is terminated.

12. The computer program product of claim 11, further comprising: When execution of the system call handler ends, the mapping of the kernel space address range added during execution of the system call handler is removed.

Images