Protection method and protection system for executable file and shared library

Abstract

A protection method and protection system for an executable file and a shared library, comprising: encrypting a part of contents of the executable file and the shared library, decrypting in real time when executing the executable file and the shared library, and prohibiting a debugged process from executing the encrypted part of contents in the executable file and the shared library, therefore, an attacker is prevented from acquiring the executable file and the part of content in the shared library.

Description

Technical Field

[0001] The present invention relates to protection technologies for executable files and shared libraries, and particularly to anti-reverse engineering, anti-debugging, or anti-detection technologies for executable files and shared libraries. Background Art

[0002] Computer-executable software is usually packaged in the form of executable files or shared libraries. The purpose of binary obfuscation for executable files or shared libraries is to increase software security and prevent reverse engineering. Through binary obfuscation, the execution logic of the software can be made more difficult to understand and analyze, making it difficult for attackers to analyze and understand the internal operation of the program. This can prevent the theft of intellectual property rights, hinder reverse engineering, or make it difficult for attackers to identify vulnerabilities to attack.

[0003] Currently, code restructuring and instruction replacement are two common code obfuscation methods.

[0004] Code restructuring is to restructure the structure of binary program code to make it more difficult to understand. This can include changing the function order of the program, adjusting the position of code blocks, inserting invalid instructions or useless code fragments to interfere with analysts' understanding of the code flow and logic of the program.

[0005] Instruction replacement is to replace the original binary instructions to make the execution path and logic of the program more obscure and difficult to understand. The above replacement can be based on fixed conversion rules or dynamically generated rules. For example, replacing binary instructions with equivalent but more complex instruction sequences, or using similar instructions for replacement to make the behavior of the program more unpredictable.

[0006] These methods can be used in combination with other obfuscation technologies, such as instruction insertion, false control flow, invalid instructions, or data insertion. Using multiple technologies in combination can increase the difficulty for analysts to understand and parse binary program code to improve software security and the ability to resist reverse engineering.

[0007] Although code restructuring and instruction replacement are common binary code obfuscation methods, they cannot completely prevent attackers from analyzing the running process, and there are the following disadvantages:

[0008] First, it increases execution time and performance overhead: Obfuscation techniques usually introduce additional computations and runtime processing, which may increase the execution time and performance overhead of the program. For example, code restructuring and instruction replacement may result in additional computations during execution to restore the original program flow and logic, which may lead to performance degradation.

[0009] Second, it increases development and maintenance costs: Obfuscating the program code may increase the complexity of development and maintenance. Obfuscation techniques may require special tools and processes to handle the obfuscated code. In addition, since the obfuscated code may be difficult to understand, more labor and time may be required during development and maintenance.

[0010] Third, reversibility: Most obfuscation techniques are reversible, which means that attackers can perform reverse engineering to restore the original program code. For example, code restructuring and instruction replacement can be reverse engineered through corresponding de-obfuscation techniques to restore the original program structure and logic.

[0011] Fourth, difficulties in pointer and data handling: Some obfuscation techniques may make it difficult to handle pointers and data. For example, instruction replacement may change the way pointers or data are used, making it more difficult to handle pointers and data.

[0012] Fifth, incomplete protection: Obfuscation techniques can increase the difficulty of analysis and reverse engineering, but they cannot provide absolute protection. With enough time and effort, experienced attackers may still be able to decode or restore the obfuscated program code. Obfuscation techniques should be used as part of a security strategy in combination with other defensive measures (such as encryption, integrity checks, and authorization mechanisms) to improve the overall security of the software.

[0013] In summary, more rigorous protection techniques are currently needed. Summary of the Invention

[0014] To solve the above problems, the present invention provides a method for protecting executable files and shared libraries, which is executed by at least one processor of an electronic device in an operating system of the electronic device. The protection method includes the following steps: The electronic device determines whether a first process is being debugged or is formed after a second executable file is executed. Wherein, if the first process is being debugged and is about to execute a first executable file, and the first executable file has been encrypted, then the first process is refused to execute the first executable file; if the first process is being debugged and is about to perform a memory mapping on a shared library, and the shared library has been encrypted, then the first process is refused to perform the memory mapping on the shared library; and, if the first process is formed after the second executable file is executed, and the second executable file has been encrypted, and a second process is about to debug the first process, then the second process is refused to debug the first process.

[0015] In an embodiment, after the first process has performed the memory mapping on the shared library, the data protection method further includes: if the shared library has been encrypted, and the second process is about to debug the first process, then the second process is refused to debug the first process.

[0016] The present invention further provides a computer-readable storage medium storing a plurality of instructions, which are read by an electronic device to execute the above protection method.

[0017] The present invention further provides a protection system for executable files and shared libraries, including: a storage device installed with an operating system; and at least one processor for executing the operating system and executing the above protection method in the operating system.

[0018] The present invention uses anti-reverse engineering and anti-debugging techniques to protect executable files and shared libraries to prevent attackers from obtaining the decrypted content in the executable files and shared libraries through reverse engineering or debugging. In addition, the present invention is also applicable to executable files and shared libraries that have been binary obfuscated. Brief Description of the Drawings

[0019] Figures 1 to 5 It is a flowchart of the protection method according to an embodiment of the present invention.

[0020] Figure 6A 、 Figure 6B and Figure 7 It is a flowchart of the protection method according to another embodiment of the present invention.

[0021] Figure 8 It is a block diagram of the protection system according to an embodiment of the present invention.

[0022] Description of the Main Component Symbols

[0023] Steps 11 - 14, 21 - 24, 31 - 34, 41 - 43, 51 - 58, 601 - 618, 71 - 78 80 Electronic device

[0024] 81 Processor

[0025] 82 Memory

[0026] 83 Storage device. Detailed implementation manners

[0027] The following uses specific specific embodiments to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0028] The protection method of the present invention is applicable to the Linux operating system. Before detailing the protection method of the present invention, the following technical details of the Linux operating system are first described to facilitate the understanding of the protection method of the present invention.

[0029] Executable file

[0030] An executable file is a computer file in binary format, which contains compiled program code and related resources and can be directly run on a computer.

[0031] In software development, program code usually goes through a compilation process to convert high - level languages (such as C, C++, Java, etc.) into machine code, which are binary instructions that can be directly executed by a processor. The compilation process includes steps such as syntax checking, semantic analysis, code generation, and optimization, and finally generates one or more executable files.

[0032] The format and structure of executable files may vary depending on the operating system and execution environment. For example, in Linux and UNIX operating systems, the common executable file format is the Executable and Linkable Format (ELF), while in the Windows operating system, the common executable file format is the Portable Executable (PE) format.

[0033] Shared library

[0034] A shared library, also known as a Dynamic Link Library (DLL), is a library that can be shared and used by multiple executable files. It is a binary file that contains functions and resources required by multiple executable files.

[0035] Common shared library formats include the ELF format for Linux and UNIX operating systems and the DLL format for Windows operating systems. Shared libraries can be provided by developers, the operating system, or a third party, and can be used through a linker or dynamic linking when the executable file is running.

[0036] ELF file format

[0037] The ELF file format is widely used for executable files, shared libraries, and object files in Linux and UNIX operating systems. It defines the structure and organization of the file to facilitate the loading, execution, and linking of program code in the operating system. The following are some of the main components of the ELF format:

[0038] ELF file header: The ELF file header is located at the beginning of the file and contains important information describing the entire ELF file, such as file type, target architecture, entry point address, section table, and flags.

[0039] Program segment: The program segment defines the layout of the executable file in memory. Each program segment describes the memory blocks required for the execution of the executable file, such as the code segment, data segment, and stack segment. Each segment has its own attributes, such as location, size, access permissions, etc.

[0040] Section table: The section table contains descriptions and attributes of all sections. It provides indexes and links to program segments, shared libraries, symbol tables, relocation tables, and other information. Each section also contains a flag field to indicate the attributes of the section.

[0041] The ELF format is scalable and flexible and supports various types of target files, including executable files, shared libraries, object files, and core dump files.

[0042] Execute an ELF executable file

[0043] When an ELF executable file is executed, it needs to be mapped into the virtual address space of the virtual memory so that the processor can execute the instructions and access the data therein. This mapping process involves two main steps: page table mapping and demand paging.

[0044] Regarding page table mapping:

[0045] Each process formed when an executable file is executed has its own virtual address space, which is usually divided into multiple pages or segments. The size of each page or segment is usually fixed, for example, 4KB.

[0046] The physical address space of the physical memory is divided into multiple page frames, which are the same size as the pages or segments. A page frame is a contiguous area in the physical memory, and its size is usually also fixed, for example, 4KB.

[0047] A page table is a data structure used to map the pages in the virtual address space to the actual page frames. Each page table entry in the page table corresponds to a page in the virtual address space. Each page table entry contains information about the mapping relationship between the page and the page frame, such as the starting address and permissions of the page frame.

[0048] When an ELF executable file is loaded into the memory, the operating system will create corresponding page table entries according to the structure of the ELF file and the information of the sections, mapping the pages in the virtual address space to the actual page frames. Thus, the instructions and data of the executable file can be accessed through the virtual address.

[0049] Regarding demand paging:

[0050] When an ELF executable file is mapped into the virtual address space, not all pages are immediately loaded into the memory. Instead, only when the content of a specific page needs to be accessed, that page will be read from a storage device such as a disk into the memory. This mechanism of loading on demand is called demand paging.

[0051] Demand paging is usually implemented through a paging mechanism. When a process executes to a page that has not been loaded into the memory, the processor will trigger a page fault interrupt. After receiving this interrupt, the operating system will be responsible for reading the page from the storage device into an idle page frame, and then updating the page table entry to point the mapping of the page to the new page frame.

[0052] The advantage of demand paging is to save memory space because the content of a page is loaded only when needed. At the same time, it also allows more executable files to be loaded into the virtual address space, and the size of the virtual address space of the processes of these executable files can exceed the actual available memory.

[0053] In summary, when an ELF executable file is executed, it is mapped into the virtual address space of the process of that executable file. This mapping process involves page table mapping and the demand paging mechanism. Page table mapping maps the pages in the virtual address space to the page frames in the actual memory, while demand paging loads the content of the pages into the memory on demand to save memory space and allow the content of more executable files to be loaded into the virtual address space.

[0054] Page cache

[0055] In the Linux kernel, the page cache is a mechanism for caching pages in the file system. It provides a cache system based on pages as the basic unit to accelerate file read and write operations. The main features of the page cache are as follows:

[0056] First, cache at the page level: The page cache caches pages in units of pages (usually 4KB in size). When a file is read, the relevant pages are read from storage devices such as hard disks into the page cache; when a file is written, the relevant pages are first cached in the page cache and then written back to the storage device later.

[0057] Second, fast reading: If a page in a file already exists in the page cache, when that page needs to be read, it can be directly read from the page cache without accessing the storage device, thus improving the reading efficiency.

[0058] Third, delayed writing: When a file is written, the relevant pages are first written into the page cache instead of being immediately written to the storage device. This delayed writing strategy can combine multiple scattered write operations and write them together, thus improving the writing efficiency.

[0059] Fourth, pre-fetch: The page cache can pre-read the pages in a file into the page cache according to a prediction algorithm to pre-load the pages that may be needed, thus reducing the latency of subsequent read operations.

[0060] Fifth, sharing: The page cache is shareable, that is, multiple processes accessing the same file can share the same page in the page cache. This can save memory and improve the sharing and cooperation efficiency of files.

[0061] The page cache plays an important role in Linux. By providing a page-based caching mechanism, it accelerates the reading and writing operations of files. It not only offers efficient reading capabilities but also optimizes writing performance through strategies such as delayed writing and prefetching. Through its sharing feature, multiple processes can share pages in the page cache to improve the efficiency of file sharing and collaboration.

[0062] Linux virtual file system

[0063] Linux divides the file system into two layers, namely the virtual file system (VFS) and the physical file system.

[0064] The Linux operating system supports multiple physical file systems, such as the common ext2, ext3, ext4, xfs, zfs file systems, etc.

[0065] The virtual file system belongs to the Linux kernel software layer and is a software abstraction layer implemented on top of the physical file system. It is used to receive file system-related system calls and forward these system calls to the system interfaces implemented by the physical file system. The Linux virtual file system includes important objects such as the index node (inode) and the directory entry (dentry).

[0066] File structure of Linux

[0067] In the Linux kernel, the file structure (struct file) is a data structure representing an open file. Each file structure in the kernel is used to manage an open file and store various information about that file. Each file structure includes at least three pointers, namely f_op, f_security, and f_inode.

[0068] f_op is a pointer to the file operation structure of the file. This file operation structure contains pointers to functions (or routines) related to various operations on the file, such as function pointers for reading, writing, seeking, etc. Through the f_op pointer, the Linux kernel can call the appropriate functions to operate on the open file.

[0069] f_security is a pointer to the security context used by the mandatory access control (MAC) module.

[0070] f_inode is a pointer to the inode of the file. The inode contains the metadata of the file, such as the size and access permissions of the file.

[0071] The file structure plays a crucial role in the Linux kernel. It allows the Linux kernel to track and manipulate information such as the open state of the file, the current access position, and permissions.

[0072] Task structure of Linux

[0073] The task_struct in the Linux kernel is a data structure used to represent a process or thread and is used to store basic information and status related to the process or thread. The following are some uses and functions of the task structure:

[0074] Thread management: The task structure is used to manage the threads in Linux. It stores information about the state, priority, and scheduling attributes of the threads. It allows the kernel to schedule and switch between threads to ensure fair and efficient execution.

[0075] Thread identification: The task structure includes fields such as the process identification number (pid) and the thread group identification number (tgid) to uniquely identify each process or thread in the operating system. These identification numbers are used for thread management, as well as communication and resource allocation between threads.

[0076] Security context: The task structure includes a pointer to the security context of the process or thread to which the task structure belongs. This security context is used by the Linux security module to record some states.

[0077] ptrace: The task structure includes a ptrace field used to record debugging information about the process or thread.

[0078] Linux security module

[0079] The Linux Security Module (LSM) is a framework in the Linux kernel for supporting various computer security models. The Linux Security Module provides the functions required for Mandatory Access Control (MAC) while minimizing modifications to the Linux kernel. This framework provides a mechanism to hook multiple security checks into the Linux kernel.

[0080] LSM provides a memory block called the security context for important objects in the Linux kernel (such as the task structure and the inode) so that implementations of various security models can store their respective information.

[0081] Implementation details: The protection method of the present invention is described below through embodiments

[0082] The protection method of the present invention can be executed by one or more processors of an electronic device such as a computer, a server, or a network attached storage (NAS) in the operating system (such as Linux) of the electronic device. The implementation details of the protection method of the present invention are described below.

[0083] Taking an ELF file as an example, first, a part of the content that needs to be protected in the ELF file can be encrypted by a tool program. The ELF file can be an executable file or a shared library. The encryption scope is usually the ".text" section storing the program code, or the ".strtab" section or "shstrtab" section storing string literals.

[0084] When a section of an ELF file is encrypted, a special label is set in the flag field of the section in the section table by using the tool program to indicate that the section has been encrypted.

[0085] When at least one section of an ELF file is encrypted, another special label is set in the header of the ELF file by using the tool program. For example, the label can be set in the flag field of the header to indicate that the ELF file has been encrypted and what method must be used for decryption.

[0086] Figures 1 to 3 It is a flowchart of the protection method according to an embodiment of the present invention. The present invention provides a MAC module in the core of the operating system to execute Figures 1 to 3 the process shown. In an embodiment, the electronic device determines whether a first process is being debugged or whether it is formed after a second executable file is executed, and respectively executes Figures 1 to 3 the process shown.

[0087] Whenever an executing process (hereinafter referred to as the current process) is about to execute an executable file, the MAC module will execute Figure 1 the process.

[0088] First, in step 11, it is checked whether the executable file has been encrypted. If the executable file has been encrypted, the process proceeds to step 12; otherwise, the process proceeds to step 14.

[0089] In step 12, it is checked whether the current process is being debugged. If the current process is being debugged, the process proceeds to step 13; otherwise, the process proceeds to step 14.

[0090] In step 13, the current process is denied from executing the executable file. Since the content of the executable file will be decrypted during execution, this denial can prevent the leakage of the content of the executable file during the debugging process.

[0091] In step 14, the current process is allowed to execute the executable file.

[0092] In addition, in step 14, the MAC module can parse the section table of the executable file to know the encryption range of the executable file (which sections have been encrypted), and then record the encryption range of the executable file in the security context of the file structure corresponding to the executable file.

[0093] Furthermore, in step 14, the MAC module can check whether the file structure of the executable file is linked to the page cache. If the file structure is linked to the page cache, the MAC module removes the executable file from the page cache to prevent other processes from obtaining the decrypted content of the executable file through the page cache.

[0094] Before executing an executable file, it is necessary to map the executable file into the virtual address space of the process formed by the executable file. In addition, before a process executes a function in a shared library, it is necessary to map the shared library into the virtual address space of the process. The above two mappings can both be simply referred to as memory mapping.

[0095] Whenever an executing process (hereinafter referred to as the current process) is about to perform a memory mapping on a shared library, the MAC module will execute Figure 2 the following process.

[0096] First, in step 21, check whether the shared library has been encrypted. If the shared library has been encrypted, the process proceeds to step 22; otherwise, the process proceeds to step 24.

[0097] In step 22, check whether the current process is being debugged. If the current process is being debugged, the process proceeds to step 23; otherwise, the process proceeds to step 24.

[0098] In step 23, the current process is denied from performing a memory mapping on the shared library. Since the content of the shared library will be decrypted during execution, this denial can prevent the leakage of the content of the shared library during the debugging process.

[0099] In step 24, the current process is allowed to perform a memory mapping on the shared library.

[0100] In addition, in step 24, the MAC module can parse the section table of the shared library to know the encryption range of the shared library (which sections have been encrypted), and then record the encryption range of the shared library in the security context of the current process and / or the file structure corresponding to the shared library.

[0101] Furthermore, in step 24, the MAC module can check whether the file structure of the shared library is linked to the page cache. If the file structure is linked to the page cache, the MAC module removes the shared library from the page cache to prevent other processes from obtaining the decrypted content of the shared library through the page cache.

[0102] Whenever another process is about to debug the foregoing current process, the MAC module will execute Figure 3 the process.

[0103] For example, in the Linux operating system, the other process must debug the current process via the ptrace system call, and the MAC module can perform mandatory access control on the system call in the operating system kernel to first use Figure 3 the process to handle the system call.

[0104] First, in step 31, check whether the current process is formed after the execution of an encrypted executable file; if so, the process proceeds to step 33; if not, the process proceeds to step 32. For example, the MAC module can perform mandatory access control on the memory mapping of the executable file in the kernel of the operating system, and when the executable file of the current process is mapped to the virtual address space of the current process, the MAC module can check whether there is a tag indicating encryption in the header of the executable file. If there is such a tag, the MAC module can set the corresponding tag in the security context of the task structure corresponding to the current process. Then, in step 31, the MAC module can check whether the corresponding tag is set in the security context to determine whether the current process originates from an encrypted executable file.

[0105] In step 32, check whether the current process has memory-mapped at least one encrypted shared library; if so, the process proceeds to step 33; if not, the process proceeds to step 34. For example, the MAC module can perform mandatory access control on the memory mapping of shared libraries in the kernel of the operating system. When a shared library is mapped to the virtual address space of the current process, the MAC module can check whether there is a tag indicating encryption in the header of the shared library. If there is such a tag, the MAC module can set the corresponding tag in the security context of the task structure corresponding to the current process. Then, in step 32, the MAC module can check whether the corresponding tag is set in the security context to determine whether the current process has memory-mapped an encrypted shared library.

[0106] In step 33, deny the other process from debugging the current process to prevent the leakage of protected content in the executable file or shared library during the debugging process.

[0107] In step 34, allow the other process to debug the current process.

[0108] The following description refers to Figure 4 When the operating system performs a memory mapping on an ELF file (executable file or shared library), the MAC module can execute Figure 4 the process.

[0109] First, in step 41, check whether the memory mapping range of the ELF file includes the encrypted part of the ELF file. If the mapping range includes the encrypted part, the process proceeds to step 42; otherwise, the process ends.

[0110] In step 42, record the entry point of the page fault handling function A provided by the original file system where the ELF file is located to the operating system kernel.

[0111] In step 43, replace the original page fault handling function A with the page fault handling function B provided by the MAC module.

[0112] The following description refers to Figure 5 When a page fault occurs in a running process, the MAC module can execute Figure 5 the process.

[0113] Figure 5 The process continues Figure 4 the process and is applicable to encrypted executable files and encrypted shared libraries stored in different types of file systems (such as ext2, ext3, ext4, zfs, xfs, etc.).

[0114] First, in step 51, a page fault occurs in the process.

[0115] In step 52, due to the page fault, the page fault handler of the operating system calls the page fault handling function B. Steps 53 to 57 that follow all belong to the page fault handling function B.

[0116] In step 53, the page fault handling function B calls the original page fault handling function A to cause the page fault handling function A to load the content of the page that caused the page fault from the file system into the virtual address space of the process.

[0117] In step 54, it is checked whether the content of the page has been successfully loaded into the virtual address space of the process. If it has been successfully loaded, the process proceeds to step 55; otherwise, the process proceeds to step 58.

[0118] In step 55, it is checked whether the content of the page that caused the page fault has been decrypted. If it has been decrypted, the process proceeds to step 58; otherwise, the process proceeds to step 56.

[0119] In step 56, according to the decryption method indicated by the header of the ELF file mapped to the page, the content of the page is decrypted. Specifically, in the content of the page, only the encrypted range recorded within the security context of the file structure of the ELF file mapped to the page is decrypted. For example, the decryption method hands over the content of the page to the trusted platform module (TPM) built into the electronic device for decryption. The TPM module is a hardware component containing a key and can use the key to encrypt and decrypt important data.

[0120] In step 57, a label is set in the page to indicate that the page has been decrypted.

[0121] In step 58, return to the page fault handler of the operating system.

[0122] Another embodiment is given below to illustrate the protection method of the present invention. In this embodiment, the electronic device is a network attached storage (NAS). NAS is a network storage device that can provide file sharing and storage services on the network. NAS is usually a dedicated hardware device that contains one or more hard disk drivers internally and runs a specific operating system and vendor-specific software and firmware to provide functions such as file management, access control, and network communication. The operating system of NAS is usually Linux.

[0123] In this embodiment, the operating system image file of the NAS (image file, including the aforementioned MAC module), other core modules, and other software and firmware programs can be compiled by a compiler into executable files or shared libraries. Then, a tool program can be used to encrypt the ".text" section of the executable files or shared libraries that need to be protected, and encryption can be selected according to whether the executable files and shared libraries are vulnerable to network attacks. These executable files and shared libraries are all ELF files and remain ELF files after encryption.

[0124] When encrypting an ELF file, the tool program can set certain bits to 1 in the flag field of the file header of the ELF file as an identification label for the decryption method. For example, if the 31st bit of the flag field is set, it means that the ELF file needs to be decrypted using the Advanced Encryption Standard (AES) algorithm; if the 30th bit of the flag field is set, it means that the ELF file needs to be decrypted using the Data Encryption Standard (DES) algorithm; if the 29th bit of the flag field is set, it means that the ELF file needs to be decrypted using the trusted platform module (TPM). In addition, the tool program can set a specific bit of the flag field of the encrypted section in the section table of the ELF file to 1 as an identification label.

[0125] The above-mentioned executable files and shared libraries of the NAS, after being encrypted by the tool program, can be packaged into an installation package for installing or updating the operating system, core modules, and other software and firmware of the NAS. After the installation or update is completed and the NAS is restarted, the MAC module in its operating system will be executed earlier than the encrypted ELF files.

[0126] In this embodiment, the NAS device is already equipped with a hardware TPM security chip, which can store keys and is used to decrypt the content when executing ELF files, and the aforementioned encryption can be encrypted with another TPM chip or encrypted with the same key by the tool program. The encryption is carried out in a secure software compilation and packaging environment to reduce the risk of key leakage.

[0127] In a certain situation, a process 1 formed by an executable file 1 in the NAS device decides to execute another executable file 2. Figure 6A And Figure 6B is a flowchart for process 1 to execute executable file 2. The following description refers to Figure 6A And Figure 6B .

[0128] First, in step 601, process 1 executes executable file 2. For example, process 1 can call system calls such as execve in Linux to execute executable file 2. The kernel of the Linux operating system is responsible for handling this system call.

[0129] In step 602, the executable file loader of the Linux kernel reads the header of executable file 2 to identify the format of executable file 2. In this embodiment, executable file 2 is in the ELF format, so the ELF loader is used to load executable file 2.

[0130] In step 603, the ELF loader requests the MAC module of the Linux kernel to perform a mandatory access check.

[0131] In step 604, the MAC module performs a mandatory access check, and its process is shown in Figure 7 Hereinafter, with reference to Figure 7 the process of this mandatory access check will be described.

[0132] First, in step 71, the MAC module first checks whether the 29th, 30th, or 31st bit of the flag field in the file header of executable file 2 is set to 1 to determine whether executable file 2 has been encrypted. If executable file 2 has been encrypted, the process proceeds to step 72. Otherwise, if executable file 2 has not been encrypted, the process proceeds to step 78.

[0133] In step 72, the MAC module checks whether process 1 is being debugged. The MAC module can check the task structure of process 1 to determine whether process 1 is being debugged. If process 1 is being debugged, the process proceeds to step 73. If process 1 is not being debugged, the process proceeds to step 74.

[0134] In step 73, the MAC module returns the value of refusing to execute executable file 2 to the ELF loader.

[0135] In this embodiment, the 29th bit of the flag field in the file header of executable file 2 is set to 1, indicating that executable file 2 has been encrypted and should be decrypted by the TPM. Additionally, process 1 is not being debugged, so the process will proceed to step 74.

[0136] In step 74, the MAC module reads the section table of executable file 2 to obtain the encryption range of executable file 2, that is, which sections of executable file 2 have been encrypted and must be decrypted, and then records this encryption range in the security context pointed to by the file structure of executable file 2.

[0137] In step 75, the MAC module replaces the memory mapping processing function M recorded in the file structure of the executable file 2 with the memory mapping processing function N provided by the MAC module.

[0138] Unless the executable file 2 is opened in a special way, the read and write operations on the executable file 2 will go through the page cache of the Linux kernel. To prevent the content of the decrypted executable file 2 from leaking through the page cache, in step 76, the MAC module removes the executable file 2 from the page cache.

[0139] In step 77, the MAC module sets a label Z in the security context pointed to by the task structure of the process 2 generated when the executable file 2 is executed, to indicate that the process 2 is executed from an encrypted executable file.

[0140] In step 78, the MAC module passes back the value allowing the execution of the executable file 2 to the ELF loader.

[0141] Back to Figure 6A In the illustrated process, in step 605, the ELF loader determines whether to continue executing the executable file 2 according to the value passed back by the MAC module. In this embodiment, the value passed back by the MAC module is to allow the execution of the executable file 2, so the ELF loader continues to execute the executable file 2.

[0142] In step 606, the ELF loader performs the memory mapping of the executable file 2. Specifically, the ELF loader maps the sections required to execute the executable file 2 into the virtual address space of the virtual memory of the process 2 generated by the executable file 2. The ELF loader performs the memory mapping in the aforementioned demand paging manner.

[0143] In step 607, the ELF loader calls the replaced memory mapping processing function N. In this embodiment, the memory mapping processing function N is implemented in the MAC module.

[0144] In step 608, the memory mapping processing function N in the MAC module calls the original memory mapping processing function M.

[0145] In step 609, the memory mapping processing function N in the MAC module replaces the page fault handling function A provided by the file system where the original executable file 2 is located to the Linux kernel with the page fault handling function B implemented in the MAC module.

[0146] In step 610, the ELF loader starts process 2 formed by the executable file 2. Process 2 is arranged to start execution from the entry point recorded in the file header of the executable file 2, and this entry point is located within the ".text" section of the executable file 2. The subsequent process steps are illustrated in Figure 6B .

[0147] In step 611, the processor of the NAS reads the instruction at the virtual memory address of the entry point to execute the instruction.

[0148] However, in step 612, a page fault occurs because the page corresponding to the virtual memory address does not exist.

[0149] In step 613, the page fault handler of the Linux kernel calls the page fault handling function B within the MAC module.

[0150] In step 614, the page fault handling function B first calls the original page fault handling function A. The page fault handling function A loads the content of the page where the instruction causing the page fault is located into the physical memory of the NAS for the processor to read.

[0151] In step 615, the page fault handling function B checks whether the content of the page has been encrypted and not yet decrypted. If the content of the page has been encrypted and not yet decrypted, it hands the page over to the TPM of the NAS for decryption.

[0152] After the TPM completes the decryption of the page, in step 616, the page fault handling function B sets a label in the page to indicate that the page has been decrypted to avoid repeated decryption of the page.

[0153] In step 617, the page fault handler of the Linux kernel wakes up process 2.

[0154] In step 618, since the page where the instruction is located has been loaded, the processor can continue to read and execute the instruction.

[0155] If another debugging process, process 3, uses the ptrace system call to debug process 2, when the kernel of the Linux operating system processes this system call, it will call another function C of the MAC module to perform a mandatory access permission check. Function C checks whether label Z has been set within the security context pointed to by the task structure corresponding to process 2, and after finding that label Z has been set within the security context, it will reject the debugging request of process 3.

[0156] The protection method of the present invention is not limited to being applied to the Linux operating system, but can also be applied to other operating systems that can support the technical solution of this protection method. Additionally, the protection method of the present invention is not limited to protecting executable files and shared libraries in the ELF format, but can also be used to protect executable files and shared libraries in other formats that support the technical solution of this protection method.

[0157] Figure 8 FIG. 4 is a block diagram of an electronic device 80 according to an embodiment of the present invention. The electronic device 80 includes a processor 81, a memory 82, and a storage device 83 that are electrically connected to each other. The storage device 83 may include at least one non-volatile memory or a data storage device such as a hard disk drive, installs an operating system, such as Linux, and is used to store executable files and shared libraries in the file system. The processor 81 may be at least one processor, which is used to execute the operating system and execute the protection method of any of the above embodiments in the operating system to prevent the content of the executable files and shared libraries from leaking. The memory 82 may be a volatile random access memory, which serves as the physical memory for the memory mapping of the executable files and shared libraries, and is used to temporarily store the data required to execute the protection method and the data generated during the execution of the protection method.

[0158] In one embodiment, the present invention provides a computer-readable storage medium. The computer-readable storage medium may be a memory, a floppy disk, a hard disk, or an optical disc, which is used to store a plurality of instructions, and the instructions can be read by an electronic device to execute the protection method of any of the above embodiments. For example, the computer-readable storage medium may be the storage device 83 in Figure 8 , the instructions may be the instructions in the aforementioned installation package, and the instructions can be read by the processor 81 of the electronic device 80 to execute the protection method of any of the above embodiments. In another embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0159] In summary, the present invention uses anti-reverse engineering and anti-debugging techniques to protect executable files and shared libraries, so as to prevent attackers from obtaining the decrypted content in the executable files and shared libraries through reverse engineering or debugging, and provides a more rigorous protection technology to solve at least one of the above disadvantages. In addition, the present invention is equally applicable to executable files and shared libraries that have been binary obfuscated.

[0160] The above embodiments are only illustrative of the principles and effects of the present invention, and are not used to limit the present invention. Any person skilled in the art can modify and change the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of the patent protection of the present invention should be as listed in the claims.

Claims

1. A method for protecting an executable file and a shared library, executed by at least one processor of an electronic device in an operating system of the electronic device, the method comprising the following steps: The electronic device determines whether a first process is being debugged or is formed after a second executable file is executed, wherein: If the first process is being debugged and is about to execute an encrypted first executable file, denying the first process from executing the first executable file; If the first process is being debugged and is about to perform memory mapping on an encrypted shared library, denying the first process from performing the memory mapping on the shared library; as well as If the first process is formed after the second executable file is executed, and the second executable file has been encrypted, and a second process is about to debug the first process, the second process is denied from debugging the first process.

2. The protection method according to claim 1, wherein: After the first process has performed the memory mapping on the shared library, the protection method further comprises: If the shared library has been encrypted, and the second process is about to debug the first process, the second process is denied the debugging of the first process.

3. The protection method according to claim 1, wherein: The method further comprises the steps of: Whether the first executable file has been encrypted is determined according to a flag field in the header of the first executable file.

4. The protection method according to claim 1, wherein: The method further comprises the steps of: Whether the shared library has been encrypted is determined according to the flag field of the header of the shared library.

5. The protection method according to claim 1, wherein: If the first process is formed after the second executable file is executed, the protection method further includes: When executing the second executable file, determining whether the second executable file has been encrypted according to a flag field in a header of the second executable file, wherein if the second executable file has been encrypted, setting a label in a security context pointed to by a task structure of the first process; and When the second process is about to debug the first process, it is checked whether the tag has been set in the security context of the first process, wherein if the tag has been set in the security context, the second process is denied to debug the first process.

6. The protection method according to claim 1, wherein: If the first process is formed after the second executable file is executed, and the second executable file has been encrypted, the protection method further includes: The first memory mapping processing function recorded in the file structure of the second executable file is replaced with a second memory mapping processing function, wherein the second memory mapping processing function includes: calling the first memory mapping processing function; and The first page fault processing function provided by the file system where the second executable file is located is replaced with the second page fault processing function.

7. The protection method according to claim 6, wherein: The second page fault processing function includes: Calling the first page fault processing function to load the page causing the page fault of the first process from the file system into the virtual address space of the first process; Checks whether the contents of the page have been decrypted; and If the content of the page has not been decrypted, decrypt the content of the page.

8. The protection method according to claim 7, wherein: The method further includes: If the content of the page has not been decrypted, the content of the page is decrypted using the decryption method indicated by the flag field of the header of the second executable file.

9. The protection method according to claim 1, wherein: The method further includes: If the first executable file has been encrypted and the first process is allowed to execute the first executable file, moving the first executable file out of the page cache of the operating system; and If the shared library has been encrypted and the first process is allowed to perform the memory mapping on the shared library, the shared library is removed from the page cache.

10. A protection system for executable files and shared libraries, comprising: a storage device having an operating system installed therein; as well as At least one processor is used to execute the operating system, and execute the protection method according to any one of claims 1 to 9 in the operating system.

Images