Page swap method, device and electronic device

Abstract

The embodiments of the present disclosure disclose a page swap method, device, and electronic device. In the page swap method described in the present disclosure, the swap-out overhead of the hybrid storage structure can be reduced by: (1) using a per-thread page table and TLB shootdown operation to track the page mapping cache and reduce the TLB shootdown overhead; (2) through the operating system kernel and the programming language runtime with garbage collection and lightweight concurrency, the swap-out is regarded as a lightweight thread scheduled by the programming language runtime with garbage collection and lightweight concurrency to perform the swap-out asynchronously.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a page swap method, device and electronic device. Background Art

[0002] Currently, the memory consumption of data center applications is increasing. Given the limited amount of dynamic random access memory (DRAM) that can be accommodated in a single server, memory-intensive applications (e.g., data analysis systems, graphics processing systems, memory caching systems, and databases) face tremendous memory pressure. Kernel swapping can alleviate this pressure by swapping out inactive pages to high-performance solid-state drives (SSDs) and remote memory. Before the kernel swaps in a page from the swap space, the kernel must first swap out the page to make room for the swapped-in page. However, page swapping is a heavyweight operation that is located on the critical path, has poor scalability, and is heavyweight.

[0003] The current kernel swap system is designed for the traditional scenario where disks (HDDs) are used as swap devices and disk access is slow. Applications cannot swap to disk frequently to avoid unacceptable performance degradation. However, this assumption does not hold in a hybrid storage architecture setting, because non-volatile memory (NVM) provides higher read / write bandwidth and lower random access latency than traditional HDD / SSD. Therefore, the swap throughput of applications under hybrid memory can be several orders of magnitude higher than swapping to disk. For example, if a certain logistic regression test is run on hybrid memory, 25% of the working set is on the DRAM side and 75% of the working set is on the NVM side. Experimental results show that each CPU core generates a major page fault every 100us during operation. High-frequency swap-in is accompanied by high-frequency swap-out to make room for newly swapped-in pages.

[0004] Swapping out is a heavyweight task in the operating system kernel page fault handling path. Each swap out first selects the victim physical page by scanning the kernel LRU (Least Recently Used) active / inactive page list, then searches for the page table entry (PTE) that references the victim physical page through reverse mapping, and finally sends an Inter-Processor Interrupt (IPI) broadcast to all online CPU cores to shoot down the Translation Lookup Buffer (TLB), write out the page to NVM, etc. This series of complex processes introduces high computational overhead. In addition, the cost of sending and responding to IPIs increases with the number of online CPU cores, leading to serious scalability issues. Frequent TLB shootdowns and refreshes also increase the TLB miss rate and hurt application performance. Summary of the invention

[0005] In order to solve the problems in the existing technology, the embodiments of the present disclosure provide a lightweight page swapping method, device and electronic device for a hybrid storage architecture, which solves the scalability problem of TLB shootdown in page swapping operations by using per-thread page tables and can reduce the computational overhead of the page swapping process.

[0006] In a first aspect, an embodiment of the present disclosure provides a page swap method, wherein the method includes:

[0007] Allocating a per-thread page table for a first type of thread to record page mappings of pages locally accessed by the first type of thread;

[0008] Allocating a global page table for the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process;

[0009] In the case of performing page fault processing when performing page swapping, detecting a sharing state of a page table entry of the global page table in the global page table, wherein the page table entry in the global page table is accompanied by a CPU core bitmap to track all CPU cores that cache the page table entry in a translation lookup buffer, thereby determining the sharing state of the page table entry;

[0010] According to the detected sharing status of the page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

[0011] In combination with the first aspect, in a first implementation of the first aspect of the present disclosure, according to the detected sharing state of the page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, including:

[0012] Based on a translation lookup buffer miss when performing a shootdown on the translation lookup buffer, traversing a per-thread page table following the same page table structure using a memory management unit in a CPU core that caches the page table entry in the translation lookup buffer;

[0013] Based on the traversal result of the memory management unit in the CPU core, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

[0014] In combination with the first aspect or the first implementation of the first aspect, in a second implementation of the first aspect of the present disclosure, the page fault processing includes at least one of the following processing:

[0015] For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of a corresponding page table entry in the per-thread page table, the per-thread page table is enabled to obtain the missing page table entry through the global page table;

[0016] For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of corresponding page table entries in the per-thread page table and the global page table, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and a page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table;

[0017] For a CPU core loaded with a global page table, when a page fault is caused by an invalid forward pointer of a page table entry pointing to the per-thread page table, the operating system kernel copies the value of the page table entry to the global page table, and marks the page table entry as shared in the CPU core bitmap of the page table entry;

[0018] For a CPU core loaded with a global page table, when a page fault is caused by a non-existent page table entry, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked using a specific forward pointer pointing to the page table entry of the per-thread page table.

[0019] In combination with the first aspect or the first implementation manner of the first aspect, in a third implementation manner of the first aspect of the present disclosure, according to the detected sharing state of the page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, performing a shootdown process on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, including:

[0020] Based on the detected fact that the page table entry of the global page table is in a private state in the global page table, the CPU core that schedules the thread involved in the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or

[0021] According to the detection that the page table entries of the global page table are in a shared state in the global page table, the CPU core that schedules the thread involved in the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core that schedules the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread.

[0022] In combination with the first aspect, in a fourth implementation manner of the first aspect of the present disclosure, the first type of thread is created and started by a runtime of a programming language with lightweight concurrency.

[0023] In combination with the fourth implementation of the first aspect, in a fifth implementation of the first aspect of the present disclosure, the method further includes:

[0024] The operating system kernel and the runtime of the programming language with lightweight concurrency are used to export the swap-out thread executed in the page fault processing as a swap-out lightweight thread, so that the swap-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, so that the swap-out lightweight thread is executed asynchronously.

[0025] In combination with the fifth implementation of the first aspect, in a sixth implementation of the first aspect of the present disclosure, the method further includes:

[0026] Inserting safe points for a plurality of steps executed by said swapping out lightweight threads;

[0027] Allocating a set of memory areas in the operating system kernel to store swap-out state information when starting the runtime;

[0028] The swapped-out lightweight thread allocates a sub-memory area from the group of memory areas and saves the execution state of the swapped-out lightweight thread to the sub-memory area;

[0029] Returning, by the application kernel, an identification as an index of the sub-memory region to the runtime at each security point;

[0030] The identifier is passed back to the operating system kernel by the runtime when rescheduling the swapped out lightweight thread.

[0031] In combination with the fifth implementation manner or the sixth implementation manner of the first aspect, in a seventh implementation manner of the first aspect of the present disclosure, the step of scheduling, by the runtime, the swapped-out lightweight thread together with the user lightweight thread managed and scheduled by the runtime includes:

[0032] The swapped-out lightweight thread is assigned, by the runtime, a lower priority than the user lightweight thread.

[0033] In combination with the first aspect or the first implementation of the first aspect, in the eighth implementation of the first aspect of the present disclosure, page swapping is performed in a hybrid storage architecture, and the hybrid storage architecture is a hybrid storage architecture that includes at least a dynamic random access memory DRAM and a non-volatile memory NVM.

[0034] In a second aspect, an embodiment of the present disclosure provides a page swap device, wherein the device includes:

[0035] a per-thread page table allocation module, configured to allocate a per-thread page table for a first type of thread to record page mappings of pages locally accessed by the first type of thread;

[0036] A global page table allocation module is configured to allocate a global page table to the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process;

[0037] a page table entry detection module, configured to detect a sharing state of a page table entry of the global page table in the global page table when performing a page fault process during page swapping, wherein a CPU core bitmap is attached to the page table entry in the global page table to track all CPU cores that cache the page table entry in a translation lookup buffer, thereby determining the sharing state of the page table entry;

[0038] The conversion lookup buffer execution shootdown processing module is configured to execute shootdown processing on the conversion lookup buffer of the CPU core that schedules the thread involved in the page fault processing based on the detected sharing status of the page table entries of the global page table in the global page table and based on the per-thread page table and the global page table involved in the page fault processing.

[0039] In a third aspect, an electronic device is provided in an embodiment of the present disclosure, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method as described in the first aspect or any one of the first to eighth implementations of the first aspect.

[0040] In a fourth aspect, an embodiment of the present disclosure provides a readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in any one of the first aspect and the first to eighth implementations of the first aspect is implemented.

[0041] In a fifth aspect, a computer program product is provided in an embodiment of the present disclosure, comprising computer instructions, which, when executed by a processor, implement the method described in the first aspect and any one of the first to eighth implementations of the first aspect.

[0042] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0043] According to the technical solution provided by the embodiment of the present disclosure, a per-thread page table is allocated to a first type of thread to record the page mapping of pages locally accessed by the first type of thread; a global page table is allocated to the process to which the first type of thread belongs to record the page mapping of pages accessed by threads in the process; when a page fault processing is performed during page swapping, the sharing status of the page table entries of the global page table in the global page table is detected, wherein the page table entries in the global page table are accompanied by a CPU core bitmap to track all CPU cores that cache the page table entries in the translation lookup buffer, thereby determining the sharing status of the page table entries; according to the detected sharing status of the page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown processing is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, the scalability problem of TLB shootdown in the page swap out operation can be solved by using a per-thread page table, and the computational overhead of the page swap out process can be reduced.

[0044] According to the technical solution provided by the embodiment of the present disclosure, based on the shared status of the detected page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, including: based on the situation that a translation lookup buffer miss occurs when the shootdown is performed on the translation lookup buffer, a memory management unit in the CPU core that caches the page table entries in the translation lookup buffer is used to traverse a per-thread page table that follows the same page table structure; based on the traversal result of the memory management unit in the CPU core, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, the scalability problem of TLB shootdown in page swapping operations can be solved by using per-thread page tables, and the computational overhead of the page swapping process can be reduced.

[0045] According to the technical solution provided by the embodiment of the present disclosure, the page fault processing includes at least one of the following processing: for a CPU core loaded with a per-thread page table, when a page fault error is caused by a lack of a corresponding page table entry in a per-thread page table, the per-thread page table is enabled to obtain the missing page table entry through the global page table; for a CPU core loaded with a per-thread page table, when a page fault error is caused by a lack of a corresponding page table entry in a per-thread page table and in the global page table, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table; for a CPU core loaded with a global page table, In the case where a page fault is caused by an invalid specific forward pointer of a page table entry pointing to the per-thread page table, the value of the page table entry is copied to the global page table through the operating system kernel, and the page table entry is marked as shared in the CPU core bitmap of the page table entry; for a CPU core loaded with a global page table, in the case where a page fault is caused by the non-existence of a page table entry, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table. The scalability problem of TLB shootdown in a page swapping operation can be solved by using a per-thread page table, and the computational overhead of the page swapping process can be reduced.

[0046] According to the technical solution provided by the embodiment of the present disclosure, according to the shared state of the detected page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core of the thread involved in the scheduling of the page fault processing, including: according to the detected page table entry of the global page table being in a private state in the global page table, the CPU core of the thread involved in the scheduling of the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or according to the detected page table entry of the global page table being in a shared state in the global page table, the CPU core of the thread involved in the scheduling of the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core of the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread, the scalability problem of TLB shootdown in the page swapping operation can be solved by using per-thread page tables, and the computational overhead of the page swapping process can be reduced.

[0047] According to the technical solution provided by the embodiment of the present disclosure, the first type of thread is created and started by the runtime of the programming language with lightweight concurrency, thereby reducing the swap-out overhead of the programming language with lightweight concurrency.

[0048] According to the technical solution provided by the embodiment of the present disclosure, the swap-out thread executed in the page fault processing is exported as a swap-out lightweight thread by utilizing the operating system kernel and the runtime of the programming language with lightweight concurrency, so that the swap-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, so that the swap-out lightweight thread is executed asynchronously, and the swap-out thread can be decoupled into some swap-out lightweight threads and co-scheduled with ordinary user lightweight threads to asynchronously perform the swap-out operation, thereby reducing the swap-out overhead of the programming language with lightweight concurrency.

[0049] According to the technical solution provided by the embodiment of the present disclosure, by inserting security points for multiple steps executed by the swapped-out lightweight thread; allocating a group of memory areas in the operating system kernel to store swap-out status information when starting the runtime; the swapped-out lightweight thread allocates a sub-memory area from the group of memory areas and saves the execution status of the swapped-out lightweight thread to the sub-memory area; returning an identifier as an index of the sub-memory area to the runtime through the application kernel at each security point; and passing the identifier back to the operating system kernel when rescheduling the swapped-out lightweight thread by the runtime, the swapped-out thread can be decoupled into some swapped-out lightweight threads, and co-scheduled with ordinary user lightweight threads to asynchronously perform swap-out operations, thereby reducing the swap-out overhead of programming languages ​​with lightweight concurrency.

[0050] According to the technical solution provided by the embodiment of the present disclosure, the swapped-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, including: the swapped-out lightweight thread is assigned a lower priority than the user lightweight thread through the runtime, so that the swapped-out thread can be decoupled into some swapped-out lightweight threads and co-scheduled with ordinary user lightweight threads to asynchronously perform the swap-out operation, thereby reducing the swap-out overhead of the programming language with lightweight concurrency.

[0051] According to the technical solution provided in the embodiment of the present disclosure, by performing page swapping in a hybrid storage architecture, the hybrid storage architecture is a hybrid storage architecture that includes at least a dynamic random access memory DRAM and a non-volatile memory NVM. For the hybrid storage architecture of DRAM and NVM, the scalability problem of TLB shootdown in the page swap operation can be solved by using a per-thread page table, and the computational overhead of the page swap process can be reduced.

[0052] According to the technical solution provided by the embodiment of the present disclosure, a per-thread page table allocation module is configured to allocate a per-thread page table for a first type of thread to record the page mapping of pages locally accessed by the first type of thread; a global page table allocation module is configured to allocate a global page table for the process to which the first type of thread belongs to record the page mapping of pages accessed by threads in the process; a page table entry detection module is configured to detect the sharing status of the page table entries of the global page table in the global page table when performing page fault processing during page swapping, wherein the page table entries in the global page table are attached with CPU core A bitmap is provided to track all CPU cores that cache the page table entries in the translation lookup buffer, so as to determine the sharing status of the page table entries; a translation lookup buffer execution shootdown processing module is configured to perform shootdown processing on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing based on the detected sharing status of the page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing. The scalability problem of TLB shootdown in page swapping operation can be solved by using per-thread page table, and the computational overhead of the page swapping process can be reduced.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0055] Figure 1 A flow chart showing a page swap method according to an embodiment of the present disclosure is shown;

[0056] Figure 2 An exemplary schematic diagram showing a page table structure including a per-thread page table and a global page table used in a lightweight page swap method of a hybrid storage architecture according to an embodiment of the present disclosure;

[0057] Figure 3 A schematic diagram showing a scenario in which a swapped-out thread is exported as a special Goroutine so that the Go runtime can schedule it together with a user Goroutine in a page swap method according to an embodiment of the present disclosure;

[0058] Figure 4 A structural block diagram of a page switching device according to an embodiment of the present disclosure is shown;

[0059] Figure 5 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0060] Figure 6It is a schematic diagram of the structure of a computer system suitable for implementing the method according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0062] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of labels, numbers, steps, behaviors, components, parts, or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other labels, numbers, steps, behaviors, components, parts, or combinations thereof exist or are added.

[0063] It should also be noted that, in the absence of conflict, the embodiments and labels in the embodiments of the present disclosure can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] In the embodiments of the present disclosure, the following concepts may be mentioned:

[0065] Non-volatile memory (NVM): Persistent byte-addressable memory. NVM offers larger memory capacity than DRAM, lower access latency (<10us) and higher bandwidth than SSD. For example, as an intermediate layer between DRAM and SSD in the storage hierarchy, NVM can be used as a high-performance swap partition when the application runs out of memory.

[0066] Programming languages ​​with garbage collection and light-weight concurrency: Many general-purpose programming languages ​​have built-in light-weight concurrency support (see definition below) and garbage collection runtime systems (see definition below). Go and Erlang are two examples of such languages. Go is a popular modern language, and we use the Go language as an example in the description of the embodiments of the present disclosure. The following embodiments also refer to programming languages ​​with garbage collection and light-weight concurrency as programming languages ​​with light-weight concurrency. Lightweight concurrency refers to the large amount of concurrency that can exist when running with lightweight threads of a programming language with light-weight concurrency, and therefore, it is called a programming language with light-weight concurrency. Compared with traditional system-level threads and processes, the biggest advantage of lightweight threads is their "lightweight", and millions or even tens of millions can be easily created without exhausting system resources; and the number of threads and processes that can be allowed to run by system resources is much less than this number.

[0067] Lightweight concurrency is provided by lightweight user-level threads managed by the language runtime. For example, Goroutine is a lightweight user-level thread (or simply lightweight thread) managed and scheduled by the Go runtime. The Go runtime creates a set of kernel threads as a mutator thread pool and schedules Goroutines onto available threads in the user-level thread pool. Throughout this document, we refer to kernel threads as "threads".

[0068] Garbage Collection (GC) in Go: In addition to mutator threads, the Go runtime also creates a set of GC threads for concurrent tracking. GC threads start tracking from some "root" objects and mark all reachable objects through reference chains. When all reachable objects are marked as active, the tracking is complete. The GC thread then compresses the active objects into a continuous memory area and releases the memory occupied by dead objects. The threads that run user code are called mutator threads, and other threads created by the runtime for auxiliary work such as GC are non-mutator threads.

[0069] Page Table: A page table is a data structure residing in memory that maps virtual memory addresses to physical addresses. In x86 and x86-64 architectures, a page table can be viewed as a radix-tree indexed by virtual addresses. The operating system provides a page table for each process, and its address is stored in the CR3 register of the CPU. When translating a virtual address to a physical address, the CPU reads the CR3 register to get the page table and retrieve the mapping.

[0070] Page Table Entry (PTE): The leaves of the radix tree are called PTEs, which store the physical memory address and auxiliary bit flags indicating the state of the physical page, such as the access bit and the dirty bit. For example, there are several available bits in a certain standard PTE structure that are not used by specific hardware and can be customized by the operating system.

[0071] Paging and Swapping: A typical operating system divides the virtual address space and physical memory space into contiguous pages of the same size (e.g., 4KB or 2MB). When the operating system is low on memory, it will page out recently unused pages to the swap space through a swap mechanism. For example, the swap space is typically located on NVM or a disk. Later, when an application accesses a page located in the swap space, it will trigger a page fault, and the operating system will fetch the page back to the main memory, which is called swap in. In the embodiments of the present disclosure, pages are sometimes also referred to as pages.

[0072] Inter-processor interrupt (IPI): A special type of hardware interrupt in a multi-core CPU. IPI is issued by one CPU core to other CPU cores, which may interrupt other CPU cores and require actions and responses from other cores.

[0073] TLB shootdown: The translation lookup buffer (TLB) is a special type of CPU cache that caches virtual to physical memory address translations. It is accessed at every memory read / write operation. To improve efficiency, the TLB is usually not maintained by hardware for consistency, but the TLB shootdown protocol is implemented by the operating system to maintain TLB consistency. When the mapping of a virtual page changes and its TLB entry becomes stale, the operating system kernel will shoot down the TLB entry in the TLBs of all CPU cores to restore consistency. In actual implementation, the kernel can directly shoot down the corresponding entry in the TLB of the current CPU core, but it is necessary to broadcast an IPI request to all remote CPU cores, let them shoot down the corresponding TLB entry or flush the entire TLB according to the additional information in the IPI, and wait for their response. The latency of IPI sending and response will increase with the increase in the number of CPU cores.

[0074] In order to solve the problems of the prior art, the inventors of the present disclosure have considered the following related solutions:

[0075] For example, an access bit-based TLB invalidation mechanism: ABIS (Access Based Invalidation System) reuses the access bit in the PTE as a hint of the PTE's shared status, avoiding IPI broadcasts during TLB shootdowns if the page is only accessed by one CPU core. ABIS constructs a fake page table to copy the page mapping when handling a page fault. With such a copy page table, it can directly insert an entry into the CPU core's TLB without setting the access bit in the corresponding PTE. If other CPU cores access the page and trigger a TLB miss, the access bit will be set and therefore serve as an indicator of a shared page. However, ABIS may cause false-positive indications when TLB entries are evicted and later reinserted. Go GC (Garbage Collection) threads can significantly amplify the problem because they track live objects at the same time and therefore share a large number of pages with mutator threads.

[0076] For example, per-core page table schemes: RadixVM (RadixVirtual Machine) provides a radix tree structure per-core page table to track page sharing status to avoid unnecessary IPI broadcasts during TLB shootdowns. It is implemented in a teaching OS that lacks many memory management features. PSPT (Partially Separated Page Table) provides a similar per-core page table design for a certain processor in an HPC (high performance computing) setting. PSPT requires programmers to carefully mark the computational area that each core page table will handle. Such detailed modifications and annotations to the program are usually impractical for non-HPC applications. In addition, both schemes do not have a global page table, so modifying the page mapping requires changing the PTEs in multiple page tables, resulting in additional overhead. In the case of using per-core page tables instead of per-thread page tables, the per-core page tables do not distinguish between different types of application threads (such as mutator threads and GC threads).

[0077] For example, the kernel swap path optimization scheme: Fastswap (Fast Page Swapping, Fastswap) optimizes page read / write during swapping by bypassing the kernel block IO software stack and utilizing the frontswap (front-end swap) interface. It reduces the software overhead when performing page IO, but the swap-out operation is still in the critical path of its page fault handling, and TLB shootdown still has scalability issues.

[0078] Considering the shortcomings of the related art solutions, the disclosed embodiment proposes a per-thread page table design to solve the scalability problem of TLB shootdown during page swapping and reduce the computational overhead of the swapping process. In addition, the disclosed embodiment proposes a method for reducing the swap-out overhead of a programming language (e.g., Go language) application with garbage collection and lightweight concurrency by decoupling the swapped-out thread into several "swapped-out lightweight threads (e.g., Goroutine)" and co-scheduling them with ordinary lightweight threads (e.g., Goroutine).

[0079] It should be noted that "lightweight" is a concept relative to "heavyweight", and the two are mainly compared in terms of the ease of use of the application framework, the service characteristics provided, etc. Moreover, the "lightweight" mentioned in the embodiments of the present disclosure refers to the "lightweight" of a programming language with garbage collection and lightweight concurrency.

[0080] According to the technical solution provided by the embodiment of the present disclosure, a per-thread page table is allocated to a first type of thread to record the page mapping of pages locally accessed by the first type of thread; a global page table is allocated to the process to which the first type of thread belongs to record the page mapping of pages accessed by threads in the process; when a page fault processing is performed during page swapping, the sharing status of the page table entries of the global page table in the global page table is detected, wherein the page table entries in the global page table are accompanied by a CPU core bitmap to track all CPU cores that cache the page table entries in the translation lookup buffer, thereby determining the sharing status of the page table entries; according to the detected sharing status of the page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown processing is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, the scalability problem of TLB shootdown in the page swap out operation can be solved by using a per-thread page table, and the computational overhead of the page swap out process can be reduced.

[0081] In order to solve the above problems, the present disclosure proposes a page swap method, an apparatus, an electronic device, a readable storage medium and a computer program product.

[0082] Figure 1 FIG. 1 is a flow chart of a page swap method according to an embodiment of the present disclosure. Figure 1 As shown, the page swap method includes steps S101, S102, S103, and S104.

[0083] In step S101, a per-thread page table is allocated to a first type of thread to record page mappings of pages locally accessed by the first type of thread.

[0084] In step S102, a global page table is allocated to the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process.

[0085] In step S103, when performing page fault processing during page swapping, the sharing status of the page table entries of the global page table in the global page table is detected, wherein the page table entries in the global page table are accompanied by a CPU core bitmap to track all CPU cores that cache the page table entries in the translation lookup buffer, thereby determining the sharing status of the page table entries.

[0086] In step S104, according to the detected sharing status of the page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

[0087] In one embodiment of the present disclosure, page swapping is performed in a hybrid storage architecture, which is a hybrid storage architecture that includes at least a dynamic random access memory DRAM and a non-volatile memory NVM. The kernel swap system of the related art is designed for traditional scenarios where a disk is used as a swap device and the disk access speed is slow. Applications cannot frequently swap to disk to avoid unacceptable performance degradation. However, this assumption does not hold true under the hybrid storage architecture mentioned in the embodiment of the present disclosure, because NVM provides higher read / write bandwidth and lower random access latency than traditional HDD / SSD. Therefore, the swap throughput of the application under hybrid memory can be several orders of magnitude higher than that of swapping to disk. For example, if a certain logistic regression test is run on the hybrid memory, 25% of the working set is located on the DRAM side and 75% of the working set is located on the NVM side. Experimental results show that during operation, each CPU core will generate a major page fault every 100us. High-frequency swapping in is accompanied by high-frequency swapping out to make room for newly swapped-in pages. In one embodiment of the present disclosure, a hybrid storage architecture may include multiple storage hierarchies, for example, NVM as an intermediate layer between DRAM and SSD in the storage hierarchy, which can be used as a high-performance swap partition when the application is short of memory.

[0088] According to the technical solution provided in the embodiment of the present disclosure, by performing page swapping in a hybrid storage architecture, the hybrid storage architecture is a hybrid storage architecture that includes at least a dynamic random access memory DRAM and a non-volatile memory NVM. For the hybrid storage architecture of DRAM and NVM, the scalability problem of TLB shootdown in the page swap operation can be solved by using a per-thread page table, and the computational overhead of the page swap process can be reduced.

[0089] In one embodiment of the present disclosure, the first type of thread is created and started by the runtime of a programming language with lightweight concurrency. In one embodiment of the present disclosure, the programming language with lightweight concurrency refers to the aforementioned programming language with garbage collection and lightweight concurrency, for example, Go language or Erlang language.

[0090] According to the technical solution provided by the embodiment of the present disclosure, the first type of thread is created and started by the runtime of the programming language with lightweight concurrency, thereby reducing the swap-out overhead of the programming language with lightweight concurrency.

[0091] In one embodiment of the present disclosure, the page swap method may refer to a page swap method for a hybrid storage architecture. In one embodiment of the present disclosure, the page swap method may refer to a lightweight page swap method for a hybrid storage architecture. Referring to the page swap method as a lightweight page swap method refers to a page swap method supported by a programming language with lightweight concurrency.

[0092] In one embodiment of the present disclosure, a hierarchical page table structure is proposed to track whether a page is accessed by multiple CPU cores. If a physical page is only accessed by a single core, the operating system kernel can skip the expensive IPI broadcast and perform TLB shootdown locally when evicting the page, thereby reducing the TLB shootdown overhead in swapping out. In order to balance memory consumption and tracking accuracy, we propose a per-thread page table for each Go mutator thread and provide a backup global page table for all (performance-unimportant) non-mutator threads. Good performance can be provided for mutator threads by ensuring the consistency of per-thread page tables and global page tables.

[0093] The following will refer to Figure 2 The page table structure used in the lightweight page swap method of the hybrid storage architecture according to one embodiment of the present disclosure is described.

[0094] Figure 2 An exemplary schematic diagram shows a page table structure including a per-thread page table and a global page table used in a lightweight page swap method of a hybrid storage architecture according to an embodiment of the present disclosure.

[0095] The following reference Figure 2 Describe the overall design of the page table structure. Figure 2 As shown, a per-thread page table is allocated for each mutator thread started by the Go runtime to record the page mappings of pages accessed locally by the mutator thread, and a "shared" global page table can be reserved for the entire process to which the mutator thread belongs to record the page mappings of pages accessed by multiple threads. The global page table is also used by non-mutator threads for address translation (see Figure 2 The operating system also keeps a flag for each core to mark whether the thread currently on the core is a mutator thread. Figure 2 As shown, the mutator thread flag is 1 and the non-mutator thread flag is 0. The per-thread page table structure and the process global page table structure can be similar to the x86 / x86-64 traditional page table structure. These two page table structures share the same process context identifier (Process-Context Identifier PCID) in the CR3 value (although Figure 2Not shown, but Figure 2 All CR3s in the global page table have the same PCID value). A CPU core bitmap is also attached to each PTE in the global page table to track all CPU cores that may cache this PTE in their TLB. Each CPU core loads the address of the per-thread page table into its CR3 register when scheduling a thread. When a TLB miss occurs, the memory management unit (MMU) in the CPU core can still traverse the per-thread page table that follows the same page table structure. In addition, the meanings of PGD, PUD, PMD, etc. in the global page table can be learned from the traditional page table structure of x86 / x86-64, and this disclosure will not go into details.

[0096] The following reference Figure 2 Describes Page Fault handling.

[0097] For a CPU core with per-thread page tables loaded in CR3, page faults can be divided into two cases: (1) the corresponding PTE is missing in the per-thread page table, but can be resolved in the global page table; (2) the PTE is missing in both.

[0098] If the corresponding PTE is valid in the global page table, it belongs to case (1), and the page fault handler will mark the PTE as shared by setting the corresponding bit in the core bitmap of the PTE in the global page table, and then fill the PTE value into the per-thread page table to complete the page fault handling. Figure 2 PTE 204 is missing from the per-thread page table in FIG. 1 , and the double-dashed line in the legend indicates that the value of PTE 203 in the global page table is filled into the per-thread page table.

[0099] If the corresponding PTE does not exist in either the per-thread page table or the global page table, then we get a real page fault, which is case (2). For case (2), the page fault handler first follows the original page fault handling logic in the operating system kernel to get a valid PTE. Then we lock the per-thread page table and the global page table, fill only this PTE into the per-thread page table, and mark the global PTE 202 with a special forward pointer to the per-thread PTE 201 (referred to by Figure 2 The page is now in the "private" state, so the IPI broadcast can be skipped when performing a TLB shootdown.

[0100] For a kernel with the global page table loaded in CR3, page faults can be divided into two cases: (1) the PTE in the global page table is a forward pointer to an entry in the per-thread page table and the PTE is invalid; (2) the PTE does not exist in the global page table.

[0101] If it is case (1), and the PTE is a forward pointer, the operating system should lock the global page table and the per-thread page table pointed to by the forward pointer, copy the PTE value into the global page table, and mark this PTE as shared in the core bitmap.

[0102] If it is case (2), the processing method is the same as the processing method of case (2) of the page fault when the CPU core with per-thread page table loaded by CR3 above. That is, if it is case (2) that the PTE does not exist in the global page table, the global page table can be locked, the page fault can be resolved by filling the PTE, and then the global page table can be unlocked.

[0103] The following reference Figure 2 Describes TLB miss handling. For a mutator thread, when the CPU core it is in encounters a TLB miss, the MMU of the CPU core will first traverse the per-thread page table of the current mutator thread. If the PTE is valid, the MMU can resolve the TLB miss by caching the PTE. If the per-thread PTE does not exist, the MMU will trigger a page fault, which is then handled by the page fault handler. For non-mutator threads, since they use the global page table directly, the MMU will traverse the global page table on a TLB miss. If the required PTE does not exist or the PTE forward pointer points to an entry in the per-thread page table, a page fault is generated and handled by the page fault handler.

[0104] The following reference Figure 2 Describe TLB shootdown processing. When an initiator core of the CPU wants to shoot down a PTE, the operating system kernel will first detect the sharing status of the PTE in the global page table. If the PTE in the global page table is private, the initiator core will shoot down the PTE from its own TLB and skip the IPI broadcast; if the PTE in the global page table is shared, the initiator core will only send an IPI request to the core recorded in the core bitmap, and the core schedules a non-mutator thread. In either case, a lot of IPI overhead can be saved.

[0105] The following reference Figure 2Describes page mapping changes. When the initiator core wants to change a PTE, it needs to maintain the consistency of its per-thread page table and the global page table. It will first lock its per-thread page table and the global page table, and detect the sharing status of the PTE of the global page table. If the PTE is private, then the CPU core only needs to change the PTE directly in its per-thread page table. The global page table forward pointer remains unchanged in this process, so it does not need to be updated. If the PTE is shared, then the initiator core needs to invalidate the PTE in the per-thread page tables of other cores that share the PTE. For those per-thread page tables that do not currently schedule threads, their modifications can be postponed until they are scheduled.

[0106] The following reference Figure 2 Describes thread scheduling. The operating system always maintains a per-core mutator thread flag each time it schedules a Go thread, such as Figure 2 As shown. Each time the OS starts a mutator thread, it allocates a newly allocated per-thread page table for the mutator thread and sets the page table address into the CR3 register of the scheduled core. Later when the OS stops the thread, the OS first locks the global page table, scans the per-thread page table, and evicts all private PTEs from the per-thread page table into the global page table by replacing the forward pointers in the global page table with the real PTE values. We use one of the customizable bits in the PTE to mark whether the PTE in the global page table is a forward pointer or a real PTE value. Note that the PTE in the global page table may not be used by any CPU core, but this does not affect correctness. Later for shared PTEs, the OS clears the corresponding bit in its CPU core bitmap, since the core bitmap can accurately track all cores that reference the PTE. At this point the page table consistency is restored. Then the OS flushes all TLB entries related to the descheduled thread from the core where the thread resides and unlocks the global page table. For non-mutator threads, they always use the global page table when they are scheduled. Likewise, the OS flushes all TLB entries from the core where the thread resides, without changing the global page table.

[0107] In one embodiment of the present disclosure, step S104 includes:

[0108] Based on a translation lookup buffer miss when performing a shootdown on the translation lookup buffer, traversing a per-thread page table following the same page table structure using a memory management unit in a CPU core that caches the page table entry in the translation lookup buffer;

[0109] Based on the traversal result of the memory management unit in the CPU core, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

[0110] In one embodiment of the present disclosure, performing a shootdown on a translation lookup buffer (TLB shootdown) may refer to shooting down a PTE in the TLB, or may refer to flushing the TLB.

[0111] According to the technical solution provided by the embodiment of the present disclosure, based on the shared status of the detected page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, including: based on the situation that a translation lookup buffer miss occurs when the shootdown is performed on the translation lookup buffer, a memory management unit in the CPU core that caches the page table entries in the translation lookup buffer is used to traverse a per-thread page table that follows the same page table structure; based on the traversal result of the memory management unit in the CPU core, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing, the scalability problem of TLB shootdown in page swapping operations can be solved by using per-thread page tables, and the computational overhead of the page swapping process can be reduced.

[0112] In one embodiment of the present disclosure, the page fault processing includes at least one of the following processing:

[0113] For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of a corresponding page table entry in the per-thread page table, the per-thread page table is enabled to obtain the missing page table entry through the global page table;

[0114] For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of corresponding page table entries in the per-thread page table and the global page table, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and a page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table;

[0115] For a CPU core loaded with a global page table, when a page fault is caused by an invalid forward pointer of a page table entry pointing to the per-thread page table, the operating system kernel copies the value of the page table entry to the global page table, and marks the page table entry as shared in the CPU core bitmap of the page table entry;

[0116] For a CPU core loaded with a global page table, when a page fault is caused by a non-existent page table entry, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked using a specific forward pointer pointing to the page table entry of the per-thread page table.

[0117] According to the technical solution provided by the embodiment of the present disclosure, the page fault processing includes at least one of the following processing: for a CPU core loaded with a per-thread page table, when a page fault error is caused by a lack of a corresponding page table entry in a per-thread page table, the per-thread page table is enabled to obtain the missing page table entry through the global page table; for a CPU core loaded with a per-thread page table, when a page fault error is caused by a lack of a corresponding page table entry in a per-thread page table and in the global page table, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table; for a CPU core loaded with a global page table, In the case where a page fault is caused by an invalid specific forward pointer of a page table entry pointing to the per-thread page table, the value of the page table entry is copied to the global page table through the operating system kernel, and the page table entry is marked as shared in the CPU core bitmap of the page table entry; for a CPU core loaded with a global page table, in the case where a page fault is caused by the non-existence of a page table entry, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table. The scalability problem of TLB shootdown in a page swapping operation can be solved by using a per-thread page table, and the computational overhead of the page swapping process can be reduced.

[0118] In one embodiment of the present disclosure, step S104 includes:

[0119] Based on the detected fact that the page table entry of the global page table is in a private state in the global page table, the CPU core that schedules the thread involved in the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or

[0120] According to the detection that the page table entries of the global page table are in a shared state in the global page table, the CPU core that schedules the thread involved in the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core that schedules the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread.

[0121] In one embodiment of the present disclosure, the first type of thread may refer to a mutator thread, and the second type of thread may refer to a thread other than a mutator thread, such as a GC thread.

[0122] According to the technical solution provided by the embodiment of the present disclosure, according to the shared state of the detected page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core of the thread involved in the scheduling of the page fault processing, including: according to the detected page table entry of the global page table being in a private state in the global page table, the CPU core of the thread involved in the scheduling of the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or according to the detected page table entry of the global page table being in a shared state in the global page table, the CPU core of the thread involved in the scheduling of the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core of the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread, the scalability problem of TLB shootdown in the page swapping operation can be solved by using per-thread page tables, and the computational overhead of the page swapping process can be reduced.

[0123] The following will refer to Figure 3 Describes the scenario in which the swapped-out thread is exported as a special Goroutine in the paging method.

[0124] Figure 3 A schematic diagram showing a scenario in which a swapped-out thread is exported as a special Goroutine in a page swap method according to an embodiment of the present disclosure so that the thread is scheduled together with the user Goroutine by the Go runtime.

[0125] In the embodiments of the present disclosure, lightweight threads are described by taking Goroutine as an example of lightweight threads.

[0126] The following reference Figure 3 Description Export the swapped-out thread (in the kernel space operating system thread) as a special Goroutine ( Figure 3 As shown, Goroutine T0 and T1 are swapped out. Figure 3 As shown, the Go runtime in user space schedules the swapped-out Goroutine together with the user Goroutine.

[0127] exist Figure 3In the illustrated embodiment, the swap-out thread is exported as a Goroutine using the operating system kernel and the Go runtime, so that the swap-out operation can be performed asynchronously. Exporting the swap-out as a Goroutine can benefit the application in three aspects. First, asynchronous swap-out can release some cold pages in advance before demand swap-in, thereby shortening the demand swap-in delay and thus shortening the overall delay. Second, the Go runtime can schedule the swap-out Goroutine when the user Goroutine cannot fully utilize CPU resources, thereby improving overall CPU utilization and performance. Third, moving the swap-out out of the critical path of request page fault handling means that the Go runtime can overlap the swap-out communication with the user calculation, thereby further improving CPU utilization and NVM bandwidth utilization.

[0128] For example, at the kernel level, a new system call `user_swapout` can be implemented to export the swapped-out thread to the Go runtime in user space. The logic of `user_swapout` is similar to the current swap-out function in the kernel, but there are two main differences. First, given that Goroutines are cooperatively scheduled, several "safe points" need to be added in `user_swapout` where the system call saves its state and returns to the user level. Therefore, the Go runtime can safely pause the swapped-out Goroutine and reallocate the CPU to other Goroutines. We divide page swapping into five steps: (1) scan the kernel active / inactive lists and select a batch of pages to swap out; (2) search for PTEs that reference these victim physical pages through reverse mapping; (3) allocate swap items and insert swap items into the swap cache; (4) unmap pages and shoot down TLB; (5) write page contents to NVM. A safe point can be inserted at the end of each step.

[0129] In order to correctly restore the swapped out Goroutine when it is rescheduled, the Goroutine state information, such as the stack and other auxiliary state information, needs to be saved at each safe point. Since swapping out is performed at the kernel level, it is not possible to store the kernel stack and data in the user-level Go runtime. Instead, when the Go runtime is started, a set of memory areas is allocated in the kernel space to store the swapped out state information. The swapped out Goroutine can allocate an area from this set of memory areas and save its execution state into it (see Figure 3The `user_swapout` system call returns an ID to the Go runtime at each safepoint, which is the index of the region where its state is stored. The Go runtime remembers this ID just for the swapped-out goroutine and passes it back to the `user_swapout` system call the next time the swapped-out goroutine is rescheduled. The system call can follow the ID to retrieve the previous state and resume execution.

[0130] At the user level, the Go runtime can be modified to co-schedule swapped-out Goroutines and user Goroutines. To prevent swapped-out Goroutines from consuming too much CPU resources and starving user Goroutines, swapped-out Goroutines can be assigned a lower priority than normal user Goroutines and only scheduled when more available user Goroutines cannot be scheduled. The scheduling policy is also customizable, and users can adjust the policy or implement their own policy to achieve higher throughput or lower latency.

[0131] In one embodiment of the present disclosure, the page swap method further includes:

[0132] The operating system kernel and the runtime of the programming language with lightweight concurrency are used to export the swap-out thread executed in the page fault processing as a swap-out lightweight thread, so that the swap-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, so that the swap-out lightweight thread is executed asynchronously.

[0133] According to the technical solution provided by the embodiment of the present disclosure, the swap-out thread executed in the page fault processing is exported as a swap-out lightweight thread by utilizing the operating system kernel and the runtime of the programming language with lightweight concurrency, so that the swap-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, so that the swap-out lightweight thread is executed asynchronously, and the swap-out thread can be decoupled into some swap-out lightweight threads and co-scheduled with ordinary user lightweight threads to asynchronously perform the swap-out operation, thereby reducing the swap-out overhead of the programming language with lightweight concurrency.

[0134] In one embodiment of the present disclosure, the page swap method further includes:

[0135] Inserting safe points for a plurality of steps executed by said swapping out lightweight threads;

[0136] Allocating a set of memory areas in the operating system kernel to store swap-out state information when starting the runtime;

[0137] The swapped-out lightweight thread allocates a sub-memory area from the group of memory areas and saves the execution state of the swapped-out lightweight thread to the sub-memory area;

[0138] Returning, by the application kernel, an identification as an index of the sub-memory region to the runtime at each security point;

[0139] The identifier is passed back to the operating system kernel by the runtime when rescheduling the swapped out lightweight thread.

[0140] According to the technical solution provided by the embodiment of the present disclosure, by inserting security points for multiple steps executed by the swapped-out lightweight thread; allocating a group of memory areas in the operating system kernel to store swap-out status information when starting the runtime; the swapped-out lightweight thread allocates a sub-memory area from the group of memory areas and saves the execution status of the swapped-out lightweight thread to the sub-memory area; returning an identifier as an index of the sub-memory area to the runtime through the application kernel at each security point; and passing the identifier back to the operating system kernel when rescheduling the swapped-out lightweight thread by the runtime, the swapped-out thread can be decoupled into some swapped-out lightweight threads, and co-scheduled with ordinary user lightweight threads to asynchronously perform swap-out operations, thereby reducing the swap-out overhead of programming languages ​​with lightweight concurrency.

[0141] In one embodiment of the present disclosure, scheduling the swapped out lightweight thread together with the user lightweight thread managed and scheduled by the runtime through the runtime includes:

[0142] The swapped-out lightweight thread is assigned, by the runtime, a lower priority than the user lightweight thread.

[0143] According to the technical solution provided by the embodiment of the present disclosure, the swapped-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, including: the swapped-out lightweight thread is assigned a lower priority than the user lightweight thread through the runtime, so that the swapped-out thread can be decoupled into some swapped-out lightweight threads and co-scheduled with ordinary user lightweight threads to asynchronously perform the swap-out operation, thereby reducing the swap-out overhead of the programming language with lightweight concurrency.

[0144] In an embodiment of the present disclosure, the per-thread page table is suitable for the x86 / x86-64 architecture because its TLB is managed by hardware. The present disclosure embodiment is directed to an application written in Golang that runs on a hybrid storage architecture where NVM can be used as a swap device.

[0145] In the page swapping method described in the present disclosure, the swapping out overhead of the hybrid storage structure can be reduced by: (1) using per-thread page tables and TLB shootdown operations to track page mapping caches and reduce TLB shootdown overhead; (2) treating the swapping out as a lightweight thread scheduled by the programming language runtime with garbage collection and lightweight concurrency through the operating system kernel and the programming language runtime with garbage collection and lightweight concurrency to perform the swapping out asynchronously.

[0146] The following reference Figure 4 A page swapping device according to an embodiment of the present disclosure is described. Figure 4 FIG. 4 shows a structural block diagram of a page switching device 400 according to an embodiment of the present disclosure. Figure 4 As shown, the page exchange device 400 includes a per-thread page table allocation module 401, a global page table allocation module 402, a page table entry detection module 403 and a conversion lookup buffer execution shootdown processing module 404.

[0147] The per-thread page table allocation module 401 is configured to allocate a per-thread page table to a first type of thread to record page mappings of pages locally accessed by the first type of thread.

[0148] The global page table allocation module 402 is configured to allocate a global page table to the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process.

[0149] The page table entry detection module 403 is configured to detect the sharing status of the page table entries of the global page table in the global page table when performing page fault processing during page swapping, wherein the page table entries in the global page table are accompanied by a CPU core bitmap to track all CPU cores that cache the page table entries in the translation lookup buffer, thereby determining the sharing status of the page table entries.

[0150] The translation lookup buffer execution shootdown processing module 404 is configured to execute shootdown processing on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing based on the detected sharing status of the page table entries of the global page table in the global page table and based on the per-thread page table and the global page table involved in the page fault processing.

[0151] According to the technical solution provided by the embodiment of the present disclosure, a per-thread page table allocation module is configured to allocate a per-thread page table for a first type of thread to record the page mapping of pages locally accessed by the first type of thread; a global page table allocation module is configured to allocate a global page table for the process to which the first type of thread belongs to record the page mapping of pages accessed by threads in the process; a page table entry detection module is configured to detect the sharing status of the page table entries of the global page table in the global page table when performing page fault processing during page swapping, wherein the page table entries in the global page table are attached with CPU core A bitmap is provided to track all CPU cores that cache the page table entries in the translation lookup buffer, so as to determine the sharing status of the page table entries; a translation lookup buffer execution shootdown processing module is configured to perform shootdown processing on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing based on the detected sharing status of the page table entries of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing. The scalability problem of TLB shootdown in page swapping operation can be solved by using per-thread page table, and the computational overhead of the page swapping process can be reduced.

[0152] Those skilled in the art will understand that referring to Figure 4 The technical solutions described can be compared with Figures 1 to 3 The described embodiments are combined to provide a reference Figures 1 to 3 The technical effects achieved by the described embodiments. Figures 1 to 3 The specific content of the description will not be repeated here.

[0153] Figure 5 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0154] The present disclosure also provides an electronic device, such as Figure 5 As shown, it includes at least one processor 501; and a memory 502 that is communicatively connected to the at least one processor 501; wherein the memory 502 stores instructions that can be executed by the at least one processor 501, and the instructions are executed by the at least one processor 501 to implement the following steps:

[0155] Allocating a per-thread page table for a first type of thread to record page mappings of pages locally accessed by the first type of thread;

[0156] Allocating a global page table for the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process;

[0157] In the case of performing page fault processing when performing page swapping, detecting a sharing state of a page table entry of the global page table in the global page table, wherein the page table entry in the global page table is accompanied by a CPU core bitmap to track all CPU cores that cache the page table entry in a translation lookup buffer, thereby determining the sharing state of the page table entry;

[0158] According to the detected sharing status of the page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

[0159] In one embodiment of the present disclosure, the performing of a shootdown process on a translation lookup buffer of a CPU core that schedules a thread involved in the page fault processing based on a per-thread page table and a global page table involved in the page fault processing according to the detected sharing state of the page table entry of the global page table in the global page table comprises:

[0160] Based on a translation lookup buffer miss when performing a shootdown on the translation lookup buffer, traversing a per-thread page table following the same page table structure using a memory management unit in a CPU core that caches the page table entry in the translation lookup buffer;

[0161] Based on the traversal result of the memory management unit in the CPU core, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

[0162] In one embodiment of the present disclosure, the page fault processing includes at least one of the following processing:

[0163] For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of a corresponding page table entry in the per-thread page table, the per-thread page table is enabled to obtain the missing page table entry through the global page table;

[0164] For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of corresponding page table entries in the per-thread page table and the global page table, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and a page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table;

[0165] For a CPU core loaded with a global page table, when a page fault is caused by an invalid forward pointer of a page table entry pointing to the per-thread page table, the operating system kernel copies the value of the page table entry to the global page table, and marks the page table entry as shared in the CPU core bitmap of the page table entry;

[0166] For a CPU core loaded with a global page table, when a page fault is caused by a non-existent page table entry, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked using a specific forward pointer pointing to the page table entry of the per-thread page table.

[0167] In one embodiment of the present disclosure, the performing of a shootdown process on a translation lookup buffer of a CPU core that schedules a thread involved in the page fault processing based on a per-thread page table and a global page table involved in the page fault processing according to the detected sharing state of the page table entry of the global page table in the global page table comprises:

[0168] Based on the detected fact that the page table entry of the global page table is in a private state in the global page table, the CPU core that schedules the thread involved in the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or

[0169] According to the detection that the page table entries of the global page table are in a shared state in the global page table, the CPU core that schedules the thread involved in the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core that schedules the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread.

[0170] In one embodiment of the present disclosure, the memory 502 further stores instructions that can be executed by at least one processor 501, and the instructions are executed by at least one processor 501 to implement the following steps:

[0171] The operating system kernel and the runtime of the programming language with lightweight concurrency are used to export the swap-out thread executed in the page fault processing as a swap-out lightweight thread, so that the swap-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, so that the swap-out lightweight thread is executed asynchronously.

[0172] In one embodiment of the present disclosure, the memory 502 further stores instructions that can be executed by at least one processor 501, and the instructions are executed by at least one processor 501 to implement the following steps:

[0173] Inserting safe points for a plurality of steps executed by said swapping out lightweight threads;

[0174] Allocating a set of memory areas in the operating system kernel to store swap-out state information when starting the runtime;

[0175] The swapped-out lightweight thread allocates a sub-memory area from the group of memory areas and saves the execution state of the swapped-out lightweight thread to the sub-memory area;

[0176] Returning, by the application kernel, an identification as an index of the sub-memory region to the runtime at each security point;

[0177] The identifier is passed back to the operating system kernel by the runtime when rescheduling the swapped out lightweight thread.

[0178] In one embodiment of the present disclosure, scheduling the swapped out lightweight thread together with the user lightweight thread managed and scheduled by the runtime through the runtime includes:

[0179] The swapped-out lightweight thread is assigned, by the runtime, a lower priority than the user lightweight thread.

[0180] In one embodiment of the present disclosure, page swapping is performed in a hybrid memory architecture, which is a hybrid memory architecture including at least a dynamic random access memory DRAM and a non-volatile memory NVM.

[0181] Figure 6 Schematic diagram of the structure of a computer system suitable for implementing the method according to each embodiment of the present disclosure. Figure 6 As shown, the computer system 600 includes a processing unit 601, which can perform various processes in the embodiments shown in the above figures according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The processing unit 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0182] The following components are connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. The drive 610 is also connected to the I / O interface 905 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 610 as needed, so that the computer program read therefrom is installed into the storage part 608 as needed. Among them, the processing unit 601 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0183] In particular, according to an embodiment of the present disclosure, the method described above with reference to the accompanying drawings can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing the method in the accompanying drawings. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611. For example, an embodiment of the present disclosure includes a readable storage medium on which computer instructions are stored, and when the computer instructions are executed by a processor, the program code for executing the method in the accompanying drawings is implemented.

[0184] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0185] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0186] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the node described in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0187] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A page swap method, wherein: The method comprises: Allocating a per-thread page table for a first type of thread to record page mappings of pages locally accessed by the first type of thread; Allocating a global page table for the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process; In the case of performing page fault processing when performing page swapping, detecting a sharing state of a page table entry of the global page table in the global page table, wherein the page table entry in the global page table is accompanied by a CPU core bitmap to track all CPU cores that cache the page table entry in a translation lookup buffer, thereby determining the sharing state of the page table entry; According to the detected sharing state of the page table entry of the global page table in the global page table, based on the per-thread page table and the global page table involved in the page fault processing, performing a shootdown process on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing; The method of performing a shootdown process on a translation lookup buffer of a CPU core that schedules a thread involved in the page fault processing based on a per-thread page table and a global page table involved in the page fault processing according to the detected shared state of the page table entry of the global page table in the global page table comprises: Based on the detected fact that the page table entry of the global page table is in a private state in the global page table, the CPU core that schedules the thread involved in the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or According to the detection that the page table entries of the global page table are in a shared state in the global page table, the CPU core that schedules the thread involved in the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core that schedules the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread.

2. The method according to claim 1, wherein: The method of performing a shootdown process on a translation lookup buffer of a CPU core that schedules a thread involved in the page fault processing based on a per-thread page table and a global page table involved in the page fault processing according to the detected shared state of the page table entry of the global page table in the global page table comprises: Based on a translation lookup buffer miss when performing a shootdown on the translation lookup buffer, traversing a per-thread page table following the same page table structure using a memory management unit in a CPU core that caches the page table entry in the translation lookup buffer; Based on the traversal result of the memory management unit in the CPU core, based on the per-thread page table and the global page table involved in the page fault processing, a shootdown process is performed on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing.

3. The method according to claim 1 or 2, wherein: The page fault processing includes at least one of the following processing: For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of a corresponding page table entry in the per-thread page table, the per-thread page table is enabled to obtain the missing page table entry through the global page table; For a CPU core loaded with a per-thread page table, when a page fault is caused by a lack of corresponding page table entries in the per-thread page table and the global page table, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and a page table entry in the global page table is marked with a specific forward pointer pointing to the page table entry of the per-thread page table; For a CPU core loaded with a global page table, when a page fault is caused by an invalid forward pointer of a page table entry pointing to the per-thread page table, the operating system kernel copies the value of the page table entry to the global page table, and marks the page table entry as shared in the CPU core bitmap of the page table entry; For a CPU core loaded with a global page table, when a page fault is caused by a non-existent page table entry, a valid page table entry is obtained through the operating system kernel, the obtained page table entry is filled into the per-thread page table, and the page table entry in the global page table is marked using a specific forward pointer pointing to the page table entry of the per-thread page table.

4. The method according to claim 1, wherein: The first type of thread is created and started by a runtime of a programming language with lightweight concurrency.

5. The method according to claim 4, wherein: The method further comprises: The operating system kernel and the runtime of the programming language with lightweight concurrency are used to export the swap-out thread executed in the page fault processing as a swap-out lightweight thread, so that the swap-out lightweight thread is scheduled together with the user lightweight thread managed and scheduled by the runtime through the runtime, so that the swap-out lightweight thread is executed asynchronously.

6. The method according to claim 5, wherein: The method further comprises: Inserting safe points for a plurality of steps executed by said swapping out lightweight threads; Allocating a set of memory areas in the operating system kernel to store swap-out state information when starting the runtime; The swapped-out lightweight thread allocates a sub-memory area from the group of memory areas and saves the execution state of the swapped-out lightweight thread to the sub-memory area; Returning, by the application kernel, an identification as an index of the sub-memory region to the runtime at each security point; The identifier is passed back to the operating system kernel by the runtime when rescheduling the swapped out lightweight thread.

7. The method according to claim 5 or 6, wherein: The step of scheduling the swapped out lightweight thread together with the user lightweight thread managed and scheduled by the runtime through the runtime includes: The swapped-out lightweight thread is assigned, by the runtime, a lower priority than the user lightweight thread.

8. The method according to claim 1 or 2, wherein: Page swapping is performed in a hybrid memory architecture, which is a hybrid memory architecture including at least a dynamic random access memory DRAM and a non-volatile memory NVM.

9. A page swapping device, wherein: The device comprises: a per-thread page table allocation module, configured to allocate a per-thread page table for a first type of thread to record page mappings of pages locally accessed by the first type of thread; A global page table allocation module is configured to allocate a global page table to the process to which the first type of thread belongs to record page mappings of pages accessed by threads in the process; a page table entry detection module, configured to detect a sharing state of a page table entry of the global page table in the global page table when performing a page fault process during page swapping, wherein a CPU core bitmap is attached to the page table entry in the global page table to track all CPU cores that cache the page table entry in a translation lookup buffer, thereby determining the sharing state of the page table entry; a translation lookup buffer execution shootdown processing module, configured to execute shootdown processing on the translation lookup buffer of the CPU core that schedules the thread involved in the page fault processing based on the detected sharing state of the page table entry of the global page table in the global page table and based on the per-thread page table and the global page table involved in the page fault processing; The conversion search buffer execution shootdown processing module is specifically configured as follows: Based on the detected fact that the page table entry of the global page table is in a private state in the global page table, the CPU core that schedules the thread involved in the page fault processing will shoot down the page table entry from its own translation lookup buffer and skip the inter-processor interrupt broadcast, or According to the detection that the page table entries of the global page table are in a shared state in the global page table, the CPU core that schedules the thread involved in the page fault processing will only send an inter-processor interrupt request to the CPU core recorded in the CPU core bitmap, and the CPU core that schedules the thread involved in the page fault processing and the CPU core recorded in the CPU core bitmap schedule the second type of thread other than the first type of thread.

10. An electronic device comprising a memory and a processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps described in any one of claims 1-8.

11. A readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method steps described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions, which, when executed by a processor, implement the method steps of any one of claims 1 to 8.

Images