Layered memory page allocation method based on process virtual address space segmentation

Abstract

The invention relates to the technical field of novel storage of a data center, in particular to a hierarchical memory page allocation method based on process virtual address space segmentation, which comprises the following steps of: constructing a segmentation-based data page allocation node selection mechanism in a hierarchical memory, and managing an allocation strategy of process data pages in a hierarchical memory architecture by utilizing the mechanism. Different address fields of the virtual address space are divided into different page access heat degrees, and physical pages are distributed from different memory nodes; a segmentation-based page table page allocation node selection mechanism is constructed in a hierarchical memory, a space allocation strategy of a process page table space in a hierarchical memory architecture is managed by utilizing the mechanism, and physical spaces are allocated to different virtual address space segmentation page table pages from different hierarchical memory nodes. Through a data page distribution node selection mechanism based on segmentation and a page table page distribution node selection mechanism based on segmentation, the access delay of a newly distributed page is reduced, and the migration frequency of the page between the nodes is reduced.

Description

Technical Field

[0001] The present invention relates to the field of novel storage technology for data centers, and in particular to a hierarchical memory page allocation method based on process virtual address space segmentation. Background Art

[0002] Emerging applications such as high-performance computing, deep learning, and graphics computing have increasing demands for the capacity and performance of computer memory resources. The growth rate of traditional DRAM density is gradually slowing down, and the cost and complexity of expanding to larger capacity are increasing, causing memory to become a major infrastructure expenditure for hyperscale data centers. Data center infrastructure needs to be optimized to access and analyze massive data sets to create business value. In recent years, tiered memory technology has built tiered memory systems and alleviated the demand for large amounts of memory capacity in data centers with lower storage unit prices and larger memory capacity. Tiered memory systems deploy multiple memory layers with different characteristics, including capacity, access latency, bandwidth, energy consumption, and volatility. The upper layer of memory has lower access latency, and the lower layer of memory has larger storage capacity. For example, persistent memory and CXL memory are used as a lower layer of memory of DRAM. Through tiered memory technology, DRAM capacity can be used for hot data that requires high-performance access, and the capacity layer can be used to process tasks that require large capacity but less performance requirements.

[0003] In order to achieve higher throughput, reliability and economy of computer systems, the NUMA (Non-Uniform Memory Access) architecture is widely used in high-performance computing and cloud computing data center servers. The NUMA architecture system divides the processor and memory into nodes for management, avoiding the scalability problem caused by the increasing memory access conflicts when multiple processors address the same memory address space, and has lower local memory access latency and better scalability. In Linux, a hierarchical memory system is built based on the NUMA architecture, and the lower-level memory with low access latency and large capacity is managed as a CPU-less NUMA node (CPU-less Node). As the new type of memory becomes part of the main memory, kernel data and user process data can allocate space from fast local nodes or remote nodes with higher access latency. The key factor of performance needs to consider not only the locality of program access, but also the characteristics of different memories.

[0004] In order to take advantage of the characteristics of hierarchical memory, many research works have been carried out. The Linux kernel uses a technique called NUMAhint-faulting to detect whether a given page needs to be migrated. The kernel periodically unmaps the process virtual address to the physical page so that subsequent access to the page in that range will trigger a page fault. When a page fault occurs, the memory management subsystem can use the location of the CPU that triggered the page fault to determine whether the page needs to be migrated to the node containing the CPU. AutoTiering takes into account more complex NUMA topologies based on AutoNUMA, allowing pages to migrate, demote, and promote between NUMA nodes with CPUs. TPP uses the default LRU mechanism in Linux to obtain the least recently used page on the node, demote it to the underlying slower memory device, and migrate the page from the underlying memory to DRAM based on the popularity of the page being accessed again in the underlying slower device.

[0005] For the memory allocation strategy of the hierarchical memory structure, Linux uses DRAM priority as its allocation strategy, that is, allocating pages from the higher-performance DRAM until it is full, and then all future allocations are made from the lower-performance underlying memory, which will cause the DRAM to be quickly occupied and space to be allocated from the slower memory, resulting in a large number of page migrations. Many page-oriented access features are used to guide the allocation of memory space. For example, TPP uses the PEBS mechanism supported in the CPU to record the behavioral characteristics of application memory access, thereby generating a heat map of memory usage on different types of pages, allocating physical space for file cache pages from memory nodes with higher underlying access latency, and allocating physical space for anonymous pages from local DRAM. Porter middleware provides appropriate memory allocation for Serverless applications. Therefore, selecting the appropriate memory allocation node to allocate space for the application process space can avoid the situation where hotspot access data is allocated from remote nodes due to insufficient DRAM space on the local node, effectively reducing the access latency of newly allocated pages and reducing the number of page migrations between nodes. Summary of the invention

[0006] In order to reduce the access delay of newly allocated pages and reduce the number of page migrations between nodes, the present invention proposes a hierarchical memory page allocation method based on process virtual address space segmentation, including constructing a segmentation-based data page allocation node selection mechanism in the hierarchical memory, using the mechanism to manage the allocation strategy of process data pages in the hierarchical memory architecture, dividing different address segments of the virtual address space into different page access hotnesses, and allocating physical pages from different memory nodes; constructing a segmentation-based page table page allocation node selection mechanism in the hierarchical memory, using the mechanism to manage the space allocation strategy of the process page table space in the hierarchical memory architecture, and allocating physical space from different hierarchical memory nodes for page table pages of different virtual address space segments.

[0007] Furthermore, a segment-based data page allocation node selection mechanism is provided in the hierarchical memory. This mechanism represents different page access hotness according to different segments of the process virtual address space, and allocates physical space pages from different NUMA nodes, where:

[0008] The virtual address space of a Linux process is divided into user space segments and kernel space segments. The user space segment includes code segments, initialized data segments, uninitialized BSS segments, heap segments, stack segments, and shared library mapping segments.

[0009] Physical space is allocated from the memory on the underlying NUMA node with higher access latency for the code segment, data segment, BSS segment, and shared library mapping segment, and physical space is allocated from the memory on the DRAM node for the heap segment and stack segment.

[0010] Furthermore, the code segment, data segment, and shared library segment are loaded from the executable binary file, and the file cache page is allocated physical space from the underlying memory node with higher access latency; the heap segment and stack segment process dynamically allocate memory during operation, and the anonymous page is allocated physical space from the local DRAM.

[0011] Furthermore, the execution process of the segment-based data page allocation node selection mechanism in the hierarchical memory includes:

[0012] 301. The instruction in the parent process or the child process executes to write the address space of the shared data page, and the PTE-level page table entry flag bit of the parent process and the child process page table is set to write protection;

[0013] 302. The CPU will generate a page fault interrupt, indicating that the currently accessed virtual address space is illegal, or the physical space corresponding to the current virtual address space is empty, and no mapping from virtual address to physical address is established;

[0014] 303. The kernel captures the interrupt and executes the page fault interrupt handler;

[0015] 304. In the page fault interrupt handler, determine whether the currently accessed virtual address belongs to the heap segment or the stack segment of the process virtual address space. If so, execute step 306; otherwise, execute step 305.

[0016] 305. Allocate physical pages for the code segment, data segment, and shared library segment from the underlying PM or CXL-memory and then execute step 307;

[0017] 306. Allocate physical pages for the heap and stack virtual address space segments from the local DRAM;

[0018] 307. Copy the content of the original physical page to the new physical page and insert it into the process page table;

[0019] 308. Re-execute the instruction and write the corresponding address space data.

[0020] Furthermore, a segmentation-based page table page allocation node selection mechanism is constructed in the hierarchical memory. This mechanism allocates physical space from the memory on the underlying NUMA node with higher access latency for the PTE-level page tables of the process code segment, data segment, BSS segment, and shared library mapping segment space, while the page table space corresponding to other segments is allocated physical space from the local node DRAM.

[0021] Furthermore, Linux uses a four-level page table to manage address mapping. When allocating space for the page tables of the Page Global Directory (PGD), Page Upper Directory (PUD), and Page Middle Directory (PMD) from the local DRAM, the PTE layer page table space is allocated from different levels of memory space according to the different virtual address space segmentations. That is, the PTE layers corresponding to the address spaces of the code segment, data segment, and shared library segment are allocated from the CPU-less node memory with larger capacity and lower access latency, and the PTE layers corresponding to the heap and stack segment address spaces are allocated from the local DRAM memory with smaller capacity and faster access speed.

[0022] Furthermore, the execution process of constructing a segment-based page table page allocation node selection mechanism in the hierarchical memory includes:

[0023] 501. The user opens a program or calls the fork() library function to create a process;

[0024] 502. The kernel receives a request to create a process system call, executes the do_fork() function to call the copy_process() function to complete the creation of the task structure task_struck and the initialization of the memory space copy_mm();

[0025] 503. copy_mm() calls dum_mm() to allocate and initialize the mm_struct structure, and allocates a physical page for the first-level page table page of the newly created process from the local DRAM;

[0026] 504. Determine whether the number of VMAs in all virtual address space areas in the parent process is greater than 0. If so, loop through and execute S505 to allocate a page table physical page copy page table. Otherwise, jump to the end.

[0027] 505. Determine whether the page table page allocated for the current VMA address space belongs to the PTE level. If so, execute step 506; otherwise, execute step S508;

[0028] 506. The page table page currently to be allocated is at the PTE level. It is determined whether it is a code segment, a data segment, or a shared library segment according to the virtual address space area VMA range. If so, execute step 507; otherwise, execute step 508.

[0029] 507. Allocate physical pages from the underlying PM or CXL-memory for page table pages belonging to the code segment, data segment, or shared library segment at the PTE level;

[0030] 508. Allocate a physical page for the page table page from the local DRAM;

[0031] 509. Insert the allocated page table page into the newly created process page table.

[0032] The present invention reduces the access delay of newly allocated pages and the number of page migrations between nodes by using a segment-based data page allocation node selection mechanism in a hierarchical memory and a segment-based page table page allocation node selection mechanism in a hierarchical memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the architecture of the hierarchical memory page allocation method based on process virtual address space segmentation of the present invention;

[0034] Figure 2 A schematic diagram of a node selection mechanism for allocating data pages based on segmentation in a hierarchical memory of the present invention;

[0035] Figure 3 A flow chart of a node selection mechanism for allocating data pages based on segmentation in a hierarchical memory of the present invention;

[0036] Figure 4 A schematic diagram of a segment-based page table page allocation node selection mechanism in a hierarchical memory of the present invention;

[0037] Figure 5 The present invention is a flow chart of the segment-based page table page allocation node selection mechanism in the hierarchical memory. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The present invention proposes a hierarchical memory page allocation method based on process virtual address space segmentation, comprising: constructing a segmentation-based data page allocation node selection mechanism in the hierarchical memory, using the mechanism to manage the allocation strategy of process data pages in the hierarchical memory architecture, dividing different address segments of the virtual address space into different page access hotnesses, and allocating physical pages from different memory nodes; constructing a segmentation-based page table page allocation node selection mechanism in the hierarchical memory, using the mechanism to manage the space allocation strategy of the process page table space in the hierarchical memory architecture, and allocating physical space from different hierarchical memory nodes for page table pages of different virtual address space segments.

[0040] In existing hierarchical memory architecture systems, DRAM capacity is used for hot data that requires high-performance access, and the capacity layer is used to process cold data that requires large capacity. For the memory allocation strategy of the hierarchical memory structure, Linux uses DRAM priority as its allocation strategy, that is, pages are allocated from the higher-performance DRAM until they are full, and all future allocations are made from the lower-performance underlying memory. This will cause the DRAM to be quickly filled up, and space will be allocated from the slower memory, resulting in a large number of page migrations.

[0041] In the Linux kernel, physical memory management is managed using the NUMA architecture, and the UMA architecture is treated as a NUMA architecture with only one node. In the NUMA architecture, the CPU and memory in the system are divided into multiple NUMA nodes, each node contains a CPU and local memory, and the CPU in the node does not need to pass through the fast bus between nodes to access the local memory, so the access to the local memory has high bandwidth and low latency. Hierarchical memory divides the physical memory into different levels according to the different performance characteristics of the physical memory, and allocates memory to the workload running in the system in different ways. In order to support hierarchical memory, Linux manages hierarchical memory PM and CXL-memory as CPU-less NUMA nodes. The management of hierarchical memory space includes the selection of the target layer when allocating physical space, and moving data between different memory levels according to the access popularity of the monitoring data. The present invention belongs to the category of target layer selection when allocating physical space.

[0042] Figure 2Schematic diagram of node selection mechanism for segment-based data page allocation in hierarchical memory. Linux organizes the process virtual address space into a collection of regions. A region is a continuous piece of virtual memory that has been allocated. These pages are related in some way. For example, the code segment, data segment, heap, shared library segment, and user stack are all different regions. Each existing virtual page is stored in a region, and virtual pages that do not belong to a region do not exist and cannot be referenced by the process. Among them, the code segment, data segment, and shared library segment are loaded from the executable binary file, and the corresponding memory space in the kernel is the file cache page. The heap and stack segments are dynamically allocated during the process's operation, and the corresponding memory space in the kernel is an anonymous page. Therefore, the physical space of the process code segment, data segment, and shared library segment is allocated from the CPU-less node memory with larger capacity and lower access latency, and the physical space of the heap and stack segments is allocated from the local DRAM memory with smaller local capacity and faster access speed.

[0043] Figure 3 Flowchart of node selection mechanism for segment-based data page allocation in hierarchical memory. The current Linux kernel uses the copy-on-write COW technology to optimize the process of copying the parent process's space when the child process is created. Specifically, when creating a child process, the kernel only copies the page table data related to the parent process's address space to the child process, and modifies the page attribute in the PTE item pointing to the physical page to read-only, so that the parent process and the child process share the same memory, avoiding the delay of copying a large number of page data during the creation process; when the parent process or the child process needs to modify the content of a physical page, the address management unit hardware MMU will trigger a memory write protection interrupt. During the interrupt processing, the kernel allocates a new physical page for the corresponding process virtual address space and establishes a page mapping, so that the parent process and the child process have their own corresponding physical space. For example, when executing a command in the terminal, a child process will be created, and then execve() will be called to execute the child process's code. When allocating physical pages, different memory levels are selected to allocate physical memory space according to the virtual address space segment where the current page is located. Figure 3 The process of segment-based data page allocation node selection mechanism in hierarchical memory is shown, including the following steps:

[0044] S301, the instruction in the parent process or the child process executes the address space of the write shared data page, and the PTE-level page table entry flag bit in the parent process and the child process page table is set to write protection;

[0045] S302, the CPU will generate a page fault interrupt, indicating that the currently accessed virtual address space is illegal, or the physical space corresponding to the current virtual address space is empty, and no mapping from virtual address to physical address is established;

[0046] S303, the kernel captures the interrupt and executes the page fault interrupt handler;

[0047] S304, in the page fault interrupt handler, determine whether the currently accessed virtual address belongs to the heap segment or stack segment of the process virtual address space. If yes, execute step S306, otherwise execute step S305;

[0048] S305, allocating physical pages for the code segment, data segment, and shared library segment from the underlying PM or CXL-memory;

[0049] S306, allocating physical pages for the heap and stack virtual address space segments from the local DRAM;

[0050] S307, copy the content of the original physical page to the new physical page, and insert it into the process page table;

[0051] S308, re-execute the instruction and write the corresponding address space data.

[0052] Figure 4 Schematic diagram of the segment-based page table page allocation node selection mechanism in hierarchical memory. The page table is used to store the mapping relationship between the process virtual address and the physical address. Every time the address translation hardware converts a virtual address into a physical address, the page table is read. The page table is an array of page table entries (PTE). In order to avoid the page table occupying too much memory space and not all mapping relationships need to be kept in memory all the time, Linux now uses a 4-level page table to manage address mapping, and only the page table structure of the virtual address space that the process has allocated is saved in memory. The page table is the core data structure used by each process in memory. When creating a process, it is necessary to allocate memory space for the process page table and copy the page table data from the parent process page table. In a multi-layer memory architecture, physical pages in a page table can be allocated from different memory levels. Considering that a page table is a 4-layer tree structure, the present invention designs different physical page allocation strategies for page tables at different levels, allocates space for the page tables of PGD, PUD, and PMD from the local DRAM, and allocates PTE layer page table space from memory spaces at different levels according to different virtual address space segmentations, wherein the PTEs corresponding to the address spaces of the code segment, data segment, and shared library segment are allocated from the CPU-less node memory with a larger capacity and a lower access latency, and the PTEs corresponding to the address spaces of the heap and stack segments are allocated from the local DRAM memory with a smaller local capacity and a faster access speed.

[0053] Figure 5Flowchart of node selection mechanism for segmentation-based page table page allocation in hierarchical memory. The user program creates a child process by calling the fork() system call, and then falls into the kernel to execute the do_fork() function to call copy_process() to complete the creation of the task structure task_struck, the initialization of the memory space copy_mm(), etc. In the copy_mm() function, call dum_mm() to allocate and initialize the mm_struct structure, and then allocate the first-level page table page, and point to the base address of the first-level page table of this process in mm_struct; then loop through all virtual address space areas VMA in the parent process, call copy_page_range() for each VMA, traverse each virtual page in the VMA, and then copy the page table of the parent process to the child process; in the page table page allocation process, according to the PTE pages corresponding to different levels of page table pages and different process virtual address space segments, memory space is allocated from different levels of hierarchical memory. Figure 5 The process of allocating node selection mechanism for segment-based page table pages in hierarchical memory includes:

[0054] S501, the user opens a program or calls the fork() library function to create a process;

[0055] S502, the kernel receives a request to create a process system call, executes the do_fork() function to call the copy_process() function to complete the creation of the task structure task_struck, the initialization of the memory space copy_mm(), etc.;

[0056] S503, copy_mm() calls dum_mm() to allocate and initialize the mm_struct structure, and allocates a physical page for the first-level page table page of the newly created process from the local DRAM;

[0057] S504, determine whether the number of all virtual address space areas VMA in the parent process is greater than 0, if so, loop through and execute step S505 to allocate the page table physical page copy page table. Otherwise, jump to the end;

[0058] S505, determining whether the page table page allocated for the current VMA address space belongs to the PTE level, if so, executing S506, otherwise executing step S508, allocating a physical page for the page table page from the local DRAM;

[0059] S506, the page table page currently to be allocated is at the PTE level, and whether it is a code segment, a data segment, or a shared library segment is determined according to the virtual address space area VMA range. If so, step S507 is executed, otherwise, step S508 is executed;

[0060] S507, allocating physical pages from the bottom PM or CXL-memory for page table pages belonging to the code segment, data segment, and shared library segment at the PTE level;

[0061] S508, allocating a physical page for the page table page from the local DRAM;

[0062] S509, inserting the allocation page table page into the newly created process page table.

[0063] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A hierarchical memory page allocation method based on process virtual address space segmentation, characterized in that: A segmentation-based data page allocation node selection mechanism is constructed in the hierarchical memory. This mechanism is used to manage the allocation strategy of process data pages in the hierarchical memory architecture, and different address segments of the virtual address space are divided into different page access hotnesses, and physical pages are allocated from different memory nodes. A segmentation-based page table page allocation node selection mechanism is constructed in the hierarchical memory. This mechanism is used to manage the space allocation strategy of the process page table space in the hierarchical memory architecture, and physical space is allocated from different hierarchical memory nodes for page table pages of different virtual address space segments.

2. The hierarchical memory page allocation method based on process virtual address space segmentation according to claim 1, characterized in that: The segment-based data page allocation node selection mechanism in hierarchical memory represents different page access hotness according to different segments of the process virtual address space, and allocates physical space pages from different NUMA nodes, where: The virtual address space of a Linux process is divided into user space segments and kernel space segments. The user space segment includes code segments, initialized data segments, uninitialized BSS segments, heap segments, stack segments, and shared library mapping segments. Physical space is allocated from the memory on the underlying NUMA node with higher access latency for the code segment, data segment, BSS segment, and shared library mapping segment, and physical space is allocated from the memory on the DRAM node for the heap segment and stack segment.

3. The hierarchical memory page allocation method based on process virtual address space segmentation according to claim 2, characterized in that: The code segment, data segment, and shared library segment are loaded from the executable binary file, and the file cache page is allocated physical space from the underlying memory node with higher access latency; The heap segment and stack segment processes dynamically allocate memory during operation and allocate physical space for anonymous pages from local DRAM.

4. A hierarchical memory page allocation method based on process virtual address space segmentation according to claim 1, 2 or 3, characterized in that: The execution process of the segment-based data page allocation node selection mechanism in hierarchical memory includes:

301. The instruction in the parent process or the child process executes the address space of the write shared data page, and the PTE-level page table entry flag bit in the parent process and the child process page table is set to write protection; 302. The CPU will generate a page fault interrupt, indicating that the currently accessed virtual address space is illegal, or the physical space corresponding to the current virtual address space is empty, and no mapping from virtual address to physical address is established; 303. The kernel captures the interrupt and executes the page fault interrupt handler; 304. In the page fault interrupt handler, determine whether the currently accessed virtual address belongs to the heap segment or the stack segment of the process virtual address space. If so, execute step 306; otherwise, execute step 305.

305. Allocate physical pages for the code segment, data segment, and shared library segment from the underlying PM (Product Management) or CXL-Mem (Compute Express Link for Memory) and then execute step 307. In the traditional architecture, PM is mainly responsible for coordinating the CPU with other components. In the CPU-less Node, PM no longer coordinates resources and manages functions around the CPU, but focuses on the collaborative work between the CPU-less node and other CPU-equipped nodes or devices to ensure the functional integrity and efficiency of the entire system.

306. Allocate physical pages for the heap and stack virtual address space segments from the local DRAM; 307. Copy the content of the original physical page to the new physical page and insert it into the process page table; 308. Re-execute the instruction and write the corresponding address space data.

5. The hierarchical memory page allocation method based on process virtual address space segmentation according to claim 1, characterized in that: A segmentation-based page table page allocation node selection mechanism is constructed in the hierarchical memory. This mechanism allocates physical space for the PTE-level page tables of the process code segment, data segment, BSS segment, and shared library mapping segment space from the memory on the underlying NUMA node with higher access latency, while the page table space corresponding to other segments is allocated physical space from the local node DRAM.

6. The hierarchical memory page allocation method based on process virtual address space segmentation according to claim 5, characterized in that: Linux uses a four-level page table to manage address mapping, and allocates space for the page tables of the PGD layer, PUD layer, and PMD layer from the local DRAM. For the PTE layer page table space allocation, it is allocated from different levels of memory space according to the different virtual address space segmentations. That is, the PTE layer corresponding to the code segment, data segment, and shared library segment address space is allocated from the CPU-less node memory with larger capacity and lower access latency, and the PTE layer corresponding to the heap and stack segment address space is allocated from the local DRAM memory with smaller capacity and faster access speed.

7. A hierarchical memory page allocation method based on process virtual address space segmentation according to claim 5 or 6, characterized in that: The execution process of building a segment-based page table page allocation node selection mechanism in hierarchical memory includes:

501. The user opens a program or calls the fork() library function to create a process; 502. The kernel receives a request to create a process system call, executes the do_fork() function to call the copy_process() function to complete the creation of the task structure task_struck and the initialization of the memory space copy_mm(); 503. copy_mm() calls dum_mm() to allocate and initialize the mm_struct structure, and allocates a physical page for the first-level page table page of the newly created process from the local DRAM; 504. Determine whether the number of VMAs in all virtual address space areas in the parent process is greater than 0. If so, loop through and execute S505 to allocate a page table physical page copy page table. Otherwise, jump to the end.

505. Determine whether the page table page allocated for the current VMA address space belongs to the PTE level. If so, execute step 506; otherwise, execute step S508; 506. The page table page currently to be allocated is at the PTE level. It is determined whether it is a code segment, a data segment, or a shared library segment according to the virtual address space area VMA range. If so, execute step 507; otherwise, execute step 508.

507. Allocate physical pages from the underlying PM or CXL-memory for page table pages belonging to the code segment, data segment, or shared library segment at the PTE level; 508. Allocate a physical page for the page table page from the local DRAM; 509. Insert the allocated page table page into the newly created process page table.

Images