A page migration method and system for detached memory

Abstract

The application discloses a page migration method and system for separated memory and belongs to the field of data storage. The page migration method comprises the following steps: a node dynamically samples the access condition of a page through NUMA Balancing, traces and marks the access record of the page; in combination with an LRU list, hot pages are identified and other pages are marked as cold pages; the identified hot pages and cold pages are respectively stored in a hot migration queue and a cold migration queue; a background kernel thread continuously obtains the hot pages and cold pages to be migrated from the hot migration queue and the cold migration queue, performs asynchronous page migration, migrates the hot pages from a remote memory node to the node, and migrates the cold pages of the node to the remote memory node; and the access of a user to the pages is decoupled during the page migration. The real high-frequency access pages are preferentially migrated to the local memory node; the hit rate of local memory access is improved, and the delay of remote memory access is reduced.

Description

Technical Field

[0001] This invention belongs to the field of data storage, and more specifically, relates to a page migration method and system for discrete memory. Background Technology

[0002] Decoupled memory is a technology that integrates multiple independent memory units in a computer cluster into a unified logical memory space. Decoupled memory systems use a decoupled memory architecture to separate computing and memory resources into independent resource pools, connected by ultra-high-speed networks, which can improve memory utilization, reduce costs, and enable elastic scaling of computing and memory resources.

[0003] Existing memory decoupling solutions based on Remote Direct Memory Access (RDMA) suffer from high latency and overhead, primarily due to page faults. In these solutions, memory is distributed across multiple remote nodes. When a program needs to access memory on these remote nodes, a page fault is triggered if the memory page is not in local memory. A page fault forces the system to load the required page from the remote node into local memory, a process involving data transfer over the network. Even though RDMA reduces some network protocol overhead, network latency remains a significant factor. Each page fault requires initiating a network request and waiting for data transfer to complete, significantly increasing access latency. Furthermore, handling page faults requires substantial processing work from the operating system. The operating system must locate the required page, initiate a remote data request, wait for the data transfer to complete, and then load the page into local memory. This process is not only time-consuming but also requires additional computational resources, further increasing system overhead.

[0004] In split-memory systems, most research focuses on mitigating the impact of Page Faults after they occur, neglecting the importance of proactively preventing their occurrence. From a proactive perspective, proactive page migration can store frequently accessed data in the local memory of compute nodes and infrequently accessed data in remote memory, thereby reducing remote access and achieving proactive Page Fault prevention. Current research on page migration for split-memory systems mainly explores methods such as adapting to traditional NUMA strategies, utilizing hardware features like CXL for network-aware adaptive migration, optimizing heat measurement techniques, reducing application pauses during migration, and improving underlying operating system mechanisms. While these studies explore various technical approaches, they lack effective solutions that systematically combine and balance these three aspects. The universality and synergistic optimization of the three key aspects—migration target selection, migration timing, and migration method design—remain insufficient.

[0005] Therefore, there is an urgent need for a new method and system that can systematically optimize object selection, migration timing, and migration methods to achieve accurate identification, low-overhead asynchronous active migration, and significantly shorten the window of unavailable pages, so as to effectively improve the performance and software quality assurance level of the split memory system. Summary of the Invention

[0006] In view of the shortcomings of related technologies, the purpose of this invention is to provide a page migration method and system for separate memory, which aims to solve the problems of the lack of a systematic combination of migration object selection, migration timing and migration method design in the existing technology, as well as the lack of versatility and collaborative optimization.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a page migration method for split memory, comprising:

[0008] In a split memory environment, this node dynamically samples page access information through NUMA Balancing, tracks and marks page access records; it then uses an LRU list for static filtering to identify hot pages and mark other pages as cold pages.

[0009] Construct a hot migration queue and a cold migration queue, and store the identified hot pages and cold pages in the hot migration queue and the cold migration queue respectively;

[0010] The background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and performs asynchronous page migration, migrating hot pages from remote memory nodes to the local node and cold pages from the local node to remote memory nodes; during page migration, user access to the page is decoupled.

[0011] Optionally, in the split memory configuration, memory is distributed across multiple nodes. Each node dynamically samples page access information using NUMABalancing, tracking and marking page access records, including:

[0012] Enabling NUMA in memory nodes prompts errors, periodically scans the memory pages of running tasks, and the kernel dynamically samples page access information;

[0013] When a running task accesses a marked page, a Minor Page Fault is triggered through the NUMA Balancing mechanism, and the accessed page in memory is sampled. The kernel records the state of the accessed page, including the accessed object of the page, including local nodes and remote nodes, for real-time monitoring of memory access patterns.

[0014] Optionally, the step of combining the LRU list for static filtering to identify hot pages and mark other pages as cold pages includes:

[0015] The activity of the marked pages is verified using the LRU list. If a page is in the Inactive LRU, it is marked as visited and added to the Active LRU list. If a page triggers a minor page error again and is in the Active LRU, it is marked as a hot page. If a page is in the Inactive LRU for more than a certain time threshold, it is marked as a cold page.

[0016] Optionally, the step of constructing a hot migration queue and a cold migration queue, and storing the identified hot pages and cold pages in the hot migration queue and the cold migration queue respectively, includes:

[0017] Construct separate hot migration queues and cold migration queues;

[0018] Pages marked as hot pages are inserted into the hot migration queue, and pages marked as cold pages are inserted into the cold migration queue.

[0019] Optionally, the background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and performs asynchronous page migration, including:

[0020] t1. The background kernel thread kmove selects a page from the hot migration queue or the cold migration queue each time, clears the dirty bits of the page, and migrates the page.

[0021] t2. Send a TLB shutdown message to all kernels that have accessed the pages being migrated;

[0022] t3. Pages being migrated are copied between remote memory and local memory;

[0023] t4. Cancel PTE mapping and check for dirty bits in the pages being migrated;

[0024] t5. Send a TLB shutdown message to all kernels that have accessed the pages being migrated;

[0025] t6. Determine if the page is dirty. If the page is dirty, proceed to step t8; otherwise, proceed to step t7.

[0026] t7. Complete this page migration and remap the migrated pages in the operating system;

[0027] t8. Abort this page migration and restore the original PTE, then proceed to step t1.

[0028] Secondly, the present invention also provides a page migration system for discrete memory, comprising:

[0029] The page marking module is used in a distributed memory architecture where memory is distributed across multiple nodes. This node dynamically samples page access information through NUMABalancing, tracks and marks page access records; it combines LRU list for static filtering to identify hot pages and mark other pages as cold pages.

[0030] The queue construction module is used to build hot and cold migration queues, and store the identified hot and cold pages in the hot and cold migration queues respectively.

[0031] The page migration module is used by the background kernel thread to continuously retrieve hot and cold pages to be migrated from the hot and cold migration queues, and perform asynchronous page migration, migrating hot pages from remote memory nodes to the current node and cold pages from the current node to remote memory nodes; during page migration, user access to pages is decoupled.

[0032] Compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0033] 1. This invention provides a page migration method for split memory. In split memory, by combining NUMA Balancing and LRU lists for hot page identification, it effectively avoids the "ping-pong effect" common in NUMA Balancing, reduces unnecessary migration operations, lowers system resource consumption, and avoids performance degradation caused by excessive migration. The introduction of LRU lists allows the system to more accurately determine which pages are truly active hot pages, effectively filtering out "cold pages" with low access frequency or temporary access, and prioritizing the migration of truly high-frequency access pages to local memory nodes. This ensures that the migrated pages are most likely to be frequently accessed, thereby improving the hit rate of local memory access, reducing the latency of remote memory access, and further improving the overall memory access efficiency. Furthermore, by constructing a dual-queue mechanism of hot and cold migration queues for buffering, and using background threads to complete the migration operation asynchronously, the execution of user programs is avoided during the entire process, thus significantly improving the response performance of user programs. At the same time, this approach can effectively reduce data access latency, further optimize the overall operating efficiency of the system, and is conducive to optimizing the system's quality of service assurance.

[0034] 2. This invention provides a page migration method for separate memory. The redesigned migration method makes page migration completely asynchronous and decoupled from user access to the page. During the migration process, the time window when the page is inaccessible is between t4 and t7 / t8. Compared with the traditional page migration, which is inaccessible from t1 to t7 throughout the entire migration process, this shortens the time when the user program cannot access the page, effectively reduces the overhead of the migration process, and reduces the impact on system performance. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a page migration method for split memory provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of hot page recognition provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram illustrating asynchronous migration using a dual-queue system, as provided in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram illustrating page access decoupling provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0040] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.

[0041] Example 1

[0042] This invention provides a page migration method for split memory, comprising:

[0043] In a split memory architecture, memory is distributed across multiple nodes. Each node dynamically samples page access information through NUMA Balancing, tracks and marks page access records, and performs static filtering using an LRU list to identify hot pages and mark other pages as cold pages.

[0044] Construct a hot migration queue and a cold migration queue, and store the identified hot pages and cold pages in the hot migration queue and the cold migration queue respectively;

[0045] The background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and performs asynchronous page migration, migrating hot pages from remote memory nodes to the local node and cold pages from the local node to remote memory nodes; during page migration, user access to the page is decoupled.

[0046] Current popular memory data tracking mechanisms mainly include three methods: software instrumentation, page table flags, and CPU hardware counting. Software instrumentation is a mechanism that tracks memory access behavior by inserting additional statistical code before and after memory access instructions. Its advantages are simplicity and high development flexibility. However, the instrumentation code itself introduces additional computational and storage overhead, significantly increasing the system load during operation and leading to performance degradation. Page table flags utilize flags in the operating system's page table (such as the accessed flag and the dirty flag) to obtain information on memory page access and modification. However, this method significantly increases performance overhead during the scanning process. Furthermore, since the flag information only reflects whether a page has been accessed or modified, the statistical accuracy is limited. CPU hardware tracks memory access through hardware-level sampling. Its advantages are extremely low performance overhead and high accuracy. However, it is heavily dependent on the processor's hardware characteristics; different processor architectures (such as x86 and ARM) may provide different performance monitoring support, resulting in poor cross-platform compatibility.

[0047] To address the lack of proactive page fault prevention in discrete memory systems and reduce remote access, this invention provides a page migration method for discrete memory. The overall architecture for implementing this page migration method is as follows: Figure 1 As shown, the page migration method specifically includes: combining NUMA Balancing and LRU algorithms, using NUMA Balancing to dynamically sample task pages, and using the LRU list for static filtering to identify hot pages; storing the identified hot and cold pages in hot migration queues and cold migration queues respectively; having the background kernel thread kmove continuously attempt to migrate asynchronously from the migration queues; during asynchronous page migration, user access to the page is decoupled, shortening the page inaccessibility time window, and achieving the purpose of proactively preventing PageFault by reducing remote access, thereby avoiding frequent PageFaults, reducing page migration overhead, and optimizing the software quality assurance of the split memory system.

[0048] In this application, when tracking hot memory data, a proposed approach combines NUMA Balancing and the LRU algorithm. NUMA Balancing is used for dynamic sampling, while the LRU list is used for static filtering to identify hot pages. NUMA Balancing is a transparent memory management mechanism in the Linux kernel that maximizes local memory utilization and reduces remote memory access overhead by dynamically adjusting the location of memory pages or the execution node of tasks. The introduction of the LRU list allows the system to more accurately determine which pages are truly active hot pages. The hierarchical mechanism of Inactive LRU and Active LRU effectively filters out cold pages with low access frequency or those accessed only temporarily, prioritizing the migration of truly high-frequency access pages to local memory nodes. This strategy ensures that the migrated pages are most likely to be frequently accessed, thereby improving the hit rate of local memory access and reducing the latency of remote memory access.

[0049] In this embodiment, the steps for hot page identification combining NUMA Balancing and LRU algorithms are as follows: Figure 2 As shown. NUMA Hint Faults are enabled in the memory node, and the memory pages of the task are scanned periodically. The kernel dynamically samples the page access situation. When the running task accesses the marked page, NUMA Balancing samples the page in memory and triggers Minor Page Fault. The kernel records the state of the accessed page, including whether the page is accessed by the local node or the remote node, to realize real-time monitoring of the memory access mode. Further, the static filtering combined with the LRU list to identify hot pages and mark other pages as cold pages specifically includes: then verifying the activity of the page through the LRU list, (1) if the page is in the Inactive LRU, it indicates that it is not active enough and will not be migrated immediately, but will be marked as accessed and put into the Active LRU list, (2) if the page is accessed again and is in the Active LRU, it indicates that its activity has increased and it is marked as a hot page; if the page is in the Inactive LRU for more than a certain time threshold, it is marked as a cold page. Construct separate hot migration queues and cold migration queues; insert pages marked as hot pages into the hot migration queues and pages marked as cold pages into the cold migration queues.

[0050] Based on the above steps, qualified hot pages can be selected to provide decisions for subsequent page migration. For example, when a task is started in CPU0, NUMA Balancing detects that page 1 in CPU1 is frequently accessed, and this page is already in Active LRU, indicating that it is a qualified hot page that needs to be migrated. At this time, page 1 will be inserted into the hot migration queue, and through subsequent operations, page 1 will be migrated to the local memory in CPU0.

[0051] In this embodiment, due to the excessive overhead of synchronous migration, asynchronous migration is implemented using dual queues. The implementation of asynchronous migration is as follows: Figure 3 As shown. Figure 3 As shown, in this embodiment, in addition to the inactive and active LRU lists in memory tracking, this project also introduces a hot migration queue and a cold migration queue.

[0052] The task runs in user space and uses NUMA balancing to periodically scan the task's memory pages. After NUMA Hint Faults in the memory node catch page exceptions, a Minor Page Fault is triggered in the kernel. At this time, the liveness of the page is verified by the LRU list. If the page is in the Inactive LRU, it means that it is not active enough and is not migrated immediately. Instead, it is marked as accessed and put into the Active LRU list. If the page is accessed again and is in the Active LRU, it means that its liveness has increased and it is marked as a hot page.

[0053] The hot migration queue stores hot pages that meet the criteria after being filtered by NUMA Balancing and LRU algorithms, while the cold migration queue stores inactive pages.

[0054] The background kernel thread continuously attempts to migrate asynchronously from two queues.

[0055] Based on the above operations, this embodiment uses a dual-queue mechanism for buffering and utilizes a background thread to asynchronously complete the migration operation. This avoids blocking the execution of user programs throughout the process, thereby significantly improving the response performance of user programs. Simultaneously, this approach effectively reduces data access latency, further optimizing the overall system operating efficiency and contributing to improved service quality assurance.

[0056] like Figure 4 As shown, the background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and performs asynchronous page migration, specifically including the following steps:

[0057] (S1) The background kernel thread kmove selects a page from either the hot or cold migration queue each time, clears the dirty bits of that page, starts a migration transaction, and migrates that page. Figure 4 Step ① in the process;

[0058] (S2) Issue a TLB close order to all kernels that have accessed the page, i.e. Figure 4 Step ② in the process;

[0059] (S3) The pages being migrated are copied between remote memory and local memory, i.e. Figure 4 Step ③ in the process;

[0060] (S4) Cancel PTE mapping and check for dirty bits in the pages being migrated, i.e. Figure 4 Step 4 in the process;

[0061] (S5) Issue a TLB shutdown message to all kernels that have accessed the pages being migrated, i.e. Figure 4 Step 5 in the process;

[0062] (S6) Determine if the page is dirty (i.e., ... Figure 4 If the page is dirty, proceed to step S8; otherwise, proceed to step S7.

[0063] (S7) The migration was successful. Submit the migration results and remap the pages. Figure 4 Step ⑦ in the process;

[0064] (S8) Abort this page migration and restore the original PTE, proceed to step S1, and wait for the next migration. Figure 4 Step ⑧ in the process.

[0065] This embodiment identifies hot pages through a page tagging module, performs asynchronous background migration through a migration queue construction module, and decouples user access to pages through a page migration module. This effectively reduces remote access, thereby achieving the goal of proactively preventing PageFaults, avoiding frequent PageFaults, reducing page migration overhead, optimizing the software quality assurance of the split memory system, and improving the stability and reliability of the system.

[0066] Example 2

[0067] The present invention also provides a page migration system for discrete memory, comprising:

[0068] The page marking module is used in a distributed memory architecture where memory is distributed across multiple nodes. This node dynamically samples page access information through NUMABalancing, tracks and marks page access records; it combines LRU list for static filtering to identify hot pages and mark other pages as cold pages.

[0069] The queue construction module is used to build hot and cold migration queues, and store the identified hot and cold pages in the hot and cold migration queues respectively.

[0070] The page migration module is used by the background kernel thread to continuously retrieve hot and cold pages to be migrated from the hot and cold migration queues, and perform asynchronous page migration, migrating hot pages from remote memory nodes to the current node and cold pages from the current node to remote memory nodes; during page migration, user access to pages is decoupled.

[0071] The page migration system for split memory provided in this embodiment of the invention is used to execute the page migration method for split memory in Embodiment 1, and has the same or similar beneficial effects.

[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A page migration method for split memory, characterized in that, include: In a split memory architecture, memory is distributed across multiple nodes. Each node dynamically samples page access information through NUMA Balancing, tracks and marks page access records, and performs static filtering using an LRU list to identify hot pages and mark other pages as cold pages. Construct a hot migration queue and a cold migration queue, and store the identified hot pages and cold pages in the hot migration queue and the cold migration queue respectively; The background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, performs asynchronous page migration, migrates hot pages from remote memory nodes to the local node, and migrates cold pages from the local node to remote memory nodes. Decouple users from page access during page migration; The background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and performs asynchronous page migration, including: t1. The background kernel thread kmove selects a page from the hot migration queue or the cold migration queue each time, clears the dirty bits of the page, and migrates the page. t2. Send a TLB shutdown message to all kernels that have accessed the pages being migrated; t3. Pages being migrated are copied between remote memory and local memory; t4. Cancel PTE mapping and check for dirty bits in the pages being migrated; t5. Send a TLB shutdown message to all kernels that have accessed the pages being migrated; t6. Determine if the page is dirty. If the page is dirty, proceed to step t8; otherwise, proceed to step t7. t7. Complete this page migration and remap the migrated pages in the operating system; t8. Abort this page migration and restore the original PTE, then proceed to step t1.

2. The method as described in claim 1, characterized in that, In the separate memory configuration, this node dynamically samples page access information using NUMABalancing, tracking and marking page access records, including: Enabling NUMA in memory nodes indicates an error; the kernel periodically scans the memory pages of running tasks and dynamically samples page access information. When a running task accesses a marked page, a Minor Page Fault is triggered through the NUMA Balancing mechanism, and the accessed page in memory is sampled. The kernel records the state of the accessed page, including the accessed object of the page, including local nodes and remote nodes, for real-time monitoring of memory access patterns.

3. The method as described in claim 1, characterized in that, The step of combining an LRU list for static filtering to identify hot pages and mark other pages as cold pages includes: The activity of the marked pages is verified using the LRU list. If a page is in the Inactive LRU, it is marked as visited and added to the Active LRU list. If a page triggers a minor page error again and is in the Active LRU, it is marked as a hot page. If a page is in the Inactive LRU for more than a certain time threshold, it is marked as a cold page.

4. The method as described in claim 3, characterized in that, The construction of the hot-to-migrate queue and the cold-to-migrate queue, and the storage of the identified hot pages and cold pages in the hot-to-migrate queue and the cold-to-migrate queue respectively, includes: Construct separate hot migration queues and cold migration queues; Pages marked as hot pages are inserted into the hot migration queue, and pages marked as cold pages are inserted into the cold migration queue.

5. A page migration system for discrete memory, characterized in that, include: The page marking module is used in a distributed memory architecture where memory is distributed across multiple nodes. This node dynamically samples page access information through NUMABalancing, tracks and marks page access records; it combines LRU list for static filtering to identify hot pages and mark other pages as cold pages. The queue construction module is used to build hot and cold migration queues, and store the identified hot and cold pages in the hot and cold migration queues respectively. The page migration module is used by the background kernel thread to continuously retrieve hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and perform asynchronous page migration, migrating hot pages from remote memory nodes to the current node, and migrating cold pages from the current node to remote memory nodes; during page migration, user access to the page is decoupled. Specifically, the page migration module performs the following steps when performing asynchronous page migration: The background kernel thread continuously retrieves hot and cold pages to be migrated from the hot migration queue and the cold migration queue, and performs asynchronous page migration, including: t1. The background kernel thread kmove selects a page from the hot migration queue or the cold migration queue each time, clears the dirty bits of the page, and migrates the page. t2. Send a TLB shutdown message to all kernels that have accessed the pages being migrated; t3. Pages being migrated are copied between remote memory and local memory; t4. Cancel PTE mapping and check for dirty bits in the pages being migrated; t5. Send a TLB shutdown message to all kernels that have accessed the pages being migrated; t6. Determine if the page is dirty. If the page is dirty, proceed to step t8; otherwise, proceed to step t7. t7. Complete this page migration and remap the migrated pages in the operating system; t8. Abort this page migration and restore the original PTE, then proceed to step t1.

Images