Data processing method, device, medium and product of persistent memory

Abstract

This invention provides a persistent memory data processing method, apparatus, medium, and product, belonging to the field of data processing. The method includes: at a first moment, taking a snapshot of a first source file comprising multiple data blocks in a file system to obtain a first snapshot file comprising multiple data blocks, wherein the multiple data blocks in the first snapshot file are all in a shared state. At a second moment, triggering a modification operation on a first data block in the first source file, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block. Then, the first snapshot file is updated, so that the new data block is in a non-shared state in the updated first snapshot file, thereby preventing the Copy-on-Write (COW) mechanism from being triggered when modifying the first data block. Afterwards, the data in the first data block of the first source file is modified according to the modification operation to obtain a second source file. This ensures the continuity of the first source file in terms of blocks.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, medium and product for persistent memory. Background Technology

[0002] With the rapid development of internet applications, the demand for data storage is constantly increasing, and high-performance, high-reliability data storage solutions have become the foundation for many internet services. Remote Dictionary Server (Redis), as an in-memory key-value database, is widely used in caching, message queues, real-time data analysis, and other fields due to its excellent read / write performance and flexible data structure. However, the high cost and limited capacity of memory resources restrict the large-scale deployment of Redis in certain application scenarios.

[0003] To address this issue, one solution involves storing a portion of Redis data in persistent memory and employing file system snapshot technology. After taking a snapshot of the source file, any modification to a data block in the source file will trigger a copy-on-write (COW) mechanism. This means allocating a new data block, copying the data from the source data block to the new data block, and then mapping the new data block to the corresponding location in the source data block.

[0004] However, after multiple copy-and-write operations, the number of data blocks in the source file becomes uncontrollable, leading to severe fragmentation. Therefore, it is necessary to propose a new persistent memory data processing method to solve the fragmentation problem of source files. Summary of the Invention

[0005] This invention provides a persistent memory data processing method, device, medium, and product to ensure the stability and continuity of source files.

[0006] In a first aspect, embodiments of the present invention provide a data processing method for persistent memory, applied to the main process of an application, wherein at least a portion of the application's data is stored in persistent memory, and the application directly accesses the persistent memory through a file system in the kernel. The method includes:

[0007] At the first moment, a snapshot is taken of the first source file in the file system to obtain the first snapshot file. The first source file contains multiple data blocks, and all the data blocks are in a shared state in the first snapshot file.

[0008] At the second moment, in response to the modification operation on the first data block in the first source file, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block. The new data block corresponds to a new address space in the file system that is different from multiple address spaces. The first data block is any one of the multiple data blocks.

[0009] Update the first snapshot file to determine that the updated first snapshot file contains new data blocks and data blocks other than the first data block among multiple data blocks, and to determine that the new data blocks are in a non-shared state in the updated first snapshot file;

[0010] Based on the modification operation, the data in the first data block of the first source file is modified to obtain the second source file.

[0011] In a second aspect, embodiments of the present invention provide a data processing apparatus for persistent memory, the apparatus comprising:

[0012] The snapshot module is used to take a snapshot of the first source file in the file system at the first moment to obtain the first snapshot file. The first source file includes multiple data blocks, and all data blocks are in a shared state in the first snapshot file.

[0013] The allocation module is used at the second moment to allocate a new data block corresponding to the first data block in response to the modification operation of the first data block in the first source file, and to copy the data in the first data block to the new data block. The new data block corresponds to a new address space in the file system that is different from multiple address spaces. The first data block is any one of the multiple data blocks.

[0014] The update module is used to update the first snapshot file to determine that the updated first snapshot file contains new data blocks and data blocks other than the first data block among multiple data blocks, and to determine that the new data blocks are in a non-shared state in the updated first snapshot file.

[0015] The modification module is used to modify the data in the first data block of the first source file to obtain the second source file based on the modification operation.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the persistent memory data processing methods described above.

[0017] Fourthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the persistent memory data processing methods described above.

[0018] Fifthly, embodiments of the present invention provide a computer program product comprising a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the persistent memory data processing methods described above.

[0019] In the solution provided by this embodiment of the invention, a first snapshot file is obtained by taking a snapshot of a first source file, which includes multiple data blocks, in the file system at a first moment. All data blocks in the first snapshot file are in a shared state. At a second moment, if a modification operation on the first data block in the first source file is triggered, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block. Then, the first snapshot file is updated, so that the new data block is in a non-shared state in the updated first snapshot file. This prevents the Copy-on-Write (COW) mechanism from being triggered when the first data block is modified. Afterwards, the data in the first data block in the first source file is modified according to the modification operation to obtain a second source file. In this way, since the modification is directly performed on the first data block in the first source file, the first source file still uses the original segments; that is, no segment addition occurs. Therefore, the continuity and stability of the segments in the first source file are guaranteed. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the composition of a file system;

[0022] Figure 2 This is a diagram illustrating the file modification process in a file system.

[0023] Figure 3 A flowchart illustrating a persistent memory data processing method provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram illustrating a persistent memory data processing procedure provided in an embodiment of the present invention;

[0025] Figure 5 A flowchart illustrating a method for writing back dirty pages in persistent memory, as provided in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram illustrating the dirty page write-back process in persistent memory provided in an embodiment of the present invention;

[0027] Figure 7 A flowchart of a persistent memory space reclamation method provided in an embodiment of the present invention;

[0028] Figure 8 A schematic diagram illustrating the space reclamation process in persistent memory provided in an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of the structure of a persistent memory data processing device provided in an embodiment of the present invention;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0034] First, some terms involved in the embodiments of this invention will be explained.

[0035] Copy on write (COW) refers to copying data only when it needs to be modified, in order to avoid unnecessary data copying. It is often used in scenarios such as creating child processes in memory and file system snapshots.

[0036] Direct Access (DAX): A mechanism for directly accessing persistent memory data.

[0037] An extent is a data structure used in mainstream file systems to manage disk space.

[0038] Dirty page write-back: Writes dirty data in the page cache back to the corresponding location on the disk to ensure data consistency.

[0039] Space reclamation: After a file is deleted from the file system, the space is returned to the file system to meet the needs of subsequent new space allocation.

[0040] Page Fault: This refers to an exception thrown by the memory management unit when a process accesses an unloaded page, which is then handled by the corresponding handler called by the operating system.

[0041] With the rapid development of internet applications, data generation and processing have reached a significant scale. To meet the ever-increasing demand for data storage, high-performance, high-reliability data storage solutions have become an indispensable foundation for many internet services. Remote Dictionary Server (Redis), as an in-memory key-value database, is widely used in caching, message queues, real-time data analysis, and other fields due to its excellent read / write performance and flexible data structure. However, the high cost and limited capacity of memory resources restrict the large-scale deployment of Redis in certain application scenarios.

[0042] To address this issue, one solution involves storing a portion of Redis data in persistent memory. To ensure data integrity and enable recovery to a specific point in time when necessary, file system snapshot technology is typically employed. This involves creating a new persistent memory instance, essentially generating a copy of the current state, which can be modified independently without affecting the original data.

[0043] like Figure 1As shown, the user application 11 can access the file system 12. For example, it can read files in persistent memory 14 through the data read / write interface (mmap) of the file system 12. mmap can map the virtual memory space of files in the file system 12 to the application 11, allowing the application to see the virtual address space of the file system and directly access persistent memory 14 through the persistent memory driver (PMEM driver) 13. The application 11 can also perform snapshot operations on files in persistent memory 14 through the snapshot interface (ioctl). The file system 12 can be an extended file system (Ext) series file system, such as the second extended file system (ext2), the third extended file system (ext3), the fourth extended file system (ext4), the eXtended file system (XFS), the B-tree file system (Butterfs, Btrfs), the file allocation table (FAT), etc.

[0044] In a file system, a file can be called a source file. A source file may contain several data blocks. A larger management granularity than data blocks can be set in the file system: segments, which are used to manage data blocks. A segment can contain multiple data blocks. When a snapshot operation is performed on a source file, a snapshot file is generated, thus recording the actual state of the data blocks in the source file at the time of the snapshot.

[0045] When a data block in the source file is modified (for example, when data in a data block is modified), the copy-on-write (COW) mechanism is triggered. Before actually modifying the data, the COW mechanism creates a new copy of the data and applies the changes to the new copy. That is, it allocates a new data block, copies the data in the modified data block in the source file to the new data block, and then maps the new data block to the corresponding position in the source file.

[0046] like Figure 2As shown, assume the source file contains 6 data blocks, labeled from top to bottom as data block 1, data block 2, data block 3, data block 4, data block 5, and data block 6. A snapshot operation is performed on the source file to obtain a snapshot file, which also contains 6 data blocks. This snapshot file is a read-only file. For the sake of distinction between the snapshot file and the source file, the data blocks in the snapshot file are labeled from top to bottom as data block 1', data block 2', data block 3', data block 4', data block 5', and data block 6'. The source file and the snapshot file share these 6 data blocks; that is, data block 1 and data block 1' point to the same address space, data block 2 and data block 2' point to the same address space, data block 3 and data block 3' point to the same address space, data block 4 and data block 4' point to the same address space, data block 5 and data block 5' point to the same address space, and data block 6 and data block 6' point to the same address space. These six data blocks reside within at least one segment, and each block corresponds to a different address space within the file system. If the application's main process then modifies data block 3 in the source file, the Copy-on-Write (COW) mechanism will be triggered because these blocks are currently shared. This mechanism allocates a new data block 7, which corresponds to a different address space within the file system than the aforementioned address spaces. For example, assuming the data in data block 3 is "abc", the data "abc" is copied from data block 3 to data block 7, meaning data block 7 also contains "abc". Then, data block 7 is mapped to the corresponding location in the source file, effectively replacing data block 3. The data "abc" in data block 7 is modified, for example, by changing "abc" to "def", thus completing the data modification operation. At this point, if... Figure 2 As shown, the source file will eventually include data block 1, data block 2, data block 7, data block 4, data block 5, and data block 6. The snapshot file remains unchanged, including data block 1', data block 2', data block 3', data block 4', data block 5', and data block 6'. Assuming the initial 6 data blocks in the source file correspond to one segment, the newly allocated data block 7 will correspond to another segment. Therefore, this data modification operation will result in an increase in segments in the source file, and the segments corresponding to consecutive data blocks will no longer be consecutive.

[0047] Specifically, when multiple copy-and-write (COW) operations are performed on a source file, the number of data blocks corresponding to the source file continuously increases because each modification creates a new data block. These newly allocated data blocks are then managed by newly allocated segments, meaning the number of segments corresponding to the source file keeps increasing and becoming discontinuous. This ultimately leads to severe fragmentation of the source file within the file system. This situation not only consumes more storage resources but also increases the complexity of data management and may even affect the efficiency of data access.

[0048] In view of this, embodiments of the present invention provide a novel data processing method for persistent memory. A snapshot file is obtained by taking a snapshot of a source file in a file system. Multiple data blocks included in the snapshot file are in a shared state, and these data blocks reside within at least one segment and correspond to different address spaces in the file system. When a modification to the first data block in the source file is triggered, the new Copy-on-Write (COW) mechanism executes as follows: a new data block corresponding to the first data block is allocated; then, the data in the first data block is copied to the new data block; subsequently, the snapshot file is updated so that it includes the new data block and all data blocks except the first data block—that is, the new data block is mapped to the snapshot file. Then, the data in the first data block of the first source file is directly modified according to the modification operation to obtain a second source file. In this way, since the modification is directly applied to the first data block in the first source file, the first source file still uses the original segment, meaning no segment addition occurs. Therefore, the severe fragmentation problem caused by multiple COW operations on the source file is solved, ensuring the continuity and stability of the first source file in terms of segments.

[0049] The persistent memory data processing method provided in the embodiments of the present invention below can be executed by the application's main process. At least a portion of the application's data is stored in persistent memory, and the application can directly access the persistent memory through the file system in the kernel. The persistent memory data processing process provided in the embodiments of the present invention will be described in detail below with reference to the following embodiments.

[0050] Figure 3 This is a flowchart illustrating a persistent memory data processing method provided in an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0051] 301. At the first moment, a snapshot is taken of the first source file in the file system to obtain the first snapshot file. The first source file contains multiple data blocks, and all the data blocks are in a shared state in the first snapshot file.

[0052] 302. At the second moment, in response to the modification operation on the first data block in the first source file, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block. The new data block corresponds to a new address space in the file system that is different from multiple address spaces. The first data block is any one of the multiple data blocks.

[0053] 303. Update the first snapshot file to determine that the updated first snapshot file contains new data blocks and data blocks other than the first data block among multiple data blocks, and to determine that the new data blocks are in a non-shared state in the updated first snapshot file.

[0054] 304. Based on the modification operation, modify the data in the first data block of the first source file to obtain the second source file.

[0055] The first moment and the second moment are two different moments. The snapshot processing of the first source file at the first moment and the modification operation of the first data block in the first source file at the second moment belong to different main processes, without a strict execution sequence, and both are frequent operations in the file system.

[0056] When a snapshot is taken of the first source file, a first snapshot file is obtained that is completely identical to the first source file. The first snapshot file can be regarded as a copy of the first source file, and multiple data blocks in the first snapshot file are shared. That is to say, the first source file and the first snapshot file share multiple data blocks, which correspond to multiple contiguous address spaces in the file system.

[0057] For example, such as Figure 4 As shown, assuming the first source file contains six data blocks, labeled from top to bottom as data block 1, data block 2, data block 3, data block 4, data block 5, and data block 6, then the first snapshot file will also contain six data blocks. For the purpose of distinguishing between the first snapshot file and the first source file, the data blocks in the first snapshot file are labeled from top to bottom as data block 1', data block 2', data block 3', data block 4', data block 5', and data block 6'. The first source file and the first snapshot file share these six data blocks; that is, data block 1 and data block 1' point to the same address space, similarly, data block 2 and data block 2', data block 3 and data block 3', data block 4 and data block 4', data block 5 and data block 5', and data block 6 and data block 6' point to the same address space.

[0058] These six data blocks correspond to different contiguous address spaces in the file system. For example, suppose data block 1 and data block 1' correspond to address spaces 100M to 200M in the file system, data block 2 and data block 2' correspond to address spaces 200M to 300M in the file system, data block 3 and data block 3' correspond to address spaces 300M to 400M in the file system, data block 4 and data block 4' correspond to address spaces 400M to 500M in the file system, data block 5 and data block 5' correspond to address spaces 500M to 600M in the file system, and data block 6 and data block 6' correspond to address spaces 600M to 700M in the file system.

[0059] If, at the second moment, a modification operation is triggered on the first data block of the first source file, a new data block corresponding to the first data block is allocated in response to this modification operation. It can be understood that the first data block is any one of the multiple data blocks in the first source file, and the new data block corresponds to a new address space in the file system, different from the multiple address spaces. Then, the data in the first data block is copied to the new data block, and the first snapshot file is updated so that the updated first snapshot file contains the new data block as well as the data blocks other than the first data block, and the new data block is in a non-shared state in the updated first snapshot file. In this way, modifying the first data block will not trigger the Copy-on-Write (COW) mechanism. Afterwards, since the first data block in the first source file is no longer in a shared state, the data in the first data block in the first source file can be directly modified according to the modification operation to obtain the second source file. In the second source file, the first data block is in a non-shared state, while the data blocks other than the first data block are still in a shared state.

[0060] Accept Figure 4 In the example, assuming the modification operation targets data block 3 in the first source file (i.e., the first data block is data block 3 out of six data blocks), then in response to this modification operation, data block 7 corresponding to data block 3 is allocated, assuming data block 7 corresponds to address space 900M-1000M in the file system. Then, the data "abc" from data block 3 is copied to data block 7, meaning the data in data block 7 is now "abc", and the first snapshot file is updated. Then... Figure 4 As shown, the updated first snapshot file includes multiple data blocks: data block 1', data block 2', data block 7 (abc), data block 4', data block 5', and data block 6'. Of these six data blocks, data blocks 1', 2', 4', 5', and 6' remain shared, while data block 7 is not shared. Next, based on the modification operation, the data 'abc' in data block 3 of the first source file is modified, for example, by changing 'abc' to 'def', resulting in the second source file. At this point, data blocks 1, 2, 4, 5, and 6 in the second source file are all shared, while data block 3 is not shared.

[0061] Based on the technical solution of this invention, at a first moment, a snapshot of a first source file comprising multiple data blocks in the file system is taken to obtain a first snapshot file also comprising multiple data blocks, and the multiple data blocks in the first snapshot file are all in a shared state. At a second moment, if a modification operation on the first data block in the first source file is triggered, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block. Then, the first snapshot file is updated, so that the new data block is in a non-shared state in the updated first snapshot file, thereby preventing the Copy-on-Write (COW) mechanism from being triggered when modifying the first data block. Afterwards, the data in the first data block in the first source file is modified according to the modification operation to obtain the second source file. It can be seen that since the first data block in the first source file is no longer in a shared state, the first data block in the first source file can be directly modified in this embodiment of the invention, and the address space of the first source file still uses the original segments, that is, no segment addition occurs, thus ensuring the continuity and stability of the first source file in terms of segments.

[0062] It's understandable that modifying the first data block of the first source file requires data modification based on the page table. Since the modified data is temporarily stored in the cache instead of persistent memory, dirty pages are generated. Because the snapshot file needs to maintain data consistency with the source file, the dirty pages of the first source file need to be written back—that is, the modified data needs to be written back to persistent memory. Typically, during the dirty page write-back process, all virtual memory regions sharing the dirty page need to be found through reverse mapping and set to write-protected mode. Then, the data in the dirty page is written back from the cache to persistent memory. During this process, if there are too many dirty pages, the source file remains write-protected for an extended period, which can affect the snapshot operation, resulting in poor snapshot efficiency.

[0063] Based on this, embodiments of the present invention also provide a dirty page write-back method to improve snapshot efficiency. Figure 5 This is a flowchart illustrating a method for writing back dirty pages in persistent memory, as provided in an embodiment of the present invention. Figure 5 As shown, the method may include the following steps:

[0064] 501. Write the modified data in the first data block to the corresponding virtual address in the page table.

[0065] 502. If it is determined that the physical address corresponding to the virtual address in persistent memory has not been allocated, a page fault is generated, and a physical address is allocated for the page fault, and the data in the modified first data block is written to the physical address.

[0066] 503. Start the dirty page write-back thread. The dirty page write-back thread asynchronously scans the intermediate directories of each page to determine the dirty page tables generated in each intermediate directory. Each intermediate directory of a page includes the addresses of multiple page tables.

[0067] 504. Persistently write the data in the dirty page table to persistent memory.

[0068] Specifically, the page table has a multi-level structure, where each page table entry (PTE) stores the mapping relationship between virtual addresses and physical addresses, and the page middle directory (PMD) contains the addresses of several page tables.

[0069] When modifying data in the first data block of the first source file, the modified data in the first data block is first written to the corresponding virtual address in the page table. If the physical address corresponding to the virtual address in persistent memory is not allocated, a page fault will occur. At this time, the allocation of the physical address in the cache will be triggered. After allocating the physical address in the cache through the page fault, the modified data in the first data block can be written to the physical address in the cache first.

[0070] Subsequently, in response to the snapshot operation triggered on the second source file at the third time step, a dirty page write-back thread is started. This dirty page write-back thread executes asynchronously with the main process. Simultaneously, in response to the snapshot operation triggered on the second source file at the third time step, a snapshot of the second source file is taken to obtain a second snapshot file. The second snapshot file contains the modified first data block and multiple data blocks other than the first data block. All data blocks in the second snapshot file are in a shared state. This allows the main process to continue the snapshot operation without waiting for the dirty page write-back to complete. The third time step is later than the second time step.

[0071] In this embodiment of the invention, after being placed in write-protected state, the dirty page write-back thread asynchronously scans the intermediate directories of each page to determine the dirty page tables generated in each intermediate directory. When a dirty page table is found, the data in the dirty page table is persistently written to persistent memory. For example... Figure 6 As shown, each page intermediate directory corresponds to 512 page table entries. After starting the dirty page write-back thread, the thread only needs to traverse the 2M-granularity page intermediate directories to determine the dirty page tables generated under the page intermediate directories, instead of traversing each of the 512 page table entries under each page intermediate directory to determine the generated dirty page tables. Therefore, it can save approximately 512 times the page table processing time. At the same time, the data in the dirty page tables under each page intermediate directory can be written to persistent memory through an asynchronous thread. Obviously, this greatly reduces the latency of snapshot operations.

[0072] In practical applications, snapshotting source files is often done frequently. This means that it's necessary to frequently allocate new data blocks corresponding to the data blocks to be modified, and also to continuously delete old snapshot files to reclaim address space in persistent memory. In a file system, address space reclamation is controlled by reference counting; that is, address space reclamation is triggered when the file's reference count reaches 0.

[0073] Specifically, Figure 7 This is a flowchart illustrating a method for reclaiming space in persistent memory, as provided in an embodiment of the present invention. Figure 7 As shown, the method may include the following steps:

[0074] 701. When it is determined that the reference count of the updated first snapshot file is zero, start the space reclamation thread to asynchronously reclaim the updated first snapshot file.

[0075] 702. At the fourth time step, in response to the modification operation on the second data block in the first source file, the data in the second data block in the first source file is directly modified according to the modification operation.

[0076] The first snapshot file includes data blocks in a shared state and data blocks in a non-shared state. When reclaiming the first snapshot file, the reclamation methods for these two different states of data blocks also differ. It can be understood that data blocks in different states can be distinguished by setting markers. For example, a data block in a shared state can be marked as 0, while a data block in a non-shared state can be marked as 1; or, a data block in a non-shared state can be marked as 0, while a data block in a shared state can be marked as 1. This embodiment of the invention does not limit the specific method of marking data blocks in different states.

[0077] Specifically, during the process of reclaiming the updated first snapshot file, for new data blocks in the updated first snapshot file that are in a non-shared state, the correspondence between the updated first snapshot file and the new address space corresponding to the new data blocks is deleted, and the new address space is placed in an idle state to notify the file system.

[0078] Because the new data block is in a non-shared state, it means that whether or not the first data block corresponding to the new data block in the first source file is modified will not affect the new data block. Therefore, for the new data block in the first snapshot file that is in a non-shared state, the mapping between the updated first snapshot file and the new address space corresponding to the new data block can be deleted directly. This will not cause a page fault.

[0079] Since the data block in the shared state is also used in the source file, if the source file continues to modify this data block, the main process will also be involved in the copy-and-write (COW) process of this data block, which will affect the reclamation of the snapshot file. Therefore, the data block in the snapshot file can only be reclaimed when it is certain that the data block in the shared state will not be modified again. Otherwise, if the snapshot file that is in the reclamation process is updated again, it will cause a conflict with the reclamation process.

[0080] Therefore, during the process of reclaiming the updated first snapshot file, for the second data block in the updated first snapshot file that is in a shared state, the correspondence between the address spaces of the updated first snapshot file and the second data block is deleted, and the second data block in the first source file needs to be modified to a non-shared state. The second data block is any data block other than the first data block among multiple data blocks, and the second data block is shared by the first source file and the updated first snapshot file.

[0081] like Figure 8 As shown, assuming that after modifying the first data block, the reference count of the updated first snapshot file is reduced to zero, the main process is triggered to reclaim the updated first snapshot file. Specifically, during the reclamation of the first snapshot file, the main process reclaims the address space corresponding to the non-shared data blocks (i.e., Figure 8 For the non-shared segments in the data block, directly delete the mapping between the address space of the first snapshot file and the data block; for the address space corresponding to the data block that is still in a shared state (i.e., Figure 8 In the shared segment, the correspondence between the address space of the shared data block and the first snapshot file can be skipped until it is determined that no additional writes have been made to the shared data block. Then, the data block is changed to a non-shared state, and the correspondence between the updated first snapshot file and the address space of the corresponding data block is deleted.

[0082] After the space reclamation thread is started, if a modification operation on the second data block in the first source file is received at the fourth time, since the correspondence between the updated first snapshot file and the address space corresponding to the second data block has been deleted, and the second data block is in a non-shared state, modifying the second data block will not have any impact on the updated first snapshot file. Therefore, the modification operation on the second data block can be performed directly without triggering the COW mechanism.

[0083] According to the technical solution of this embodiment, the updated first snapshot file is asynchronously reclaimed by starting a space reclamation thread, and different reclamation methods are used for data blocks in the updated first snapshot file that are in a shared state and a non-shared state, thereby reducing the latency caused by page misses.

[0084] Figure 9This is a schematic diagram of a data processing device for persistent memory provided in an embodiment of the present invention. Figure 9 As shown, the persistent memory data processing device 90 includes:

[0085] The snapshot module 91 is used to take a snapshot of the first source file in the file system at the first moment to obtain the first snapshot file. The first snapshot file contains multiple data blocks, and the multiple data blocks are all in a shared state in the first snapshot file.

[0086] The allocation module 92 is used at the second moment to allocate a new data block corresponding to the first data block in response to the modification operation of the first data block in the first source file, and to copy the data in the first data block to the new data block. The new data block corresponds to a new address space in the file system that is different from multiple address spaces. The first data block is any one of the multiple data blocks.

[0087] The update module 93 is used to update the first snapshot file to determine that the updated first snapshot file contains new data blocks and data blocks other than the first data block among multiple data blocks, and to determine that the new data blocks are in a non-shared state in the updated first snapshot file.

[0088] Modification module 94 is used to modify the data in the first data block of the first source file to obtain the second source file based on the modification operation.

[0089] In one feasible implementation, the modification module 94 can specifically be used to write the data in the modified first data block to the corresponding virtual address in the page table. If it is determined that the physical address corresponding to the virtual address in persistent memory is not allocated, a page fault is generated, and a physical address is allocated for the page fault, and the data in the modified first data block is written to the physical address.

[0090] In one feasible implementation, the persistent memory data processing device 90 further includes a scanning module for initiating a dirty page write-back thread to asynchronously scan each page intermediate directory to determine the dirty page tables generated in each page intermediate directory, wherein each page intermediate directory includes the addresses of multiple page tables. Also included is a writing module for persistently writing data from the dirty page tables into persistent memory.

[0091] Specifically, the scanning module can respond to the snapshot operation triggered by the second source file at the third moment and start the dirty page write-back thread. The third moment is later than the second moment.

[0092] In one feasible implementation, the persistent memory data processing device 90 further includes a reclamation module for initiating a space reclamation thread when it is determined that the reference count of the updated first snapshot file is zero, so as to asynchronously reclaim the updated first snapshot file through the space reclamation thread.

[0093] Specifically, the recycling module can be used to delete the mapping between the updated first snapshot file and the new address space corresponding to the new data block that is in a non-shared state, and place the new address space in an idle state to notify the file system. For the second data block in the updated first snapshot file that is in a shared state, the mapping between the updated first snapshot file and the address space corresponding to the second data block is deleted, and the second data block in the first source file is modified to a non-shared state. The second data block is any data block other than the first data block among multiple data blocks, and the second data block is shared by the first source file and the updated first snapshot file.

[0094] In one feasible implementation, the modification module 94 is also used to, at a fourth time, in response to a modification operation on the second data block in the first source file, directly modify the data in the second data block in the first source file according to the modification operation.

[0095] Figure 9 The apparatus shown can perform the steps in the persistent memory data processing method in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0096] This invention also provides an electronic device, such as... Figure 10 As shown, the electronic device 20 may include a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to at least implement the persistent memory data processing method provided in the foregoing embodiments.

[0097] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the persistent memory data processing method provided in the foregoing embodiments.

[0098] In addition, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the persistent memory data processing method provided in the foregoing embodiments.

[0099] The device embodiments described above are merely illustrative, and the network elements described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using necessary general-purpose hardware platforms, or it can be implemented through a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method for persistent memory, characterized in that, Applied to the main process of an application, wherein at least a portion of the application's data is stored in persistent memory, and the application directly accesses the persistent memory through a file system in the kernel, the method includes: At a first moment, a snapshot is taken of the first source file in the file system to obtain a first snapshot file. The first source file includes multiple data blocks, and the multiple data blocks are all in a shared state in the first snapshot file. The multiple data blocks are located in at least one segment and correspond to different address spaces in the file system. At the second moment, in response to the modification operation on the first data block in the first source file, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied into the new data block. The new data block corresponds to a new address space in the file system that is different from the plurality of address spaces. The first data block is any one of the plurality of data blocks. The first snapshot file is updated to determine that the updated first snapshot file contains the new data block and the data blocks other than the first data block among the plurality of data blocks, and to determine that the new data block is in a non-shared state in the updated first snapshot file; According to the modification operation, the data in the first data block of the first source file is modified to obtain the second source file. The address space of the first data block in the second source file in the file system is the same as before the modification, and the second source file does not have any increase in segments compared to the first source file.

2. The method according to claim 1, characterized in that, Modifying the data in the first data block of the first source file includes: Write the modified data from the first data block into the corresponding virtual address in the page table; If it is determined that the physical address corresponding to the virtual address in persistent memory is not allocated, a page fault is generated, and the physical address is allocated to the page fault, and the data in the modified first data block is written to the physical address.

3. The method according to claim 2, characterized in that, The method further includes: Start a dirty page write-back thread, and asynchronously scan each page intermediate directory to determine the dirty page table generated in each page intermediate directory. Each page intermediate directory includes the addresses of multiple page tables. The data in the dirty page table is persistently written to the persistent memory.

4. The method according to claim 3, characterized in that, The initiation of the dirty page write-back thread includes: In response to a snapshot operation triggered on the second source file at a third time point, the dirty page write-back thread is started, the third time point being later than the second time point.

5. The method according to claim 1, characterized in that, The method further includes: When it is determined that the reference count of the updated first snapshot file is zero, a space reclamation thread is started to asynchronously reclaim the updated first snapshot file.

6. The method according to any one of claims 1-5, characterized in that, The process of reclaiming the updated first snapshot file includes: For the new data block in the updated first snapshot file that is in a non-shared state, delete the correspondence between the updated first snapshot file and the new address space corresponding to the new data block, and put the new address space into an idle state to notify the file system; For the second data block in the updated first snapshot file that is in a shared state, delete the correspondence between the address space of the updated first snapshot file and the second data block, and modify the second data block in the first source file to a non-shared state. The second data block is any data block other than the first data block among the plurality of data blocks, and the second data block is shared by the first source file and the updated first snapshot file.

7. The method according to claim 6, characterized in that, After initiating the space reclamation thread, the method further includes: At the fourth moment, in response to the modification operation on the second data block in the first source file, the data in the second data block in the first source file is directly modified according to the modification operation.

8. An electronic device, characterized in that, include: The device includes a memory, a processor, and a communication interface; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7.

9. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 7.

Images