Data processing method and device for persistent memory, medium and product

Abstract

The embodiment of the invention provides a data processing method and device for a persistent memory, a medium and a product, and belongs to the field of data processing. The method comprises the steps that at a first moment, a first source file comprising a plurality of data blocks in a file system is snapshot to obtain a first snapshot file comprising the data blocks, and the data blocks in the first snapshot file are all in a shared state; and at a second moment, triggering a modification operation on a first data block in the first source file, distributing a new data block corresponding to the first data block, copying data in the first data block into the new data block, and then updating the first snapshot file to enable the new data block to be in an unshared state in the updated first snapshot file, therefore, a COW mechanism is not triggered when the first data block is modified, and then the data in the first data block in the first source file is modified according to the modification operation to obtain the second source file. Therefore, the continuity of the first source file on the block is ensured.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a method, device, medium and product for processing data in a persistent memory. Background Art

[0002] With the rapid development of Internet applications, the demand for data storage is also growing. High-performance and high-reliability data storage solutions have become the basic support for many Internet services. As an in-memory key-value database, Remote Dictionary Server (Redis) is widely used in cache, message queue, real-time data analysis and other fields due to its excellent read and write performance and flexible data structure. However, due to the high cost of memory resources and its limited capacity, the large-scale deployment of Redis in some application scenarios is limited.

[0003] To solve this problem, in one solution, part of Redis data is stored in persistent memory, and the snapshot technology of the file system is used. After taking a snapshot of the source file, the copy on write (COW) mechanism is triggered when the data block in the source file is modified. That is, a new data block is allocated and the data in the source data block is copied to the new data block, and then the new data block is mapped to the corresponding position of the source data block.

[0004] However, after the source file has been processed by COW for many times, the number of data blocks in the source file will become uncontrollable, resulting in serious fragmentation problems. Therefore, it is necessary to propose a new data processing method for persistent memory to solve the fragmentation problem of source files. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method, device, medium and product of a persistent memory, so as to ensure the stability and continuity of source files.

[0006] In a first aspect, an embodiment of the present invention provides a method for processing data in a persistent memory, which is applied to a main process of an application program, wherein at least part of the data of the application program is stored in the persistent memory, and the application program directly accesses the persistent memory through a file system in a kernel, and the method comprises: At a first moment, a snapshot process is performed on a first source file in the file system to obtain a first snapshot file, wherein the first source file includes a plurality of data blocks, and the plurality of data blocks are all in a shared state in the first snapshot file; At a second moment, in response to a modification operation on a first data block in a first source file, a new data block corresponding to the first data block is allocated, data in the first data block is copied to the new data block, the new data block corresponds to a new address space in the file system that is different from the multiple address spaces, and the first data block is any one of the multiple data blocks; updating the first snapshot file to determine that the updated first snapshot file contains the new data block and data blocks other than the first data block among the multiple data blocks, and determining that the new data block is in a non-shared state in the updated first snapshot file; According to the modification operation, data in the first data block in the first source file is modified to obtain a second source file.

[0007] In a second aspect, an embodiment of the present invention provides a data processing device for a persistent memory, the device comprising: A snapshot module, used for taking a snapshot of a first source file in the file system at a first moment to obtain a first snapshot file, wherein the first source file includes a plurality of data blocks, and the plurality of data blocks are all in a shared state in the first snapshot file; an allocation module, configured to allocate, in response to a modification operation on a first data block in a first source file at a second moment, a new data block corresponding to the first data block, copy data in the first data block to the new data block, the new data block corresponding to a new address space in the file system that is different from the multiple address spaces, the first data block being any one of the multiple data blocks; An updating module, used for updating the first snapshot file to determine that the updated first snapshot file contains the new data block and data blocks other than the first data block among the multiple data blocks, and to determine that the new data block is in a non-shared state in the updated first snapshot file; The modification module is used to modify the data in the first data block in the first source file according to the modification operation to obtain the second source file.

[0008] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data processing method of the persistent memory as described above.

[0009] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data processing method of persistent memory as described above.

[0010] In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program, and when the computer program product is executed by a processor of an electronic device, the processor can at least implement the data processing method of the persistent memory as described above.

[0011] In the solution provided by the embodiment of the present invention, a first snapshot file including multiple data blocks is obtained by taking a snapshot of a first source file including multiple data blocks in a file system at a first moment, and multiple data blocks in the first snapshot file are in a shared state. At a second moment, a modification operation on the first data block in the first source file is triggered, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block, and then the first snapshot file is updated, so that the new data block is in a non-shared state in the updated first snapshot file, so that the COW mechanism is not triggered when the first data block is modified, and then the data in the first data block in the first source file is modified according to the modification operation to obtain the second source file. In this way, since the first data block in the first source file is directly modified, the first source file still uses the original segment, that is, no segment is added, thereby ensuring the continuity and stability of the first source file in the segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic diagram of the composition of a file system; Figure 2 A schematic diagram of a file modification process in a file system; Figure 3 A flowchart of a data processing method for persistent memory provided by an embodiment of the present invention; Figure 4 A schematic diagram of a data processing process of a persistent memory provided by an embodiment of the present invention; Figure 5 A flowchart of a method for writing back dirty pages in persistent memory provided by an embodiment of the present invention; Figure 6 A schematic diagram of a dirty page write-back process in a persistent memory provided by an embodiment of the present invention; Figure 7 A flowchart of a method for reclaiming space in persistent memory provided by an embodiment of the present invention; Figure 8A schematic diagram of a space recovery process in a persistent memory provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the structure of a data processing device for persistent memory provided by an embodiment of the present invention; Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0017] First, some terms involved in the embodiments of the present invention are explained.

[0018] Copy on write (COW): refers to copying data only when it needs to be modified to avoid unnecessary data copying. It is often used in scenarios such as creating child processes in memory and file system snapshots.

[0019] Direct access (Data Access eXensions, referred to as DAX): a mechanism for directly accessing persistent memory data.

[0020] Extent: A data structure used in mainstream file systems to manage disk space.

[0021] Dirty page writeback: Write dirty data in the page cache back to the corresponding location on the disk to ensure data consistency.

[0022] Space reclamation: After deleting files from a file system, the space is returned to the file system to meet the needs of subsequent new space allocation.

[0023] Page Fault: refers to the exception thrown by the memory management unit when a process accesses an unloaded page, and the operating system calls the corresponding handler for processing.

[0024] With the rapid development of Internet applications, data generation and processing have reached a large scale. In order to meet the growing demand for data storage, high-performance and high-reliability data storage solutions have become an indispensable basic support for many Internet services. As an in-memory key-value database, Remote Dictionary Server (Redis) is widely used in cache, message queue, real-time data analysis and other fields due to its excellent read and write performance and flexible data structure. However, due to the high cost of memory resources and its limited capacity, the large-scale deployment of Redis in some application scenarios is limited.

[0025] To solve this problem, in one solution, part of Redis data is stored in persistent memory. In order to ensure data integrity and be able to restore to the state at a specific point in time when necessary, the snapshot technology of the file system is usually used to create a new persistent memory instance in the persistent memory, that is, to generate a copy of the current state. This copy can be modified independently without affecting the original data.

[0026] like Figure 1As shown, the user application 11 can access the file system 12, for example, read the file in the persistent memory 14 through the data read and write interface (mmap) of the file system 12, and mmap can map the virtual memory space of the file in the file system 12 to the application 11, so that the application can see the virtual address space of the file system, and directly access the persistent memory 14 through the virtual address space through the persistent memory driver (PMEM Driver) 13. The application 11 can also take a snapshot of the file in the persistent memory 14 through the snapshot interface (ioctl). Among them, the file system 12 can be an extended file system (Ext) series file system, such as the second extended file system (ext2), the third extended file system (ext3) and the fourth extended file system (ext4), the extended file system (eXtended File System, XFS), the B-tree file system (Butterfs, Btrfs), the file allocation table (FAT), etc.

[0027] A file in a file system can be called a source file. A source file can include several data blocks. In the file system, a larger management granularity than a data block can be set: a segment, for managing data blocks. A segment can include multiple data blocks. When a snapshot operation is performed on a source file, a snapshot file corresponding to the source file is generated, so that the actual situation of the data blocks in the source file at the time of the snapshot is recorded through the snapshot file.

[0028] When a data block in the source file is modified (for example, the data in the data block is modified), the copy on write (COW) mechanism will be triggered. The COW mechanism will create a new copy of the data before actually modifying the data, and apply the changes to the new copy of the data, that is, allocate a new data block and copy the data in the modified data block in the source file to the new data block, and then map the new data block to the corresponding position of the source file.

[0029] like Figure 2As shown, it is assumed that the source file includes 6 data blocks, which are recorded as data block 1, data block 2, data block 3, data block 4, data block 5 and data block 6 in order from top to bottom. A snapshot operation is performed on the source file to obtain a snapshot file, which also includes 6 data blocks. The snapshot file is a read-only file. In this article, only to distinguish the snapshot file from the source file, the data blocks contained in the snapshot file are recorded as data block 1', data block 2', data block 3', data block 4', data block 5' and data block 6' in order from top to bottom. Among them, the source file and the snapshot file share these 6 data blocks, that is, data block 1 and data block 1' point to the same address space, data block 2 and data block 2' point to the same address space, data block 3 and data block 3' point to the same address space, data block 4 and data block 4' point to the same address space, data block 5 and data block 5' point to the same address space, and data block 6 and data block 6' point to the same address space. These 6 data blocks are located in at least one segment, and the 6 data blocks correspond to multiple different address spaces in the file system. If the main process of the application triggers the modification of data block 3 in the source file at this time, then since these data blocks are currently in a shared state, the execution of the COW mechanism will be triggered: a new data block 7 is allocated, and the new data block 7 corresponds to a new address space in the file system that is different from the above-mentioned multiple address spaces. For example, assuming that the data in data block 3 is abc, the data abc in data block 3 is copied to data block 7, that is, the data in data block 7 is also abc, and then data block 7 is mapped to the corresponding position of the source file, that is, replacing data block 3, and the data abc in data block 7 in the source file is modified, for example, the data abc in data block 7 is modified to def, thereby completing the data modification operation. At this time, Figure 2 As shown in , the source file will eventually include data blocks 1, 2, 7, 4, 5, and 6, while the snapshot file remains unchanged: including data blocks 1', 2', 3', 4', 5', and 6'. Assuming that the initial 6 data blocks in the source file correspond to one segment, the newly allocated data block 7 will correspond to another segment, so this data modification operation will result in an increase in the number of segments in the source file, and the segments corresponding to consecutive data blocks will be discontinuous.

[0030] Specifically, after multiple COW operations are performed on the source file, the number of data blocks corresponding to the source file will continue to increase because each modification will create a new data block, and the newly allocated data block will be managed by the newly allocated segment, that is, the number of segments corresponding to the source file will continue to increase and be discontinuous, which will eventually lead to serious fragmentation problems of the source file in the file system. This situation not only consumes more storage resources, but also increases the complexity of data management and even affects the efficiency of data access.

[0031] In view of this, an embodiment of the present invention provides a new data processing method for persistent memory, which obtains a snapshot file by taking a snapshot of a source file in a file system, wherein multiple data blocks included in the snapshot file are all in a shared state, and multiple data blocks are located in at least one segment and correspond to different multiple address spaces in the file system. When triggering the modification of the first data block in the source file, the execution process of the new COW mechanism is: allocating a new data block corresponding to the first data block, then copying the data in the first data block to the new data block, and then updating the snapshot file so that the snapshot file includes the new data block and the data blocks other than the first data block in the multiple data blocks, that is, mapping the new data block to the snapshot file, and then directly modifying the data in the first data block in the first source file according to the modification operation to obtain the second source file. In this way, since the first data block in the first source file is directly modified, the first source file still uses the original segment, that is, no segment increase occurs, thereby solving the serious fragmentation problem of the source file caused by multiple COW operations, and ensuring the continuity and stability of the first source file in the segment.

[0032] The data processing method of the persistent memory provided in the embodiments of the present invention below can be executed by the main process of the application, at least part of the data of the application is stored in the persistent memory, and the application can directly access the persistent memory through the file system in the kernel. The data processing process of the persistent memory provided in the embodiments of the present invention is described in detail below in combination with the following embodiments.

[0033] Figure 3 A flowchart of a data processing method for persistent memory provided by an embodiment of the present invention. Figure 3 As shown, the method may include the following steps: 301. At a first moment, a snapshot process is performed on a first source file in a file system to obtain a first snapshot file, wherein the first source file includes a plurality of data blocks, and the plurality of data blocks are in a shared state in the first snapshot file; 302. At a second moment, in response to a modification operation on a first data block in a first source file, a new data block corresponding to the first data block is allocated, and data in the first data block is copied to the new data block, the new data block corresponds to a new address space in the file system that is different from the multiple address spaces, and the first data block is any one of the multiple data blocks.

[0034] 303 . Update the first snapshot file to determine that the updated first snapshot file contains the new data block and data blocks other than the first data block among the multiple data blocks, and determine that the new data block is in a non-shared state in the updated first snapshot file.

[0035] 304. According to the modification operation, modify the data in the first data block in the first source file to obtain a second source file.

[0036] Among them, the first moment and the second moment are two different moments. The snapshot processing of the first source file at the first moment and the modification operation of the first data block in the first source file at the second moment belong to different main processes, there is no strict execution sequence, and both are frequent operations in the file system.

[0037] When the first source file is snapshotted, a first snapshot file that is completely consistent with the first source file is obtained. The first snapshot file can be regarded as a copy file of the first source file, and multiple data blocks in the first snapshot file are in a shared state. In other words, the first source file and the first snapshot file share multiple data blocks, and the multiple data blocks correspond to different multiple continuous address spaces in the file system.

[0038] For example, Figure 4 As shown, assuming that the first source file includes six data blocks, which are recorded as data block 1, data block 2, data block 3, data block 4, data block 5 and data block 6 from top to bottom, then the first snapshot file will also include six data blocks. In this article, only to distinguish the first snapshot file from the first source file, the data blocks contained in the first snapshot file are recorded as data block 1', data block 2', data block 3', data block 4', data block 5' and data block 6' from top to bottom. The first source file and the first snapshot file share these six data blocks, that is, data block 1 and data block 1' point to the same address space, similarly, data block 2 and data block 2' point to the same address space, data block 3 and data block 3' point to the same address space, data block 4 and data block 4' point to the same address space, data block 5 and data block 5' point to the same address space, and data block 6 and data block 6' point to the same address space.

[0039] These six data blocks correspond to different multiple continuous address spaces in the file system. For example, assume that data block 1 and data block 1' correspond to 100M to 200M in the address space in the file system, data block 2 and data block 2' correspond to 200M to 300M in the address space in the file system, data block 3 and data block 3' correspond to 300M to 400M in the address space in the file system, data block 4 and data block 4' correspond to 400M to 500M in the address space in the file system, data block 5 and data block 5' correspond to 500M to 600M in the address space in the file system, and data block 6 and data block 6' correspond to 600M to 700M in the address space in the file system.

[0040] If a modification operation for the first data block of the first source file is triggered at the second moment, a new data block corresponding to the first data block is allocated in response to the modification operation. It can be understood that the first data block is any one of the multiple data blocks in the first source file, and the new data block corresponds to a new address space in the file system that is different from the multiple address spaces. Afterwards, the data in the first data block is copied to the new data block, and then the first snapshot file is updated, so that the updated first snapshot file contains the new data block and the data blocks other than the first data block among the multiple data blocks, and the new data block is in a non-shared state in the updated first snapshot file. In this way, the COW mechanism will not be triggered when the first data block is modified. Afterwards, since the first data block in the first source file is no longer in a shared state at this time, the data in the first data block in the first source file can be directly modified according to the modification operation to obtain the second source file. In the second source file, the first data block is in a non-shared state, while the data blocks other than the first data block among the multiple data blocks are still in a shared state.

[0041] Undertake Figure 4 In the example in , assuming that the modification operation is to modify data block 3 in the first source file, that is, the first data block is data block 3 among the six data blocks, then in response to the modification operation, data block 7 corresponding to data block 3 is allocated, assuming that data block 7 corresponds to 900M to 1000M in the address space in the file system. Afterwards, the data abc in data block 3 is copied to data block 7, that is, the data in data block 7 is abc, and then the first snapshot file is updated. Then Figure 4 As shown, the updated first snapshot file includes multiple data blocks, namely, data block 1', data block 2', data block 7 (abc), data block 4', data block 5' and data block 6', and among these six data blocks, data block 1', data block 2', data block 4', data block 5' and data block 6' are still in a shared state, while data block 7 is in a non-shared state. Afterwards, according to the modification operation, the data abc in data block 3 of the first source file is modified, for example, the data abc is modified to def, and the second source file is obtained. At this time, the data blocks 1, data blocks 2, data blocks 4, data blocks 5 and data blocks 6 included in the second source file are all in a shared state, while data block 3 is in a non-shared state.

[0042] Based on the technical solution of the embodiment of the present invention, by taking a snapshot of a first source file including multiple data blocks in a file system at a first moment, a first snapshot file also including multiple data blocks is obtained, and multiple data blocks in the first snapshot file are in a shared state. At a second moment, a modification operation on the first data block in the first source file is triggered, a new data block corresponding to the first data block is allocated, and the data in the first data block is copied to the new data block, and then the first snapshot file is updated, so that the new data block is in a non-shared state in the updated first snapshot file, so that the COW mechanism will not be triggered when the first data block is modified, and then the data in the first data block in the first source file is modified according to the modification operation to obtain the second source file. It can be seen that since the first data block in the first source file is no longer in a shared state, the first data block in the first source file can be directly modified in the embodiment of the present invention, and the address space of the first source file still uses the original segment, that is, no segment is added, thereby ensuring the continuity and stability of the first source file in the segment.

[0043] It can be understood that in the process of modifying the first data block of the first source file, data modification needs to be performed based on the page table, and since the modified data is temporarily stored in the cache rather than in the persistent memory, dirty pages will be generated, and since the snapshot file needs to maintain data consistency with the source file, it is necessary to write back the dirty pages of the first source file, that is, write the modified data back to the persistent memory. Usually, in the process of writing back dirty pages, it is necessary to find all virtual memory areas that share the dirty page through reverse mapping, set them to a write-protected state, and then write the data in the dirty page from the cache back to the persistent memory. In this process, if there are too many dirty pages and the source file is in a write-protected state for a long time, it will affect the snapshot operation of the source file, resulting in poor snapshot efficiency.

[0044] Based on this, an embodiment of the present invention further provides a dirty page write-back method to improve snapshot efficiency. Figure 5 A flowchart of a method for writing back dirty pages in persistent memory provided by an embodiment of the present invention. Figure 5 As shown, the method may include the following steps: 501. Write the modified data in the first data block into a corresponding virtual address in the page table.

[0045] 502. If it is determined that the physical address corresponding to the virtual address in the persistent memory is not allocated, a page fault is generated, a physical address is allocated to the page fault, and the data in the modified first data block is written to the physical address.

[0046] 503. Start a dirty page write-back thread, and use the dirty page write-back thread to asynchronously scan each page intermediate directory to determine a dirty page table generated in each page intermediate directory, wherein each page intermediate directory includes addresses of multiple page tables.

[0047] 504. Persistently write the data in the dirty page table into the persistent memory.

[0048] Specifically, the page table is a multi-level structure, in which each page table entry (pte) stores the mapping relationship between the virtual address and the physical address, and the page middle directory (pmd) contains the addresses of several page tables.

[0049] When modifying the data in the first data block in the first source file, first write the modified data in the first data block to the corresponding virtual address in the page table. If the physical address corresponding to the virtual address in the persistent memory is not allocated, a page fault will occur, and at this time, the allocation of the physical address in the cache will be triggered. After the physical address in the cache is allocated through the page fault, the data in the modified first data block can be written to the physical address in the cache first.

[0050] Afterwards, in response to the snapshot operation triggered on the second source file at the third moment, a dirty page writeback thread is started, and the dirty page writeback thread is executed asynchronously with the main process. At the same time, in response to the snapshot operation triggered on the second source file at the third moment, a snapshot is taken of the second source file to obtain a second snapshot file, the second snapshot file contains the modified first data block and data blocks other than the first data block among the multiple data blocks, and each data block in the second snapshot file is in a shared state. In this way, the main process can continue the snapshot operation without waiting for the dirty page writeback to be completed. Among them, the third moment is later than the second moment.

[0051] In the embodiment of the present invention, after being placed in the write protection state, the dirty page write-back thread asynchronously scans each page intermediate directory to determine the dirty page table generated under each page intermediate directory, and when the dirty page table is found, the data in the dirty page table is persistently written to the persistent memory. Figure 6 As shown in the figure, each page middle directory corresponds to 512 page table entries. After starting the dirty page write-back thread, the dirty page write-back thread only needs to traverse the page middle directory of 2M granularity to determine the dirty page page table generated in the page middle directory, without traversing the 512 page table entries in each page middle directory one by one to determine the dirty page page table, thus saving about 512 times of page table processing time. At the same time, the data in the dirty page page table in each page middle directory can be written to the persistent memory through an asynchronous thread. Obviously, this greatly reduces the latency of the snapshot operation.

[0052] In actual applications, snapshot operations on source files are usually very frequent, which means that new data blocks corresponding to the data blocks to be modified need to be frequently allocated, and old snapshot files need to be continuously deleted to reclaim the address space in the persistent memory. In the file system, the recycling of address space is controlled by reference counting, that is, when the reference count of a file is reduced to 0, the recycling of address space will be triggered.

[0053] Specifically, Figure 7 A flowchart of a method for reclaiming space in a persistent memory provided by an embodiment of the present invention. Figure 7 As shown, the method may include the following steps: 701. When it is determined that the reference count of the updated first snapshot file is zero, start a space recovery thread to asynchronously recover the updated first snapshot file through the space recovery thread.

[0054] 702. At a fourth moment, in response to a modification operation on the second data block in the first source file, directly modify the data in the second data block in the first source file according to the modification operation.

[0055] The first snapshot file includes data blocks in a shared state and data blocks in a non-shared state. When the first snapshot file is recycled, there are certain differences in the recycling methods adopted for the data blocks in these two different states. It can be understood that data blocks in different states can be distinguished by setting marks. For example, a data block in a shared state is marked as 0, and a data block in a non-shared state is marked as 1; or, a data block in a non-shared state is marked as 0, and a data block in a shared state is marked as 1. The embodiment of the present invention does not limit the specific method of marking data blocks in different states.

[0056] Specifically, during the process of recycling the updated first snapshot file, for the new data blocks in the updated first snapshot file that are in a non-shared state, the correspondence between the updated first snapshot file and the new address space corresponding to the new data blocks is deleted, and the new address space is placed in an idle state to notify the file system.

[0057] Because the new data block is in a non-shared state, it means that whether the first data block corresponding to the new data block in the first source file is modified or not will not affect the new data block. Therefore, for the new data block in a non-shared state in the first snapshot file, the correspondence between the updated first snapshot file and the new address space corresponding to the new data block can be directly deleted. In this case, no page fault will be caused.

[0058] Since the data block in a shared state is also used in the source file, if the source file continues to modify this data block, the main process will also be involved in the COW process of this data block, which will affect the recovery of the snapshot file. Therefore, it is necessary to make sure that the data block in a shared state will not be modified again before the data block in the snapshot file can be recovered. Otherwise, if the snapshot file in the recovery process is updated again, it will conflict with the recovery process.

[0059] Therefore, in the process of recovering the updated first snapshot file, for the second data block in the updated first snapshot file that is in a shared state, the correspondence between the address space corresponding to the updated first snapshot file and the second data block is deleted, and the second data block in the first source file needs to be modified to a non-shared state. The second data block is any data block among multiple data blocks except the first data block, and the second data block is shared by the first source file and the updated first snapshot file.

[0060] like Figure 8 As shown in FIG. 1 , assuming that after the first data block is modified, the reference count of the updated first snapshot file is reduced to zero, at this time, the main process is triggered to recycle the updated first snapshot file. Specifically, in the process of reclaiming the first snapshot file, the main process Figure 8 The corresponding relationship between the first snapshot file and the address space corresponding to the data block is directly deleted; for the address space corresponding to the data block that is still in the shared state (i.e. Figure 8 ), the correspondence between the address space of the data block in the shared state and the first snapshot file can be skipped first, until it is determined that there is no additional writing to the data block in the shared state, the data block is modified to a non-shared state, and the correspondence between the updated first snapshot file and the address space corresponding to the data block is deleted.

[0061] After starting the space recovery thread, if at the fourth moment, a modification operation is received on the second data block in the first source file, at this time, since the correspondence between the updated first snapshot file and the address space corresponding to the second data block has been deleted, and the second data block is in a non-shared state, modifying the second data block will not have any impact on the updated first snapshot file. Therefore, the second data block can be modified directly without triggering the COW mechanism.

[0062] According to the technical solution of this embodiment, the updated first snapshot file is asynchronously recovered by starting a space recovery thread, and different recovery methods are used for data blocks in a shared state and a non-shared state in the updated first snapshot file, thereby reducing the delay caused by page deletion.

[0063] Fig. 9A structural diagram of a data processing device for persistent memory provided by an embodiment of the present invention. Fig. 9 As shown, the data processing device 90 of the persistent memory includes: A snapshot module 91 is used to take a snapshot of a first source file in the file system at a first moment to obtain a first snapshot file, wherein the first snapshot file includes a plurality of data blocks, and the plurality of data blocks are in a shared state in the first snapshot file; an allocation module 92, configured to allocate, in response to a modification operation on a first data block in a first source file at a second moment, a new data block corresponding to the first data block, copy data in the first data block to the new data block, the new data block corresponding to a new address space in the file system that is different from the multiple address spaces, the first data block being any one of the multiple data blocks; An updating module 93 is used to update the first snapshot file to determine that the updated first snapshot file contains the new data block and data blocks other than the first data block among the multiple data blocks, and to determine that the new data block is in a non-shared state in the updated first snapshot file; The modification module 94 is used to modify the data in the first data block in the first source file according to the modification operation to obtain the second source file.

[0064] In a feasible implementation, the modification module 94 can be specifically used to write the data in the modified first data block to the corresponding virtual address in the page table. If it is determined that the physical address corresponding to the virtual address in the persistent memory is not allocated, a page fault is generated, a physical address is allocated to the page fault, and the data in the modified first data block is written to the physical address.

[0065] In a feasible implementation, the data processing device 90 of the persistent memory further includes a scanning module, which is used to start a dirty page write-back thread, and asynchronously scan each page intermediate directory through the dirty page write-back thread to determine the dirty page table generated under each page intermediate directory, wherein each page intermediate directory includes addresses of multiple page tables. And a writing module, which is used to persistently write the data in the dirty page table into the persistent memory.

[0066] Specifically, the scanning module may start a dirty page write-back thread in response to a snapshot operation triggered on the second source file at a third moment, and the third moment is later than the second moment.

[0067] In a feasible implementation, the data processing device 90 of the persistent memory further includes a recycling module, which is used to start a space recycling thread when it is determined that the reference count of the updated first snapshot file is zero, so as to asynchronously recycle the updated first snapshot file through the space recycling thread.

[0068] Specifically, the recycling module can be used to delete the correspondence between the updated first snapshot file and the new address space corresponding to the new data block in the unshared state for the new data block in the updated first snapshot file, and put the new address space into an idle state to notify the file system. For the second data block in the shared state in the updated first snapshot file, delete the correspondence between the updated first snapshot file and the address space corresponding to the second data block, and modify the second data block in the first source file to a unshared state. The second data block is any data block among the multiple data blocks except the first data block, and the second data block is shared by the first source file and the updated first snapshot file.

[0069] In a feasible implementation, the modification module 94 is further configured to, at a fourth moment, in response to a modification operation on the second data block in the first source file, directly modify the data in the second data block in the first source file according to the modification operation.

[0070] Fig. 9 The device shown can execute the steps in the data processing method of persistent memory in the aforementioned embodiment. The detailed execution process and technical effects can be found in the description of the aforementioned embodiment, which will not be repeated here.

[0071] The embodiment of the present invention also provides an electronic device, such as Fig.10 As shown, the electronic device 20 may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, and when the executable code is executed by the processor 21, the processor 21 can at least implement the data processing method of the persistent memory provided in the above embodiment.

[0072] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data processing method of the persistent memory provided in the aforementioned embodiment.

[0073] In addition, an embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor of an electronic device, the processor can at least implement the data processing method of the persistent memory provided in the aforementioned embodiment.

[0074] The device embodiments described above are merely illustrative, and the network elements described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement the solution without creative effort.

[0075] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method for persistent memory, characterized in that: Applied to the main process of an application, at least part of the data of the application is stored in a persistent memory, and the application directly accesses the persistent memory through a file system in the kernel, and the method includes: At a first moment, a snapshot process is performed on a first source file in the file system to obtain a first snapshot file, wherein the first source file includes a plurality of data blocks, and the plurality of data blocks are all in a shared state in the first snapshot file; At a second moment, in response to a modification operation on a first data block in the first source file, a new data block corresponding to the first data block is allocated, and data in the first data block is copied to the new data block, wherein the new data block corresponds to a new address space in the file system that is different from the multiple address spaces, and the first data block is any one of the multiple data blocks; updating the first snapshot file to determine that the updated first snapshot file contains the new data block and data blocks other than the first data block among the plurality of data blocks, and determining that the new data block is in a non-shared state in the updated first snapshot file; According to the modification operation, data in the first data block in the first source file is modified to obtain a second source file.

2. The method according to claim 1, characterized in that The modifying the data in the first data block in the first source file includes: Writing the modified data in the first data block into the corresponding virtual address in the page table; If it is determined that the physical address corresponding to the virtual address in the persistent memory is not allocated, a page fault is generated, the physical address is allocated to the page fault, and the modified data in the first data block is written to the physical address.

3. The method according to claim 2, characterized in that The method further comprises: Starting a dirty page write-back thread, and asynchronously scanning each page intermediate directory through the dirty page write-back thread to determine a dirty page page table generated under each page intermediate directory, wherein each page intermediate directory includes addresses of multiple page tables; The data in the dirty page table is persistently written into the persistent memory.

4. The method according to claim 3, characterized in that The starting of the dirty page write-back thread includes: In response to a snapshot operation triggered on the second source file at a third time, the dirty page write-back thread is started, and the third time is later than the second time.

5. The method according to claim 1, characterized in that The method further comprises: When it is determined that the reference count of the updated first snapshot file is zero, a space reclamation thread is started to asynchronously reclaim the updated first snapshot file through the space reclamation thread.

6. The method according to any one of claims 1 to 5, characterized in that: The process of recovering the updated first snapshot file includes: For the new data block in the updated first snapshot file that is in a non-shared state, deleting the correspondence between the updated first snapshot file and the new address space corresponding to the new data block, and placing the new address space in an idle state to notify the file system; For the second data block in the updated first snapshot file that is in a shared state, the corresponding relationship between the address space corresponding to the updated first snapshot file and the second data block is deleted, and the second data block in the first source file is modified to a non-shared state. The second data block is any data block among the multiple data blocks except the first data block, and the second data block is shared by the first source file and the updated first snapshot file.

7. The method according to claim 6, characterized in that After starting the space recovery thread, the method further includes: At a fourth moment, in response to a modification operation on the second data block in the first source file, the data in the second data block in the first source file is modified directly according to the modification operation.

8. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the method as claimed in any one of claims 1 to 7.

9. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to perform the method as claimed in any one of claims 1 to 7.

10. A computer program product, characterized in that include: A computer program, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 7.

Images