High-speed data processing methods, apparatus, devices, and media based on persistent memory
By partitioning and caching in dynamic memory and using persistent memory for I/O merging and alignment, the latency and lifespan issues caused by multiple writes in high-speed solid-state drive block storage are solved, achieving high-efficiency data storage performance and extended device lifespan.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2023-12-11
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, high-speed solid-state drive block storage requires two write operations when writing data, which increases latency, reduces write efficiency, and shortens the lifespan of NAND solid-state drives due to their characteristics. At the same time, the non-aligned access method of the file system causes performance waste.
By partitioning and caching cluster data in dynamic memory, using persistent memory for caching and IO merging and alignment, the merged data is flushed to the solid-state drive in one go. Combined with the data-unlossable feature of persistent memory, the data storage process is optimized.
It improved the performance of data storage services, reduced IO latency, extended device lifespan, and enhanced write efficiency and storage space utilization.
Smart Images
Figure CN117806989B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a high-speed data processing method, apparatus, device and medium based on persistent memory. Background Technology
[0002] Currently, in the design of high-speed solid-state drive (SSD) block storage products, data from cloud hosts in the computing cluster is first located or allocated to the corresponding block nodes via hashing, and then the data is cached in different pages. Next, the system needs to perform two write operations: one to write the business data to the corresponding data area of the SSD; and the other to append the metadata to the SSD in log format (wAppend).
[0003] There are currently three problems with this situation:
[0004] 1. This process requires two write operations to the solid-state drive (SSD). The write latency of SSDs built on NAND flash memory is typically tens of microseconds, so two write operations will result in a latency of tens or even nearly a hundred microseconds. This will limit the overall performance of high-speed block storage.
[0005] 2. NAND solid-state drives require data to be written in blocks and erase operations are required before writing. These characteristics reduce writing efficiency and shorten the lifespan of the drive.
[0006] 3. User data received by the file system is often not strictly aligned to a fixed size. In order to simplify the design of upper-layer software and make full use of storage space, the storage system allows users to update data with arbitrary bytes, but uses aligned IO to read and write storage hardware. This access method will lead to huge performance waste.
[0007] Therefore, improving the performance of data storage services has become an urgent technical problem to be solved. Summary of the Invention
[0008] This application provides a high-speed data processing method, apparatus, device, and storage medium based on persistent memory, aiming to improve the performance of data storage services.
[0009] In a first aspect, this application provides a high-speed data processing method based on persistent memory, the method comprising:
[0010] After receiving cluster data from the server, the cluster data is partitioned and cached based on at least one memory page in dynamic memory to obtain memory page data;
[0011] Based on persistent memory, the memory page data corresponding to each memory page is cached, and the memory page data is merged and aligned by I / O to obtain aligned data;
[0012] After completing the IO merging and alignment of the memory page data, the aligned data is flushed to the solid-state drive (SSD) for storage.
[0013] Obtain the IO data type, IO data volume, and access frequency of each IO request within a preset period;
[0014] Based on the IO data type corresponding to each IO request, determine the IO data volume threshold and the frequency threshold corresponding to each IO request.
[0015] If the amount of IO data in the current IO request is less than the IO data amount threshold, and the access frequency is less than the frequency threshold, then the cluster data corresponding to the current IO request cached in the persistent memory is deleted.
[0016] Secondly, this application also provides a high-speed data processing device based on persistent memory, the high-speed data processing device based on persistent memory comprising:
[0017] The memory page caching module is used to, after receiving cluster data from the server, partition and cache the cluster data based on at least one memory page in dynamic memory to obtain memory page data;
[0018] The persistent memory caching module is used to cache the memory page data corresponding to each memory page based on persistent memory, and to perform IO merging and alignment on each memory page data to obtain aligned data;
[0019] The solid-state drive (SSD) storage module is used to flush the aligned data to the SSD after completing the IO merging and alignment of the memory page data, so as to store the aligned data through the SSD.
[0020] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the high-speed data processing method based on persistent memory as described above.
[0021] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the high-speed data processing method based on persistent memory as described above.
[0022] This application provides a high-speed data processing method, apparatus, device, and storage medium based on persistent memory. The method includes, after receiving cluster data from a server, partitioning and caching the cluster data based on at least one memory page in dynamic memory to obtain memory page data; caching the memory page data corresponding to each memory page based on persistent memory, and performing IO merging and alignment on the memory page data to obtain aligned data; after completing the IO merging and alignment of the memory page data, flushing the aligned data to a solid-state drive (SSD) for storage. Through this method, this application partitions and caches cluster data using memory pages in dynamic memory, and then utilizes the characteristic of persistent memory that does not lose data when power is off to store data and metadata in DRAM in persistent memory. Through an IO alignment strategy, unaligned data in persistent memory is temporarily stored in a PageCache, and after the data volume reaches the basic alignment size, the data is flushed to the SSD in the background all at once. This improves IO bandwidth, reduces IO latency, extends device lifespan, and thus improves the performance of data storage services. Attached Figure Description
[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 A flowchart illustrating the first embodiment of a high-speed data processing method based on persistent memory provided in this application;
[0025] Figure 2 This application provides a schematic diagram of the data storage process for computing cluster data.
[0026] Figure 3 A schematic diagram of the high-speed data processing flow for persistent memory provided in this application;
[0027] Figure 4 A flowchart illustrating the second embodiment of the high-speed data processing method based on persistent memory provided in this application;
[0028] Figure 5 This is a schematic diagram of the structure of a first embodiment of a high-speed data processing device based on persistent memory provided in this application;
[0029] Figure 6 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.
[0030] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0033] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0034] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of a high-speed data processing method based on persistent memory provided in this application.
[0035] like Figure 1 As shown, the high-speed data processing method based on persistent memory includes steps S101 to S103.
[0036] S101. After receiving the cluster data from the server, the cluster data is partitioned and cached based on at least one memory page in the dynamic memory to obtain the memory page data.
[0037] In one embodiment, such as Figure 2 As shown, after receiving or collecting cluster data from the server, the data storage server uses an algorithm to allocate the cluster data to the corresponding block nodes in the dynamic memory DRAM and caches it in the memory page.
[0038] In one embodiment, the server may be a computing cluster, and the cluster data may be performance data, network data, etc. of the computing cluster.
[0039] In one embodiment, DRAM (Dynamic Random Access Memory) typically uses one transistor and one capacitor to represent one bit. Its dynamic nature stems primarily from the fact that the values in the memory cells of the memory array are represented by small charges that slowly leak from the circuitry over time. Therefore, the values in the memory cells change over time, rather than being static. Because the electrical values represented in the memory cells dissipate over time, the values in the memory cells must be refreshed periodically to ensure data validity.
[0040] Furthermore, the metadata corresponding to each memory page is obtained, and the metadata is transferred to the persistent memory so that the metadata can be stored or updated through the persistent memory.
[0041] Metadata refers to the descriptive information of the cluster data, including data type, IO request, and data address.
[0042] In one embodiment, the metadata of a block node or memory page is updated to the corresponding metadata area in persistent memory. This metadata includes data type, I / O requests, and the storage address of each type of data within the page, facilitating rapid lookup.
[0043] S102. Based on persistent memory, cache the memory page data corresponding to each memory page, and perform IO merging and alignment on each memory page data to obtain aligned data;
[0044] Further, the data objects corresponding to each IO request are obtained; when there are multiple IO requests to be merged whose data objects are consistent, the IO requests to be merged are merged based on the request time corresponding to each IO request to be merged to obtain the alignment data.
[0045] In one embodiment, when a data cache request corresponding to the current IO request is received, if historical data corresponding to the current IO request exists in the persistent memory, the current data of the current IO request is used to overwrite the historical data to obtain the aligned data.
[0046] In one embodiment, such as Figure 3 As shown, the plug list is used to fully merge the I / O generated within a process, while the elevator queue is used to handle the merging of I / O between processes, thereby improving I / O bandwidth, reducing I / O latency, and extending device life.
[0047] In one embodiment, multiple I / O requests are merged into a single I / O request for processing. I / O merging is achieved through a page cache. The cache in persistent memory is written back in units of pages or blocks. It is not that the number of I / O requests submitted to the page cache by the application needs to be submitted to the general block layer several times during the write-back. In this way, small I / O requests that are not contiguous in submission time but contiguous in space can be merged into the same cache page for processing.
[0048] In one embodiment, if the application previously generated IO that is already in the page cache, and the same IO is subsequently generated, then it is only necessary to overwrite the old IO in the page cache with the later IO. If the application frequently operates on the same location of the file, it is only necessary to submit the last IO to the underlying storage device.
[0049] In one embodiment, cached data written by the application to persistent memory can serve subsequent read operations. When reading data, the application first searches for a match and returns the result directly. If the match is not found, the data is read from the underlying layer and saved to the page cache. The next time the data is read, it can be found from the page cache.
[0050] In this embodiment, a three-layer logic of cache, plug, and elevator is implemented from top to bottom on persistent memory, maximizing the merging of I / O generated by the application. The page cache leads the way, performing both data caching and I / O merging, primarily focusing on merging small I / O blocks. Because memory pages are used for caching, the maximum merged I / O unit is the page size. Of course, for large I / O blocks, the page cache will also break them down into page-sized units. The plug list strives to merge I / O generated within a process, as I / O generated within a process is more correlated from the device's perspective. The elevator queue primarily handles the merging of inter-process I / O.
[0051] S103. After completing the IO merging and alignment of the memory page data, the aligned data is flushed to the solid-state drive (SSD) for storage.
[0052] In one embodiment, data from the computing cluster is allocated to the corresponding block nodes according to the relevant algorithm and cached in the Page. At the same time, the metadata of the Page / Block is also updated in place in the corresponding data area.
[0053] In one embodiment, unaligned data is temporarily stored in persistent memory. Leveraging the larger capacity of persistent memory, the data is cached and merged until the data volume reaches basic alignment. Both data caching and I / O merging are performed in persistent memory, primarily merging small I / O blocks. Because memory pages are used for caching, the maximum merged I / O unit is the page size. For large I / O blocks, the page cache will also break them down into page units. Simultaneously, user data received by the file system is aligned to a fixed size, using aligned I / O for storage, thereby improving write performance. Data is then flushed to the SSD in the background in one go. Maximizing the write block size reduces write amplification, improving NAND SSD write efficiency and lifespan.
[0054] S104. Obtain the IO data type, IO data volume, and access frequency of each IO request within a preset period for each IO request.
[0055] S105. Based on the IO data type corresponding to each IO request, determine the IO data volume threshold and the frequency threshold corresponding to each IO request.
[0056] S106. If the amount of IO data in the current IO request is less than the IO data amount threshold and the access frequency is less than the frequency threshold, then delete the cluster data corresponding to the current IO request cached in the persistent memory.
[0057] In one embodiment, an improved cache eviction policy is implemented on persistent memory. Different IO size thresholds are set for different business types based on the frequency of data access. In this way, the cache hit rate under various business scenarios can be improved by adjusting the IO size and usage frequency.
[0058] Further, the page number of each memory page is obtained; when a client requests data access to target data, the target data is queried in each memory page based on the data access request to determine the current memory page to which the target data is hit; and the page number corresponding to the current memory page is stored based on the data storage queue.
[0059] Furthermore, if the memory of the data storage queue is insufficient, the page number at the bottom layer of the data storage queue is removed, and the page number is stored at the front of the data storage queue.
[0060] In one embodiment, an improved cache eviction policy is implemented on persistent memory, using a special stack to store the page numbers of each currently used page. A customized strategy and algorithm determine whether to further flush data from persistent memory to the solid-state drive.
[0061] For example, when a new process accesses a page, such as reading data from persistent memory, that page number is pushed onto the top of the stack, and other page numbers are shifted to the bottom. If memory is insufficient, the page number at the bottom of the stack is removed. In this way, the top of the stack always contains the number of the most recently accessed page, while the bottom contains the page number of the least recently accessed page. Based on the frequency of data access, different IO size thresholds are set for different business types. If the IO size is large but the access frequency is low, the IO is discarded. This improves cache hit rate in various business scenarios by controlling IO size and usage frequency.
[0062] In one embodiment, a cache eviction policy is implemented on persistent memory, improving the cache hit rate. This enhances block storage performance, significantly reduces data read / write latency, and also improves data storage performance.
[0063] In this embodiment, leveraging the characteristic of persistent memory that does not lose data when power is off, the data and metadata in DRAM are first stored in persistent memory. An IO alignment strategy is provided, temporarily storing unaligned data in the PageCache. Once the data volume reaches the basic alignment size, the data is flushed to the solid-state drive in the background all at once. By optimizing the data persistence scheme, the "write amplification" problem of previous NAND solid-state drives is effectively solved, thereby avoiding the reduction in device lifespan caused by repeated writes and erases.
[0064] This embodiment provides a high-speed data processing method based on persistent memory. This method uses memory pages in dynamic memory to partition and cache cluster data. Leveraging the fact that persistent memory does not lose data when power is off, data and metadata from DRAM are first stored in persistent memory. Through an IO alignment strategy, unaligned data in persistent memory is temporarily stored in a PageCache. Once the data volume reaches the basic alignment size, the data is flushed to the solid-state drive in the background. This improves IO bandwidth, reduces IO latency, extends device lifespan, and ultimately enhances the performance of data storage services.
[0065] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the high-speed data processing method based on persistent memory provided in this application.
[0066] In this embodiment, based on the above... Figure 1 In the illustrated embodiment, after receiving the cluster data from the server, the process further includes:
[0067] S201. Obtain the data type and data volume of each data to be stored in the cluster data;
[0068] S202. Based on the data volume threshold corresponding to each of the data types, classify each of the data to be stored to obtain a first dataset and a second dataset, so as to cache the first dataset in the dynamic memory and cache the second dataset in the persistent memory.
[0069] The first dataset includes the data to be stored whose data type is a sensitive data type and whose data volume is less than the data volume threshold, and the second dataset includes the data to be stored whose data volume is greater than or equal to the data volume threshold.
[0070] In one embodiment, larger memory allocations are placed on persistent memory configured as system RAM, while smaller allocations tend to reside in DRAM; optimal performance is achieved by selecting the ratio of data stored in persistent memory and DRAM. That is, a large persistent memory capacity is used to store user data, while DRAM is used to store small, latency-sensitive data.
[0071] In one embodiment, although persistent memory has slightly lower latency than DRAM, it has a capacity that is tens of times larger and a speed that is an order of magnitude faster than NVMe SSDs. Compared to increasing memory capacity, using persistent memory reduces the overall holding cost.
[0072] In one embodiment, important but infrequently accessed data is stored in persistent memory. The non-volatility and high performance of persistent memory are used to ensure the performance of data storage services without significantly reducing data access performance (compared to DRAM).
[0073] Furthermore, based on the PMDK memory mapping method, persistent memory is accessed to cache the memory page data in the dynamic memory into the persistent memory.
[0074] Building upon this, the PMDK (Persistent Memory Development Kit) is used to access persistent memory via memory mapping. In this mode, persistent memory is directly connected to the processor via the memory bus. Applications can bypass the operating system and access data in persistent memory directly from user space without drivers, system calls, interrupts, or context switching. Therefore, the system has virtually no software overhead. This allows for more granular and precise control over the entire write process and effectively improves the system's write performance.
[0075] In one embodiment, by using the function library and tools provided by PMDK, memory units can be accessed and updated individually without performing garbage collection, thus improving the efficiency of storage space utilization.
[0076] In this embodiment, the memory allocation strategy is optimized: by selecting the ratio of data stored in persistent memory and DRAM to achieve optimal performance, a larger memory allocation is placed on persistent memory, and a smaller allocation is reserved for DRAM; this achieves the goal of using a large persistent memory capacity to store user data, while using DRAM to store small, latency-sensitive data. This improves data access performance; storing important but infrequently accessed data in persistent memory enhances the performance of data storage services. PMDK is used to access persistent memory via memory mapping, bypassing drivers, system calls, interrupts, and context switching, thus improving application access performance.
[0077] In this embodiment, on the one hand, the memory allocation strategy is optimized to improve the performance of data access and data storage services.
[0078] Please see Figure 5 , Figure 5 This is a schematic diagram of a first embodiment of a high-speed data processing device based on persistent memory provided in this application. This high-speed data processing device based on persistent memory is used to execute the aforementioned high-speed data processing method based on persistent memory. The high-speed data processing device based on persistent memory can be configured in a server.
[0079] like Figure 5 As shown, the high-speed data processing device 300 based on persistent memory includes: a memory page cache module 301, a persistent memory cache module 302, and a solid-state drive storage module 303.
[0080] The memory page caching module 301 is used to, after receiving cluster data from the server, partition and cache the cluster data based on at least one memory page in dynamic memory to obtain memory page data.
[0081] The persistent memory cache module 302 is used to cache the memory page data corresponding to each memory page based on persistent memory, and to perform IO merging and alignment on each memory page data to obtain aligned data;
[0082] Solid-state drive storage module 303 is used to flush the aligned data to the solid-state drive after completing the IO merging and alignment of the memory page data, so as to store the aligned data through the solid-state drive;
[0083] The IO request data acquisition module 304 is used to acquire the IO data type, IO data volume and access frequency of each IO request within a preset period for each IO request.
[0084] The IO request access frequency determination module 305 is used to determine the IO data volume threshold and the frequency threshold corresponding to each IO request based on the IO data type corresponding to each IO request.
[0085] The IO request deletion module 306 is used to delete the cluster data corresponding to the current IO request cached in persistent memory if the IO data volume of the current IO request is less than the IO data volume threshold and the access frequency is less than the frequency threshold.
[0086] In one embodiment, the high-speed data processing device 300 based on persistent memory further includes a metadata storage module, comprising:
[0087] The metadata acquisition and storage unit is used to acquire the metadata corresponding to each memory page data and transfer the metadata data to the persistent memory so as to store or update the metadata through the persistent memory; wherein, the metadata is the descriptive information of the cluster data, including data type, IO request and data address.
[0088] In one embodiment, the high-speed data processing device 300 based on persistent memory further includes a page access data caching module, comprising:
[0089] The page number acquisition unit is used to acquire the page number of each memory page;
[0090] The current memory page determination unit is used to, upon receiving a data access request from a client for target data, query the target data in each of the memory pages based on the data access request, and determine the current memory page to which the target data is hit;
[0091] The page number queue storage unit is used to store the page number corresponding to the current memory page based on the data storage queue.
[0092] In one embodiment, the page number queue storage unit includes:
[0093] The page number removal subunit is used to remove the page number at the bottom of the data storage queue and store the page number at the front of the data storage queue if the memory of the data storage queue is insufficient.
[0094] In one embodiment, the high-speed data processing device 300 based on persistent memory further includes a data filtering module, comprising:
[0095] A data information acquisition unit is used to acquire the data type and data volume of each piece of data to be stored in the cluster data;
[0096] A data classification unit is used to classify each of the data to be stored based on the data volume threshold corresponding to each of the data types, and to obtain first data whose data type belongs to a sensitive data type and whose data volume is less than the data volume threshold, and second data whose data volume is greater than or equal to the data volume threshold.
[0097] A data storage unit is used to store the second data, which has a large amount of data, in the persistent memory, and to store the first data, which has a small amount of data, in the dynamic memory.
[0098] In one embodiment, the persistent memory cache module 302 includes:
[0099] The data object acquisition unit is used to acquire the data object corresponding to each IO request;
[0100] An IO request merging unit is used to merge the IO requests to be merged based on the request time of each IO request to be merged when the data objects corresponding to multiple IO requests to be merged are consistent, in order to obtain the aligned data.
[0101] An IO request data overwrite unit is used to, when receiving a data cache request corresponding to the current IO request, if historical data corresponding to the current IO request exists in the persistent memory, overwrite the historical data with the current data of the current IO request to obtain the aligned data.
[0102] In one embodiment, the high-speed data processing device 300 based on persistent memory further includes:
[0103] The data access module is used to access persistent memory based on the PMDK memory mapping method, so as to cache the memory page data in the dynamic memory into the persistent memory.
[0104] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the aforementioned embodiments of the high-speed data processing method based on persistent memory, and will not be repeated here.
[0105] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.
[0106] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.
[0107] See Figure 6The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.
[0108] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any high-speed data processing method based on persistent memory.
[0109] The processor provides computing and control capabilities, supporting the operation of the entire computer device.
[0110] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any high-speed data processing method based on persistent memory.
[0111] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0112] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.
[0113] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:
[0114] After receiving cluster data from the server, the cluster data is partitioned and cached based on at least one memory page in dynamic memory to obtain memory page data;
[0115] Based on persistent memory, the memory page data corresponding to each memory page is cached, and the memory page data is merged and aligned by I / O to obtain aligned data;
[0116] After completing the IO merging and alignment of the memory page data, the aligned data is flushed to the solid-state drive (SSD) for storage.
[0117] Obtain the IO data type, IO data volume, and access frequency of each IO request within a preset period;
[0118] Based on the IO data type corresponding to each IO request, determine the IO data volume threshold and the frequency threshold corresponding to each IO request.
[0119] If the amount of IO data in the current IO request is less than the IO data amount threshold, and the access frequency is less than the frequency threshold, then the cluster data corresponding to the current IO request cached in the persistent memory is deleted.
[0120] In one embodiment, after receiving cluster data from the server, the processor partitions and caches the cluster data based on at least one memory page in dynamic memory. After obtaining the memory page data, the processor is further configured to:
[0121] Obtain the metadata corresponding to each memory page and transfer the metadata to the persistent memory so as to store or update the metadata through the persistent memory;
[0122] Metadata refers to the descriptive information of the cluster data, including data type, IO request, and data address.
[0123] In one embodiment, after receiving cluster data from the server, the processor partitions and caches the cluster data based on at least one memory page in dynamic memory. After obtaining the memory page data, the processor is further configured to:
[0124] Obtain the page number of each memory page;
[0125] Upon receiving a data access request from a client for target data, the system queries the target data in each memory page based on the data access request to determine the current memory page to which the target data is located.
[0126] The page number corresponding to the current memory page is stored using a data storage queue.
[0127] In one embodiment, when the processor implements the storage of the page number corresponding to the current memory page based on the data storage queue, it is configured to:
[0128] If the memory of the data storage queue is insufficient, the page number at the bottom of the data storage queue is removed and the page number is stored at the front of the data storage queue.
[0129] In one embodiment, after receiving cluster data from the server, the processor is further configured to:
[0130] Obtain the data type and data volume of each piece of data to be stored in the cluster data;
[0131] Based on the data volume threshold corresponding to each of the data types, each of the data to be stored is classified to obtain first data whose data type belongs to a sensitive data type and whose data volume is less than the data volume threshold, and second data whose data volume is greater than or equal to the data volume threshold.
[0132] The second data, which has a large amount of data, is stored in the persistent memory, while the first data, which has a small amount of data, is stored in the dynamic memory.
[0133] In one embodiment, when the processor implements the persistent memory-based method of caching the memory page data corresponding to each memory page and performing I / O merging and alignment on each memory page data to obtain aligned data, it is configured to:
[0134] Retrieve the data object corresponding to each IO request;
[0135] When multiple IO requests to be merged correspond to the same data object, the IO requests to be merged are merged based on the request time of each IO request to be merged to obtain the alignment data.
[0136] Alternatively, when a data cache request corresponding to the current IO request is received, if historical data corresponding to the current IO request exists in the persistent memory, the current data of the current IO request is used to overwrite the historical data to obtain the aligned data.
[0137] In one embodiment, the processor is configured to run a computer program stored in memory and also to implement:
[0138] Based on the PMDK-based memory mapping method, persistent memory is accessed to cache the memory page data in the dynamic memory into the persistent memory.
[0139] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the high-speed data processing methods based on persistent memory provided in the embodiments of this application.
[0140] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.
[0141] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A high-speed data processing method based on persistent memory, characterized in that, The method includes: After receiving cluster data from the server, the cluster data is partitioned and cached based on at least one memory page in dynamic memory to obtain memory page data; Based on persistent memory, the memory page data corresponding to each memory page is cached, and the memory page data is merged and aligned by I / O to obtain aligned data; After completing the IO merging and alignment of the memory page data, the aligned data is flushed to the solid-state drive (SSD) for storage. Obtain the IO data type, IO data volume, and access frequency of each IO request within a preset period; Based on the IO data type corresponding to each IO request, determine the IO data volume threshold and the frequency threshold corresponding to each IO request. If the amount of IO data in the current IO request is less than the IO data amount threshold, and the access frequency is less than the frequency threshold, then the cluster data corresponding to the current IO request cached in the persistent memory is deleted.
2. The high-speed data processing method based on persistent memory according to claim 1, characterized in that, After receiving cluster data from the server, the process of partitioning and caching the cluster data based on at least one memory page in dynamic memory, and then obtaining the memory page data, further includes: Obtain the metadata corresponding to each memory page and transfer the metadata to the persistent memory so as to store or update the metadata through the persistent memory; Metadata refers to the descriptive information of the cluster data, including data type, IO request, and data address.
3. The high-speed data processing method based on persistent memory according to claim 1, characterized in that, After receiving cluster data from the server, the process of partitioning and caching the cluster data based on at least one memory page in dynamic memory, and then obtaining the memory page data, further includes: Obtain the page number of each memory page; Upon receiving a data access request from a client for target data, the system queries the target data in each memory page based on the data access request to determine the current memory page to which the target data is located; The page number corresponding to the current memory page is stored using a data storage queue.
4. The high-speed data processing method based on persistent memory according to claim 3, characterized in that, The step of storing the page number corresponding to the current memory page based on the data storage queue includes: If the memory of the data storage queue is insufficient, the page number at the bottom of the data storage queue is removed and the page number is stored at the front of the data storage queue.
5. The high-speed data processing method based on persistent memory according to claim 1, characterized in that, After receiving the cluster data from the server, the process also includes: Obtain the data type and data volume of each piece of data to be stored in the cluster data; Based on the data volume threshold corresponding to each of the data types, each of the data to be stored is classified to obtain first data whose data type belongs to a sensitive data type and whose data volume is less than the data volume threshold, and second data whose data volume is greater than or equal to the data volume threshold. The second data, which has a large amount of data, is stored in the persistent memory, while the first data, which has a small amount of data, is stored in the dynamic memory.
6. The high-speed data processing method based on persistent memory according to claim 1, characterized in that, The method of caching the memory page data corresponding to each memory page based on persistent memory, and performing IO merging and alignment on the memory page data to obtain aligned data includes: Retrieve the data object corresponding to each IO request; When multiple IO requests to be merged correspond to the same data object, the IO requests to be merged are merged based on the request time of each IO request to be merged to obtain the alignment data. Alternatively, when a data cache request corresponding to the current IO request is received, if historical data corresponding to the current IO request exists in the persistent memory, the current data of the current IO request is used to overwrite the historical data to obtain the aligned data.
7. The high-speed data processing method based on persistent memory according to claim 1, characterized in that, The method further includes: Based on the PMDK-based memory mapping method, persistent memory is accessed to cache the memory page data in the dynamic memory into the persistent memory.
8. A high-speed data processing device based on persistent memory, characterized in that, The high-speed data processing device based on persistent memory includes: The memory page caching module is used to, after receiving cluster data from the server, partition and cache the cluster data based on at least one memory page in dynamic memory to obtain memory page data; The persistent memory caching module is used to cache the memory page data corresponding to each memory page based on persistent memory, and to perform IO merging and alignment on each memory page data to obtain aligned data; The solid-state drive storage module is used to flush the aligned data to the solid-state drive after completing the IO merging and alignment of the memory page data, so as to store the aligned data through the solid-state drive; The IO request data acquisition module is used to acquire the IO data type, IO data volume, and access frequency of each IO request within a preset period for each IO request. The IO request access frequency determination module is used to determine the IO data volume threshold and the frequency threshold corresponding to each IO request based on the IO data type corresponding to each IO request. The IO request deletion module is used to delete the cluster data corresponding to the current IO request cached in the persistent memory if the IO data volume of the current IO request is less than the IO data volume threshold and the access frequency is less than the frequency threshold.
9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the high-speed data processing method based on persistent memory as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the high-speed data processing method based on persistent memory as described in any one of claims 1 to 7.