A method and apparatus for ensuring persistent memory data crash consistency

Abstract

The application belongs to the technical field of computer storage, and specifically discloses a method and device for ensuring crash consistency of persistent memory data. The method comprises the following steps: removing a log operation for ensuring crash consistency of memory data in a system; after the system crashes and restarts, performing a log-free crash recovery process; the log-free crash recovery process is used for sequentially scanning each encoding strip stored in a memory space, so that when data crash inconsistency occurs in the encoding strip, original check data of the encoding strip is used for data recovery. Through the application, the write performance of the persistent memory system can be effectively enhanced, the delay and throughput of the system can be significantly improved, and the overall performance of the system can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer storage, and more particularly, relates to a method and device for ensuring crash consistency of persistent memory data. BACKGROUND

[0002] Persistent memory (PM) is a promising new memory technology that aims to provide the performance of dynamic random-access memory (DRAM) and the capacity of a disk for persistent storage systems. System applications can directly execute load / store instructions on byte-addressable persistent memory. Although persistent memory provides data persistence, the correctness of data is seriously affected if an unexpected crash occurs, resulting in data inconsistency.

[0003] In the prior art, to solve the above technical defects, the traditional crash consistency method is to introduce a log mechanism to record the historical data of memory operations in the persistent memory system, especially write operations, and to ensure that the system recovers data to a consistent state after a crash through log rollback and other functions. However, this log mechanism significantly increases the write operation, which limits the system write performance and further leads to a decline in system performance. SUMMARY

[0004] In view of the defects of the prior art, the purpose of the present application is to enhance the write performance of the persistent memory system and improve the system performance, aiming to solve the problem that the log mechanism in the prior art for ensuring crash consistency of the system significantly increases the write operation, which limits the system write performance and further leads to a decline in system performance.

[0005] To achieve the above purpose, in a first aspect, the present application provides a method for ensuring crash consistency of persistent memory data, comprising:

[0006] removing the log operation in the system for ensuring crash consistency of memory data;

[0007] after the system crashes and restarts, performing a log-free crash recovery process;

[0008] the log-free crash recovery process is used to sequentially scan each encoding strip stored in the memory space, so as to perform data recovery using the original check data of the encoding strip stored in the case of data crash inconsistency in the encoding strip.

[0009] Optionally, the original check data includes the original checksum of the original check block and the original data block; and the specific steps of scanning the encoding strip stored in the memory space include:

[0010] reading the encoding strip into a memory buffer;

[0011] In the memory buffer, all original data blocks in the encoding strip are traversed to calculate new checksums of the original data blocks;

[0012] Each of the new checksums is matched with a corresponding original checksum, and based on the matching result, the number of data block columns with inconsistent data in the encoding strip is determined;

[0013] In the case where the number of data block columns is greater than zero and not greater than the number of check block columns in the encoding strip, each original check block in the check block column is decoded to obtain a plurality of new data blocks;

[0014] Based on the plurality of new data blocks and the original checksums of each of the original data blocks, data recovery is performed on the encoding strip.

[0015] Optionally, the data recovery on the encoding strip based on the plurality of new data blocks and the original checksums of each of the original data blocks comprises:

[0016] Determine the checksum of each of the new data blocks;

[0017] Match the checksum of each of the new data blocks with each of the original checksums;

[0018] In the case where there is a matching successful new data block, the matching successful new data block is replaced by the corresponding original data block in the encoding strip;

[0019] Encode the matching successful new data block to obtain a corresponding new check block;

[0020] In the case where the new check block does not match the original check block, the new check block is used to replace the corresponding original check block in the encoding strip to complete the data recovery of the encoding strip.

[0021] Optionally, after the matching of each of the new checksums with the corresponding original checksum and the determination of the number of data block columns with inconsistent data in the encoding strip based on the matching result, the method further comprises:

[0022] In the case where the number of data block columns with inconsistent data in the encoding strip is zero, new check blocks are determined based on all original data blocks in the encoding strip;

[0023] In the case where the new check block does not match the original check block, the new check block is used to replace the corresponding original check block in the encoding strip to complete the data recovery of the encoding strip.

[0024] Optionally, after the matching each of the new checksums with the corresponding original checksums and determining the number of data block columns with inconsistent data in the encoding strip based on the matching results, the method further comprises:

[0025] In the case that the number of data block columns is greater than the number of check block columns in the encoding strip, popping up a warning message;

[0026] Scanning a next encoding strip of the encoding strips to perform data recovery on the next encoding strip.

[0027] Optionally, after the removing the log operation for ensuring the crash consistency of the in-memory data, the method further comprises:

[0028] After receiving a data block update request of the encoding strip, caching the data block update request and update data of at least one target data block to a memory buffer; the target data block is a data block in the encoding strip required to be updated by the data block update request;

[0029] In response to the data block update request, calculating a data block difference value of each of the target data blocks corresponding to the target byte cache line granularity in the memory buffer, and determining a check block difference value based on each of the data block difference values;

[0030] Performing a first operation of asynchronously writing update data of the data block to execute the update data of each of the target data blocks to the corresponding memory address in the data block address order and in the target byte cache line granularity;

[0031] Calling a first memory barrier instruction to persist the first operation.

[0032] Optionally, after the calling the first memory barrier instruction to persist the first operation, the method further comprises:

[0033] Determining a new check block corresponding to each of the check block difference values in the target byte cache line granularity;

[0034] Performing a second operation of asynchronously writing update data of the check block to execute the data of each of the new check blocks to the corresponding memory address in the target byte cache line granularity;

[0035] Calling a second memory barrier instruction to persist the second operation.

[0036] In a second aspect, the application provides a device for ensuring the crash consistency of the persistent memory data, comprising:

[0037] A log removal module configured to remove a log operation for ensuring the crash consistency of the in-memory data;

[0038] a data recovery module, configured to perform a logless crash recovery process after a system crash restart.

[0039] The logless crash recovery process is configured to sequentially scan each encoding strip stored in the memory space, and perform data recovery by using original check data of the encoding strip stored in the case of data crash inconsistency in the encoding strip.

[0040] In a third aspect, the present application provides an electronic device, comprising: at least one memory configured to store a program; and at least one processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0041] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0042] In a fifth aspect, the present application provides a computer program product, and when the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0043] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0044] The method and device for ensuring data crash consistency of persistent memory provided by the present application introduce a logless crash recovery process, remove the log operation for ensuring data crash consistency in the system, and after a system crash restart, the logless crash recovery process is executed. The process will sequentially scan each encoding strip stored in the memory space, and in the case of data crash inconsistency in the encoding strip, the original check data of the encoding strip stored is used for data recovery. Thus, the system completely uses check data to replace the log, reuses the existing multiple reliable redundant data in the system to ensure data crash consistency, can effectively enhance the write performance of the persistent memory system, significantly improve the delay and throughput of the system, and improve the overall performance of the system. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is one of the flowcharts of the method for ensuring data crash consistency of persistent memory provided by the embodiments of the present application;

[0046] Figure 2Figure 2 is a flowchart of a method for guaranteeing crash consistency of persistent memory data according to an embodiment of the present application;

[0047] Figure 3 Figure 1 is a flowchart of data updating in a method for guaranteeing crash consistency of persistent memory data according to an embodiment of the present application;

[0048] Figure 4 Figure 2 is a flowchart of data updating in a method for guaranteeing crash consistency of persistent memory data according to an embodiment of the present application;

[0049] Figure 5 Figure 3 is a structural diagram of an apparatus for guaranteeing crash consistency of persistent memory data according to an embodiment of the present application;

[0050] Figure 6 Figure 4 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0052] The terms "first" and "second" and the like in the specification and claims herein are used to distinguish different objects, rather than to describe a specific order. For example, the first memory barrier instruction and the second memory barrier instruction are used to distinguish different functions of the memory barrier instruction, rather than to describe a specific order of the memory barrier instruction.

[0053] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration, in any non-limiting and non-exhaustive sense. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of terms such as "exemplary" or "for example" is merely intended to present concepts in a concrete manner.

[0054] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more, for example, a plurality of new data blocks means two or more new data blocks, and the like.

[0055] For the convenience of understanding, the relevant technical terms involved in the embodiments of the present application will be explained and described below.

[0056] (1) Coding strip

[0057] The original stored data is divided into a plurality of data blocks, and then one or more check blocks are generated by performing specific encoding on the data blocks. The data blocks and the check blocks form a stripe, which can also be described as an encoded stripe.

[0058] The data blocks are actual data units in the storage or transmission process, and they contain specific data information that needs to be saved or transmitted by the user. Each data block contains a certain amount of data. The check blocks are redundant data used to verify the integrity of the data blocks, which are usually calculated by a certain algorithm (such as cyclic redundancy check) on the data blocks.

[0059] (2) Crash consistency

[0060] Crash consistency is a very important concept in file systems, database systems, and other systems that need to persist data. It refers to the ability of a system to guarantee the consistency and integrity of data after a system crash, power failure, or other unforeseen failure, and to recover to a legal and operable state, ensuring that after the system restarts or recovers, the data will not be damaged, lost, or inconsistent.

[0061] (3) Memory barrier instruction

[0062] Memory barrier instruction, also known as memory fence, memory barrier, etc., is a type of synchronization barrier instruction. It is a synchronization point in the operation of random access to memory by CPU or compiler, so that all read and write operations before this point are executed before the operation after this point can be started.

[0063] The embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application.

[0064] Figure 1 is one of the flowcharts of the method for guaranteeing crash consistency of persistent memory data provided by the embodiments of the present application, as shown in Figure 1 , comprising steps 110 and 120.

[0065] In step 110, the log operation for guaranteeing crash consistency of memory data in the system is removed.

[0066] In step 120, after the system crashes and restarts, a log-free crash recovery process is performed.

[0067] The log-free crash recovery process is used to sequentially scan each encoded stripe stored in the memory space, so as to perform data recovery using the original check data of the stored encoded stripe in the case of data crash inconsistency in the encoded stripe.

[0068] It can be understood that the system described in the embodiments of the present application is a persistent memory storage system.

[0069] Specifically, the log operation described in the embodiments of the present application refers to a log operation in the original system for ensuring the crash consistency of the memory data, mainly including log addition and deletion operations in the data update process and log write operations.

[0070] The log-free crash recovery process described in the embodiments of the present application is used to recover the data to a consistent state after the system crashes in the case of completely removing the log mechanism.

[0071] In the embodiments of the present application, in step 110, the log operation for ensuring the crash consistency of the memory data in the system is first removed, that is, the log addition and deletion operations in the data update process of the original system, the log write operation and the like are removed, so as to reduce the log overhead.

[0072] It should be noted that directly removing the log will affect the correctness of the data. In order to not affect the reliability and crash consistency of the system in the case of no log, the log-free crash recovery process is introduced in the embodiments of the present application.

[0073] Further, in the embodiments of the present application, in step 120, after the system crashes and restarts, the log-free crash recovery process is executed, and under the process, each encoded strip stored in the memory space will be scanned in sequence according to the storage address of the memory space. In the scanning process, if it is found that there is a data crash inconsistency in a certain encoded strip, the original check data of the encoded strip can be used to recover the inconsistent data in the encoded strip to its previous consistent state by decoding and calculating the parity check and combining the data in the encoded strip in the consistent state, so as to realize the crash consistency recovery of the data.

[0074] The method for ensuring the crash consistency of the persistent memory data in the embodiments of the present application introduces the log-free crash recovery process, removes the log operation for ensuring the crash consistency of the memory data in the system, executes the log-free crash recovery process after the system crashes and restarts, and the process will scan each encoded strip stored in the memory space in sequence, so as to use the stored original check data of the encoded strip to recover the data when the data crash inconsistency occurs in the encoded strip. Thus, the log is completely replaced by the check data in the system, the existing various reliability redundant data in the system is reused to ensure the data crash consistency, the write performance of the persistent memory system can be effectively enhanced, the delay and throughput of the system can be significantly improved, and the overall performance of the system can be improved.

[0075] Based on the content of the above embodiments, as an optional embodiment, the original check data includes original check blocks and original checksums of the original data blocks; the specific steps of scanning the encoded stripe stored in the memory space include:

[0076] reading the encoded stripe into a memory buffer;

[0077] in the memory buffer, traversing all the original data blocks in the encoded stripe to calculate new checksums of the original data blocks;

[0078] matching each new checksum with a corresponding original checksum, and based on the matching result, determining the number of data block columns in the encoded stripe that are inconsistent;

[0079] in the case where the number of data block columns is greater than zero and not greater than the number of check block columns in the encoded stripe, decoding each original check block in the check block column to obtain a plurality of new data blocks;

[0080] based on the plurality of new data blocks and the original checksums of the original data blocks, performing data recovery on the encoded stripe.

[0081] It should be noted that the data block column refers to the column storage data in the encoded stripe composed of one or more data blocks, and each data block has its corresponding data block column. Similarly, the check block column is also a column storage data composed of one or more check blocks. In addition, for a single check block storage system, such as a storage system based on a redundant array of independent disks (RAID), its encoded stripe contains multiple data blocks and one check block, so its encoded stripe contains multiple data block columns and a single check block column; and for a multi-check block storage system, such as a storage system based on erasure coding technology, its encoded stripe can contain multiple data blocks and multiple check blocks, so its encoded stripe can contain multiple data block columns and multiple check block columns.

[0082] Specifically, in the embodiments of the present application, in order to ensure that the system cannot obtain the location information of potential crash inconsistency due to lack of log records, an optimistic speculative recovery technique is proposed.

[0083] More specifically, in the process of performing the logless crash recovery procedure, each encoded stripe stored in the memory space will be scanned in turn, starting from the logical head of the configured full address space, in units of encoded stripes (a series of data blocks and corresponding check blocks). In the scanning process, for any encoded stripe, the process first reads the encoded stripe into a memory buffer, and then traverses all the original data blocks in the encoded stripe in the memory buffer to calculate whether the checksum of the internal persistent record matches the content of the data block.

[0084] Specifically, according to each original data block, the original data block is operated according to the calculation method of the original checksum, such as using exclusive OR operation, cumulative summation, or cyclic redundancy check, to calculate the new checksum of each original data block.

[0085] Further, in the embodiments of the present application, the new checksum of each original data block is matched with the original checksum corresponding to each original data block, to determine whether they are consistent, and according to the matching result of each original data block, it can be determined which data blocks have inconsistent data, and then the number of data block columns with inconsistent data in the encoded stripe can be determined according to the data block column with inconsistent data. Then, according to the size relationship between the number of data block columns and the number of check block columns in the encoded stripe, it is determined whether the data inconsistency of the data block exceeds the repair capability.

[0086] Further, in the embodiments of the present application, if it is determined that the number of data block columns with inconsistent data is greater than zero and not greater than the number of check block columns in the encoded stripe, it means that there is data block crash inconsistency, and the check data can be used for data recovery, and the system data block data inconsistency does not exceed the repair capability, at this time, further decoding calculation of different paths can be performed on each original checksum in the check block column, wherein each check block corresponds to a decoding calculation path, to obtain the corresponding data block of each original checksum decoding calculation, thereby obtaining a plurality of new data blocks.

[0087] In the embodiments of the present application, the plurality of new data blocks obtained by the above decoding and the original checksum of each original data block in the check data are used for re-encoding calculation to obtain new check blocks, thereby completing the data recovery procedure of the encoded stripe.

[0088] The method of the embodiments of the present application fully reuses the check blocks and checksums in the system, uses the form of scanning recovery to completely replace the function of the traditional log, effectively enhances the write performance of the system, and at the same time, can detect all crash inconsistency conditions and repair inconsistent data blocks within its repair capability, thereby improving the overall performance of the system.

[0089] Based on the content of the above embodiments, as an optional embodiment, data recovery is performed on the encoding strip based on the new data blocks and the original checksums of the original data blocks, including:

[0090] determining the checksums of the new data blocks;

[0091] matching the checksums of the new data blocks with the original checksums;

[0092] in the case where there is a matching successful new data block, replacing the corresponding original data block in the encoding strip with the matching successful new data block;

[0093] encoding the matching successful new data block to obtain a corresponding new checksum block;

[0094] in the case where there is a new checksum block that does not match the original checksum block, replacing the corresponding original checksum block in the encoding strip with the new checksum block to complete the data recovery of the encoding strip.

[0095] Specifically, in the embodiments of the present application, after the new data blocks are calculated by decoding the original checksum blocks, the new data blocks can be processed according to the calculation method of the original checksums, such as using exclusive-OR operation, cumulative summation or cyclic redundancy check, to calculate the checksums of the new data blocks.

[0096] Further, in the embodiments of the present application, the checksums of the new data blocks obtained are matched with the original checksums to determine whether there is a checksum consistent with the original checksum in the checksums of the new data blocks. In the case where there is a matching successful new data block, it means that the checksum of the new data block is consistent with the original checksum. At this time, the matching successful new data block is replaced with the corresponding original data block in the encoding strip according to the corresponding spatial address, thereby realizing the recovery of the data block data in the encoding strip.

[0097] In the embodiments of the present application, after the checksums of the new data blocks are compared and matched with the original checksums, in the case where there is no matching successful new data block, that is, the checksums of the new data blocks are all different from the original checksums, there is no new data block that passes the checksum, at this time, it means that the number of inconsistent total crashes on the data blocks and the checksum blocks exceeds the repair capability, then a warning information is popped up.

[0098] Further, in the embodiments of the present application, the system will continue to scan the next encoding strip of the encoding strip, and the next encoding strip is recovered or a warning is thrown in the same way as described above. Thus, by judging whether the checksums of the new data blocks match the original checksums, if not, it is determined that the total number of inconsistent crashes on the data blocks and the check blocks in the system encoding strip exceeds the repair capability, and a warning information is promptly popped up to the front desk, further reminding the user of the recovery of the memory data, so as to facilitate the user to respond in time and improve the user experience.

[0099] Further, in the embodiments of the present application, the new data blocks that match successfully are encoded to obtain corresponding new check blocks. At this time, the new check blocks are compared with the original check blocks, and if there is a mismatch, the new check blocks can replace the corresponding original check blocks in the encoding strip, thereby realizing the recovery of the check block data in the encoding strip and finally completing the data recovery of the encoding strip. If each new check block matches each original check block, it means that the original check block is correct data, and the data recovery of the original check block is not needed.

[0100] The method of the embodiments of the present application compares and matches the original checksums with the checksums of the new data blocks to judge whether the new data blocks are correct data blocks, and then executes the re-encoding process on the new data blocks that match successfully to recover the check block data and finally complete the data recovery of the entire encoding strip, which can effectively realize the data crash consistency under the condition of no log.

[0101] Based on the above-mentioned embodiments, as an optional embodiment, after matching each new checksum with the corresponding original checksum and determining the number of data block columns with inconsistent data in the encoding strip based on the matching result, the method further comprises:

[0102] In the case where the number of data block columns with inconsistent data in the encoding strip is zero, the new check blocks are determined based on all the original data blocks in the encoding strip.

[0103] In the case where the new check blocks do not match the original check blocks, the new check blocks are used to replace the corresponding original check blocks in the encoding strip to complete the data recovery of the encoding strip.

[0104] Specifically, in the embodiments of the present application, in the case where the number of data block columns with inconsistent data in the encoding strip is zero, it can be considered that the data blocks in the encoding strip have not occurred data inconsistency, and the original data blocks are correct data. At this time, the data recovery of the data of each original data block is not needed.

[0105] Further, in the embodiment of the present application, the determination of whether the original check block has data inconsistency can be achieved by re-encoding calculation of all original data blocks in the encoding strip to determine new check blocks, and then comparing and matching the new check blocks with the corresponding original check blocks. In the case where the new check blocks do not match the original check blocks, it indicates that the data of the original check block is inaccurate. At this time, the new check blocks obtained can be used to replace the corresponding original check blocks in the encoding strip to complete the data recovery of the entire encoding strip.

[0106] The method of the embodiment of the present application can effectively realize data crash consistency under the condition of no log by judging that the original data block has no data inconsistency, comparing and matching the original check block with the new check block obtained by re-encoding the original data block, using the new check block to recover the check block data in the case of mismatch, and finally completing the data recovery of the entire encoding strip.

[0107] Based on the content of the above embodiment, as an optional embodiment, after matching each new check sum with the corresponding original check sum and determining the number of data block columns with data inconsistency in the encoding strip based on the matching result, the method further comprises:

[0108] In the case where the number of data block columns is greater than the number of check block columns in the encoding strip, a warning information is popped up;

[0109] The next encoding strip of the encoding strip is scanned to perform data recovery on the next encoding strip.

[0110] Specifically, in the embodiment of the present application, after determining the number of data block columns with data inconsistency in the encoding strip, if it is judged that the number of data block columns is greater than the number of check block columns in the encoding strip, it indicates that the number of data blocks with data inconsistency is too large, and the system data block data inconsistency condition exceeds the repair capability. At this time, a warning information is popped up for the user to view, informing the user that the encoding strip repair fails.

[0111] Further, in the embodiment of the present application, the system will continue to scan the next encoding strip of the encoding strip, and perform data recovery or throw a warning on the next encoding strip in the same way as described above.

[0112] The method of the embodiment of the present application can determine that the system data block data inconsistency condition exceeds the repair capability by judging that the number of data block columns is greater than the number of check block columns in the encoding strip, and timely pop up a warning information to the foreground to remind the user of the recovery condition of the memory data, so as to facilitate the user to respond in time and improve the user experience.

[0113] Figure 2Figure 2 is a flowchart of a method for guaranteeing crash consistency of persistent memory data according to an embodiment of the present application. Figure 2 As shown in the figure, in an embodiment of the present application, when the system unexpectedly crashes and restarts, the system enters a no-log crash recovery process, which includes the following steps:

[0114] First, starting from the logical head of the configured full address space, scan in units of an encoding stripe (a series of data blocks and corresponding check blocks), read a complete encoding stripe into a memory buffer.

[0115] Then, determine whether the original checksum matches the data block: traverse all the original data blocks in the encoding stripe, and determine whether the original checksum of the internal persistent record matches the checksum recalculated from the original data block content. If not, it means that there is a crash inconsistency on the original data block, otherwise, it means that there is no crash inconsistency on the original data block.

[0116] Next, after scanning all the original data blocks in the encoding stripe, if there is no crash inconsistency, i.e., the number of data blocks with inconsistency is zero, then enter the following re-encoding process; if the number of data blocks with crash inconsistency is greater than the number of check blocks, it means that an error beyond the repair capability has occurred, an alert is thrown, and it is determined whether all encoding stripes have been scanned. If not, the next encoding stripe is scanned in the above manner; if the number of data blocks with crash inconsistency is greater than zero and not greater than the number of check blocks, it means that the data block has a crash inconsistency and is within the system repair capability, then enter the following repair process.

[0117] Next, perform the re-encoding process: use the correct data block data to re-encode to obtain new check blocks, compare the read original check blocks, and determine whether the new check blocks match. If not, it means that there is a data inconsistency check block, and the correct new check block is written back. After the operation is completed, it is determined whether all encoding stripes have been scanned. If not, the next encoding stripe is scanned in the above manner.

[0118] Then, perform the repair process: calculate the new data blocks through multi-path decoding (M check blocks correspond to M decoding calculation paths), and check the checksums inside the new data blocks obtained after decoding with the original checksums. If any of them passes the check, the recovery is successful, the new data block is written back, and then the re-encoding process is entered. If none of them passes the check, it means that the total number of crash inconsistencies on the data block and the check block exceeds the repair capability, an alert message is thrown, and it is determined whether all encoding stripes have been scanned. If not, the next encoding stripe is scanned in the above manner.

[0119] In the embodiments of the present application, after the log operation for ensuring the crash consistency of the memory data is removed, in order to avoid the system reliability being affected by the lack of logs due to the removal of the log addition and deletion operations in the updating process, a log-free crash consistency updating process is further provided.

[0120] Based on the content of the above embodiments, as an optional embodiment, after the log operation for ensuring the crash consistency of the memory data is removed, the method further comprises:

[0121] After receiving the data block updating request of the encoding strip, the data block updating request and the updating data of at least one target data block are cached to the memory buffer; the target data block is a data block in the encoding strip required to be updated by the data block updating request;

[0122] In response to the data block updating request, in the memory buffer, the data block difference values corresponding to each target data block are calculated according to the target byte cache line granularity, and the corresponding check block difference values are determined based on each data block difference value;

[0123] The first operation of asynchronously writing the updating data of the data block is performed, so as to asynchronously write the updating data of each target data block to the corresponding memory address in the order of the data block address and according to the target byte cache line granularity;

[0124] The first memory barrier instruction is called to persist the first operation.

[0125] Specifically, the data block updating request described in the embodiments of the present application refers to a data updating request initiated by a user in the foreground, or a data updating request initiated by a periodic data updating instruction set internally by the system.

[0126] The target byte cache line granularity described in the embodiments of the present application refers to the smallest unit of data transmission between the cache and the memory in the data updating process. It can be determined according to the cache line granularity of the computer architecture. For example, the cache line granularity of the X86 computer architecture is 64 bytes, and the target byte cache line granularity can be 64 bytes.

[0127] The first operation described in the embodiments of the present application refers to the operation of asynchronously writing the updating data of the data block.

[0128] The first memory barrier instruction described in the embodiments of the present application is used to persist the above-mentioned first operation. Through the first memory barrier instruction, the execution commands before and after it are not interchangeable, ensuring that the first operation of asynchronously writing the updating data of the data block is persistent, so as to order the persistent memory write requests executed before and after it, and avoid potential multi-crash inconsistency problems caused by concurrent updating.

[0129] Figure 3This is one of the schematic diagrams of the data update process in the method for ensuring consistency of persistent memory data in the event of a crash, as provided in the embodiments of this application. Figure 3 As shown in the embodiments of this application, upon receiving a data block update request for the encoded stripe, the objects to be updated on the mobile phone are stored in a memory buffer. This means that the data block update request and the update data of at least one target data block to be updated are cached in the memory buffer. Then, in response to the data block update request, the cache is traversed at a target byte cache line granularity, such as 64 bytes. The update data of each target data block is calculated in the memory buffer and compared with its corresponding original data to obtain the data block difference (delta data) for each target data block. Simultaneously, the corresponding check block difference (deltaparity) can be calculated using the erasure coding finite field calculation and the data block difference.

[0130] Furthermore, in the embodiments of this application, a first operation of asynchronously writing updated data to data blocks is performed, so as to asynchronously write the updated data of each target data block to the corresponding memory address in order of data block address and at the target byte cache line granularity. That is, according to the asynchronous persistent write mechanism, based on the data block address order, the system can update the updated data of each target data block to the corresponding memory address one by one in 64-byte cache line granularity until the update processing of each target data block is completed.

[0131] Furthermore, in the embodiments of this application, a memory barrier instruction request is initiated, a first memory barrier instruction is invoked, and the operation of asynchronously writing data blocks to updated data is persisted to ensure the persistence of all new data block asynchronous write requests.

[0132] The method in this application introduces fine-grained memory barrier technology. An additional memory barrier instruction request is initiated between consecutive update instructions of data blocks and parity blocks at the same memory address offset. This allows the memory barrier instruction to order the persistent memory write requests executed before and after it, thereby avoiding potential multi-crash inconsistency problems caused by concurrent updates. At the same time, by parallelizing encoding calculation and asynchronous persistent write requests at the smallest byte cache line granularity, each cache line can be flushed to persistent memory as early as possible without waiting for the encoding of the entire data block to be completed, which can further improve the instruction parallelism in the update process.

[0133] Based on the above embodiments, as an optional embodiment, after invoking the first memory barrier instruction to persist the first operation, the method further includes:

[0134] Based on the difference between each check block, determine the corresponding new check block according to the target byte cache line granularity;

[0135] performing a second operation of a check block asynchronous write update data to asynchronously write data of each new check block to a corresponding memory address in a target byte cache line granularity;

[0136] invoking a second memory barrier instruction to persist the second operation.

[0137] Specifically, the second operation described in the embodiments of the present application refers to an operation of check block asynchronous write update data.

[0138] The second memory barrier instruction described in the embodiments of the present application is used to persist the above-mentioned second operation. Similarly, through the second memory barrier instruction, the execution command in front of it and the execution command behind it can be made non-interchangeable, ensuring the operation of check block asynchronous write update data to be persistent, and avoiding potential multi-crash inconsistency problems caused by concurrent updates.

[0139] Continuing to refer to Figure 3 In the embodiments of the present application, in the process of check block data update, each new check block is quickly calculated by using each check block difference value and corresponding original check block data in a target byte cache line granularity, and then a new check block asynchronous persistent write operation is performed, that is, a second operation of check block asynchronous write update data is performed to asynchronously write data of each new check block to a corresponding memory address in a target byte cache line granularity. That is, according to the asynchronous persistent write mechanism, based on the check block address sequence, the system can use the check block difference value in the corresponding scale to determine the corresponding new check block data in a 64-byte cache line granularity, and then update the new check block data to the corresponding memory address, and so on, sequentially update the update data of each calculated check block to the corresponding memory address, until all check block data is updated.

[0140] Further, in the embodiments of the present application, a memory barrier instruction request is initiated, a second memory barrier instruction is invoked to persist the operation of check block asynchronous write update data, so as to ensure the persistence of all new check block asynchronous write requests.

[0141] The method of the embodiments of the present application ensures that the data blocks and check blocks with the same offset involved in each update process do not appear crash inconsistency problem at the same time, and improves the reliability and efficiency of data update of the persistent memory system.

[0142] Figure 4 is a second flowchart of data update in a method for ensuring crash consistency of persistent memory data provided by the embodiments of the present application, as shown in Figure 4 Figure 4 ​(a) is a schematic diagram of a traditional log-based memory data update process, and (b) is a schematic diagram of a logless crash consistent data update process in embodiments of the present application. As shown in (a) and (b), the data update process in the present application is similar to the traditional log-based memory data update process, but the present application is different from the traditional log-based memory data update process in that the present application does not use a log to record the data update process. Figure 4 As can be seen in the foregoing, taking a single check block as an example, in the original traditional technology, the encoding calculation is uniformly performed first, and then the calculation result is written back. The pipeline encoding and persistent writing technology of the present application advances a part of the calculation operations originally performed after the memory barrier to the window before the memory barrier, so as to alleviate the blocking effect of the memory barrier and hide the update delay. Moreover, the encoding calculation and the asynchronous persistent writing request are parallelized at a granularity of a minimum of 64 bytes, so that each cache line can be flushed to the persistent memory as early as possible without waiting for the encoding of the entire data block to be completed, thereby greatly improving the update efficiency of the memory data.

[0143] In some embodiments, considering that the time-consuming operation of scanning the global space is required in the crash recovery process of the present application, which will affect the event of the foreground service recovery, the system instant recovery function is further provided in the state of temporary reliability guarantee degradation through the read-time re-verification strategy, and the specific process is as follows:

[0144] Step S1, a plurality of independent background threads are first started to perform the scan repair in parallel between the encoding strips.

[0145] Step S2, when there is a read request access in the foreground, the system reads out the corresponding data block and checksum, and performs the verification matching in advance to prevent potential crash inconsistency. If the matching is successful, the data is returned; if the matching is not successful, it indicates that the current data block is erroneous, and the background repair is waited for, and an error is returned.

[0146] Step S3, when there is a write request access in the foreground, the normal logless crash consistency update process described above is performed.

[0147] Step S4, after the global scan of the logless crash recovery process is completed, the reliability guarantee level of the system is restored to the state before the crash.

[0148] It should be noted that in the recovery process of the traditional technology, in addition to recording the contents of the update region, the log also records the positions of the regions, so as to completely detect and recover the inconsistency. However, the logless method of the present application does not record this information, and thus cannot perform normal detection and recovery. Therefore, in the crash recovery process of the present application, an optimistic speculative recovery method described above is used to effectively scan the entire persistent memory space and restore to a data consistent state. This method reuses the checksum and the check redundancy decoding to achieve comprehensive detection and recovery of potential inconsistency. In addition, the speculative recovery can also achieve instant recovery service through temporary reliability degradation.

[0149] In the update process, because the data area and the check area can be updated simultaneously in one update process, multiple areas can be inconsistent. Therefore, in the update process, the application uses a fine-grained memory barrier instruction to impose a more stringent write order, thereby preventing concurrent updates to multiple areas in a single update. By separating the concurrent update of the data block and the check block at the same offset value, the repair capability of the check is ensured to meet the potential inconsistency range. Secondly, in order to solve the problem of concurrent update caused by the introduction of additional memory barrier in the update, the application proposes a pipeline encoding and persistent writing technology, which reduces the blocking effect of the memory barrier and hides the update delay by flushing and encoding calculation in advance, so that the CPU can execute the foreground calculation while the asynchronous persistent writing request in the background execution is submitted to the persistent memory in parallel, thereby improving the parallelism of instruction execution.

[0150] In addition, the application further evaluates the effect by executing real workloads on the transactional key-value storage engine built on the system. The test results on a multi-channel server platform deployed with real persistent memory devices show that the application can reduce the traffic to the persistent memory by 33-43%, significantly improve the latency and throughput of the system, and increase the update transaction throughput by up to 3.4 times.

[0151] The device for ensuring the crash consistency of persistent memory data provided by the application is described below, and the device for ensuring the crash consistency of persistent memory data described below can be referred to each other.

[0152] Figure 5 is a structural schematic diagram of the device for ensuring the crash consistency of persistent memory data provided by the embodiment of the application, as Figure 5 shown, comprising:

[0153] The log removal module 510 is configured to remove the log operation for ensuring the crash consistency of memory data in the system.

[0154] The data recovery module 520 is configured to execute a log-free crash recovery process after the system crashes and restarts.

[0155] The log-free crash recovery process is configured to sequentially scan each encoding strip stored in the memory space, and perform data recovery using the original check data of the encoding strip in the case of data crash inconsistency in the encoding strip.

[0156] It should be understood that the above device is used to execute the method in the above embodiment, and the corresponding program modules in the device have similar implementation principles and technical effects to the description in the above method. The working process of the device can refer to the corresponding process in the above method, which will not be described here.

[0157] The device for ensuring the crash consistency of the persistent memory data of the embodiment of the application introduces a no-log crash recovery process, removes the log operation for ensuring the crash consistency of the memory data in the system, and executes the no-log crash recovery process after the system crashes and restarts. The process will sequentially scan each encoding strip stored in the memory space, and when the data crash inconsistency occurs in the encoding strip, the original check data of the stored encoding strip is used for data recovery. Thus, the system completely uses the check data to replace the log, reuses the existing multiple reliable redundant data in the system to ensure the data crash consistency, can effectively enhance the write performance of the persistent memory system, significantly improves the delay and throughput of the system, and improves the overall performance of the system.

[0158] Based on the method in the above embodiment, an electronic device is provided in the embodiment of the application, as shown in the figure, the electronic device can include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, the memory 630 complete mutual communication through the communication bus 640. The processor 610 can call the logic instruction in the memory 630 to execute the method in the above embodiment. Figure 6

[0159] In addition, the logic instruction in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0160] Based on the method in the above embodiment, a computer readable storage medium is provided in the embodiment of the application, and the computer readable storage medium stores a computer program. When the computer program runs on the processor, the processor executes the method in the above embodiment.

[0161] Based on the method in the above embodiment, a computer program product is provided in the embodiment of the application, and when the computer program product runs on the processor, the processor executes the method in the above embodiment.

[0162] ​It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0163] The method steps in the embodiments of the present application can be implemented in the form of hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0164] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0165] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0166] Those skilled in the art easily understand that the above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of guaranteeing persistent memory data crash consistency, the method comprising: The method comprises: removing log operations in a system for ensuring memory data crash consistency; after a crash restart of the system, performing a log-free crash recovery process; the log-free crash recovery process is used to sequentially scan each encoding strip stored in the memory space, and in the case of data crash inconsistency in the encoding strip, data recovery is performed using the original check data of the encoding strip stored in the encoding strip; wherein the original check data comprises original check blocks and original checksums of original data blocks; the specific steps of scanning the encoding strip stored in the memory space comprise: reading the encoding strip into a memory buffer; traversing all original data blocks in the encoding strip in the memory buffer to calculate the new checksums of each original data block; matching each new checksum with the corresponding original checksum, and determining the number of data block columns with inconsistent data in the encoding strip based on the matching result; in the case where the number of data block columns is greater than zero and not greater than the number of check block columns in the encoding strip, decoding each original check block in the check block column to obtain a plurality of new data blocks; based on the plurality of new data blocks and the original checksums of each original data block, performing data recovery on the encoding strip; wherein the data recovery on the encoding strip based on the plurality of new data blocks and the original checksums of each original data block comprises: determining the checksums of each new data block; matching the checksums of each new data block with each original checksum; in the case where there is a matching successful new data block, replacing the corresponding original data block in the encoding strip with the matching successful new data block; encoding the matching successful new data block to obtain a corresponding new check block; in the case where the new check block does not match the original check block, replacing the corresponding original check block in the encoding strip with the new check block to complete the data recovery of the encoding strip.

2. The method of guaranteeing persistent memory data crash consistency according to claim 1, wherein, after the matching of each new checksum with the corresponding original checksum and the determination of the number of data block columns with inconsistent data in the encoding strip based on the matching result, the method further comprises: in the case where the number of data block columns with inconsistent data in the encoding strip is zero, determining a new check block based on all original data blocks in the encoding strip; in the case where the new check block does not match the original check block, replacing the corresponding original check block in the encoding strip with the new check block to complete the data recovery of the encoding strip.

3. The method of guaranteeing persistent memory data crash consistency according to claim 1, wherein, after the matching of each new checksum with the corresponding original checksum and the determination of the number of data block columns with inconsistent data in the encoding strip based on the matching result, the method further comprises: in the case where the number of data block columns is greater than the number of check block columns in the encoding strip, popping up a warning message; scanning the next encoding strip of the encoding strip to perform data recovery on the next encoding strip.

4. The method of guaranteeing persistent memory data crash consistency according to any one of claims 1-3, characterized in that, After the log operation for ensuring memory data crash consistency is removed, the method further comprises: After receiving the data block update request of the encoded stripe, the data block update request and update data of at least one target data block are cached to a memory buffer; the target data block is a data block in the encoded stripe required to be updated by the data block update request; In response to the data block update request, in the memory buffer, a data block difference value corresponding to each target data block is calculated according to a target byte cache line granularity, and a corresponding check block difference value is determined based on each data block difference value; A first operation of asynchronously writing update data of a data block is performed to asynchronously write update data of each target data block to a corresponding memory address in a data block address order and according to the target byte cache line granularity; A first memory barrier instruction is called to persist the first operation.

5. The method of guaranteeing persistent memory data crash consistency according to claim 4, wherein, After the first memory barrier instruction is called to persist the first operation, the method further comprises: According to the target byte cache line granularity, a new check block corresponding to each check block difference value is determined based on each check block difference value; A second operation of asynchronously writing update data of a check block is performed to asynchronously write data of each new check block to a corresponding memory address according to the target byte cache line granularity; A second memory barrier instruction is called to persist the second operation.

6. An apparatus to guarantee persistent memory data crash consistency, the apparatus comprising: Comprise: A log removal module for removing a log operation for ensuring memory data crash consistency in a system; A data recovery module for performing a log-free crash recovery process after the system crashes and restarts; The log-free crash recovery process is used to sequentially scan each encoded stripe stored in a memory space, and in the case of data crash inconsistency in the encoded stripe, data recovery is performed using stored original check data of the encoded stripe; The original check data includes original check blocks and original checksums of original data blocks; the specific steps of scanning the encoded stripe stored in the memory space comprise: The encoded stripe is read into a memory buffer; In the memory buffer, all original data blocks in the encoded stripe are traversed to calculate new checksums of each original data block; Each new checksum is matched with a corresponding original checksum, and based on the matching result, the number of data block columns with inconsistent data in the encoded stripe is determined; In the case where the number of data block columns is greater than zero and not greater than the number of check block columns in the encoded stripe, each original check block in the check block column is decoded to obtain a plurality of new data blocks; Based on the plurality of new data blocks and the original checksums of each original data block, data recovery is performed on the encoded stripe; The data recovery based on the plurality of new data blocks and the original checksums of each original data block comprises: The checksum of each new data block is determined; The checksum of each new data block is matched with each original checksum; In the case that there is a new data block of matching success, replacing the corresponding original data block in the encoding strip with the new data block of matching success; encoding the new data block of matching success to obtain a corresponding new check block; In the case that the new check block does not match the original check block, replacing the corresponding original check block in the encoding strip with the new check block to complete the data recovery of the encoding strip.

7. An electronic device, comprising: Comprise: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method as claimed in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: When the computer program runs on the processor, the processor is caused to execute the method as claimed in any one of claims 1-5.

Images