Cache optimization-based write command processing method, terminal and storage medium
By splitting the writing commands and generating verification data, the problem of excessive cache resource consumption in the RAID architecture is solved, and the response efficiency and user experience of RAID cards in multi-user concurrent IO scenarios are improved.
Patent Information
- Application Number
- CN202411301395.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-27
AI Technical Summary
In the RAID architecture, with the increase of host IO requests and the increase of disk count, cache resource consumption has increased significantly, resulting in poor IO response delay and user experience, especially in the multi-user concurrent IO operation scenario, which affects the efficiency of data processing and reliability.
By monitoring the cache margin of the disk array, analyze the required cache of write commands. If the cache margin exceeds the cache margin, split the write command into a subcommand, gradually process and generate verification data to reduce cache consumption, and ensure that each subcommand does not overflow Processing under cache situations, thereby improving system efficiency and stability.
Effectively reduce the consumption of RAID card cache, improve the ability of RAID card to handle multiple host IOs, ensure the reliability and flexibility of cache resources, and improve the overall operating efficiency and user experience of the system.
Smart Images
Figure CN119200987A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of disk arrays, and in particular relates to a write command processing method based on cache optimization, a terminal and a storage medium. Background Art
[0002] In the RAID architecture, checksum data plays a core role in ensuring data security and reliability. It not only serves as the key to data recovery in the event of a disk failure or system error, but also ensures data integrity and continuous availability. The checksum data generation process is precise and efficient. It accompanies the execution of the host write IO operation and performs complex calculations immediately after the data is cached by the RAID card, thereby generating critical checksum information.
[0003] However, as host IO requests become increasingly large and the number of disks in the RAID array increases, the consumption of cache resources in this process has also increased significantly. Cache, as a valuable resource on the RAID card, has a limited capacity, which directly limits its flexibility in dealing with large-scale data processing. In order to balance performance and cost, the design and manufacture of RAID cards must carefully consider the configuration of cache capacity to avoid unnecessary waste of resources and cost increases.
[0004] Under the NVMe protocol framework, the introduction of the MaxPayloadSize parameter provides a new idea for optimizing data transmission efficiency. This parameter defines the maximum amount of data that a PCIe device can carry in a single transmission. If it is set to 256KB, any data transmission exceeding this threshold will be intelligently split into multiple 256KB data packets for transmission. Although this mechanism helps manage the complexity and efficiency of data transmission, it may also impose certain restrictions on the host's ability to handle large IOs, especially in scenarios with multiple users performing concurrent IO operations, which may cause IO response delays and thus affect the smoothness of the user experience. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a write command processing method, a terminal and a storage medium based on cache optimization to solve the above-mentioned technical problems.
[0006] In a first aspect, the present invention provides a write command processing method based on cache optimization, comprising: Monitor the cache margin of the disk array and analyze the required cache for write commands; Confirming that the required cache exceeds the cache margin, splitting the sub-command whose required cache does not exceed the cache margin from the write command; Moving the host data corresponding to the subcommand to the cache of the disk array, generating the verification data corresponding to the subcommand using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily storing the verification data in the cache; The host data corresponding to the subcommand is written to the corresponding disk, and the write command is executed by step-by-step splitting based on the updated cache margin until the last subcommand of the write command is written to the disk, and the verification data corresponding to the last subcommand is written to the corresponding disk.
[0007] In an optional implementation, monitoring the cache margin of the disk array and analyzing the required cache of the write command include: Use real-time monitoring tools to monitor the cache margin of the disk array; Read the write command to be executed from the waiting queue of the host, and obtain the required number of logical blocks of the write command and the size of the check disk; The required cache of the write command is obtained by calculating the sum of twice the required number of logical blocks and the size of the check disk.
[0008] In an optional embodiment, the method further comprises: Determine whether the required cache exceeds the cache margin: If not, it is determined that the write command does not need to be split, and the write command is moved to the cache of the disk array; If so, it is determined that the write command needs to be split.
[0009] In an optional implementation manner, if it is confirmed that the required cache exceeds the cache margin, a sub-command in which the required cache does not exceed the cache margin is split from the write command, including: The theoretical required number of logic blocks is calculated based on a calculation formula for the required cache and the cache margin, wherein the calculation formula includes: 2×nlb+chunk<=c+k; Among them, nlb represents the number of logical blocks of host data corresponding to the subcommand, chunk represents the size of the check block, c represents the cache margin, and k represents the cache margin.
[0010] In an optional embodiment, the method further comprises: Collecting historical processing data, wherein the historical processing data includes the required number of logical blocks and the size of the check block of the write command that has been processed; Generate a corresponding relationship between the number of required logic blocks and the size of the check blocks based on the historical processing data; The size of the check block is calculated based on the corresponding relationship and the known number of required logic blocks.
[0011] In an optional implementation, the host data corresponding to the subcommand is moved to a cache of a disk array, verification data corresponding to the subcommand is generated using original data on the disk corresponding to the subcommand, original verification data, and the host data, and the verification data is temporarily stored in the cache, including: Applying for a first cache area, a second cache area, and a third cache area from the cache of the disk array; Temporarily storing the host data corresponding to the subcommand in the first cache area; Read the original data and original check data at the disk location corresponding to the subcommand; Temporarily storing the original data in a second cache area; Temporarily storing the original verification data in a third cache area; Performing an XOR calculation on the host data and the original data to obtain intermediate verification data corresponding to the original data; Temporarily storing the intermediate verification data in a second cache area; Performing an XOR calculation on the intermediate verification data and the original verification data to obtain new verification data; The new verification data is updated to the third cache area in a replacement manner.
[0012] In an optional implementation, writing the host data corresponding to the subcommand to the corresponding disk, and executing the write command by stepwise splitting based on the updated cache margin until the last subcommand of the write command is written to the disk, and writing the verification data corresponding to the last subcommand to the corresponding disk, includes: Writing the host data corresponding to the subcommand into the corresponding disk location, and recording the completion status of the subcommand in the task queue of the disk array; Based on the completion status of the subcommand, obtaining the cache requirements of the remaining write commands and the current cache margin of the disk array; Determine whether the cache requirement of the remaining write commands exceeds the current cache margin: If yes, split the new subcommand from the remaining write command, and perform verification and disk writing on the new subcommand; If not, the remaining write commands are moved to the cache of the disk array, and the remaining write commands are checked and written to disk.
[0013] In an optional implementation, the method for verifying and writing a new subcommand to a disk includes: Update the new host data corresponding to the new subcommand to the first cache area; Performing an XOR calculation on the new host data and the original data to obtain new intermediate verification data corresponding to the original data; Updating the new intermediate verification data to the second cache area; Performing an XOR calculation on the new intermediate verification data and the verification data stored in the third cache area to obtain the latest verification data; Updating the latest verification data to the third cache area in a replacement manner; Writing new host data corresponding to the new subcommand into a corresponding disk location, and recording the completion status of the new subcommand in a task queue of the disk array; confirming that there is no unprocessed host data in the write command, determining that the write command processing is complete, and writing the verification data currently stored in the third cache area to the disk; The host's to-be-executed queue feeds back prompt information indicating that the write command has been executed.
[0014] In a second aspect, the present invention provides a write command processing system based on cache optimization, comprising: Parameter monitoring module, used to monitor the cache margin of the disk array and analyze the required cache of write commands; A splitting execution module, used for confirming that the required cache exceeds the cache margin, and then splitting the sub-commands whose required cache does not exceed the cache margin from the write command; A verification processing module, used to move the host data corresponding to the subcommand to the cache of the disk array, generate verification data corresponding to the subcommand using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily store the verification data in the cache; A step-by-step splitting module is used to write the host data corresponding to the sub-command to the corresponding disk, and execute the write command by step-by-step splitting based on the updated cache margin until the last sub-command of the write command is completed and written to the corresponding disk, and the verification data corresponding to the last sub-command is written to the corresponding disk.
[0015] In a third aspect, a terminal is provided, including: processor, memory, wherein: The memory is used to store computer programs. The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned terminal method.
[0016] According to a fourth aspect, a computer storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the methods described in the above aspects.
[0017] The beneficial effect of the present invention is that the write command processing method, terminal and storage medium based on cache optimization provided by the present invention can effectively reduce the consumption of RAID card cache by splitting the original large host write IO into multiple small sub-IOs and gradually calculating each sub-IO. Each split sub-IO itself consumes less cache, so that under the same cache quantity, the RAID card can process the IO of multiple hosts at the same time, and split and process the multiple host IOs respectively.
[0018] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.
[0021] Figure 2 It is a schematic diagram of the processing process of a write command of a method according to an embodiment of the present invention.
[0022] Figure 3 The figure is a schematic flow chart of write command analysis of a method according to an embodiment of the present invention.
[0023] Figure 4 It is a schematic flow chart of subcommand processing of a method according to an embodiment of the present invention.
[0024] Figure 5 is a schematic block diagram of a system according to an embodiment of the present invention.
[0025] Figure 6 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0028] The key terms appearing in the present invention are explained below.
[0029] RAID, Redundant Arrays of Independent Disks. A disk array is a huge disk group composed of many independent disks. The data is divided into many sections and stored on each disk. It can greatly increase the storage capacity, improve the system input and output request processing capabilities, and improve data reliability through distributed data storage technology, parallel access means and information redundancy technology.
[0030] IO, input and output, I is the abbreviation of Input and O is the abbreviation of Output.
[0031] RCW, Reconstruct Write, is usually called uppercase. It is a method of calculating new checksum data by reading data at other locations other than the IO location written to the disk array this time and the new data of this IO.
[0032] RMW, Read Modify Write, is usually called lowercase. It is a method of calculating new check data by reading the old data, old check data, and new data of the IO position written to the disk array this time.
[0033] FSW, Full Stripe Write, is often referred to as full write. The amount of data written to the disk array is exactly equal to the stripe size, so that the new verification data can be calculated without the need for additional reading of old data.
[0034] Stripe is a collection of position-related strip headers on different partitions of the array, and is the unit for organizing blocks on different partitions.
[0035] Stripe Head: A uniformly sized lateral unit that further divides a stripe horizontally.
[0036] chunk, chunk. Data blocks of equal size and adjacent addresses are grouped vertically within a stripe.
[0037] PCIe, Peripheral Component Interconnect Express, is a high-speed serial computer expansion bus standard.
[0038] MaxPayloadSize, maximum payload size. Parameters of the PCIe layer, which defines the maximum payload size that the PCIe interface can transmit. For NVMe I / O operations, if the amount of data exceeds MaxPayloadSize, it needs to be transmitted in multiple TLPs (Transaction Layer Packets).
[0039] slba, start logical block address, is a logical block address specified in NVMe commands to indicate the start of a data transfer operation (such as read or write). Each logical block usually contains a certain amount of data, such as 512 bytes or 4K bytes.
[0040] nlb, number of Logical Blocks, is a parameter used to specify the number of logical blocks to be operated in a single data transfer command.
[0041] The write command processing method based on cache optimization provided by the embodiment of the present invention is executed by a computer device. Accordingly, the write command processing system based on cache optimization runs in the computer device.
[0042] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a write command processing system based on cache optimization. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.
[0043] like Figure 1 As shown, the method includes: Step 110, monitoring the cache margin of the disk array and analyzing the required cache of the write command; Step 120, confirming that the required cache exceeds the cache margin, splitting the write command into sub-commands whose required cache does not exceed the cache margin; Step 130, moving the host data corresponding to the subcommand to the cache of the disk array, generating the verification data corresponding to the subcommand using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily storing the verification data in the cache; Step 140, write the host data corresponding to the subcommand to the corresponding disk, and execute the write command step by step based on the updated cache margin until the last subcommand of the write command is written to the disk, and write the verification data corresponding to the last subcommand to the corresponding disk.
[0044] Please refer to Figure 2 In one embodiment of the present invention, a write command processing method based on cache optimization is provided, which specifically includes the following steps: Step 110: Monitor the cache margin of the disk array and analyze the required cache of the write command.
[0045] Monitor the cache status of the disk array in real time and accurately record the amount of remaining space in the current cache (cache margin). Then, conduct an in-depth analysis of the write command to be executed to evaluate the minimum cache space required to complete it, i.e., the required cache. This analysis process involves considering many aspects, such as the amount of data involved in the write command, the type of data, and the possible need to generate verification data.
[0046] Step 120: confirm that the required cache exceeds the cache margin, and then split the sub-commands whose required cache does not exceed the cache margin from the write command.
[0047] Once it is found that the required cache of a write command exceeds the remaining space of the current cache, the splitting mechanism is started. This mechanism intelligently splits the original write command into multiple smaller sub-commands, which form a command stream. In addition, the splitting process is carried out step by step rather than synchronously. The required cache of each sub-command does not exceed the current cache margin. This splitting process is designed to ensure that each sub-command can be processed without overflowing the cache, thereby improving the overall operating efficiency and stability of the system.
[0048] Step 130: Move the host data corresponding to the subcommand to the cache of the disk array, generate the verification data corresponding to the subcommand using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily store the verification data in the cache. For each of the split subcommands, the corresponding host data is moved from the host memory to the cache of the disk array. Then, the new checksum data is calculated using the data in the cache and the original data and checksum data on the disk. These newly generated checksum data are crucial to ensure the integrity and reliability of the data. They will be temporarily stored in the cache so that they can be written to the disk together with the host data later.
[0049] Step 140: Write the host data corresponding to the subcommand to the corresponding disk, and execute the write command step by step based on the updated cache margin until the last subcommand of the write command is written to the disk, and write the verification data corresponding to the last subcommand to the corresponding disk After confirming that all data required by the subcommand (including host data and verification data) are prepared and stored in the cache, write these data to the corresponding disk. After completing the write operation of each subcommand, update the cache margin information and check whether there are any remaining write commands to be processed. If so, the system will continue to split and execute the remaining write commands according to the new cache margin. This process will continue until the original write command is completely disassembled and executed, and the host data and verification data corresponding to the last subcommand are successfully written to the disk. At this point, the entire write operation is truly completed.
[0050] To facilitate understanding of the present invention, the following further describes the write command processing method based on cache optimization provided by the present invention based on the principle of the write command processing method based on cache optimization of the present invention and the process of processing the write command in the embodiment.
[0051] Specifically, the write command processing method based on cache optimization includes: S1. Host profile command management.
[0052] The host prepares to write an IO command; the host writes the host IO command to the host's I / O queue, and notifies the RAID card to read the command from the host's I / O queue.
[0053] S2, RAID reads and analyzes the write command, such as Figure 3 shown.
[0054] S201. Monitor the cache margin of the disk array using a real-time monitoring tool.
[0055] Implement monitoring tools such as iostat, dmesg in Linux, Performance Monitor in Windows, etc.
[0056] S202: Read the write command to be executed from the queue to be executed of the host, and obtain the required number of logical blocks of the write command and the size of the check disk.
[0057] The RAID card reads the host write command and analyzes the write command. The IO command of the write command host contains slba and nlb information. After obtaining the IO command, the RAID card calculates the required cache based on the slba and nlb information.
[0058] S203: Calculate the sum of twice the required number of logical blocks and the size of the check disk to obtain the required cache of the write command.
[0059] In this embodiment, the calculation and comparison units are both IO units, that is, the number of logic blocks.
[0060] The original write command uses the RMW verification method, and the required cache a = 2×nlb + p_nlb, where p_nlb is the size of the verification disk that needs to be updated when using RMW. The original write command uses RCW or FSW, and the required cache b is the full stripe size.
[0061] Assume that the RAID card cache margin is c, compare c with a and c. When a is greater than c and b is greater than c, it means that the RAID card cache margin is not enough to continue to execute the original write command, and the write command needs to be split into multiple sub-commands.
[0062] If c is greater than a, and c is greater than b, it means that the cache is sufficient and the write command can be processed in a conventional manner. For the original write command that is not split, the RAID card can use one of the write methods of RMW, RCW, and FSW to move all the host data to the RAID cache, calculate the verification data for the host data in the RAID card cache, and then write to the disk. RMW, RCW, and FSW are relatively mature, and the present invention will not repeat them.
[0063] S3. Execute write command splitting.
[0064] The theoretical required number of logic blocks is calculated based on a calculation formula for the required cache and the cache margin, wherein the calculation formula includes: 2×nlb+chunk<=c+k; Among them, nlb represents the number of logical blocks of host data corresponding to the subcommand, chunk represents the size of the check block, c represents the cache margin, and k represents the cache margin.
[0065] Collect historical processing data, the historical processing data including the required number of logical blocks and the size of the check block of the completed write command; generate a corresponding relationship between the required number of logical blocks and the size of the check block based on the historical processing data; calculate the size of the check block based on the corresponding relationship and the known required number of logical blocks. Based on this corresponding relationship, the above calculation formula can be solved to obtain the theoretical required number of logical blocks, that is, the theoretical nlb of the subcommand.
[0066] Specifically, the granularity of the original IO split is smaller than c, and the size nlb of the sub-IO after splitting satisfies (2× nlb + chunk size <= c).
[0067] For example, in Table 1, the host write command (1-8) with slba = 1, nlb = 8, the cache size required by RCW is 2×8 + 4 = 20, and the cache size required by RCW is 128. When the RAID card cache size is 10, the granularity of the original IO segmentation is 6, and the segmentation sub-commands are IO1 (1, 2, 3), IO2 (4, 5), IO3 (6, 7), and IO4 (8). IO1 satisfies (2×3+ 4) <= 10, IO2 satisfies (2×2 +4) <= 10, IO3 satisfies (2×2 +4) <= 10, and IO4 satisfies (2×1 +4) <= 10.
[0068] Table 1 Distribution of write commands with slab=1, nlb=8 on RAID5 with 32 disks and a chunk size of 4 logical blocks
[0069] S4. Process subcommands.
[0070] For the first split subcommand, the processing method is as follows: Apply for the first cache area, the second cache area and the third cache area from the cache of the disk array; temporarily store the host data corresponding to the subcommand in the first cache area; read the original data and the original verification data at the disk position corresponding to the subcommand; temporarily store the original data in the second cache area; temporarily store the original verification data in the third cache area; perform XOR calculation on the host data and the original data to obtain intermediate verification data corresponding to the original data; temporarily store the intermediate verification data in the second cache area; perform XOR calculation on the intermediate verification data and the original verification data to obtain new verification data; and update the new verification data to the third cache area in a replacement manner.
[0071] For the remaining write commands other than the first subcommand, the processing method is as follows: The host data corresponding to the subcommand is written into the corresponding disk location, and the completion status of the subcommand is recorded in the task queue of the disk array; based on the completion status of the subcommand, the cache requirements of the remaining write commands and the current cache margin of the disk array are obtained; it is determined whether the cache requirements of the remaining write commands exceed the current cache margin: if so, a new subcommand is split from the remaining write command, and the new subcommand is verified and written to the disk; if not, the remaining write command is moved to the cache of the disk array, and the remaining write command is verified and written to the disk.
[0072] Methods for verifying and writing new subcommands to disk include: The new host data corresponding to the new subcommand is updated to the first cache area; the new host data and the original data are XORed to obtain new intermediate verification data corresponding to the original data; the new intermediate verification data is updated to the second cache area; the new intermediate verification data and the verification data stored in the third cache area are XORed to obtain the latest verification data; the latest verification data is updated to the third cache area in a replacement manner; the new host data corresponding to the new subcommand is written to the corresponding disk location, and the completion status of the new subcommand is recorded in the task queue of the disk array; it is confirmed that there is no unprocessed host data for the write command, it is determined that the write command processing is completed, and the verification data currently stored in the third cache area is written to the disk; and the host's to-be-executed queue is fed back with a prompt message indicating that the write command execution is completed.
[0073] Specifically, for the original IO that is split into multiple sub-IOs, the RAID card moves the host data corresponding to the sub-IO to the internal cache, and at the same time, the RAID card reads the old data and old verification data on the disk corresponding to the sub-IO to the internal cache. Then, the host data, old data, and old verification data corresponding to the sub-IO are XORed to obtain new verification data that is temporarily stored in the internal cache, and the host data corresponding to the sub-IO is written to the disk. The RAID card repeats the above-mentioned process of moving the host data corresponding to the sub-IO and reading the old data on the disk for other sub-IOs, and recalculates the new verification data using the host data corresponding to the sub-IO on the RAID card, the old data on the disk, and the verification data updated in the cache in the previous step, and gradually completes the writing of the host data corresponding to each sub-IO to the disk, and the continuous updating of the verification data in the cache. Finally, the final updated verification data is written to the disk.
[0074] The following is an example of a specific subcommand processing. Figure 4 As shown: From a logical perspective, ΔDx refers to the update of the checksum caused by changes in certain data under the storage structure of the completed RAID group. In order to save computing costs and reduce update time, the algorithm is used to update ΔDx and calculate the checksum update value affected by the updated data and then update the checksum value. The calculation of ΔDx is completed by XORing the new data Dx' of the host and the old data Dx in the corresponding disk: As shown in the figure above, the entire data migration calculation process is as follows: 1. Dx ' is the data corresponding to the sub-command, and the RAID card is placed in cache A after migration.
[0075] 2. Dx is the original data in the disk location corresponding to the subcommand, which is taken out and placed in cache B. P is the original checksum data in the checksum disk corresponding to the subcommand, which is taken out and placed in cache C.
[0076] 3. Perform XOR calculation on Dx' and Dx to obtain the intermediate verification data ΔDx corresponding to Dx.
[0077] 4. Write ΔDx into cache B.
[0078] 5. Perform XOR calculation on ΔDx and the original verification data in the cache C to obtain the new verification data P' corresponding to Dx.
[0079] 6. Write P' into cache C and clear the original verification data P in cache C.
[0080] 7. Write the host data Dx' of the subcommand to disk.
[0081] The processing methods of other split subcommands refer to the processing method of the first subcommand and make some changes, including: changing the original verification data in step 5 to the verification data from the previous subcommand written into the cache C. Finally, the last updated verification data P' is written to the disk.
[0082] The XOR operation can only calculate one ΔDx value at a time. If multiple ΔDx values need to be calculated (i.e., multiple sub-IOs are processed step by step), the XOR operation calculates one ΔDx at a time, and then updates the verification data involved by its ΔDx, and updates P' in sequence.
[0083] In some embodiments, the cache-optimized write command processing system may include multiple functional modules composed of computer program segments. The computer programs of the various program segments in the cache-optimized write command processing system may be stored in a memory of a computer device and executed by at least one processor to execute (see Figure 1 Description) Function based on cache-optimized write command processing.
[0084] In this embodiment, the cache-optimized write command processing system can be divided into multiple functional modules according to the functions it performs, such as Figure 5 As shown. The functional modules of the system 500 may include: a parameter monitoring module 510, a splitting execution module 520, a verification processing module 530 and a step-by-step splitting module 540. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.
[0085] Parameter monitoring module, used to monitor the cache margin of the disk array and analyze the required cache of write commands; A splitting execution module, used for confirming that the required cache exceeds the cache margin, and then splitting the sub-commands whose required cache does not exceed the cache margin from the write command; A verification processing module, used to move the host data corresponding to the subcommand to the cache of the disk array, generate verification data corresponding to the subcommand by using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily store the verification data in the cache; A step-by-step splitting module is used to write the host data corresponding to the sub-command to the corresponding disk, and execute the write command by step-by-step splitting based on the updated cache margin until the last sub-command of the write command is completed and written to the corresponding disk, and the verification data corresponding to the last sub-command is written to the corresponding disk.
[0086] Optionally, as an embodiment of the present invention, monitoring the cache margin of the disk array and analyzing the required cache of the write command include: Use real-time monitoring tools to monitor the cache margin of the disk array; Read the write command to be executed from the waiting queue of the host, and obtain the required number of logical blocks of the write command and the size of the check disk; The required cache of the write command is obtained by calculating the sum of twice the required number of logical blocks and the size of the check disk.
[0087] Optionally, as an embodiment of the present invention, it further includes: Determine whether the required cache exceeds the cache margin: If not, it is determined that the write command does not need to be split, and the write command is moved to the cache of the disk array; If so, it is determined that the write command needs to be split.
[0088] Optionally, as an embodiment of the present invention, it is confirmed that the required cache exceeds the cache margin, and a sub-command whose required cache does not exceed the cache margin is split from the write command, including: The theoretical required number of logic blocks is calculated based on a calculation formula for the required cache and the cache margin, wherein the calculation formula includes: 2×nlb+chunk<=c+k; Among them, nlb represents the number of logical blocks of host data corresponding to the subcommand, chunk represents the size of the check block, c represents the cache margin, and k represents the cache margin.
[0089] Optionally, as an embodiment of the present invention, it further includes: Collecting historical processing data, wherein the historical processing data includes the required number of logical blocks and the size of the check block of the write command that has been processed; Generate a corresponding relationship between the number of required logic blocks and the size of the check blocks based on the historical processing data; The size of the check block is calculated based on the corresponding relationship and the known number of required logic blocks.
[0090] Optionally, as an embodiment of the present invention, the host data corresponding to the subcommand is moved to a cache of a disk array, the verification data corresponding to the subcommand is generated using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and the verification data is temporarily stored in the cache, including: Applying for a first cache area, a second cache area, and a third cache area from the cache of the disk array; Temporarily storing the host data corresponding to the subcommand in the first cache area; Read the original data and original check data at the disk location corresponding to the subcommand; Temporarily storing the original data in a second cache area; Temporarily storing the original verification data in a third cache area; Performing an XOR calculation on the host data and the original data to obtain intermediate verification data corresponding to the original data; Temporarily storing the intermediate verification data in a second cache area; Performing an XOR calculation on the intermediate verification data and the original verification data to obtain new verification data; The new verification data is updated to the third cache area in a replacement manner.
[0091] Optionally, as an embodiment of the present invention, writing the host data corresponding to the subcommand to the corresponding disk, and executing the write command by step-by-step splitting based on the updated cache margin until the last subcommand of the write command is written to the disk, and writing the verification data corresponding to the last subcommand to the corresponding disk includes: Writing the host data corresponding to the subcommand into the corresponding disk location, and recording the completion status of the subcommand in the task queue of the disk array; Based on the completion status of the subcommand, obtaining the cache requirements of the remaining write commands and the current cache margin of the disk array; Determine whether the cache requirement of the remaining write commands exceeds the current cache margin: If yes, split the new subcommand from the remaining write command, and perform verification and disk writing on the new subcommand; If not, the remaining write commands are moved to the cache of the disk array, and the remaining write commands are checked and written to disk.
[0092] Optionally, as an embodiment of the present invention, the method for verifying and writing a new subcommand to a disk includes: Update the new host data corresponding to the new subcommand to the first cache area; Performing an XOR calculation on the new host data and the original data to obtain new intermediate verification data corresponding to the original data; Updating the new intermediate verification data to the second cache area; Performing an XOR calculation on the new intermediate verification data and the verification data stored in the third cache area to obtain the latest verification data; Updating the latest verification data to the third cache area in a replacement manner; Writing new host data corresponding to the new subcommand into a corresponding disk location, and recording the completion status of the new subcommand in a task queue of the disk array; confirming that there is no unprocessed host data in the write command, determining that the write command processing is complete, and writing the verification data currently stored in the third cache area to the disk; The host's to-be-executed queue feeds back prompt information indicating that the write command has been executed.
[0093] Figure 6 A schematic diagram of the structure of a terminal 600 provided in an embodiment of the present invention. The terminal 600 can be used to execute the write command processing method based on cache optimization provided in an embodiment of the present invention.
[0094] The terminal 600 may include: a processor 610, a memory 620 and a communication unit 630. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0095] The memory 620 can be used to store the execution instructions of the processor 610. The memory 620 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 620 are executed by the processor 610, the terminal 600 can execute some or all of the steps in the following method embodiments.
[0096] The processor 610 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 620, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 610 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.
[0097] The communication unit 630 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.
[0098] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0099] Therefore, the present invention can effectively reduce the consumption of the RAID card cache by splitting the original large host write IO into multiple small sub-IOs and performing Δ step-by-step calculation on each sub-IO. Each split sub-IO itself consumes less cache, so under the same cache quantity, the RAID card can process the IO of multiple hosts at the same time, and split and process the multiple host IOs respectively. The technical effect that can be achieved by this embodiment can be seen in the description above, and will not be repeated here.
[0100] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention.
[0101] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.
[0102] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.
[0103] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0104] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0105] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.
Claims
1. A write command processing method based on cache optimization, characterized in that: include: Monitor the cache margin of the disk array and analyze the required cache for write commands; Confirming that the required cache exceeds the cache margin, splitting the sub-command whose required cache does not exceed the cache margin from the write command; Moving the host data corresponding to the subcommand to the cache of the disk array, generating the verification data corresponding to the subcommand using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily storing the verification data in the cache; The host data corresponding to the subcommand is written to the corresponding disk, and the write command is executed by step-by-step splitting based on the updated cache margin until the last subcommand of the write command is written to the disk, and the verification data corresponding to the last subcommand is written to the corresponding disk.
2. The method according to claim 1, characterized in that Monitor the cache margin of the disk array and analyze the required cache for write commands, including: Use real-time monitoring tools to monitor the cache margin of the disk array; Read the write command to be executed from the waiting queue of the host, and obtain the required number of logical blocks of the write command and the size of the check disk; The required cache of the write command is obtained by calculating the sum of twice the required number of logical blocks and the size of the check disk.
3. The method according to claim 2, characterized in that The method further comprises: Determine whether the required cache exceeds the cache margin: If not, it is determined that the write command does not need to be split, and the write command is moved to the cache of the disk array; If so, it is determined that the write command needs to be split.
4. The method according to claim 1, characterized in that: Confirming that the required cache exceeds the cache margin, splitting the sub-command whose required cache does not exceed the cache margin from the write command includes: The theoretical required number of logic blocks is calculated based on a calculation formula for the required cache and the cache margin, wherein the calculation formula includes: 2×nlb+chunk<=c+k; Among them, nlb represents the number of logical blocks of host data corresponding to the subcommand, chunk represents the size of the check block, c represents the cache margin, and k represents the cache margin.
5. The method according to claim 4, characterized in that The method further comprises: Collecting historical processing data, the historical processing data including the required number of logical blocks and the size of the check block of the completed write command; Generate a corresponding relationship between the number of required logic blocks and the size of the check blocks based on the historical processing data; The size of the check block is calculated based on the corresponding relationship and the known number of required logic blocks.
6. The method according to claim 1, characterized in that The method comprises: moving the host data corresponding to the subcommand to the cache of the disk array, generating the verification data corresponding to the subcommand by using the original data on the disk corresponding to the subcommand, the original verification data and the host data, and temporarily storing the verification data in the cache, including: Applying for a first cache area, a second cache area, and a third cache area from the cache of the disk array; Temporarily storing the host data corresponding to the subcommand in the first cache area; Read the original data and original check data at the disk location corresponding to the subcommand; Temporarily storing the original data in a second cache area; Temporarily storing the original verification data in a third cache area; Performing an XOR calculation on the host data and the original data to obtain intermediate verification data corresponding to the original data; Temporarily storing the intermediate verification data in a second cache area; Performing an XOR calculation on the intermediate verification data and the original verification data to obtain new verification data; The new verification data is updated to the third cache area in a replacement manner.
7. The method according to claim 6, characterized in that Writing the host data corresponding to the subcommand to the corresponding disk, and executing the write command by step-by-step splitting based on the updated cache margin until the last subcommand of the write command is written to the disk, and writing the verification data corresponding to the last subcommand to the corresponding disk, including: Writing the host data corresponding to the subcommand into the corresponding disk location, and recording the completion status of the subcommand in the task queue of the disk array; Based on the completion status of the subcommand, obtaining the cache requirements of the remaining write commands and the current cache margin of the disk array; Determine whether the cache requirement of the remaining write commands exceeds the current cache margin: If yes, split the new subcommand from the remaining write command, and perform verification and disk writing on the new subcommand; If not, the remaining write commands are moved to the cache of the disk array, and the remaining write commands are checked and written to disk.
8. The method according to claim 7, characterized in that Methods for verifying and writing new subcommands to disk include: Update the new host data corresponding to the new subcommand to the first cache area; Performing an XOR calculation on the new host data and the original data to obtain new intermediate verification data corresponding to the original data; Updating the new intermediate verification data to the second cache area; Performing an XOR calculation on the new intermediate verification data and the verification data stored in the third cache area to obtain the latest verification data; Updating the latest verification data to the third cache area in a replacement manner; Writing new host data corresponding to the new subcommand into a corresponding disk location, and recording the completion status of the new subcommand in a task queue of the disk array; confirming that there is no unprocessed host data in the write command, determining that the write command processing is complete, and writing the verification data currently stored in the third cache area to the disk; The host's to-be-executed queue feeds back prompt information indicating that the write command has been executed.
9. A terminal, characterized in that: include: A memory, used for storing a write command processing program based on cache optimization; A processor, configured to implement the steps of the write command processing method based on cache optimization as described in any one of claims 1 to 8 when executing the write command processing program based on cache optimization.
10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a cache-optimized write command processing program, which, when executed by a processor, implements the steps of the cache-optimized write command processing method according to any one of claims 1 to 8.