Hole arrangement method and device, electronic equipment and storage medium

Abstract

The invention belongs to the field of computer storage, and particularly discloses a hole arrangement method and device, electronic equipment and a storage medium, and the method comprises the following steps: for a strip group with holes, selecting a scheme which consumes less disk I / O in a first scheme and a second scheme, and arranging hole areas; according to the first scheme, when a new data request arrives, new data is written into a hole area; according to the second scheme, effective data in each logic block in the stripe group is sorted into a continuous data space in the stripe group to be stored, and when a new data request arrives, new data are sequentially written into a cavity area outside the continuous data space, by means of the method, disk I / O operation caused by data migration can be reduced, and the data migration efficiency is improved. Therefore, the system performance and reliability are improved.

Description

Technical Field

[0001] The present application belongs to the field of computer storage, and more specifically, to a void sorting method, device, electronic device and storage medium. Background Art

[0002] In modern data storage systems, due to the continuous deletion of data, a large amount of invalid data will appear in the storage medium, that is, the unfilled holes left by the deleted or moved data. These invalid data occupy storage space and reduce storage efficiency.

[0003] The hole cleaning method in the prior art generally reads out all the remaining valid data in all stripes and re-encodes them into new stripes. However, since this method involves the migration of a large amount of data and the update of verification data, it will cause a large amount of disk I / O, affecting the performance and reliability of the system. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a hole defragmentation method, device, electronic device and storage medium, aiming to solve the problem of low system performance and reliability caused by a large amount of disk I / O caused by the migration of a large amount of data and the update of verification data in the hole defragmentation method of the prior art.

[0005] To achieve the above purpose, the present application provides a void finishing method, comprising:

[0006] For stripe groups with holes, choose the solution that consumes less disk I / O between the first and second solutions to clean up the hole areas;

[0007] The first solution is to write new data into the empty area when a new data request arrives;

[0008] The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

[0009] According to the hole cleaning method provided by the present invention, when a new data request arrives, writing the new data into the hole area includes:

[0010] When a new data request arrives, the new data is directly used to overwrite the invalid data in the hole area;

[0011] Update the check block corresponding to the stripe covered by the new data.

[0012] According to the hole sorting method provided by the present invention, the valid data in each logical block in the stripe group is sorted into the continuous data space in the stripe group for storage, and when a new data request arrives, the new data is sequentially written into the hole area outside the continuous data space, including:

[0013] Count the number of valid data blocks in each storage node in each logical block respectively, and take the minimum value of all valid data blocks as the length of the continuous data space;

[0014] Based on the length of the continuous data space, a continuous area with the largest number of valid data blocks is selected as the continuous data space;

[0015] Migrating valid data blocks outside the continuous data space to locations where invalid data blocks are located in the continuous data space;

[0016] When a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space;

[0017] Update the check block corresponding to the stripe covered by the new data.

[0018] According to the hole cleaning method provided by the present invention, in the first solution and the second solution, a solution consuming less disk I / O is selected, and before cleaning the hole area, the method further includes:

[0019] The disk I / O consumed by using the first solution to organize the hole area and the disk I / O consumed by using the second solution to organize the hole area are simulated and calculated respectively.

[0020] According to the hole clearing method provided by the present invention, the updating of the check block corresponding to the stripe covered by the new data includes:

[0021] Read the unupdated data blocks and the newly written data blocks in the stripe, calculate based on the unupdated data blocks and the newly written data blocks to obtain a new check block, and then write the new check block to the stripe.

[0022] According to the hole clearing method provided by the present invention, the updating of the check block corresponding to the stripe covered by the new data includes:

[0023] The old check block and the data block being updated are read, a new check block is obtained by calculation based on the old check block and the data block being updated, and then the new check block is written to the stripe.

[0024] In a second aspect, the present application provides a cavity sorting device, comprising:

[0025] A finishing module, for selecting a solution that consumes less disk I / O between the first solution and the second solution for a stripe group with holes, and finishing the hole area;

[0026] The first solution is to write new data into the empty area when a new data request arrives;

[0027] The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

[0028] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the void sorting method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the hole sorting method described in the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the hole sorting method described in the first aspect or any possible implementation of the first aspect.

[0031] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0032] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0033] (1) By comparing the disk I / O of the two solutions, the solution that consumes less disk I / O is selected and the empty area is sorted, which can minimize the disk I / O and thus improve the system performance and reliability.

[0034] (2) By directly writing new data into the existing hole area, the disk I / O operations caused by data migration can be reduced, thereby improving system performance and reliability.

[0035] (3) By organizing valid data into continuous data space, the hole area can be made continuous, thereby improving the write performance of writing new data into the hole area. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the process of the void finishing method provided in the embodiment of the present application;

[0038] Figure 2 It is a schematic diagram of the first solution provided in the embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the second solution provided in the embodiment of the present application;

[0040] Figure 4 It is a schematic diagram of the structure of a cavity finishing device provided in an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0044] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0045] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0046] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0047] First, the following contents are introduced:

[0048] Hole compaction algorithm is a crucial technology for optimizing the storage performance and efficiency of distributed storage systems. In modern data storage systems, due to the continuous deletion of data, a large amount of invalid data will appear in the storage medium, that is, the unfilled holes left by the deleted or moved data. These invalid data occupy storage space and reduce storage efficiency. Recycling these invalid data requires data migration and update of verification data, which causes a large amount of disk I / O and affects the normal service performance of the system.

[0049] To solve this problem, hole defragmentation algorithms were introduced into storage systems. These algorithms aim to improve system service performance by reorganizing the data in the storage medium and reducing the disk I / O caused by data migration and verification data updates when recovering invalid data. By organizing scattered data blocks into continuous data space, the system can read and write data more efficiently, thereby improving overall I / O performance and response speed.

[0050] However, the hole cleaning method in the prior art generally reads out all the remaining valid data in all stripes and re-encodes them into new stripes. However, since this method involves the migration of a large amount of data and the update of verification data, it will cause a large amount of disk I / O, affecting the performance and reliability of the system.

[0051] In response to the above-mentioned defects, the purpose of this application is to provide a mixed flow water pump fault monitoring method, device, electronic equipment and storage medium, aiming to solve the problem that the fault monitoring method of the related technology cannot meet the requirements of maintaining continuous and efficient operation of the water pump unit and the monitoring effect is poor.

[0052] In a storage system using RS (n, k) erasure codes, each file is divided into k original data blocks, and then these data blocks are converted into n total data blocks through the encoding matrix. In this process, the number of total data blocks is m+k, where m is the number of check blocks. These total data blocks form a "strip". These data blocks are distributed on n different storage nodes, and the system can tolerate the failure of any nk nodes.

[0053] Next, combine Figure 1-Figure 3 The void finishing method provided in the embodiments of the present application is introduced.

[0054] Figure 1is a flow chart of the void finishing method provided in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0055] Step 100, for a stripe group with holes, between the first solution and the second solution, a solution that consumes less disk I / O is selected to sort out the hole area;

[0056] Among them, the first scheme is to write new data into the empty area when a new data request arrives;

[0057] The second solution is to organize the valid data in each logical block in the stripe group into the continuous data space in the stripe group for storage, and when a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space.

[0058] The hole defragmentation method provided in the present application proposes two schemes for defragmenting the hole areas for stripe groups with holes. In order to minimize the disk I / O, before defragmentation, it is first necessary to determine the disk I / O consumed by the two schemes for defragmenting the target stripe group respectively, so as to select a scheme that consumes less disk I / O for defragmentation.

[0059] The first solution provided by the present application is hole-filling writing, that is, whenever a new data request arrives, the new data is directly written into the existing hole area. By directly using invalid space to store new data, data migration can be minimized, thereby reducing disk I / O and improving the overall system performance and reliability.

[0060] The second solution provided in the present application is the arrangement within the logical block, that is, the valid data within the logical block is arranged together by moving, so that the hole area is continuous, so that when new data is written into the hole area, it is also continuous. Compared with the first solution in which new data is written into discontinuous hole areas, the second solution can improve the writing performance of new data.

[0061] In some embodiments, the first solution specifically includes:

[0062] Step 200, when a new data request arrives, the new data directly overwrites the invalid data in the hole area;

[0063] Step 210: Update the check block corresponding to the stripe covered by the new data.

[0064] When using the first solution for hole defragmentation, after new data covers invalid data in the hole area, in order to ensure the consistency of the update, the system needs to update the check block accordingly, that is, update the check block corresponding to the stripe covered by the new data.

[0065] Figure 2 is a schematic diagram of the first solution provided in the embodiment of the present application, such as Figure 2 As shown, in one embodiment of the present application, S1-S i N1-N6 are the stripes in the stripe group. Taking S1 as an example, there are three valid data blocks and one invalid data block in S1. The invalid data block is located at the location of storage node N4. Therefore, the new data is directly written to the location of the invalid block where N4 is located, and then the check block in S1 is updated.

[0066] In some embodiments, the second solution specifically includes:

[0067] Step 300, respectively counting the number of valid data blocks in each storage node in each logical block, and taking the minimum value of all valid data blocks as the length of the continuous data space;

[0068] Step 310, based on the length of the continuous data space, select a continuous area with the largest number of valid data blocks as the continuous data space;

[0069] Step 320, migrating the valid data blocks outside the continuous data space to the location of the invalid data blocks in the continuous data space;

[0070] Step 330, when a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space;

[0071] Step 340: Update the check block corresponding to the stripe covered by the new data.

[0072] Figure 3 is a schematic diagram of the second solution provided in the embodiment of the present application, such as Figure 3 As shown, in one embodiment of the present application, the number of valid data blocks of each storage node N1-N4 is 3, 2, 3, and 3, so the minimum number of valid data blocks 2 in all storage nodes is selected as the length of the continuous data space, that is, 2 stripes. By traversing, it can be found that among the two consecutive stripes, the number of valid data blocks in S3 and S4 is the largest, so S3 and S4 are used as continuous data spaces, and then the two valid data blocks in S2 are filled into the positions of the invalid data blocks of S3 and S4 respectively. Next, the new data is written sequentially to the positions of the invalid data blocks in each stripe, and a reasonable update method is selected to update the check block.

[0073] In some embodiments, before step 100, the void finishing method further includes:

[0074] The disk I / O consumed by using the first solution to organize the hole area and the disk I / O consumed by using the second solution to organize the hole area are simulated and calculated respectively.

[0075] Optionally, in order to determine the disk I / O consumed by the two schemes for respectively organizing the target stripe group, simulation calculations can be used to simulate the disk I / O consumed by the first scheme for organizing the hole area and the disk I / O consumed by the second scheme for organizing the hole area.

[0076] In one embodiment of the present application, the stripe group to be defragmented may be used as input, and the overhead may be calculated using two compression algorithms in sequence, and the one with the smallest disk I / O overhead may be selected as the best hole defragmentation scheme.

[0077] In some embodiments, step 210 or step 340 specifically includes:

[0078] Read the unupdated data blocks and the newly written data blocks in the stripe, calculate based on the unupdated data blocks and the newly written data blocks to obtain a new check block, and then write the new check block to the stripe.

[0079] In some embodiments, step 210 or step 340 specifically includes:

[0080] The old check block and the data block being updated are read, a new check block is obtained by calculation based on the old check block and the data block being updated, and then the new check block is written to the stripe.

[0081] Optionally, two check block update methods may be used to update the check block, including:

[0082] Read-Construct-Write (RCW) reads the unupdated data blocks and the newly written data blocks in the stripe, calculates new check blocks based on the unupdated data blocks and the newly written data blocks, and then writes the new check blocks to the stripe.

[0083] Read-Modify-Write (RMW) reads the old check block and the data block being updated, calculates the new check block based on the old check block and the data block being updated, and then writes the new check block to the stripe.

[0084] No matter whether the first solution or the second solution is used to organize the hole area, the check block needs to be updated eventually. When updating the check block, the above RCW or RMW can be used for updating.

[0085] In one embodiment of the present application, Figure 2When updating the check block in S1, if RMW is used to calculate the check block, the update I / O is 5, including 1 I / O to read the old check block, 2 I / O to read the data block being updated, and 2 I / O to write the new check block; if RCW is used to calculate the check block, the update I / O is 5, including 3 I / O to read the unupdated data block in the stripe and 2 I / O to write the new check block; the smaller of the two is taken, so the I / O to update the check block is 5.

[0086] Combine the following Figure 4 The cavity finishing device provided by the present invention is described. The cavity finishing device described below and the cavity finishing method described above can correspond to each other.

[0087] Figure 4 is a schematic diagram of the structure of the cavity finishing device provided in the embodiment of the present application, such as Figure 4 As shown, the device includes a finishing module 410, wherein:

[0088] The arrangement module 410 is used for selecting a solution that consumes less disk I / O between the first solution and the second solution for a stripe group with holes, and arranging the hole area;

[0089] The first solution is to write new data into the empty area when a new data request arrives;

[0090] The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

[0091] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0092] Based on the method in the above embodiment, Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, an embodiment of the present application provides an electronic device, which may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the void sorting method in the above embodiment.

[0093] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the void sorting method described in each embodiment of the present application.

[0094] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the hole sorting method in the above embodiment.

[0095] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the hole sorting method in the above embodiment.

[0096] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0098] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0099] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0100] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for finishing voids, characterized in that: include: For stripe groups with holes, choose the solution that consumes less disk I / O between the first and second solutions to clean up the hole areas; The first solution is to write new data into the empty area when a new data request arrives; The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

2. The method for finishing voids according to claim 1, characterized in that: When a new data request arrives, the new data is written into the empty area, including: When a new data request arrives, the new data is directly used to overwrite the invalid data in the hole area; Update the check block corresponding to the stripe covered by the new data.

3. The void finishing method according to claim 1, characterized in that: The method of arranging valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, sequentially writing the new data into an empty area outside the continuous data space, comprises: Count the number of valid data blocks in each storage node in each logical block respectively, and take the minimum value of all valid data blocks as the length of the continuous data space; Based on the length of the continuous data space, a continuous area with the largest number of valid data blocks is selected as the continuous data space; Migrating valid data blocks outside the continuous data space to locations where invalid data blocks are located in the continuous data space; When a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space; Update the check block corresponding to the stripe covered by the new data.

4. The method for finishing voids according to claim 1, characterized in that: In the first solution and the second solution, the solution consuming less disk I / O is selected, and before the hole area is sorted, the method further includes: The disk I / O consumed by using the first solution to organize the hole area and the disk I / O consumed by using the second solution to organize the hole area are simulated and calculated respectively.

5. The method for finishing voids according to claim 2 or 3, characterized in that: The updating of the check block corresponding to the stripe covered by the new data includes: Read the unupdated data blocks and the newly written data blocks in the stripe, calculate based on the unupdated data blocks and the newly written data blocks to obtain a new check block, and then write the new check block to the stripe.

6. The method for finishing voids according to claim 2 or 3, characterized in that: The updating of the check block corresponding to the stripe covered by the new data includes: The old check block and the data block being updated are read, a new check block is obtained by calculation based on the old check block and the data block being updated, and then the new check block is written to the stripe.

7. A cavity finishing device, characterized in that: include: A finishing module, for selecting a solution that consumes less disk I / O between the first solution and the second solution for a stripe group with holes, and finishing the hole area; The first solution is to write new data into the empty area when a new data request arrives; The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the void sorting method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is enabled to execute the hole sorting method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is enabled to execute the hole sorting method according to any one of claims 1 to 6.

Images