Hole arrangement method and device, electronic equipment and storage medium

Abstract

The application belongs to the field of computer storage, and specifically discloses a hole arrangement method and device, an electronic device and a storage medium. The method comprises the following steps: for a strip group with holes, in a first scheme and a second scheme, a scheme with less disk I / O consumption is selected to arrange the hole region; wherein the first scheme is that when a new data request arrives, the new data is written into the hole region; and the second scheme is that effective data in each logical block in the strip group is arranged to be stored in continuous data space in the strip group, and when a new data request arrives, the new data is sequentially written into the hole region outside the continuous data space. Through the method, disk I / O operations caused by data migration can be reduced, so that the system performance and reliability are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer storage, and more particularly relates to a hole arrangement method and device, an electronic device and a storage medium. BACKGROUND

[0002] In a modern data storage system, due to continuous deletion of data, a large amount of invalid data, i.e. holes left by data that has been deleted or moved and not filled, will appear in the storage medium. These invalid data occupy storage space and reduce storage efficiency.

[0003] The hole arrangement method in the prior art generally reads all the remaining valid data in all stripes and re-encodes them into new stripes. However, this method involves migration of a large amount of data and updating of check data, thus causing a large amount of disk I / O and affecting the performance and reliability of the system. SUMMARY

[0004] In view of the defects of the prior art, the present application aims to provide a hole arrangement method and device, an electronic device and a storage medium, and aims to solve the problem of low performance and reliability of the system caused by a large amount of disk I / O due to the migration of a large amount of data and the updating of check data in the hole arrangement method of the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides a hole arrangement method, comprising:

[0006] For a stripe group with holes, in the first and second schemes, the scheme with less disk I / O consumption is selected to arrange the hole region.

[0007] The first scheme is to write new data into the hole region when a new data request arrives.

[0008] The second scheme is to arrange the valid data in each logical block in the stripe group to the continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially to the hole region outside the continuous data space.

[0009] According to the hole arrangement method provided by the present application, when a new data request arrives, the new data is written into the hole region, comprising:

[0010] When a new data request arrives, the new data directly covers the invalid data in the hole region.

[0011] Update the check block corresponding to the stripe of the covered new data.

[0012] According to the hole cleaning method provided by the present invention, the valid data in each logical block in the stripe group is cleaned up into a continuous data space in the stripe group for storage, and when a new data request arrives, the new data is sequentially written into the hole area outside the continuous data space, including:

[0013] Count the number of valid data blocks in each storage node in each logical block respectively, and take the minimum value of all valid data blocks as the length of the continuous data space;

[0014] Based on the length of the continuous data space, selecting a continuous area with the largest number of valid data blocks as the continuous data space;

[0015] Migrating valid data blocks outside the continuous data space to locations where invalid data blocks are located within the continuous data space;

[0016] When a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space;

[0017] Update the check block corresponding to the stripe covered by the new data.

[0018] According to the hole cleaning method provided by the present invention, in the first solution and the second solution, the solution that consumes less disk I / O is selected, and before cleaning the hole area, the method further includes:

[0019] The disk I / O consumed by using the first solution to organize the hole area and the disk I / O consumed by using the second solution to organize the hole area are simulated and calculated respectively.

[0020] According to the hole clearing method provided by the present invention, updating the check block corresponding to the stripe covered by the new data includes:

[0021] Read the unupdated data blocks and the newly written data blocks in the stripe, calculate based on the unupdated data blocks and the newly written data blocks to obtain a new parity block, and then write the new parity block to the stripe.

[0022] According to the hole clearing method provided by the present invention, updating the check block corresponding to the stripe covered by the new data includes:

[0023] The old parity block and the data block being updated are read, a new parity block is calculated based on the old parity block and the data block being updated, and then the new parity block is written to the stripe.

[0024] In a second aspect, the present application provides a cavity sorting device, comprising:

[0025] A finishing module is used to select a solution that consumes less disk I / O between the first solution and the second solution for a stripe group with holes, and to finish the hole area;

[0026] The first solution is to write new data into the empty area when a new data request arrives;

[0027] The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

[0028] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the void sorting method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the void sorting method described in the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the void sorting method described in the first aspect or any possible implementation of the first aspect.

[0031] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0032] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the existing technologies:

[0033] (1) By comparing the disk I / O of the two solutions, the solution that consumes less disk I / O is selected and the empty area is sorted, which can minimize the disk I / O and thus improve the system performance and reliability.

[0034] (2) By directly writing new data into the existing empty area, the disk I / O operations caused by data migration can be reduced, thereby improving system performance and reliability.

[0035] (3) By organizing valid data into continuous data space, the hole area can be made continuous, thereby improving the write performance of writing new data into the hole area. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 1 is a flow chart of a cavity finishing method provided in an embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of the first solution provided in the embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the second solution provided in the embodiment of the present application;

[0040] Figure 4 Schematic diagram of the structure of the cavity finishing device provided in an embodiment of the present application;

[0041] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0044] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0045] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0046] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0047] Firstly, the following is introduced:

[0048] For optimizing the storage performance and efficiency of distributed storage systems, hole compaction algorithms are a crucial technology. In modern data storage systems, due to continuous deletion of data, a large amount of invalid data, i.e. holes left by data that has been deleted or moved and not filled, appears in the storage medium. These invalid data occupy storage space, reducing storage efficiency, and recycling these invalid data involves data migration and updating of check data, thus causing a large amount of disk I / O, affecting the normal service performance of the system.

[0049] To solve this problem, hole compaction algorithms are introduced into the storage system, which aims to reduce the disk I / O caused by data migration and updating of check data when recycling invalid data, thereby improving the service performance of the system, and by organizing scattered data blocks into continuous data space, the system can more effectively perform data reading and writing, thereby improving the overall I / O performance and response speed.

[0050] However, the hole compaction method in the prior art generally reads all the remaining valid data in all stripes and re-encodes them into new stripes, but this method involves a large amount of data migration and updating of check data, thus causing a large amount of disk I / O, affecting the performance and reliability of the system.

[0051] In view of the above defects, the purpose of the present application is to provide a mixed flow pump fault monitoring method and device, electronic equipment and storage medium, aiming to solve the problem that the fault monitoring method of the related art cannot meet the requirement of maintaining the continuous and efficient operation of the water pump unit, and the monitoring effect is poor.

[0052] In a storage system using RS(n, k) erasure code, each file is divided into k original data blocks, and then the data blocks are converted into n total data blocks through an encoding matrix. In this process, the number of total data blocks is m+k, where m is the number of check blocks. These total data blocks form a "stripe". These data blocks are distributed on n different storage nodes, and the system can tolerate any n-k node failures.

[0053] Next, combined with Figures 1-3 The hole compaction method provided in the embodiments of the present application is introduced.

[0054] Figure 1is a flowchart of a hole arrangement method provided by the embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0055] In the first scheme and the second scheme, for a strip group with holes, a scheme consuming less disk I / O is selected to arrange the hole region.

[0056] The first scheme is to write new data into the hole region when a new data request arrives.

[0057] The second scheme is to arrange the valid data in each logical block in the strip group to the continuous data space in the strip group for storage, and when a new data request arrives, the new data is sequentially written into the hole region outside the continuous data space.

[0058] The hole arrangement method provided by the present application proposes two schemes to arrange the hole region for a strip group with holes. In order to minimize the disk I / O, the disk I / O consumed by the two schemes for arranging the target strip group is determined before arrangement, so as to select a scheme consuming less disk I / O for arrangement.

[0059] The first scheme provided by the present application is hole filling writing, that is, whenever a new data request arrives, the new data is directly written into the existing hole region. By directly using the invalid space to store new data, the data migration can be minimized, thereby reducing the disk I / O and improving the overall performance and reliability of the system.

[0060] The second scheme provided by the present application is logical block arrangement, that is, by moving the valid data in the logical block, the valid data is arranged together, so that the hole region is continuous. Thus, when new data is written into the hole region, it is also continuous. Compared with the first scheme in which new data is written into discontinuous hole regions, the second scheme can improve the write performance of new data.

[0061] In some embodiments, the first scheme specifically comprises:

[0062] Step 200: When a new data request arrives, the new data directly covers the invalid data in the hole region.

[0063] Step 210: Update the check block corresponding to the strip of the covered new data.

[0064] When the first scheme is used for hole arrangement, after the new data covers the invalid data in the hole region, in order to ensure the consistency of the update, the system needs to update the corresponding check block, that is, update the check block corresponding to the strip of the covered new data.

[0065] Figure 2 is a schematic diagram of the first scheme provided by the embodiment of the present application, as shown inFigure 2 As shown, in one embodiment of the present application, S1-S i For each stripe in the stripe group, N1-N6 are storage nodes. Taking S1 as an example, there are three valid data blocks and one invalid data block in S1. The invalid data block is located at the location of storage node N4, so the new data is directly written to the location of the invalid block where N4 is located, and then the check block in S1 is updated.

[0066] In some embodiments, the second solution specifically includes:

[0067] Step 300: Count the number of valid data blocks in each storage node in each logical block, and take the minimum value of all valid data blocks as the length of the continuous data space;

[0068] Step 310: Based on the length of the continuous data space, a continuous area with the largest number of valid data blocks is selected as the continuous data space;

[0069] Step 320, migrating the valid data blocks outside the continuous data space to the locations of the invalid data blocks within the continuous data space;

[0070] Step 330 , when a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space;

[0071] Step 340: Update the check block corresponding to the stripe covered by the new data.

[0072] Figure 3 is a schematic diagram of the second solution provided in the embodiment of the present application, such as Figure 3 As shown, in one embodiment of the present application, the number of valid data blocks of each storage node N1-N4 is 3, 2, 3, and 3, so the minimum number of valid data blocks 2 in all storage nodes is selected as the length of the continuous data space, that is, 2 stripes. By traversing, it can be found that among the two consecutive stripes, the number of valid data blocks in S3 and S4 is the largest, so S3 and S4 are used as continuous data spaces, and then the two valid data blocks in S2 are filled into the positions of the invalid data blocks of S3 and S4 respectively. Next, the new data is written sequentially to the positions of the invalid data blocks in each stripe, and a reasonable update method is selected to update the check block.

[0073] In some embodiments, before step 100, the cavity finishing method further includes:

[0074] The disk I / O consumed by using the first solution to clean up the hole area and the disk I / O consumed by using the second solution to clean up the hole area are simulated and calculated respectively.

[0075] Optionally, in order to determine the disk I / O consumed by the two schemes for organizing the target stripe group respectively, simulation calculations can be used to simulate the disk I / O consumed by the first scheme for organizing the hole area and the disk I / O consumed by the second scheme for organizing the hole area.

[0076] In one embodiment of the present application, the stripe group to be defragmented may be used as input, and the overhead may be calculated using two compression algorithms in sequence. The one with the smallest disk I / O overhead may be selected as the optimal hole defragmentation scheme.

[0077] In some embodiments, step 210 or step 340 specifically includes:

[0078] Read the unupdated data blocks and the newly written data blocks in the stripe, calculate based on the unupdated data blocks and the newly written data blocks to obtain a new parity block, and then write the new parity block to the stripe.

[0079] In some embodiments, step 210 or step 340 specifically includes:

[0080] The old parity block and the data block being updated are read, a new parity block is calculated based on the old parity block and the data block being updated, and then the new parity block is written to the stripe.

[0081] Optionally, two check block update methods can be used to update the check block, including:

[0082] Read-Construct-Write (RCW) reads the unchanged data blocks and newly written data blocks in the stripe, calculates a new parity block based on the unchanged data blocks and the newly written data blocks, and then writes the new parity block to the stripe.

[0083] Read-Modify-Write (RMW) reads the old parity block and the data block being updated, calculates a new parity block based on the old parity block and the data block being updated, and then writes the new parity block to the stripe.

[0084] Regardless of whether the first or second solution is used to organize the hole area, the check block needs to be updated eventually. When updating the check block, the above-mentioned RCW or RMW can be used for updating.

[0085] In one embodiment of the present application, Figure 2When updating the parity block in S1, if RMW is used to calculate the parity block, the update I / O is 5, including 1 I / O to read the old parity block, 2 I / O to read the data block being updated, and 2 I / O to write the new parity block. If RCW is used to calculate the parity block, the update I / O is 5, including 3 I / O to read the unupdated data block in the stripe and 2 I / O to write the new parity block. The smaller of the two is taken, so the I / O to update the parity block is 5.

[0086] The following combination Figure 4 The cavity finishing device provided by the present invention is described. The cavity finishing device described below and the cavity finishing method described above can correspond to each other.

[0087] Figure 4 This is a schematic diagram of the structure of the cavity finishing device provided in the embodiment of the present application. Figure 4 As shown, the apparatus includes a finishing module 410, wherein:

[0088] The trimming module 410 is configured to trim the stripe group with holes by selecting the solution that consumes less disk I / O between the first solution and the second solution to trim the hole area.

[0089] The first solution is to write new data into the empty area when a new data request arrives;

[0090] The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space.

[0091] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0092] Based on the method in the above embodiment, Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, an embodiment of the present application provides an electronic device, which may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the void cleaning method in the above embodiment.

[0093] In addition, the logic instructions in the memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the void sorting method described in each embodiment of the present application.

[0094] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the void sorting method in the above embodiment.

[0095] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the hole cleaning method in the above embodiment.

[0096] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0097] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0098] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0099] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0100] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for finishing voids, characterized in that: include: For stripe groups with holes, choose the one that consumes less disk I / O between the first and second options to clean up the hole areas. The first solution is to write new data into the empty area when a new data request arrives; The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space; The method of arranging valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and sequentially writing the new data into a hole area outside the continuous data space when a new data request arrives, includes: Count the number of valid data blocks in each storage node in each logical block respectively, and take the minimum value of all valid data blocks as the length of the continuous data space; Based on the length of the continuous data space, selecting a continuous area with the largest number of valid data blocks as the continuous data space; Migrating valid data blocks outside the continuous data space to locations where invalid data blocks are located within the continuous data space; When a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space; Update the check block corresponding to the stripe covered by the new data.

2. The method for finishing voids according to claim 1, wherein: When a new data request arrives, the new data is written into the empty area, including: When a new data request arrives, the new data is directly overwritten with the invalid data in the hole area; Update the check block corresponding to the stripe covered by the new data.

3. The method for finishing voids according to claim 1, wherein: In the first and second solutions, the solution that consumes less disk I / O is selected. Before tidying up the hole area, the method further includes: The disk I / O consumed by using the first solution to organize the hole area and the disk I / O consumed by using the second solution to organize the hole area are simulated and calculated respectively.

4. The method for tidying up the voids according to claim 1 or 2, wherein: The updating of the check block corresponding to the stripe of the overwritten new data includes: Read the unupdated data blocks and the newly written data blocks in the stripe, calculate based on the unupdated data blocks and the newly written data blocks to obtain a new parity block, and then write the new parity block to the stripe.

5. The method for tidying up the voids according to claim 1 or 2, wherein: The updating of the check block corresponding to the stripe of the overwritten new data includes: The old parity block and the data block being updated are read, a new parity block is calculated based on the old parity block and the data block being updated, and then the new parity block is written to the stripe.

6. A cavity sorting device, characterized in that: include: A trimming module is used to trim the stripe group with holes by selecting the solution that consumes less disk I / O between the first solution and the second solution to trim the hole area; The first solution is to write new data into the empty area when a new data request arrives; The second solution is to organize the valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and when a new data request arrives, write the new data sequentially into the empty area outside the continuous data space; The method of arranging valid data in each logical block in the stripe group into a continuous data space in the stripe group for storage, and sequentially writing the new data into a hole area outside the continuous data space when a new data request arrives, includes: Count the number of valid data blocks in each storage node in each logical block respectively, and take the minimum value of all valid data blocks as the length of the continuous data space; Based on the length of the continuous data space, selecting a continuous area with the largest number of valid data blocks as the continuous data space; Migrating valid data blocks outside the continuous data space to locations where invalid data blocks are located within the continuous data space; When a new data request arrives, the new data is sequentially written into the empty area outside the continuous data space; Update the check block corresponding to the stripe covered by the new data.

7. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the void defragmentation method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is enabled to execute the hole defragmentation method according to any one of claims 1 to 5.

9. A computer program product, characterized in that When the computer program product runs on a processor, the processor is enabled to execute the hole defragmentation method according to any one of claims 1 to 5.

Images