Free space recovery method and device for file storage system

Abstract

The invention discloses a free space recovery method and device for a file storage system, and relates to the field of data distribution. The method comprises the following steps: when free space recovery is triggered, obtaining free space of a file storage system, taking a complete data block in a storage pool corresponding to the free space as a to-be-recovered data block, and recovering the to-be-recovered data block. According to the method, after all the free spaces of the file storage system are obtained, the complete data block in the storage pool corresponding to each free space is used as the data block to be recycled, then the whole data block to be recycled is directly recycled, and the recycling efficiency of the free spaces is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data allocation, and in particular to a free space recovery method and device for a file storage system. Background Art

[0002] As the usage of the file storage system increases, the space in the storage pool is occupied, and the number of available blocks during writing decreases. When a file is deleted, the file storage system will mark the space released after the file is deleted. At this time, the storage pool cannot perceive the release of space in real time. The storage pool will only perceive the above-mentioned released space when new data is written or free space is recycled.

[0003] There are generally two ways to manage storage pool space:

[0004] (1) Overwrite method, that is, when new data is written, the free space is traversed in the storage order (write order), and the data is written into the first traversed space; the reason why it is called "overwrite" is that: for example, data is written into space A at time T0, data is written into space B at time T1, and the data in space A is released at time T2, then the data at time T3 is written into space A.

[0005] Because this method needs to be traversed sequentially before writing, it is generally suitable for scenarios where data is written serially.

[0006] (2) Append write method, that is, reallocate space each time data is written, and adopt innovative space garbage collection algorithm. According to the usage of storage system and business needs, the garbage collection strategy is dynamically adjusted to adapt to different workloads and data patterns.

[0007] The method is as follows: all free space in the storage pool is obtained and reclaimed through periodic automatic triggering (such as polling or monitoring when the storage pool usage capacity reaches a threshold, etc.), so that when writing data, it is immediately stored in the previously obtained free space. This method supports synchronous batch writing of data and is generally suitable for scenarios where data is written in parallel.

[0008] With the diversification of data, the above method (2) has become the mainstream storage pool space management method in the market.

[0009] However, method (2) has low performance of the file storage system when working;

[0010] At the same time, the efficiency of method (2) in reclaiming free space through polling traversal is not as good as the instant recycling of method (1). Summary of the invention

[0011] In view of the defects existing in the prior art, the technical problem solved by the present invention is: how to improve the efficiency of recycling free space when writing data in parallel.

[0012] To achieve the above objectives, in the first aspect, an embodiment of the present application provides a free space recovery method for a file storage system, the method comprising the following steps: when free space recovery is triggered, after obtaining the free space of the file storage system, the complete data blocks in the storage pool corresponding to the free space are used as data blocks to be recovered and are recovered.

[0013] In combination with the first aspect, in one implementation, the process of obtaining free space of the file storage system includes: scanning to obtain available space as free space, and forming a free space list according to the free space information.

[0014] In combination with the first aspect, in an implementation manner, the process of using the complete data blocks in the storage pool corresponding to the free space as the data blocks to be reclaimed includes: determining the complete data blocks corresponding to the free space in the free space list.

[0015] In combination with the first aspect, in one implementation, the process of reclaiming the data blocks to be reclaimed includes: after acquiring the data blocks to be reclaimed corresponding to all free spaces in the file storage system, sending data block reclaiming information to the storage pool.

[0016] In combination with the first aspect, in one implementation, the process of determining the to-be-reclaimed data blocks corresponding to all free spaces in the file storage system is as follows: all available spaces have been scanned.

[0017] In combination with the first aspect, in one implementation, the process of reclaiming the data block to be reclaimed includes: when the capacity of the free space reaches a specified threshold, sending data block reclaim information to the storage pool.

[0018] In combination with the first aspect, in one implementation, the process of determining whether the capacity of the free space reaches a specified threshold includes:

[0019] The free space in the free space list reaches the set amount of space;

[0020] Or the number of recovered data blocks reaches the set number of data blocks.

[0021] In combination with the first aspect, in one implementation, the complete data block is a data block in which the free space accounts for substantially 100%.

[0022] In combination with the first aspect, in one implementation, the triggering condition for free space reclamation includes at least one of the following three conditions:

[0023] Automatically trigger at a fixed time;

[0024] Receive a data deletion message;

[0025] Received space release information.

[0026] In a second aspect, an embodiment of the present application provides a free space recovery device for a file storage system, the device comprising a file storage system and a storage pool;

[0027] The file storage system is used to: when executing the triggering free space recovery provided by the first aspect, after obtaining the free space of the file storage system, use the complete data blocks in the storage pool corresponding to the free space as the process of the data blocks to be recovered; according to the process of recovering the data blocks to be recovered provided by the first aspect, send data block recovery information to the storage pool;

[0028] The storage pool is used to recycle the corresponding data block after receiving the data block recycle information.

[0029] Compared with the prior art, the advantages of the present invention are:

[0030] (1) Compared with the append writing method in the prior art, after obtaining all the free space in the file storage system, the present application will use the complete data block in the storage pool corresponding to each free space as the data block to be recycled, and then directly recycle the entire data block to be recycled. There is no need to migrate the "old" data in the data block to be recycled, and there is no need to write the "old data" into other data blocks with free space.

[0031] At the same time, compared with the overwrite writing method in the prior art, overwrite writing usually has the problem of write amplification. For example, to write 64 bytes of data, you must first read out 512 bytes (the minimum read and write unit supported by the storage system), update the 64 bytes of data to be newly written, and then overwrite the 512 bytes. As mentioned above, the present application can directly recycle the entire data block to be recycled.

[0032] Therefore, whether compared with the append writing method or the overwrite writing method, the present application can significantly improve the efficiency of free space recovery.

[0033] (2) The present application only needs to send a small amount of recovery information to the storage pool to complete the operation of recovering free space, thereby realizing the operation of batch recovery of all data blocks. The advantage is that compared with the file storage system generating a corresponding notification message each time a free space is obtained, the number of times the recovery information is sent is greatly reduced, thereby reducing the pressure on the file transfer system and improving the performance of the file transfer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 is a flow chart of a free space recovery method for a file storage system in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the hardware structure of a free space recovery device for a file storage system involved in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0039] First, the research and development principle of this application is briefly described.

[0040] The applicant studied the performance of the file storage system and the reasons for the low recovery efficiency of the method (2) in the prior art and found that:

[0041] When method (2) obtains all the free space of the file storage system, since the free space is discretely distributed, each time a free space is obtained, the file storage system will generate a corresponding notification message, which will generate a large number of notification messages, thereby increasing the working pressure of the file storage system and affecting its performance.

[0042] When obtaining free space in the storage pool, method (2) first obtains the available space of the file storage system (i.e., space without data), then locates the free space corresponding to the available space in the storage pool, and then reclaims space based on the located free space.

[0043] However, the space in the storage pool exists in the form of data blocks. According to the above method, the available space of the file storage system corresponds to data blocks in multiple storage pools, and there is a situation where it only corresponds to part of the space in the data block (that is, it only corresponds to part of the data block, part of the space in the data block needs to be reclaimed, and there is data in the other part of the space that cannot be reclaimed). In this case, the recycling method is: first migrate the "old" data in the data block that needs to be recycled, then recycle the data block, and then write the "old data" again into other data blocks with free space.

[0044] On this basis, in order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0045] In a first aspect, an embodiment of the present application provides a free space recovery method for a file storage system, the method comprising the following steps: when free space recovery is triggered, after obtaining the free space of the file storage system, a complete data block in a storage pool corresponding to each free space is used as a data block to be recovered and recovered. A complete data block is a data block that is all free space (the free space ratio is basically 100%) and does not store any written data.

[0046] Compared with the append writing method in the prior art, after the present application obtains all the free space of the file storage system, the complete data block in the storage pool corresponding to each free space will be used as the data block to be recycled, and then the entire data block to be recycled can be directly recycled. There is no need to migrate the "old" data in the data block to be recycled, and write the "old data" to other data blocks with free space.

[0047] At the same time, compared with the overwrite writing method in the prior art, overwrite writing usually has the problem of write amplification. For example, to write 64 bytes of data, you must first read out 512 bytes (the minimum read and write unit supported by the storage system), update the 64 bytes of data to be newly written, and then overwrite the 512 bytes. As mentioned above, the present application can directly recycle the entire data block to be recycled.

[0048] Therefore, whether compared with the append writing method or the overwrite writing method, the present application can significantly improve the efficiency of free space recovery.

[0049] In one embodiment, the triggering condition for free space reclamation in the above method includes at least one of the following three conditions:

[0050] (1) Automatically triggered at regular intervals, such as the trim operation service periodically polling the file storage system;

[0051] (2) Receiving data (file) deletion information, such as when the user actively deletes the data;

[0052] (3) Receive space release information, such as releasing hot data space through tiered storage.

[0053] In one embodiment, the process of obtaining free space of the file storage system in the above method includes: scanning to obtain available space as free space, and forming a free space list with all the scanned free space information.

[0054] On this basis, the process of using the complete data block in the storage pool corresponding to each free space as the data block to be reclaimed in the above method includes: determining the complete data block corresponding to each free space in the free space list.

[0055] In one embodiment, the process of reclaiming the data block to be reclaimed in the above method is any one of the following two:

[0056] (1) After obtaining the to-be-reclaimed data blocks corresponding to all the free spaces in the file storage system, data block reclaiming information (i.e., information on reclaiming all the to-be-reclaimed data blocks) is sent to the storage pool to control the storage pool to reclaim the data blocks.

[0057] On this basis, the determination process of obtaining the to-be-reclaimed data blocks corresponding to all free spaces in the file storage system is as follows: all available spaces have been scanned.

[0058] This method only needs to send recycling information to the storage pool once to realize the operation of batch recycling of all data blocks. The advantage is that compared with the file storage system generating a corresponding notification message every time a free space is obtained, the number of times recycling information is sent is greatly reduced, thereby reducing the pressure on the file transfer system and improving the performance of the file transfer system.

[0059] (2) When the capacity of the free space reaches a specified threshold, data block reclaim information is sent to the storage pool to control the storage pool to reclaim data blocks.

[0060] The process of determining whether the capacity of the free space reaches the specified threshold can be:

[0061] The free space in the free space list reaches the set amount of space;

[0062] Or the number of recovered data blocks reaches the set number of data blocks.

[0063] This method sends a small amount of recycling information to the storage pool to implement the operation of batch recycling of all data blocks. The advantage is that compared with the file storage system generating a corresponding notification message every time a free space is obtained, the number of times recycling information is sent is greatly reduced, thereby reducing the pressure on the file transfer system and improving the performance of the file transfer system.

[0064] It can be seen that method (1) takes a relatively long time to obtain all free spaces, and is suitable for scenarios where a large amount of data needs to be written in parallel; method (2) takes a relatively short time to obtain a specified threshold number of to-be-reclaimed data blocks, and is suitable for scenarios where a relatively small amount of data needs to be written in parallel. In actual use, you can choose the appropriate method according to different needs.

[0065] The above method is described below through a specific embodiment.

[0066] The usage scenario of this embodiment is a parallel writing scenario, and the amount of data written in parallel each time is moderate. Figure 1 As shown, the steps of this embodiment include:

[0067] S1: When space recovery is triggered (timing automatic triggering, or data deletion, or space release information when space release is required), go to S2.

[0068] S2: Scan and obtain available space as free space, form a free space list with all the scanned free space, and go to S3.

[0069] S3: Determine in the free space list the complete data block corresponding to each free space as the data block to be reclaimed, and go to S4.

[0070] S4: When the free space in the free space list reaches the set amount of space, data block recovery information is sent to the storage pool, and then the process goes to S5.

[0071] S5: The storage pool reclaims the corresponding data block according to the data block reclaim information.

[0072] In a second aspect, an embodiment of the present application further provides a free space recovery device for a file storage system, which is used to implement the method provided in the first aspect.

[0073] Specifically, the device includes a file storage system and a storage pool.

[0074] The file storage system is used to: when free space recovery is triggered, after obtaining the free space of the file storage system, take the complete data blocks in the storage pool corresponding to each free space as the data blocks to be recovered, and notify the storage pool.

[0075] The storage pool is used to recycle the corresponding data block after receiving the notification from the file storage system.

[0076] Among them, the functional implementation of each module in the above-mentioned free space recovery device for file storage system corresponds to each step in the above-mentioned free space recovery method embodiment for file storage system, and its functions and implementation processes are no longer repeated here.

[0077] In a third aspect, an embodiment of the present application provides a free space recovery device for a file storage system. The free space recovery device for a file storage system may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0078] Reference Figure 2 , Figure 2 The hardware structure diagram of the free space recovery device for the file storage system involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the free space recovery device for the file storage system may include a processor, a memory, a communication interface, and a communication bus.

[0079] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0080] The communication interface includes an input / output (I / O) interface, a physical interface, and a logical interface, etc., which are used to realize the interconnection of devices inside the free space recovery device for the file storage system, and an interface for realizing the interconnection between the free space recovery device for the file storage system and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, an optical fiber interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0081] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0082] The processor may be a general-purpose processor, and the general-purpose processor may call a free space recovery program for a file storage system stored in a memory, and execute a free space recovery method for a file storage system provided in an embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the free space recovery program for a file storage system is called may refer to the various embodiments of the free space recovery method for a file storage system of the present application, and will not be described in detail herein.

[0083] Those skilled in the art will understand that Figure 2 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0084] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0085] The computer-readable storage medium of the present application stores a free space recovery program for a file storage system, wherein when the free space recovery program for a file storage system is executed by a processor, the steps of the free space recovery method for a file storage system as described above are implemented.

[0086] Among them, the method implemented when the free space recovery program for the file storage system is executed can refer to the various embodiments of the free space recovery method for the file storage system of the present application, and will not be repeated here.

[0087] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0089] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0090] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0091] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0092] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0094] The above are only specific implementations of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the embodiments of the present invention, and these modifications or replacements should be included in the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention shall be based on the protection scope of the claims.

Claims

1. A free space recovery method for a file storage system, characterized in that: The method comprises the following steps: when free space recovery is triggered, after obtaining the free space of the file storage system, the complete data blocks in the storage pool corresponding to the free space are taken as data blocks to be recovered and recovered.

2. The free space recovery method for a file storage system according to claim 1, characterized in that: The process of obtaining free space of the file storage system includes: scanning to obtain available space as free space, and forming a free space list according to free space information.

3. The free space recovery method for a file storage system according to claim 2, characterized in that: The process of using the complete data blocks in the storage pool corresponding to the free space as the data blocks to be reclaimed includes: determining the complete data blocks corresponding to the free space in the free space list.

4. The free space recovery method for a file storage system according to claim 2, characterized in that: The process of reclaiming the data blocks to be reclaimed includes: after obtaining the data blocks to be reclaimed corresponding to all free spaces in the file storage system, sending data block reclaim information to the storage pool.

5. The free space recovery method for a file storage system according to claim 4, characterized in that: The determination process of obtaining the to-be-reclaimed data blocks corresponding to all free spaces in the file storage system is as follows: all available spaces have been scanned.

6. The free space recovery method for a file storage system according to claim 2, characterized in that: The process of reclaiming the data block to be reclaimed includes: when the capacity of the free space reaches a specified threshold, sending data block reclaim information to the storage pool.

7. The free space recovery method for a file storage system according to claim 6, characterized in that: The process of determining whether the capacity of the free space reaches the specified threshold includes: The free space in the free space list reaches the set amount of space; Or the number of recovered data blocks reaches the set number of data blocks.

8. The free space recovery method for a file storage system according to any one of claims 1 to 7, characterized in that: The complete data block is a data block in which the free space accounts for substantially 100%.

9. The free space recovery method for a file storage system according to any one of claims 1 to 7, characterized in that: The triggering condition for free space recovery includes at least one of the following three conditions: Timing automatic triggering; Receive a data deletion message; Received space release information.

10. A free space recovery device for a file storage system, the device comprising a file storage system and a storage pool, characterized in that: The file storage system is used for: when executing the triggering free space recovery described in any one of claims 1 to 9, after obtaining the free space of the file storage system, taking the complete data blocks in the storage pool corresponding to the free space as the process of the data blocks to be recovered; according to the process of recovering the data blocks to be recovered described in any one of claims 4 to 7, sending data block recovery information to the storage pool; The storage pool is used to recycle the corresponding data block after receiving the data block recycle information.

Images