Garbage collection method and device based on distributed mass small file aggregation

Abstract

The invention relates to a garbage collection method and device based on distributed mass small file aggregation, and belongs to the technical field of distributed storage. Comprising the steps of obtaining a to-be-deleted aggregated large file; extracting the small files meeting the pre-judgment condition as the small files in the same period; aggregating the same-period small files into same-period aggregated large files according to a first preset storage space; judging whether the storage space of the extracted to-be-deleted aggregated large file is smaller than a second preset storage space or not, if so, marking the extracted to-be-deleted aggregated large file as a to-be-merged aggregated large file, and merging the to-be-merged aggregated large file by taking the storage space smaller than the first preset storage space as a constraint and combining a binary adaptive merging algorithm and a greedy algorithm to obtain a to-be-merged aggregated large file; obtaining a target aggregation large file; otherwise, outputting the extracted aggregation large file as a target aggregation large file; and deleting the same-period aggregation large file and the target aggregation large file to a garbage recycle bin, and releasing the storage space of the same-period aggregation large file and the target aggregation large file.

Description

Technical Field

[0001] The present invention belongs to the field of distributed storage technology, and in particular relates to a garbage collection method and device based on distributed massive small file aggregation. Background Art

[0002] In distributed systems, massive small file aggregation is an optimization strategy used to address the challenges brought by the storage and management of a large number of small files. In many large-scale storage systems, there are a large number of massive small files, which occupy a large amount of storage space.

[0003] Garbage collection based on distributed massive small file aggregation is a recycling technology specifically for a large number of small files in distributed storage systems. In these systems, small files not only consume a lot of storage space, but also have a negative impact on the performance of the file system. This garbage collection method intelligently aggregates these small files and recycles them at the appropriate time, effectively improving storage utilization and system performance, improving storage efficiency, and effectively releasing storage space.

[0004] However, when the existing massive small file aggregation is writing business, the small files that meet the aggregation strategy are no longer written into their corresponding OBJs, but are written into a special type of file (hereinafter referred to as aggregated large files) in a tightly arranged manner (OBJ alignment). When reading files, the source file OBJ is no longer read, but the aggregated small file data is read from the OBJ of the aggregated large file. Each aggregated large file can store up to 5120 small files and the maximum size of the aggregated large file is 512M. However, after the aggregated large file deletes the small files, the aggregated large file cannot promptly and effectively release the space occupied by the deleted small files, resulting in a waste of space resources and an inability to effectively utilize the storage space. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide a garbage collection method and device based on distributed massive small file aggregation, which can effectively improve the utilization of storage space and system performance, improve storage efficiency, effectively release storage space, and reduce the complexity of storage space management.

[0006] In a first aspect of the present invention, a garbage collection method based on distributed massive small file aggregation is proposed, the method comprising:

[0007] S1, obtain multiple aggregated large files to be deleted;

[0008] S2, extracting small files that meet the pre-judgment conditions from each of the aggregated large files to be deleted as small files of the same period;

[0009] S3, aggregating small files with the same life cycle into large files with the same life cycle according to the first preset storage space;

[0010] S4, determining whether there is an aggregated large file to be deleted that is smaller than the second preset storage space in the storage space of the extracted aggregated large file to be deleted; if so, marking the aggregated large file to be deleted that is smaller than the second preset storage space as an aggregated large file to be merged, and proceeding to step S5; otherwise, outputting the extracted aggregated large file to be deleted as a target aggregated large file, and proceeding to step S6;

[0011] S5, taking the storage space smaller than the first preset storage space as a constraint, combining the binary adaptive merging algorithm and the greedy algorithm to merge the large aggregated files to obtain a target large aggregated file;

[0012] S6, deleting the large files aggregated in the same period and the target large files aggregated in the same period to a garbage collection bin, thereby releasing storage space for the large files aggregated in the same period and the target large files aggregated in the same period.

[0013] Furthermore, the pre-judgment conditions include the same creator, the same creation time period, the same write frequency, the same read frequency and the same storage hard disk type.

[0014] Further, the to-be-deleted aggregated large files that are smaller than the second preset storage space are marked as to-be-merged aggregated large files, specifically:

[0015] In the metadata, the flag of the to-be-deleted aggregated large files smaller than the second preset storage space is modified to flag=1, so as to mark the to-be-deleted aggregated large files smaller than the second preset storage space as to-be-merged aggregated large files.

[0016] Furthermore, S5 specifically includes:

[0017] S501, obtaining storage space for the large aggregated files to be merged:

[0018] S=[S 1 ,S 2 ,S 3 ,...,S n ]

[0019] Among them, S represents the storage space of the large files to be merged and aggregated, S i represents the storage space size of the i-th large file to be merged and aggregated, and n represents the total number of large files to be merged and aggregated, where i=1…n;

[0020] S502, calculating the occupied space of each to-be-merged aggregated large file relative to the remaining space of the first preset storage space:

[0021] R i =512-S i

[0022] Among them, R iIndicates the remaining space of the i-th large aggregated file to be merged relative to the first preset storage space;

[0023] S503, with the minimum remaining space as the goal, a merging objective function is established by combining the binary adaptive merging algorithm and the greedy algorithm. The merging objective function is specifically:

[0024]

[0025] Among them, S j represents the storage space size of the jth large file to be merged, || represents the absolute value symbol, ∑ represents the sum symbol, and min represents the minimum value;

[0026] S504, determining the optimal combination of the large aggregated files to be merged by a dynamic programming algorithm in combination with the objective function, and outputting a merge indicator variable of the large aggregated files to be merged representing the optimal combination;

[0027] S505 , merging the to-be-merged aggregated large files according to the to-be-merged aggregated large file merging indicator variable to obtain a target aggregated large file.

[0028] Furthermore, S504 is specifically:

[0029] The following formula is used to solve the indicator variable for merging large aggregated files to be merged:

[0030] DP[Z]=max(DP[Z],DP[ZS i ]+S i )

[0031]

[0032] Among them, DP represents dynamic programming array, x i,j Indicates the merge indicator variable for the large aggregated files to be merged. max indicates the maximum value. i,j =1 means merge S j and S i , x i,j =0 means no merging S j and S i , Z represents the total remaining space.

[0033] In a second aspect of the present invention, a garbage collection device based on distributed massive small file aggregation is proposed, which is used to implement any of the garbage collection methods based on distributed massive small file aggregation in the first aspect, and the device includes:

[0034] An acquisition module is used to acquire multiple large aggregated files to be deleted;

[0035] An extraction module is used to extract small files that meet the pre-judgment conditions from each large aggregate file to be deleted as small files of the same period;

[0036] An aggregation module, used for aggregating small files with the same life cycle into large files with the same life cycle according to a first preset storage space;

[0037] A judgment module is used to judge whether there is an aggregated large file to be deleted that is smaller than a second preset storage space in the storage space of the extracted aggregated large file to be deleted. If so, the aggregated large file to be deleted that is smaller than the second preset storage space is marked as an aggregated large file to be merged and enters the merging module; otherwise, the extracted aggregated large file to be deleted is output as a target aggregated large file and enters the deletion module;

[0038] A merging module, used to merge the large files to be merged and aggregated by combining a binary adaptive merging algorithm and a greedy algorithm, with a constraint of being smaller than a first preset storage space, to obtain a target aggregated large file;

[0039] The deletion module is used to delete the large files aggregated in the same period and the target large files aggregated in the same period to the garbage recycling bin, thereby releasing the storage space of the large files aggregated in the same period and the target large files aggregated in the same period.

[0040] Furthermore, the pre-judgment conditions include the same creator, the same creation time period, the same write frequency, the same read frequency and the same storage hard disk type.

[0041] Further, the to-be-deleted aggregated large files that are smaller than the second preset storage space are marked as to-be-merged aggregated large files, specifically:

[0042] In the metadata, the flag of the to-be-deleted aggregated large files smaller than the second preset storage space is modified to flag=1, so as to mark the to-be-deleted aggregated large files smaller than the second preset storage space as to-be-merged aggregated large files.

[0043] Furthermore, the merging module is specifically used for:

[0044] Get the storage space for the large files to be merged and aggregated:

[0045] S=[S 1 ,S 2 ,S 3 ,...,S n ]

[0046] Among them, S represents the storage space of the large files to be merged and aggregated, S i represents the storage space size of the i-th large file to be merged and aggregated, and n represents the total number of large files to be merged and aggregated, where i=1…n;

[0047] Calculate the remaining space of the first preset storage space relative to the occupied space of each large file to be merged and aggregated:

[0048] R i =512-S i

[0049] Among them, R i Indicates the remaining space of the i-th large aggregated file to be merged relative to the first preset storage space;

[0050] Taking the minimum remaining space as the goal, the merging objective function is established by combining the binary adaptive merging algorithm and the greedy algorithm. The specific merging objective function is:

[0051]

[0052] Among them, S j represents the storage space size of the jth large file to be merged, || represents the absolute value symbol, ∑ represents the sum symbol, and min represents the minimum value;

[0053] Combined with the objective function, the optimal combination of the large aggregated files to be merged is determined through a dynamic programming algorithm, and a merge indicator variable of the large aggregated files to be merged representing the optimal combination is output;

[0054] The large aggregate files to be merged are merged according to the merge indicator variable of the large aggregate files to be merged to obtain the target large aggregate file.

[0055] Furthermore, in combination with the objective function, the optimal combination of the large aggregated files to be merged is determined through a dynamic programming algorithm, and the merge indicator variable of the large aggregated files to be merged representing the optimal combination is output as follows:

[0056] The following formula is used to solve the indicator variable for merging large aggregated files to be merged:

[0057] DP[Z]=max(DP[Z],DP[ZS i ]+S i )

[0058]

[0059] Among them, DP represents dynamic programming array, x i,j Indicates the merge indicator variable for the large aggregated files to be merged. max indicates the maximum value. i,j =1 means merge S j and S i , x i,j =0 means no merging S j and S i , Z represents the total remaining space.

[0060] A third aspect of the present invention provides a garbage collection system based on distributed aggregation of massive small files, comprising: a memory and one or more processors.

[0061] One or more application programs are stored in the memory, and the one or more application programs are suitable for being executed by one or more processors to implement the garbage collection method based on distributed massive small file aggregation of the first aspect.

[0062] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which can be loaded and executed by a processor to implement the garbage collection method based on distributed massive small file aggregation according to the first aspect.

[0063] The beneficial effects of the present invention are as follows:

[0064] In an embodiment of the present invention, a plurality of aggregated large files to be deleted are first obtained and the life cycle of small files is predicted according to a prediction condition. Then, the aggregated large files aggregated with the same life cycle are judged to determine whether there are aggregated large files to be deleted that are smaller than a second preset storage space in the storage space of the extracted aggregated large files to be deleted. If so, the aggregated large files to be deleted that are smaller than the second preset storage space are marked as aggregated large files to be merged. Otherwise, the extracted aggregated large files to be deleted are output as target aggregated large files, and the merging strategy of the aggregated large files to be merged is optimized by combining a binary adaptive merging algorithm and a greedy algorithm. The merged aggregated large files are deleted after meeting the conditions. The releasable occupied space can be quickly calculated and evaluated, and the problem that the occupied space of multiple aggregated large files to be deleted cannot be released in time after the small files are deleted is avoided. Efficient storage space configuration is achieved. By deleting the aggregated large files and the target aggregated large files in the same period to the garbage recycling bin, the storage resources can be quickly and timely released after the files are deleted, making the operation of the entire system more efficient and economical, thereby effectively improving the storage utilization rate and system performance, and improving storage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0066] Figure 1 It is a flow chart of a garbage collection method based on distributed massive small file aggregation provided by an embodiment of the present invention;

[0067] Figure 2 It is a structural schematic diagram of a garbage collection device based on distributed massive small file aggregation provided by an embodiment of the present invention;

[0068] Figure 3 It is a structural schematic diagram of a garbage collection system based on distributed massive small file aggregation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0070] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0071] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0073] The present invention proposes a garbage collection method and device based on distributed massive small file aggregation, in order to solve the problem that after deleting small files from existing aggregated large files, the aggregated large files cannot timely and effectively release the space occupied by the deleted small files, resulting in a waste of space resources and an inability to effectively utilize the storage space.

[0074] Method Embodiment

[0075] Reference Figure 1 , showing a flow chart of a garbage collection method based on distributed massive small file aggregation provided by an embodiment of the present invention.

[0076] An embodiment of the present invention provides a garbage collection method based on distributed massive small file aggregation, the method comprising:

[0077] Specifically, the method includes steps S1 to S6:

[0078] S1, obtain multiple large aggregated files to be deleted.

[0079] Among them, the aggregated large file to be deleted refers to an already aggregated larger file containing multiple small files. These files need to be deleted due to the end of their life cycle or other reasons. By pre-obtaining the aggregated large file to be deleted, the garbage collection operation can be concentrated on a small number of larger files instead of processing a large number of small files one by one. This method helps to optimize the entire garbage collection process and ultimately improve the storage space recovery speed and system performance.

[0080] S2, extracting small files that meet the pre-judgment conditions from the aggregated large files to be deleted as small files of the same period.

[0081] Among them, small files in the same cycle refer to small files generated or used in the same life cycle. These files have similar attributes such as creators, creation time periods, read and write frequencies, or storage hard disk types. These files are regarded as a whole and are processed and aggregated in a unified manner.

[0082] It should be noted that by extracting these small files with the same life cycle and aggregating them, the fragmentation phenomenon in the file system can be effectively reduced, and the management and performance overhead caused by processing a large number of scattered small files one by one can be reduced. After aggregating small files of the same period into large files, the system can not only perform garbage collection more efficiently, but also optimize the utilization of storage space and improve the system's I / O performance and overall efficiency.

[0083] In a possible implementation, the pre-judgment conditions include the same creator, the same creation time period, the same write frequency, the same read frequency, and the same storage hard disk type.

[0084] S3, aggregating small files in the same life cycle into large files in the same life cycle according to the first preset storage space.

[0085] Among them, the first preset storage space refers to a storage capacity threshold pre-set in a system, which is used to limit the size of the aggregated file. The same-cycle aggregated large file refers to a large file that is formed by merging multiple small files belonging to the same life cycle through an aggregation operation. These small files are usually similar in terms of file creation time, read and write frequency, storage media, etc., and within the same life cycle, by setting a reasonable storage space upper limit (such as 512MB), it is ensured that the aggregated large file will not be too large to avoid burdening the file system. This size limit helps to optimize the system's I / O operations and reduce the storage pressure caused by large files.

[0086] It should be noted that those skilled in the art can set the size of the first preset storage space according to actual needs, and the present invention does not limit this.

[0087] Specifically, the first preset storage space is 512M.

[0088] S4, determine whether there is any other aggregated large file to be deleted that is smaller than the second preset storage space in the storage space of the extracted aggregated large file to be deleted. If so, mark the aggregated large file to be deleted that is smaller than the second preset storage space as an aggregated large file to be merged, and proceed to step S5; otherwise, output the extracted aggregated large file to be deleted as the target aggregated large file, and proceed to step S6.

[0089] Among them, the second preset storage space refers to another storage capacity threshold set in the system, which is used to determine whether the extracted large aggregate files to be deleted need to be further merged. The large aggregate files to be merged refer to those large aggregate files that are marked as needing further merger in the garbage collection process. These files are usually smaller than a set threshold in size, so they cannot be processed independently or deleted directly. Instead, they need to be merged with other files into larger aggregate files so that subsequent garbage collection operations can be performed more efficiently.

[0090] It should be noted that by introducing the second preset storage space, small files that need to be further merged can be effectively screened out to avoid fragmentation problems. The merging operation of smaller files can not only improve storage efficiency, but also reduce the management burden of the file system. Through this strategy, the system can handle files of different sizes more flexibly and ensure that small files do not occupy storage space for a long time.

[0091] It should be noted that those skilled in the art can set the size of the second preset storage space according to actual needs, and the present invention is not limited thereto.

[0092] Specifically, the first preset storage space is 256M.

[0093] In a possible implementation manner, the aggregated large files to be deleted that are smaller than the second preset storage space are marked as aggregated large files to be merged, specifically:

[0094] In the metadata, the flag of the to-be-deleted aggregated large files smaller than the second preset storage space is modified to flag=1, so as to mark the to-be-deleted aggregated large files smaller than the second preset storage space as to-be-merged aggregated large files.

[0095] S5, taking the storage space smaller than the first preset storage space as a constraint, combining the binary adaptive merging algorithm and the greedy algorithm to merge the large files to be merged, and obtaining the target large file.

[0096] Among them, the first preset storage space is a file size threshold pre-set in the system, which is used to limit the maximum storage size of the target aggregated large file after the merger. The binary adaptive merge algorithm is an algorithm for merging files. It uses the idea of ​​binary search to divide files into different groups according to size, gradually narrow the merge range, and find a suitable file combination. The greedy algorithm is an optimization algorithm that makes the current best choice each time in order to achieve the optimal solution globally.

[0097] It should be noted that by combining the binary adaptive merge algorithm and the greedy algorithm, the optimal merge strategy can be quickly found when processing a large number of small files. The binary adaptive algorithm effectively narrows the range of files to be merged, making the merge operation more efficient. The greedy algorithm ensures maximum storage space utilization by selecting the best file to merge each time. This combined algorithm can balance efficiency and accuracy, reduce file fragmentation in the system, improve storage space utilization, and also reduce the complexity of file system management.

[0098] In a possible implementation, S5 specifically includes:

[0099] S501, obtaining storage space for the large aggregated files to be merged:

[0100] S=[S 1 ,S 2 ,S 3 ,...,S n ]

[0101] Among them, S represents the storage space of the large files to be merged and aggregated, S i represents the storage space size of the i-th large file to be merged and aggregated, and n represents the total number of large files to be merged and aggregated, where i=1…n.

[0102] S502, calculating the occupied space of each to-be-merged aggregated large file relative to the remaining space of the first preset storage space:

[0103] R i =512-S i

[0104] Among them, R i Indicates the remaining space of the i-th large aggregate file to be merged relative to the first preset storage space.

[0105] S503, with the minimum remaining space as the goal, a merging objective function is established by combining the binary adaptive merging algorithm and the greedy algorithm. The merging objective function is specifically:

[0106]

[0107] Among them, S j represents the storage space size of the jth large aggregated file to be merged, || represents the absolute value symbol, ∑ represents the summation symbol, and min represents the minimum value.

[0108] S504, combining the objective function, determining the optimal combination of the large aggregated files to be merged through a dynamic programming algorithm, and outputting a merge indicator variable of the large aggregated files to be merged representing the optimal combination.

[0109] Among them, the objective function is a mathematical formula or algorithm used to measure the best solution to a problem. Dynamic programming is an algorithmic idea that divides complex problems into smaller sub-problems and stores the solutions to the sub-problems to avoid repeated calculations, thereby efficiently finding the global optimal solution. This refers to those large aggregated files that have been marked as needing to be merged. Usually, they are small in size (such as less than 256MB) and need to be further merged to improve storage efficiency. The merge indicator variable is a marker used to indicate which files need to be merged. These output variables guide the system to perform specific file merging operations in subsequent operations. The variables are usually binary, 1 indicates that a merge is required, and 0 indicates no merge.

[0110] In a possible implementation manner, S504 specifically includes:

[0111] The following formula is used to solve the indicator variable for merging large aggregated files to be merged:

[0112] DP[Z]=max(DP[Z],DP[ZS i ]+S i )

[0113]

[0114] Among them, DP represents dynamic programming array, x i,j Indicates the merge indicator variable for the large aggregated files to be merged. max indicates the maximum value. i,j =1 means merge Sj and S i , x i,j =0 means no merging S j and S i , Z represents the total remaining space.

[0115] S505 , merging the to-be-merged aggregated large files according to the to-be-merged aggregated large file merging indicator variable to obtain a target aggregated large file.

[0116] The target aggregate large file is the final result of the merge operation. Multiple smaller files to be merged are merged to generate a larger file. The size of this target aggregate large file is close to the upper limit set by the system (such as 512MB) and is a file that is ready for subsequent operations (such as deletion or further management).

[0117] It should be noted that through dynamic programming and merging indicator variables, the system can efficiently process a large number of small files and aggregate them into larger target files. This process first accurately calculates the size and remaining space of each file to be merged, combines the objective function to determine the optimal file combination, and finally generates a larger file through the merging operation. This reduces the waste of computing and storage resources, improves the overall performance of garbage collection and the fluency of file operations.

[0118] S6, deleting the large files aggregated in the same period and the target large files aggregated in the same period to a garbage collection bin, thereby releasing storage space for the large files aggregated in the same period and the target large files aggregated in the same period.

[0119] The garbage bin is an area in the system used to temporarily store files to be deleted, similar to the recycle bin in the operating system. Before the files are completely removed from the storage system, they are placed in the garbage bin and wait for final deletion. By moving the same-cycle aggregated large files and the target aggregated large files into the garbage bin and freeing up storage space, the system can efficiently recycle files that are no longer in use and reduce storage pressure.

[0120] In actual application, small files with the same life cycle are first aggregated into large files through pre-judgment conditions, and these small files with the same life cycle are deleted to release storage space. For small files with different life cycles, the system marks whether they need to be further merged based on the file size. Files marked as needing to be merged will be merged into a large aggregate file of nearly 512MB, and then deleted to release storage space. The whole process aims to reduce fragmentation and improve storage efficiency by aggregating and merging small files, and finally recycle all files marked for deletion.

[0121] When predicting whether it is the same cycle condition, first classify and predict according to the different attributes of small files (such as creator, creation time, write frequency, read frequency, storage medium type, etc.). These attributes include whether the small file is stored on an SSD disk or an HDD disk, the creator and creation time of the file, and the read and write frequency of the file. Based on these conditions, the system will filter out small files that meet a specific life cycle (for example, sfile1 to sfile5120), and aggregate similar small files through these attributes for subsequent processing or recycling.

[0122] During the merging process, the system first obtains the size of the aggregated large file in the recycling directory, and then determines whether the size of the aggregated large file is less than 256MB. If the file size is less than 256MB, it is marked as Flag = 1, indicating that the file needs to be further merged; if the file size is greater than or equal to 256MB, it is marked as Flag = 0, indicating that the file does not need to be merged and can be directly processed.

[0123] The beneficial effects of the present invention are as follows:

[0124] In an embodiment of the present invention, a plurality of aggregated large files to be deleted are first obtained and the life cycle of small files is predicted according to a prediction condition. Then, the aggregated large files aggregated with the same life cycle are judged to determine whether there are aggregated large files to be deleted that are smaller than a second preset storage space in the storage space of the extracted aggregated large files to be deleted. If so, the aggregated large files to be deleted that are smaller than the second preset storage space are marked as aggregated large files to be merged. Otherwise, the extracted aggregated large files to be deleted are output as target aggregated large files, and the merging strategy of the aggregated large files to be merged is optimized by combining a binary adaptive merging algorithm and a greedy algorithm. The merged aggregated large files are deleted after meeting the conditions. The releasable occupied space can be quickly calculated and evaluated, and the problem that the occupied space of multiple aggregated large files to be deleted cannot be released in time after the small files are deleted is avoided. Efficient storage space configuration is achieved. By deleting the aggregated large files and the target aggregated large files in the same period to the garbage recycling bin, the storage resources can be quickly and timely released after the files are deleted, making the operation of the entire system more efficient and economical, thereby effectively improving the storage utilization rate and system performance, and improving storage efficiency.

[0125] Device Embodiment

[0126] Reference Manual Attached Figure 2 , showing a structural schematic diagram of a garbage collection device based on distributed massive small file aggregation provided by an embodiment of the present invention.

[0127] Another specific embodiment of the present invention discloses a garbage collection device based on distributed massive small file aggregation, which is used to implement any of the garbage collection methods based on distributed massive small file aggregation in the method embodiments. The garbage collection device 20 based on distributed massive small file aggregation includes:

[0128] The acquisition module 201 is used to acquire multiple aggregated large files to be deleted.

[0129] The extraction module 202 is used to extract small files that meet the pre-judgment condition from each of the aggregated large files to be deleted as small files of the same period.

[0130] The aggregation module 203 is used to aggregate small files with the same life cycle into large files with the same life cycle according to the first preset storage space.

[0131] The judgment module 204 is used to judge whether there is an aggregated large file to be deleted that is smaller than the second preset storage space in the storage space of the extracted aggregated large file to be deleted. If so, the aggregated large file to be deleted that is smaller than the second preset storage space will be marked as an aggregated large file to be merged and enter the merging module. Otherwise, the extracted aggregated large file to be deleted will be output as the target aggregated large file and enter the deletion module.

[0132] The merging module 205 is used to merge the large files to be merged and aggregated, with the storage space being less than the first preset storage space as a constraint, by combining the binary adaptive merging algorithm and the greedy algorithm to obtain a target aggregated large file.

[0133] The deletion module 206 is used to delete the large files aggregated in the same period and the target large files aggregated in the same period to the garbage collection bin, thereby releasing the storage space of the large files aggregated in the same period and the target large files aggregated in the same period.

[0134] In a possible implementation, the pre-judgment conditions include the same creator, the same creation time period, the same write frequency, the same read frequency, and the same storage hard disk type.

[0135] In a possible implementation manner, the aggregated large files to be deleted that are smaller than the second preset storage space are marked as aggregated large files to be merged, specifically:

[0136] In the metadata, the flag of the to-be-deleted aggregated large files smaller than the second preset storage space is modified to flag=1, so as to mark the to-be-deleted aggregated large files smaller than the second preset storage space as to-be-merged aggregated large files.

[0137] In a possible implementation, the merging module 206 is specifically configured to:

[0138] Get the storage space for the large files to be merged and aggregated:

[0139] S=[S 1,S 2 ,S 3 ,...,S n ]

[0140] Among them, S represents the storage space of the large files to be merged and aggregated, S i represents the storage space size of the i-th large file to be merged and aggregated, and n represents the total number of large files to be merged and aggregated, where i=1…n;

[0141] Calculate the remaining space of the first preset storage space relative to the occupied space of each large file to be merged and aggregated:

[0142] R i =512-S i

[0143] Among them, R i Indicates the remaining space of the i-th large aggregated file to be merged relative to the first preset storage space;

[0144] Taking the minimum remaining space as the goal, the merging objective function is established by combining the binary adaptive merging algorithm and the greedy algorithm. The specific merging objective function is:

[0145]

[0146] Among them, S j represents the storage space size of the jth large file to be merged, || represents the absolute value symbol, ∑ represents the sum symbol, and min represents the minimum value;

[0147] Combined with the objective function, the optimal combination of the large aggregated files to be merged is determined through a dynamic programming algorithm, and a merge indicator variable of the large aggregated files to be merged representing the optimal combination is output;

[0148] The large aggregate files to be merged are merged according to the merge indicator variable of the large aggregate files to be merged to obtain the target large aggregate file.

[0149] In a possible implementation, in combination with the objective function, the optimal combination of the large aggregated files to be merged is determined by a dynamic programming algorithm, and the merge indicator variable of the large aggregated files to be merged representing the optimal combination is output as follows:

[0150] The following formula is used to solve the indicator variable for merging large aggregated files to be merged:

[0151] DP[Z]=max(DP[Z],DP[ZS i ]+S i )

[0152]

[0153] Among them, DP represents dynamic programming array, x i,j Indicates the merge indicator variable for the large aggregated files to be merged. max indicates the maximum value. i,j =1 means merge S j and S i , x i,j =0 means no merging S j and S i , Z represents the total remaining space.

[0154] The device provided in the embodiment of the present invention can implement each process and step in the above method embodiment, and can obtain the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0155] The beneficial effects of the present invention are as follows:

[0156] In an embodiment of the present invention, a plurality of aggregated large files to be deleted are first obtained and the life cycle of small files is predicted according to a prediction condition. Then, the aggregated large files aggregated with the same life cycle are judged to determine whether there are aggregated large files to be deleted that are smaller than a second preset storage space in the storage space of the extracted aggregated large files to be deleted. If so, the aggregated large files to be deleted that are smaller than the second preset storage space are marked as aggregated large files to be merged. Otherwise, the extracted aggregated large files to be deleted are output as target aggregated large files, and the merging strategy of the aggregated large files to be merged is optimized by combining a binary adaptive merging algorithm and a greedy algorithm. The merged aggregated large files are deleted after meeting the conditions. The releasable occupied space can be quickly calculated and evaluated, and the problem that the occupied space of multiple aggregated large files to be deleted cannot be released in time after the small files are deleted is avoided. Efficient storage space configuration is achieved. By deleting the aggregated large files and the target aggregated large files in the same period to the garbage recycling bin, the storage resources can be quickly and timely released after the files are deleted, making the operation of the entire system more efficient and economical, thereby effectively improving the storage utilization rate and system performance, and improving storage efficiency.

[0157] System Example

[0158] Reference Manual Attached Figure 3 , showing a structural schematic diagram of a garbage collection system based on distributed massive small file aggregation provided by an embodiment of the present invention.

[0159] The present invention proposes a garbage collection system 30 based on distributed massive small file aggregation, including: a memory 303 and one or more processors 301.

[0160] One or more application programs are stored in the memory 303 , and the one or more application programs are suitable for being executed by one or more processors 301 to implement the garbage collection method based on distributed aggregation of massive small files of the method embodiment.

[0161] The garbage collection system 30 based on distributed mass small file aggregation includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, through a bus 302.

[0162] The structure of the garbage collection system 30 based on distributed massive small file aggregation does not constitute a limitation on the embodiment of the present invention.

[0163] Processor 301 may be a CPU, a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0164] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI bus or an EISA bus, etc. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0165] The memory 303 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0166] Computer Readable Storage Medium Embodiments

[0167] The present invention proposes a computer-readable storage medium on which a computer program is stored. The computer program can be loaded and executed by a processor to implement a garbage collection method based on distributed massive small file aggregation according to the first aspect.

[0168] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A garbage collection method based on distributed massive small file aggregation, characterized in that: The garbage collection method based on distributed massive small file aggregation includes: S1, obtain multiple aggregated large files to be deleted; S2, extracting small files that meet the pre-judgment conditions from each of the aggregated large files to be deleted as small files of the same period; S3, aggregating the small files with the same life cycle into a large file with the same life cycle according to the first preset storage space; S4, determining whether there is an aggregated large file to be deleted that is smaller than the second preset storage space in the storage space of the extracted aggregated large file to be deleted; if so, marking the aggregated large file to be deleted that is smaller than the second preset storage space as an aggregated large file to be merged, and proceeding to step S5; otherwise, outputting the extracted aggregated large file to be deleted as a target aggregated large file, and proceeding to step S6; S5, merging the to-be-merged aggregated large files by combining a binary adaptive merging algorithm and a greedy algorithm with a constraint that the storage space is smaller than the first preset storage space, to obtain a target aggregated large file; S6, deleting the large files aggregated in the same period and the target large files aggregated in the same period to a garbage collection bin, thereby releasing storage space of the large files aggregated in the same period and the target large files aggregated in the same period.

2. The garbage collection method based on distributed massive small file aggregation according to claim 1 is characterized in that: The pre-judgment conditions include the same creator, the same creation time period, the same write frequency, the same read frequency and the same storage hard disk type.

3. The distributed garbage collection method for mass small file aggregation according to claim 1 is characterized in that: The step of marking the to-be-deleted aggregated large files smaller than the second preset storage space as to-be-merged aggregated large files is specifically: In the metadata, the flag of the to-be-deleted aggregated large files smaller than the second preset storage space is modified to flag=1, so that the to-be-deleted aggregated large files smaller than the second preset storage space are marked as to-be-merged aggregated large files.

4. The distributed garbage collection method for mass small file aggregation according to claim 1 is characterized in that: The S5 specifically includes: S501, obtaining the storage space of the large file to be merged and aggregated: <h2 style=";text-align:left;direction:ltr">S=[S1,S2,S3,...,S<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ] Among them, S represents the storage space of the large files to be merged and aggregated, S i represents the storage space size of the i-th large file to be merged and aggregated, and n represents the total number of large files to be merged and aggregated, where i=1…n; S502: Calculate the space occupied by each large file to be merged and aggregated relative to the remaining space of the first preset storage space: R i =512-S i Among them, R i Indicates the remaining space of the i-th large aggregated file to be merged relative to the first preset storage space; S503, taking the minimum remaining space as the goal, a merging objective function is established by combining the binary adaptive merging algorithm and the greedy algorithm. The merging objective function is specifically: Among them, S j represents the storage space size of the jth large file to be merged, || represents the absolute value symbol, ∑ represents the sum symbol, and min represents the minimum value; S504, combining the objective function, determining the optimal combination of the large aggregated files to be merged through a dynamic programming algorithm, and outputting a merge indicator variable of the large aggregated files to be merged representing the optimal combination; S505: Merge the to-be-merged aggregated large files according to the to-be-merged aggregated large files merging indicator variable to obtain the target aggregated large file.

5. The distributed garbage collection method for mass small file aggregation according to claim 1 is characterized in that: The S504 is specifically as follows: The indicator variable for merging the large aggregated files to be merged is solved by the following formula: DP[Z]=max(DP[Z],DP[Z-S i ]+S i ) Among them, DP represents dynamic programming array, x i,j Indicates the merge indicator variable for the large aggregated files to be merged. max indicates the maximum value. i,j =1 means merge S j and S i , x i,j =0 means no merging S j and S i , Z represents the total remaining space.

6. A garbage collection device based on distributed massive small file aggregation, characterized in that: The garbage collection device based on distributed massive small file aggregation includes: An acquisition module is used to acquire multiple large aggregated files to be deleted; An extraction module is used to extract small files that meet the pre-judgment conditions from each large aggregate file to be deleted as small files of the same period; An aggregation module, used for aggregating the small files with the same life cycle into a large file with the same period according to a first preset storage space; A judgment module is used to judge whether there is an aggregated large file to be deleted that is smaller than a second preset storage space in the storage space of the extracted aggregated large file to be deleted. If so, the aggregated large file to be deleted that is smaller than the second preset storage space is marked as an aggregated large file to be merged and enters the merging module; otherwise, the extracted aggregated large file to be deleted is output as a target aggregated large file and enters the deletion module; A merging module, configured to merge the large aggregated files to be merged, with the size smaller than the first preset storage space as a constraint, by combining a binary adaptive merging algorithm and a greedy algorithm, to obtain a target large aggregated file; The deletion module is used to delete the large files aggregated in the same period and the target large files aggregated in the same period to a garbage collection bin, thereby releasing the storage space of the large files aggregated in the same period and the target large files aggregated in the same period.

7. The garbage collection device based on distributed massive small file aggregation according to claim 6 is characterized in that: The pre-judgment conditions include the same creator, the same creation time period, the same write frequency, the same read frequency and the same storage hard disk type.

8. The garbage collection device based on distributed massive small file aggregation according to claim 6 is characterized in that: The step of marking the to-be-deleted aggregated large files smaller than the second preset storage space as to-be-merged aggregated large files is specifically: In the metadata, the flag of the to-be-deleted aggregated large files smaller than the second preset storage space is modified to flag=1, so that the to-be-deleted aggregated large files smaller than the second preset storage space are marked as to-be-merged aggregated large files.

9. The garbage collection device based on distributed massive small file aggregation according to claim 6 is characterized in that: The merging module is specifically used for: Get the storage space for the large file to be merged and aggregated: <h2 style=";text-align:left;direction:ltr">S=[S1,S2,S3,...,S<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ] Among them, S represents the storage space of the large files to be merged and aggregated, S i represents the storage space size of the i-th large file to be merged and aggregated, and n represents the total number of large files to be merged and aggregated, where i=1…n; Calculate the space occupied by each large file to be merged and aggregated relative to the remaining space of the first preset storage space: R i =512-S i Among them, R i Indicates the remaining space of the i-th large aggregated file to be merged relative to the first preset storage space; Taking the minimum remaining space as the goal, a merging objective function is established by combining the binary adaptive merging algorithm and the greedy algorithm. The merging objective function is specifically: Among them, S j represents the storage space size of the jth large file to be merged, || represents the absolute value symbol, ∑ represents the sum symbol, and min represents the minimum value; In combination with the objective function, an optimal combination of the large aggregated files to be merged is determined by a dynamic programming algorithm, and a merge indicator variable of the large aggregated files to be merged representing the optimal combination is output; The to-be-merged aggregated large files are merged according to the to-be-merged aggregated large file merging indicator variable to obtain the target aggregated large file.

10. The garbage collection device based on distributed massive small file aggregation according to claim 9, characterized in that: In combination with the objective function, the optimal combination of the large aggregated files to be merged is determined by a dynamic programming algorithm, and the merge indicator variable of the large aggregated files to be merged representing the optimal combination is outputted as follows: The indicator variable for merging the large aggregated files to be merged is solved by the following formula: DP[Z]=max(DP[Z],DP[Z-S i ]+S i ) Among them, DP represents dynamic programming array, x i,j Indicates the merge indicator variable for the large aggregated files to be merged. max indicates the maximum value. i,j =1 means merge S j and S i , x i,j =0 means no merging S j and S i , Z represents the total remaining space.

Images