A method and device for reorganizing aggregated objects

Abstract

The present invention provides a method and device for reorganizing aggregate objects, the method comprising: determining hollowing rates of N aggregate objects, wherein the hollowing rate of the i-th aggregate object is the ratio of the total number of junk files included in the i-th aggregate object to the total number of files included in the i-th aggregate object, i and N are positive integers, i ranges from 1 to N; according to the hollowing rates of the N aggregate objects, determining M aggregate objects whose hollowing rates are greater than or equal to a preset threshold value among the N aggregate objects, M being an integer greater than or equal to 2; reorganizing the M aggregate objects to generate a new aggregate object. Therefore, disk space can be recovered and the purpose of improving disk utilization can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for reorganizing aggregated objects. Background Art

[0002] With the rapid development of the Internet (especially mobile Internet), Internet of Things, cloud computing, big data, etc., data is growing explosively. The data generated by various applications such as social networks, mobile communications, online video and audio, e-commerce, sensor networks, scientific experiments, etc. not only have huge storage capacity, but also have significant characteristics such as various data types, large data size changes, and fast flow. They can often generate a large number of small files in the tens of millions, hundreds of millions, even billions, and tens of billions, and more often a large number of large and small files are mixed and stored. Due to the huge challenges in metadata information management, access performance, storage efficiency, etc., the storage problem of lots of small files (LOSF) has become a recognized problem in industry and academia.

[0003] To address the LOSF storage problem, a storage optimization solution combining merged storage and index files is proposed. Figure 1 On the left is the metadata information of each small file, that is, the index file, including which large object it is aggregated in, its position in the large object, and its own file size. Figure 1 The right side shows the actual file data, that is, the large object after merging and storage. Mobj1 and MobjN are both large objects after aggregation, where each large object stores many small files.

[0004] The optimization scheme combining merged storage and index files is very effective for processing the aggregation of massive small files. However, under this scheme, the operation of modifying or deleting small files may not be able to reclaim storage space in time. For example, if a small file has been aggregated in a large object and then modified or deleted, the metadata information of the small file will be deleted. Figure 1 The relevant information on the left will be deleted. The small file in the large object itself becomes garbage data and is not deleted. If the metadata information of multiple small files in the large object are deleted, there will be a lot of garbage data in the aggregated large object. Summary of the invention

[0005] The present invention provides an aggregate object reorganization method and device, which are used to solve the problem that the operation of modifying or deleting a small file may not be able to reclaim the storage space in time.

[0006] In a first aspect, the present invention provides a method for reorganizing aggregate objects, comprising: determining a hollowing rate of N aggregate objects, wherein the hollowing rate of the i-th aggregate object is the ratio of the total number of junk files included in the i-th aggregate object to the total number of files included in the i-th aggregate object, i and N are positive integers, and i ranges from 1 to N; based on the hollowing rates of the N aggregate objects, determining M aggregate objects among the N aggregate objects whose hollowing rates are greater than or equal to a preset threshold, where M is an integer greater than or equal to 2; and reorganizing the M aggregate objects to generate a new aggregate object.

[0007] Therefore, it is possible to clean up the junk files in the aggregate object after deleting the files, thereby achieving the purpose of reducing the junk files in the disk, reclaiming the disk space, and improving the disk utilization.

[0008] In a possible design, the number of junk files included in the i-th aggregation object is determined according to a bit map of the i-th aggregation object, the total number of bits included in the bit map of the i-th aggregation object is equal to the total number of files included in the i-th aggregation object, and each file in the i-th aggregation object corresponds to a bit in the bit map of the i-th aggregation object;

[0009] In the bit map of the i-th aggregate object, possible values ​​of each bit include a first value and a second value, wherein the first value indicates that the file corresponding to the bit is the junk file, and the second value indicates that the file corresponding to the bit is not the junk file.

[0010] Therefore, the purpose of reclaiming disk space quickly and efficiently is achieved through bitmap operations.

[0011] In one possible design, determining the hollowing rates of N aggregate objects includes: when determining the hollowing rate of the ith aggregate object, performing the following steps: periodically updating the bitmap of the ith aggregate object based on indexes of files included in the ith aggregate object and metadata information of the ith aggregate object, wherein the metadata information of the ith aggregate object includes metadata information of each file included in the ith aggregate object; and determining the hollowing rate of the ith aggregate object based on the updated bitmap of the ith aggregate object.

[0012] Compared with the traditional forward index, the reverse index used for marking small file junk data reduces the process of traversing all small files once, greatly reduces the number of disk accesses, and makes the subsequent junk marking process more efficient.

[0013] In a possible design, the bitmap of the i-th aggregation object is periodically updated according to the index of the file included in the i-th aggregation object and the metadata information of the i-th aggregation object, including: judging in turn whether the metadata information of the file corresponding to each index is deleted; when the metadata information of the file corresponding to the index is deleted, replacing the value of the corresponding bit of the file corresponding to the index in the bitmap of the i-th aggregation object from the second value to the first value; when the metadata information corresponding to the index is not deleted, the value of the corresponding bit of the file corresponding to the index in the bitmap of the i-th aggregation object remains unchanged and is the second value.

[0014] In one possible design, determining the hollowing rate of N aggregated objects includes: when determining the hollowing rate of the i-th aggregated object, performing the following steps: after deleting metadata information of a first file, updating a bit map of the i-th aggregated object, wherein the first file is a file in the i-th aggregated object; and determining the hollowing rate of the i-th aggregated object based on the updated bit map of the i-th aggregated object.

[0015] Therefore, the efficiency of adding empty objects to the heap can be greatly improved, thereby further improving the efficiency of aggregate object integration.

[0016] In one possible design, updating the bitmap of the i-th aggregate object includes: replacing the value of the corresponding bit of the first file in the bitmap of the i-th aggregate object from the second value to the first value.

[0017] In a possible design, it also includes: before reorganizing the M aggregate objects to generate a new aggregate object, determining that the M aggregate objects meet a preset condition: the preset condition includes at least one of the following: the sum of the hollowing rates of the M aggregate objects is greater than or equal to a second threshold; or, the average value of the hollowing rates of the M aggregate objects is greater than or equal to a third threshold; or, M≥Q, Q is a preset value, Q<N, and Q is an integer greater than or equal to 2.

[0018] In a second aspect, the present application further provides a device. The device can execute the above method design. The device can be a chip or circuit capable of executing the function corresponding to the above method, or a device including the chip or circuit.

[0019] In a possible implementation, the device includes: a memory for storing computer executable program code; and a processor, the processor being coupled to the memory. The program code stored in the memory includes instructions, and when the processor executes the instructions, the device or a device equipped with the device executes the method in any of the above possible designs.

[0020] The device may further include a communication interface, which may be a transceiver, or, if the device is a chip or a circuit, the communication interface may be an input / output interface of the chip, such as an input / output pin.

[0021] In one possible design, the device includes corresponding functional units, which are respectively used to implement the steps in the above method. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more units corresponding to the above functions.

[0022] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program runs on a device, the method in any possible design described above is executed.

[0023] In addition, the technical effects brought about by any implementation method in the second to third aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of metadata information of an aggregate object and files included in the aggregate object provided by an embodiment of the present invention;

[0025] Figure 2A One of the schematic diagrams of aggregation objects provided in an embodiment of the present invention;

[0026] Figure 2B The second schematic diagram of the aggregation object provided by the embodiment of the present invention;

[0027] Figure 3 A flowchart summarizing the method for reorganizing aggregated objects provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a bit map provided in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram showing that the metadata information of the aggregated object provided by the embodiment of the present invention includes a reverse index of a file;

[0030] Figure 6 A schematic diagram of the maximum heap arrangement of aggregate objects provided by an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of a specific process of the method for reorganizing aggregated objects provided by an embodiment of the present invention;

[0032] Figure 8 One of the structural schematic diagrams of a device provided by an embodiment of the present invention;

[0033] Fig. 9 The second structural diagram of a device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0036] By analyzing the characteristics of small files, massive small file storage focuses on solving the following two problems:

[0037] First, the organization and management of metadata information for massive small files: For data in the tens of billions, if each file's metadata is calculated as 100B, the total amount of metadata data will reach 1TB, which far exceeds the memory size of a single server at present. If local persistent device storage is used, the metadata information query and addressing of each file access request must be efficiently met. In order to avoid single-point metadata information query and addressing, a backup metadata information node is also required. At the same time, a single set of metadata information servers also becomes a bottleneck for the expansion of the entire cluster scale. Alternatively, an independent storage cluster can be used to store and manage metadata information. When the status of a data storage node changes, the corresponding metadata information should be notified in a timely manner to make the change.

[0038] Second, the storage and management of local disk files (local storage engine): For common Linux file systems, reading a file usually requires three disk IOs (reading directory metadata information into memory, loading the file's inode into memory, and finally reading the actual file content); according to the current mainstream 2TB to 4TB serial hard disks (Serial ATA, SATA), 2kw to 4kw 100KB files can be stored. Due to the large number of files, it is impossible to cache the inode information of all directories and files into memory. It is difficult to achieve the ideal state that only one disk IO is required for each image reading, and the long tail phenomenon makes the hot spot cache have no obvious effect; when the request is addressed to a specific disk, how to reduce the number of file access IO times and efficiently respond to requests (especially reads) has become another problem that must be solved.

[0039] For the above two problems, a common solution is to use an optimized storage solution that combines merged storage with index files. This solution has many advantages, such as: the size of the merged large object is usually 64MB, or even larger, so the number of merged large objects stored on a single disk is much smaller than the number of original small files, and their inode information can all be cached in memory, which reduces unnecessary disk IO. In addition, index files usually have less data volume, and generally only include the identifier of the merged large object where the small file is located, its offset in the merged large object, and the size of the small file. These index files can also be loaded into the memory. When reading small files, the index files in the memory are accessed first, and then the actual file data is accessed according to the merged large object and offset read from the index file, so that the purpose of reading the file data can be achieved by reading the disk IO only once. At the same time, when a single small file is stored independently, the file system usually saves all its metadata information, such as its size, owner, creation date, modification date, access rights and other all other file information, and some of this information may not be used by the business. Therefore, when merging small files into large objects, the metadata information of small files can be filtered according to actual business needs before merging, which further reduces the space occupied by the metadata information of small files.

[0040] like Figure 2A The figure shows one of the schematic diagrams of the aggregate object. Each square corresponds to a file. White indicates that the file is not a junk file, also known as a non-junk file, also known as a valid file. Black indicates that the file is a junk file. The initially generated aggregate object does not include junk files.

[0041] like Figure 2BThe second schematic diagram of the aggregate object is shown. After the aggregate object is generated, as the business progresses, the files in the aggregate object may be constantly modified and deleted. As mentioned above, these files in the aggregate object will become junk files. At this time, the aggregate object includes several junk files and several non-junk files.

[0042] The method provided by the present invention can be used to quickly calculate the void rate of aggregated objects in real time, and then the junk files in the aggregated objects can be cleaned up efficiently and quickly to recycle storage space. Figure 3 As shown, the present invention provides an aggregate object reorganization method, which can be executed by a server or a processor, and the method includes:

[0043] Step 300: The server determines hollowing rates of N aggregate objects, where the hollowing rate of the i-th aggregate object is the ratio of the total number of junk files included in the i-th aggregate object to the total number of files included in the i-th aggregate object, i and N are positive integers, and i ranges from 1 to N.

[0044] Considering that the sizes of small files are similar in the case of massive small files, the number of junk files can be used as an indicator to measure the hollowing rate of aggregated objects. Therefore, the definition is:

[0045] α=n / N; where α is the hollowing rate of the aggregate object, n is the total number of junk files, and N is the total number of files included in the aggregate object.

[0046] In a possible design, the number of junk files included in the ith aggregate object is determined according to a bitmap of the ith aggregate object, the total number of bits included in the bitmap of the ith aggregate object is equal to the total number of files included in the ith aggregate object, and each file in the ith aggregate object corresponds to a bit in the bitmap of the ith aggregate object; in the bitmap of the ith aggregate object, the possible values ​​of each bit include a first value and a second value, wherein the first value indicates that the file corresponding to the bit is a junk file, and the second value indicates that the file corresponding to the bit is not a junk file, or is a valid file, or is a non-junk file.

[0047] The basic idea of ​​the bitmap is to use a bit to mark the value (Value) corresponding to a certain element (Key). Since bits are used as units to store data, storage space can be greatly saved. Here, the present invention uses one bit to represent the data of a file, and a bit value of 0 marks the file as a junk file, and a bit value of 1 marks the file as not a junk file, or a non-junk file, or a valid file. Alternatively, a bit value of 0 marks the file as not a junk file, or a non-junk file, or a valid file, and a bit value of 1 marks the file as a junk file. Exemplarily, Figure 4 for Figure 2B The corresponding bit map, where the bit value is 1, marking the file as a junk file, and the bit value is 0, marking the file as a valid file.

[0048] Therefore, the purpose of reclaiming disk space quickly and efficiently is achieved through bitmap operations. In addition, thousands or even tens of thousands of small files are generally stored in aggregate objects. The use of bitmaps not only greatly reduces the memory consumption generated during the marking process, but also all operations on the bitmap can be completed through bit operations, making the operation efficiency of the bitmap extremely high during the marking and cleaning process, and the hollowing rate of aggregate objects can be calculated in real time, and aggregate objects with large hollowing rates can be quickly and efficiently screened out. Compared with background thread scanning calculations, the efficiency is greatly improved without consuming system resources.

[0049] For example, if an aggregate object aggregates N small files, and generally an int occupies 32 bits in a computer system, then you only need to apply for a length of An int array is initialized to 0. For example, if an int array tmp is applied for an aggregate object, then:

[0050] tmp[0]: indicates the 0th to 31st small files

[0051] tmp[1]: indicates the 32nd to 63rd small files

[0052] tmp[2]: indicates the 64th to 95th small files ......

[0054] It also provides a bitmap read and write interface to implement atomic operations and support multi-threaded reading and writing.

[0055] Furthermore, the hollowing rates of N aggregated objects may be determined by, but are not limited to, the following two solutions, which are described below by taking the hollowing rate of the i-th aggregated object as an example:

[0056] Solution 1 (also known as the backend solution):

[0057] The server periodically updates the bitmap of the i-th aggregate object according to the index of the file included in the i-th aggregate object and the metadata information of the i-th aggregate object, where the metadata information of the i-th aggregate object includes the metadata information of each file included in the i-th aggregate object, and determines the hollowing rate of the i-th aggregate object according to the updated bitmap of the i-th aggregate object.

[0058] Exemplarily, when the bitmap of the i-th aggregate object is periodically updated according to the index of the file included in the i-th aggregate object and the metadata information of the i-th aggregate object, it is determined in turn whether the metadata information of the file corresponding to each index is deleted. When the metadata information of the file corresponding to the index is deleted, the value of the corresponding bit of the file corresponding to the index in the bitmap of the i-th aggregate object is replaced from the second value to the first value. When the metadata information corresponding to the index is not deleted, the value of the corresponding bit of the file corresponding to the index in the bitmap of the i-th aggregate object remains unchanged and is the second value.

[0059] For example, the metadata information of the aggregate object includes the following: Figure 1 The content shown can also be added to record the list of all files aggregated into the aggregate object as the index of the files included in the aggregate object, also known as the reverse index of the file, which serves as the basis for subsequent tag cleaning, such as Figure 5 shown.

[0060] The metadata information of the aggregate object is stored in the kv database, and the query efficiency is very high. The query time complexity of the metadata information of a single file is O(1), and the query time complexity of all files merged in the entire aggregate object is O(N). Figure 4 As shown, if the metadata information of a file does not exist, the bit position corresponding to the small file is set to 1. If it exists, it remains unchanged and is still set to 0. That is, 1 is used to represent junk files in the aggregate object, and 0 is used to represent valid files in the aggregate object.

[0061] Therefore, by periodically scanning the reverse indexes of all aggregated objects through the background thread and updating the bitmap corresponding to the aggregated objects, the hollowing rate of the aggregated objects can be statistically calculated. In addition, when a large number of small files are merged into an aggregated object, the list of stored small files is saved in the aggregated object as the reverse index of the small files. Compared with the traditional forward index, the reverse index used for marking small file junk data reduces the process of traversing all small files once, greatly reduces the number of disk accesses, and makes the subsequent junk marking process more efficient.

[0062] Solution 2 (front-end solution):

[0063] After deleting the metadata information of the first file, the bitmap of the i-th aggregation object is updated, the first file is a file in the i-th aggregation object, and the hollowing rate of the i-th aggregation object is determined according to the updated bitmap of the i-th aggregation object.

[0064] Exemplarily, when updating the bitmap of the i-th aggregate object, the value of the corresponding bit of the first file in the bitmap of the i-th aggregate object is replaced from the second value to the first value.

[0065] When the first file is deleted or modified, the metadata information of the first file will be deleted. When the first file is modified, the aggregate object where the modified first file is located is different from the aggregate object where the first file is located before the modification. After the aggregate object is generated, the files in the aggregate object can be read, but the files in the aggregate object cannot be modified or deleted.

[0066] Therefore, the front-end updates the bitmap of the aggregate object and calculates the hollowing rate of the aggregate object while performing the deletion operation. The bitmap operation is very efficient. Compared with the background thread scheduled scanning in Solution 1, the hollowing rate of the aggregate object is calculated while performing the deletion operation, which can greatly improve the efficiency of adding hollow objects to the heap, thereby further improving the efficiency of aggregate object integration.

[0067] Step 310: The server determines, according to the hollowing rates of the N aggregated objects, M aggregated objects whose hollowing rates are greater than or equal to a preset threshold among the N aggregated objects, where M is an integer greater than or equal to 2.

[0068] For example, define the aggregate object hollowing rate threshold β. When α≥β, add the aggregate object to the aggregate object reorganization maximum heap. The maximum heap is sorted by the hollowing rate of the aggregate object. The top element of the heap is the aggregate object with the largest hollowing rate, so garbage cleaning and data reorganization should be prioritized. The schematic diagram of the maximum heap arrangement of aggregate objects is as follows: Figure 6 shown. Figure 6 Where α1 to α7 represent the hollowing rates of the seven aggregated objects, and:

[0069] α1≥α2&α1≥α3

[0070] α2≥α4&α2≥α5

[0071] α3≥α6&α3≥α7

[0072] In terms of implementation, the aggregate object hollowing rate threshold β can be dynamically adjusted according to the number of elements in the maximum heap of aggregate object reorganization. When there are too few elements in the maximum heap, the aggregate object hollowing rate threshold β can be reduced to speed up the entry of hollow aggregate objects into the heap and speed up the recovery of garbage data; when there are too many elements in the maximum heap, the aggregate object hollowing rate threshold β can be increased to slow down the entry of hollow aggregate objects into the heap and reduce the system pressure that may be caused during the reorganization of aggregate objects.

[0073] The maximum heap is used in the process of aggregate object reorganization. The top element of the heap is the aggregate object with the largest void rate. In this way, each time the top element of the heap is taken from the maximum heap, the aggregate object with the largest void rate will be processed first. This speeds up the recovery and cleanup of disk file fragments, improves disk space utilization, and can dynamically adjust the speed of garbage cleaning according to the number of elements in the maximum heap.

[0074] Step 320: The server reorganizes the M aggregate objects to generate a new aggregate object.

[0075] Before reorganizing the M aggregate objects to generate a new aggregate object, it is determined that the M aggregate objects meet a preset condition, wherein the preset condition includes at least one of the following:

[0076] The sum of the hollowing rates of the M aggregated objects is greater than or equal to the second threshold; or, the average value of the hollowing rates of the M aggregated objects is greater than or equal to the third threshold; or, M≥Q, Q is a preset value, Q<N, Q is an integer greater than or equal to 2.

[0077] Exemplarily, if the maximum heap is not empty, aggregate objects are taken out from the top of the maximum heap in sequence. When the sum of valid files included in the multiple aggregate objects taken out reaches the aggregation threshold (e.g., the second threshold) of the aggregate object, the aggregate objects are reorganized and these valid files are reintegrated into the new aggregate object. At the same time, metadata information of the new aggregate object is generated. The specific flow chart of the embodiment of the present invention can be referred to as follows: Figure 7 shown.

[0078] In addition, when a new aggregate object is generated, the aggregate object taken from the top of the maximum heap is deleted to achieve garbage data cleaning and recycling.

[0079] By adopting the method provided by the present invention, junk files in the aggregate object after the files are deleted can be cleaned up, thereby achieving the purpose of reducing junk files in the disk, recovering disk space, and improving disk utilization.

[0080] The patent of this invention is applicable to cloud storage services for massive files, and is particularly applicable to cloud storage scenarios for massive small files. When used in cloud storage scenarios for small files, the invention can effectively improve the efficiency of disk fragmentation cleaning and recycling, effectively reduce disk IO operations during garbage collection, and reduce the system consumption generated by the garbage collection process to a minimum, effectively reducing the impact on upper-level businesses.

[0081] The division of units in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present invention may be integrated into a processor, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] The embodiment of the present invention further provides a device 800, see Figure 8 As shown, it includes: a processing module 810 and a transceiver module 820.

[0083] The transceiver module 820 may include a receiving unit and a sending unit. The processing module 810 is used to control and manage the actions of the device 800. The transceiver module 820 is used to support the communication between the device 800 and other devices. Optionally, the device 800 may also include a storage unit, which is used to store program codes and data of the device 800.

[0084] Optionally, each module in the device 800 may be implemented by software.

[0085] Optionally, the processing module 810 may be a processor or a controller, for example, a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor may also be a combination that implements a computing function, for example, including a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The transceiver module 820 may be a communication interface, a transceiver or a transceiver circuit, etc., wherein the communication interface is a general term, and in a specific implementation, the communication interface may include multiple interfaces, and the storage unit may be a memory.

[0086] The processing module 810 calls the transceiver module 820 to execute:

[0087] Determine hollowing rates of N aggregate objects, where the hollowing rate of the ith aggregate object is the ratio of the total number of junk files included in the ith aggregate object to the total number of files included in the ith aggregate object, i and N are positive integers, and i ranges from 1 to N; determine M aggregate objects among the N aggregate objects whose hollowing rates are greater than or equal to a preset threshold according to the hollowing rates of the N aggregate objects, where M is an integer greater than or equal to 2; and reorganize the M aggregate objects to generate new aggregate objects.

[0088] The embodiment of the present invention also provides another device 900, see Fig. 9 As shown, including:

[0089] Communication interface 901, memory 902 and processor 903;

[0090] The communication device 900 communicates with other devices through the communication interface 901, such as sending and receiving messages; the memory 902 is used to store program instructions; the processor 903 is used to call the program instructions stored in the memory 902 and execute the program according to the obtained method.

[0091] The processor 903 calls the program instructions stored in the communication interface 901 and the memory 902 to execute: determining the hollowing rates of N aggregate objects, wherein the hollowing rate of the ith aggregate object is the ratio of the total number of junk files included in the ith aggregate object to the total number of files included in the ith aggregate object, i and N are positive integers, and i ranges from 1 to N; based on the hollowing rates of the N aggregate objects, determining M aggregate objects among the N aggregate objects whose hollowing rates are greater than or equal to a preset threshold, where M is an integer greater than or equal to 2; and reorganizing the M aggregate objects to generate new aggregate objects.

[0092] The embodiment of the present invention does not limit the specific connection medium between the communication interface 901, the memory 902 and the processor 903, such as a bus, which can be divided into an address bus, a data bus, a control bus, etc.

[0093] In the embodiments of the present invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0094] In an embodiment of the present invention, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory may also be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in an embodiment of the present invention may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0095] An embodiment of the present invention further provides a computer-readable storage medium, comprising a program code. When the program code is executed on a computer, the program code is used to enable the computer to execute the steps of the method provided in the embodiment of the present invention.

[0096] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0100] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0101] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for reorganizing an aggregate object, characterized in that: include: Determine hollowing rates of N aggregate objects, wherein the hollowing rate of the i-th aggregate object is a ratio of the total number of junk files included in the i-th aggregate object to the total number of files included in the i-th aggregate object, i and N are positive integers, and i ranges from 1 to N; According to the hollowing rates of the N aggregated objects, determine M aggregated objects whose hollowing rates are greater than or equal to a preset threshold among the N aggregated objects, where M is an integer greater than or equal to 2; The M aggregate objects are reorganized to generate a new aggregate object; wherein, the reorganization of the M aggregate objects to generate a new aggregate object includes: sequentially taking out aggregate objects from the top of a maximum heap of aggregate object reorganization, and when the sum of valid files respectively included in the taken-out multiple aggregate objects reaches the aggregation threshold of the aggregate object, reorganizing the aggregate objects, wherein the maximum heap of aggregate object reorganization includes the M aggregate objects, and the M aggregate objects are sorted in descending order according to the hollowing rate, and the top element in the maximum heap of aggregate object reorganization is the aggregate object with the largest hollowing rate among the M aggregate objects.

2. The method according to claim 1, characterized in that The number of junk files included in the i-th aggregation object is determined according to the bitmap of the i-th aggregation object, the total number of bits included in the bitmap of the i-th aggregation object is equal to the total number of files included in the i-th aggregation object, and each file in the i-th aggregation object corresponds to a bit in the bitmap of the i-th aggregation object; In the bit map of the i-th aggregate object, possible values ​​of each bit include a first value and a second value, wherein the first value indicates that the file corresponding to the bit is the junk file, and the second value indicates that the file corresponding to the bit is not the junk file.

3. The method according to claim 2, characterized in that Determine the hollowing rate of N aggregate objects, including: When determining the hollowing rate of the i-th aggregate object, the following steps are performed: Periodically updating a bitmap of the i-th aggregation object according to indexes of files included in the i-th aggregation object and metadata information of the i-th aggregation object, where the metadata information of the i-th aggregation object includes metadata information of each file included in the i-th aggregation object; The hollowing rate of the i-th aggregation object is determined according to the updated bitmap of the i-th aggregation object.

4. The method according to claim 3, characterized in that Periodically updating a bitmap of the i-th aggregate object according to indexes of files included in the i-th aggregate object and metadata information of the i-th aggregate object, including: Determine in turn whether the metadata information of the file corresponding to each index has been deleted; When metadata information of the file corresponding to the index is deleted, replacing the value of the corresponding bit of the file corresponding to the index in the bitmap of the i-th aggregation object from the second value to the first value; When the metadata information corresponding to the index is not deleted, the value of the corresponding bit of the file corresponding to the index in the bitmap of the i-th aggregation object remains unchanged, which is the second value.

5. The method according to claim 2, characterized in that Determine the hollowing rate of N aggregate objects, including: When determining the hollowing rate of the i-th aggregate object, the following steps are performed: After deleting metadata information of a first file, updating a bitmap of the i-th aggregate object, the first file being a file in the i-th aggregate object; The hollowing rate of the i-th aggregation object is determined according to the updated bitmap of the i-th aggregation object.

6. The method according to claim 5, characterized in that Updating the bitmap of the i-th aggregate object includes: The value of the corresponding bit of the first file in the bitmap of the i-th aggregation object is replaced from the second value to the first value.

7. An aggregate object reorganization device, characterized in that: The device includes a processor and an interface circuit, wherein the interface circuit is used to receive signals from other devices outside the device and transmit them to the processor or send signals from the processor to other devices outside the device, and the processor is used to implement the method as described in any one of claims 1 to 6 through logic circuits or execution code instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images