Space recovery method and device based on reverse index, equipment and storage medium

Abstract

The invention provides a reverse index-based space recovery method, apparatus and device, and a storage medium, and relates to the technical field of computers. According to the method, a forward index pointing to a combined block is created for each storage file, and meanwhile, a unique corresponding reverse index is created for each combined block. When the space recovery event is triggered, the effective file list is directly obtained by reading the reverse index, so that the space recovery efficiency is remarkably improved; and reading valid file data from the first combined block according to the valid file list, and migrating the valid file data to a currently available second combined block. After data migration is completed, the forward index is updated, the first combined block and the reverse index of the first combined block are deleted, and utilization of storage resources is optimized. According to the method, through refined index management and on-demand recovery strategies, the space recovery efficiency is greatly improved, storage resource consumption in application scenes such as financial insurance and medical inquiry is reduced, and the storage performance and expandability of a storage system are enhanced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a space reclamation method, apparatus, device and storage medium based on reverse index. Background Technology

[0002] In the fields of distributed storage, databases, and big data processing, especially in financial and medical data management, existing technologies commonly employ "small file merging" to optimize the storage and access performance of massive amounts of small files. This technology aggregates multiple small files or data blocks into a larger data block (called a "merged block") for storage, and creates a forward index for each small file, recording its offset and length within the merged block. This approach effectively reduces the total amount of metadata and the I / O pressure on the underlying storage system, while improving sequential read and write performance.

[0003] However, this merged storage strategy introduces a new challenge: space reclamation efficiency. When small files within a merged block are frequently deleted or modified (causing old data to become invalid), a large amount of "fragmented" free space is generated within the merged block. To reclaim this space, a garbage collection (GC) process needs to be performed to migrate the still valid small files to the new merged block, and then release the storage space occupied by the original merged block.

[0004] Current mainstream space reclamation technologies mainly rely on global scanning and traversal, which has significant drawbacks: existing solutions typically only trigger garbage collection for a merged block when the overall effective data percentage falls below a certain threshold. While simple, this mechanism is not precise enough. It cannot detect the "hot" and "cold" data distribution within a merged block. A merged block may contain mostly rarely accessed "cold" data (such as historical financial transaction records or archived medical images), but to reclaim a small amount of space, it triggers a data migration of the entire block, initiating unnecessary space reclamation operations. During garbage collection, to find all still valid small files in a specific merged block, the system must perform a global metadata scan. The reclaimer needs to traverse the entire forward index metadata, filtering out all forward indices pointing to that merged block, to determine the list of valid files that need to be migrated. When the system stores billions or even tens of billions of small files (such as financial transaction details or medical image files), this global traversal operation not only consumes a large amount of computing resources, but also has very low space reclamation efficiency due to the long global scan time and the long cycle from triggering reclamation to completing space release.

[0005] Therefore, in the fields of financial and medical data storage, how to quickly and accurately locate and merge effective small files within a block to achieve efficient space reclamation has become an urgent technical problem to be solved. This is of great significance for ensuring the efficient processing of financial transaction systems and the stable operation of medical information systems. Summary of the Invention

[0006] This application provides a space reclamation method, apparatus, device, and storage medium based on reverse indexing, aiming to solve the technical problem of low space reclamation efficiency caused by the use of global scanning to locate valid small files within the merging block in existing space reclamation technologies.

[0007] Firstly, this application provides a space reclamation method based on reverse indexes, the space reclamation method based on reverse indexes comprising the following steps: Create a forward index pointing to the merged block for each storage file in the storage space, and create a unique corresponding reverse index for each merged block; When the space reclamation event of the first merge block is triggered, the reverse index of the first merge block is read to obtain a list of valid files; Read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the currently available second merge block in the storage space; After the migration of the valid file data is completed, the forward index corresponding to the valid stored data is updated, and the first merged block and the reverse index corresponding to the first merged block are deleted.

[0008] Secondly, this application also provides a space reclamation device based on a reverse index, the space reclamation device based on a reverse index comprising: An index building module is used to create a forward index pointing to a merged block for each storage file in the storage space, and to create a unique corresponding reverse index for each merged block; The space reclamation triggering module is used to read the reverse index of the first merged block and obtain a list of valid files when the space reclamation event of the first merged block is triggered. The data migration module is used to read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the second merge block currently available in the storage space; The index update module is used to update the forward index corresponding to the valid storage data after the valid file data migration is completed, and to delete the first merge block and the reverse index corresponding to the first merge block.

[0009] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the space reclamation method based on reverse index as described above.

[0010] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the space reclamation method based on reverse index as described above.

[0011] This application provides a space reclamation method, apparatus, computer device, and storage medium based on reverse indexes. The method creates a forward index pointing to a merged block for each stored file, and simultaneously creates a unique corresponding reverse index for each merged block, providing a foundation for rapid file location and management. When a space reclamation event is triggered, a list of valid files is directly obtained by reading the reverse index, avoiding the time-consuming global metadata scan of traditional methods and significantly improving the efficiency of space reclamation. Valid file data is read from the first merged block based on the list of valid files and migrated to the currently available second merged block, processing only valid data and reducing unnecessary I / O operations. After data migration is complete, the forward index is updated and the first merged block and its reverse index are deleted, releasing storage space and further optimizing the utilization of storage resources. This method, through refined index management and an on-demand reclamation strategy, greatly improves space reclamation efficiency, reduces storage system resource consumption, and enhances the storage performance and scalability of the storage system. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a space reclamation method based on reverse index in one embodiment of the present invention; Figure 2 A flowchart illustrating an embodiment of a space reclamation method based on reverse indexes provided in this application; Figure 3 This application provides a schematic diagram of the structure of a merged block reverse index; Figure 4 This is a schematic diagram of a space reclamation device based on a reverse index provided in this application. Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0014] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0017] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The space reclamation method based on reverse index provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. When the server receives a space reclamation request from the client, it can create a forward index pointing to a merge block for each storage file in the storage space, and a unique corresponding reverse index for each merge block. When a space reclamation event of the first merge block is triggered, the server reads the reverse index of the first merge block to obtain a list of valid files. It then reads the valid file data corresponding to the list of valid files from the first merge block and migrates the valid file data to a currently available second merge block in the storage space. After the migration of the valid file data is completed, the server updates the forward index corresponding to the valid storage data and deletes the first merge block and the reverse index corresponding to the first merge block.

[0020] This invention addresses the problem of low space reclamation efficiency in existing space reclamation technologies for financial and medical data storage applications, where global scanning is used to locate valid small files within merge blocks. Instead, it creates a forward index pointing to the merge block for each stored file and a unique corresponding reverse index for each merge block, providing a foundation for rapid file location and management. When a space reclamation event is triggered, the list of valid files is directly obtained by reading the reverse index, avoiding the time-consuming global metadata scan of traditional methods and significantly improving space reclamation efficiency. Valid file data is read from the first merge block based on the valid file list and migrated to the currently available second merge block, processing only valid data and reducing unnecessary I / O operations. After data migration is complete, the forward index is updated and the first merge block and its reverse index are deleted, releasing storage space and further optimizing storage resource utilization. This method, through refined index management and an on-demand reclamation strategy, greatly improves space reclamation efficiency, reduces storage system resource consumption, and enhances the storage performance and scalability of the storage system.

[0021] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0022] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of a space reclamation method based on reverse indexing provided in this application.

[0023] like Figure 2 As shown, the space reclamation method based on reverse index includes steps S101 to S104.

[0024] S101. Create a forward index pointing to the merged block for each storage file in the storage space, and create a unique corresponding reverse index for each merged block.

[0025] In one embodiment, in the storage system, each storage file creates a forward index pointing to the merged block it belongs to, while a unique reverse index is created for each merged block.

[0026] The forward index is an index structure in a storage system used to quickly locate a file's position within a merge block. It records the offset, length, and identifier of the merge block for each stored file. The forward index is the fundamental index structure for file storage and access; through it, the storage system can quickly find the file's specific location within a merge block, thus enabling efficient file read operations.

[0027] An inverted index is an index structure created for each merge block, used to record the set of unique identifiers for all valid files within that merge block. The inverted index is a dynamic collection that is updated in real-time as file writes, deletions, and other operations occur. Its main function is to quickly determine the list of all valid files in a merge block during garbage collection (GC), avoiding the inefficient global metadata scan required in traditional methods, and providing a direct file list for efficient space reclamation.

[0028] In a storage system, when a new merge block is created, an empty set of inverted indexes is initialized for it along with the creation of the forward index. When a file is written, a forward index pointing to the merge block is created, and the file identifier is added to the merge block's inverted index set. When a file is deleted, the valid flag for that file in the forward index table is marked as invalid, and the corresponding file identifier is removed from the merge block's inverted index set.

[0029] Specifically, when the storage system receives a file write request, it selects a currently available merge block using a block selection algorithm (such as based on space utilization, load balancing, etc.). Storage space is allocated within the selected merge block, the file's offset and length are recorded, and a unique identifier (such as a UUID) is generated for the file. A record for the file is created in the forward index table, recording the file identifier, merge block identifier, offset, and length. Simultaneously, a reverse index is created for each merge block, initially empty. When a file is written to a merge block, its identifier is added to the merge block's reverse index. Similarly, when a file is deleted, the forward index table is queried based on the file identifier to find the merge block identifier, offset, and length containing the file. The valid flag for the file in the forward index table is marked as invalid; simultaneously, the file's identifier is removed from the reverse index set of that merge block.

[0030] In one embodiment, upon receiving a write request for a file to be stored, a third available merge block is determined from the storage space using a block selection algorithm; the file to be stored is written into the third merge block, and the inverted index corresponding to the third merge block is updated based on the file identifier of the file to be stored, thereby completing the write operation of the file to be stored.

[0031] When a small file is written and merged into a merge block, in addition to creating its forward index, the storage system atomically adds the small file's identifier to the reverse index of the merge block to which it belongs.

[0032] Specifically, after creating a forward index for each stored file and a reverse index for each merged block in the storage space, the storage system, upon receiving a write request for a file to be stored, initiates a block selection algorithm. This algorithm selects a currently available third merged block from the storage space based on certain rules (such as space utilization and load balancing). The file to be stored is then written to the selected third merged block, its file identifier is obtained, and the reverse index corresponding to the third merged block is updated based on this identifier, completing the write operation. This process ensures that during file writing, the forward and reverse indexes can reflect the relationship between the file and the merged block in a timely and accurate manner, providing a reliable foundation for subsequent space reclamation and other operations.

[0033] In one embodiment, upon receiving a deletion request for a file to be deleted, the fourth merge block containing the file to be deleted is queried; after the file to be deleted is successfully deleted, the valid flag bit corresponding to the file to be deleted in the forward index of the fourth merge block is marked as an invalid flag bit, and the file identifier corresponding to the file to be deleted in the reverse index of the fourth merge block is deleted, thereby completing the deletion operation of the file to be deleted.

[0034] When a small file is deleted or its data becomes invalid, the storage system atomically removes its identifier from the corresponding merge block inverted index after marking its forward index as invalid.

[0035] Specifically, after creating a forward index for each storage file and a reverse index for each merge block in the storage space, the storage system can query the fourth merge block where the file to be deleted is located when it receives a deletion request for a file to be deleted.

[0036] After a file is successfully deleted, the corresponding record for the file to be deleted is located in the forward index of the fourth merge block. Its valid flag is then set to invalid, and the file identifier of the file to be deleted is removed from the reverse index of the fourth merge block, completing the deletion operation. This process ensures that the forward and reverse indexes reflect the file deletion status in a timely manner during file deletion, providing accurate information for subsequent space reclamation.

[0037] In file write and delete operations, updates to both forward and reverse indexes must be atomic to avoid data inconsistency issues. For example, transactions or distributed locks can be used to ensure the atomicity of index updates: during file writes, first lock the set of reverse indexes for the merged block, update the forward index table, then add the file identifier to the set of reverse indexes, and finally release the lock.

[0038] S102. When the space reclamation event of the first merged block is triggered, the reverse index of the first merged block is read to obtain a list of valid files.

[0039] Generally, space reclamation events are triggered based on certain preset conditions, such as the proportion of used space in the merged block being lower than a threshold, excessive fragmentation rate, low effective file density, or the probability of file failure predicted by a machine learning model.

[0040] In one embodiment, the storage space can periodically or in real time detect space reclamation monitoring indicators of the first merged block (such as the proportion of used space, fragmentation rate, effective file density, GC priority score, etc.). If these indicators meet preset reclamation conditions (e.g., the proportion of used space is less than 80%, or the fragmentation rate is greater than 50% and the effective file density is less than 30%), a space reclamation event is triggered.

[0041] Furthermore, the space reclamation monitoring indicators of the first merged block are detected; when the space reclamation monitoring indicators of the first merged block meet the preset space reclamation conditions, the space reclamation event of the first merged block is triggered.

[0042] In one embodiment, the first merged block can be any merged block in the storage system, or it can be all merged blocks currently existing in the storage system. The storage system can detect space reclamation monitoring indicators for each merged block one by one. Space reclamation monitoring indicators can include at least one of the following: used space ratio, fragmentation rate, effective file density, space reclamation GC priority score, etc.

[0043] The used space ratio refers to the ratio of used space in a merged block to the total space. This metric is used to evaluate the space utilization of a merged block. A low used space ratio indicates that there is a lot of free space in the merged block, which may need to be reclaimed.

[0044] Specifically, the storage system can periodically scan the metadata of the merged blocks, count the total space allocated to files, and calculate the used space ratio = (allocated space / total space of the merged blocks) × 100%. For example, if the total space of the first merged block B1 is 100MB and the allocated space is 60MB, then the used space ratio is 60%.

[0045] Fragmentation rate refers to the degree of fragmentation of free space within a merged block. A high fragmentation rate means that there are a large number of small free spaces within the merged block, which are difficult to utilize effectively and thus affect storage efficiency.

[0046] Specifically, the storage system can scan the free space of the merged blocks, count the number and total size of fragments in the free space, and calculate the fragmentation rate = (total fragmented space size / total merged block space) × 100%. For example, if the total space in the first merged block B1 is 100MB, and there are multiple small blocks of free space, with a total free space size of 20MB, then the fragmentation rate is 20%.

[0047] Effective file density refers to the ratio of the space occupied by effective files in a merge block to the total space. This metric reflects the density of effective data in the merge block. Low effective file density may indicate a large amount of invalid or fragmented space in the merge block.

[0048] Specifically, the storage system can calculate the effective file density by determining the space occupied by valid files in the merged block, which is calculated as (effective file space / total merged block space) × 100%. For example, if the total space in the first merged block B1 is 100MB and the effective file space is 40MB, then the effective file density is 40%.

[0049] The GC priority score for space reclamation is a score calculated by comprehensively considering multiple factors (such as the proportion of used space, fragmentation rate, and effective file density), and is used to evaluate the space reclamation priority of merged blocks. A high score indicates that the merged block requires higher priority for space reclamation.

[0050] Specifically, the GC priority score for space reclamation can be calculated based on preset weights and rules, comprehensively calculating a weighted score based on indicators such as the proportion of used space, fragmentation rate, and effective file density. For example, assuming the weight of the proportion of used space is 0.4, the weight of the fragmentation rate is 0.3, and the weight of the effective file density is 0.3, if the proportion of used space is 60%, the fragmentation rate is 20%, and the effective file density is 40%, then the GC priority score would be:

[0051] The preset space reclamation conditions are thresholds or rules set according to system requirements and performance goals, used to determine when to trigger a space reclamation event.

[0052] Furthermore, when one or more of the space reclamation monitoring indicators, such as the used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score, meet the space reclamation conditions, the space reclamation event of the first merged block is triggered.

[0053] In one embodiment, the space reclamation condition can be any single indicator condition among the used space ratio, fragmentation rate, and effective file density: if the used space ratio is less than a preset occupancy threshold (e.g., 80%), the space utilization of the merged block is considered low, with a large amount of free space, requiring space reclamation and triggering a space reclamation event. Alternatively, if the fragmentation rate is higher than a first threshold (e.g., 50%), a space reclamation event is triggered. Alternatively, if the effective file density is less than a second threshold (e.g., 30%), a space reclamation event is triggered. For example, assuming the preset used space ratio threshold is 80%, and the used space ratio of the first merged block B1 is 60%, then a space reclamation event is triggered.

[0054] In one embodiment, space reclamation conditions can also be combined metric conditions. These combined metric conditions can be a combination of any two of the following: used space ratio, fragmentation rate, and effective file density; or a combination of all three. The space reclamation GC priority score is obtained by weighted summation of the other metrics and can be considered a combination of all metrics.

[0055] Taking the combination of fragmentation rate and effective file density as an example, if the fragmentation rate is higher than the first threshold (e.g., 50%) and the effective file density is lower than the second threshold (e.g., 30%), it indicates that there is a large amount of fragmented free space within the merged block, and the effective files are relatively scattered. Space reclamation can bring significant benefits, triggering a space reclamation event. For example, the fragmentation rate threshold can be preset to 50%, and the effective file density threshold to 30%. The fragmentation rate of the first merged block B1 is 20%, and the effective file density is 40%, which does not meet the triggering conditions; however, if the fragmentation rate is 60% and the effective file density is 25%, then a space reclamation event is triggered.

[0056] For GC priority scoring, if the GC priority score is higher than the preset scoring threshold (e.g., 70 points), a space reclamation event is triggered. For example, if the preset GC priority scoring threshold is 70 points, the score calculated for the first merged block B1 is 44 points, and no space reclamation event is triggered; however, if the score reaches 75 points, it indicates that the merged block has a high space reclamation priority in the current system environment, and a space reclamation event is triggered.

[0057] In one embodiment, when a space reclamation event is triggered by the GC priority scores of multiple merged blocks, all merged blocks that meet the triggering conditions can be placed in a priority queue and sorted according to their GC priority scores. The system can then process space reclamation events sequentially according to priority, ensuring that resources are allocated reasonably.

[0058] This embodiment provides a clear basis for judgment of the storage system through specific space reclamation conditions, making the triggering of space reclamation more flexible and adaptable to the needs of different scenarios.

[0059] Generally, preset thresholds can be dynamically adjusted based on the real-time load and storage usage of the storage system. For example, when the storage system load is low, the threshold for the percentage of used space can be appropriately lowered to more actively reclaim space; when the load is high, the threshold can be increased to reduce the impact of reclamation operations on performance.

[0060] In another embodiment, for monitoring and triggering the space reclamation event of the first merged block, the storage system can predict the space usage trend of the merged block in the future not only based on current monitoring indicators but also by combining machine learning models. If the prediction results show that the fragmentation rate or effective file density of the merged block will soon reach the triggering condition, the space reclamation event is triggered in advance to prevent the fragmentation problem from worsening further.

[0061] Further, historical spatial monitoring data of the first merged block within a preset historical period is obtained; the historical spatial monitoring data within the preset historical period is analyzed using a pre-trained machine learning model to predict the file failure probability of the first merged block in the next detection period; when the file failure probability is greater than a preset probability threshold, a space reclamation event of the first merged block is triggered.

[0062] In one embodiment, historical spatial monitoring data refers to multiple spatial monitoring records of the merged block within a preset historical period (such as the past week, month, etc.), including indicators such as the proportion of used space, fragmentation rate, and effective file density. These data reflect the space usage and changes of the merged block within a historical time period.

[0063] This study uses a pre-trained machine learning model to analyze historical spatial monitoring data and predict the probability of merged blocks becoming invalid within future monitoring periods. The machine learning model learns file failure patterns and trends from historical data, thus predicting future failures in advance. The machine learning model can choose from model structures such as linear regression, decision trees, random forests, and neural networks (e.g., LSTM). Historical data can be used to train the model, with the training objective being to predict file failure probabilities.

[0064] File expiration probability refers to the likelihood that a file in the merge block will be deleted or marked as expired in the next detection cycle. A high expiration probability indicates that the merge block may generate a lot of invalid space in the future, requiring space reclamation in advance.

[0065] Specifically, the storage system collects historical space monitoring data of the first merged block within a preset historical period (such as the past week, month, etc.), including information such as the proportion of used space, fragmentation rate, and effective file density. The acquired historical space monitoring data is input into a pre-trained machine learning model, which analyzes the data and predicts the probability of file failure of the first merged block in the next detection period based on the characteristics and patterns of the historical data.

[0066] If the predicted probability of file failure exceeds a preset probability threshold (e.g., 70%), it is assumed that the merged block may generate a large number of invalid files in the near future, exacerbating space fragmentation. This allows for the early triggering of space reclamation events. This approach makes space reclamation more proactive and intelligent, enabling early handling of potential space fragmentation issues and preventing storage efficiency degradation due to file failure.

[0067] In one embodiment, in a distributed storage system, the detection of space reclamation monitoring metrics can be distributed across multiple nodes. Each node is responsible for detecting metrics of a portion of the merged blocks and reporting the results to a central coordinator. The central coordinator, based on the data reported by each node, comprehensively determines whether a space reclamation event has been triggered and coordinates the reclamation operations of each node.

[0068] In one embodiment, when a merged block (the first merged block) triggers a space reclamation event, the storage system directly reads the inverted index of that merged block, thereby quickly obtaining a complete list of valid files without additional computation. This list of valid files is a list containing all valid file identifiers from the merged block. These file identifiers are directly derived from the inverted index, ensuring the accuracy and completeness of the list.

[0069] The inverted index maintains a dynamic set for each merge block, recording the unique identifiers of all valid files within that merge block. For example... Figure 3 As shown, the six files are placed in a merge block (merge block 1). During the space reclamation process, the inverted index corresponding to the merge block is directly scanned to find all valid files (file 1 to file 6), providing a fast and direct list of valid files. This eliminates the need for a global metadata scan, avoiding the inefficient operation of a global metadata scan to determine valid files in traditional methods, and greatly improving reclamation efficiency.

[0070] In this embodiment, by using a reverse index, the time-consuming global metadata traversal is eliminated during space reclamation. A complete list of valid files can be obtained with just a single fast query of the reverse index. This improves the execution speed of the space reclamation task by several orders of magnitude, enabling the rapid release of idle storage space. Furthermore, it avoids the large amount of CPU computation and I / O operations generated by large-scale scanning of the metadata engine during space reclamation. This significantly reduces the overall load on the storage system, allowing computing resources and I / O bandwidth to better serve front-end business data read and write requests, thereby improving the overall throughput and stability of the storage system.

[0071] S103. Read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the currently available second merge block in the storage space.

[0072] Based on the list of valid files provided by the inverted index, the storage system precisely locates and reads the data of each valid file from the first merge block according to the offset and length information recorded in the forward index. A second merge block currently available in the storage space is selected to store the migrated valid file data. The valid file data read from the first merge block is written to the selected second merge block. During the migration process, the storage system only processes valid data, avoiding unnecessary operations on invalid data and improving migration efficiency.

[0073] Specifically, a list of valid files is read from the reverse index of the first merge block to ensure its accuracy and completeness. The list of valid files is then traversed, and the forward index is queried based on the file identifier to obtain the offset and length of each file within the first merge block. Based on the offset and length, file data is read from the first merge block. The read valid file data is then written to the second merge block one by one.

[0074] Understandably, when migrating valid file data to the second merge block, the storage system does not simply select an available merge block for migration, but rather formulates a reasonable data migration strategy through a series of evaluations and calculations.

[0075] Further, all currently available candidate merge blocks in the storage space are traversed, and at least one data migration evaluation index corresponding to each candidate merge block is calculated; based on the data migration requirements of the valid file data, a data migration strategy for the valid file data is determined; based on the data migration strategy, the at least one data migration evaluation index corresponding to each candidate merge block is weighted and calculated to obtain a migration priority score corresponding to each candidate merge block; based on the migration priority score corresponding to each candidate merge block, at least one second merge block is determined, so that the valid file data is migrated to the second merge block according to the data migration strategy.

[0076] In one embodiment, the selection of the second merge block can be based on various strategies, such as space utilization, load balancing, fragmentation rate, etc.

[0077] Specifically, the storage system traverses all currently available candidate merge blocks in the storage space and calculates at least one data migration evaluation metric for each candidate merge block, such as space utilization, fragmentation rate, and load balancing. Space utilization refers to the ratio of used space in the merge block to the total space, used to assess whether the remaining space in the merge block is sufficient; fragmentation rate refers to the degree of fragmentation of free space in the merge block, used to assess the storage efficiency of the merge block; load balancing refers to the I / O load of the merge block, used to assess whether data needs to be migrated to merge blocks with lower loads to optimize system performance; other metrics can also be added based on specific needs, such as read / write latency and data access frequency of the merge block.

[0078] Based on the data migration requirements of valid file data, a data migration strategy is formulated. This strategy consists of rules designed to guide the selection of target merge blocks. The strategy can be based on metrics such as space utilization, fragmentation rate, and load balancing, or it can be customized to meet specific business needs. For example, if optimizing storage space utilization is required, the data migration strategy might prioritize merging blocks with lower space utilization; if reducing fragmentation rate is required, it might prioritize merging blocks with lower fragmentation rates to achieve load balancing and other objectives. A migration strategy could prioritize merging blocks with space utilization below 70% and fragmentation rate below 15%.

[0079] Based on the data migration strategy, the evaluation metrics of each candidate merge block are weighted and calculated to obtain a migration priority score for each candidate merge block. A higher score indicates that the merge block is more suitable as the second merge block. For example, if the weight of space utilization is 0.4, the weight of fragmentation rate is 0.3, and the weight of load balancing is 0.3, then the migration priority score for each candidate merge block is calculated based on the corresponding metric values. For instance, the weight allocation is 0.6 for used space ratio and 0.4 for fragmentation rate; assuming that candidate merge block B2 has a used space ratio of 60% and a fragmentation rate of 10%, the score is calculated as: Score = (60 × 0.6) + (10 × 0.4) = 40. Based on the migration priority score, the merge block with the highest score is selected as the second merge block.

[0080] Based on the migration priority score, the candidate merge block with the highest score is selected as the second merge block. Then, according to the data migration strategy, the valid file data is migrated to the determined second merge block.

[0081] In some cases, multiple second merge blocks can be selected for data migration to further optimize storage space and load balancing. For example, the two highest-rated merge blocks, B3 and B4, can be selected as the second merge blocks, with some valid file data migrated to B3 and the other portion to B4.

[0082] In one embodiment, during the migration of valid file data, it is necessary to update the relevant index information to ensure that the index information after data migration is accurate. For example, the data of valid file F1 is written into the second merge block B2, the forward index is updated to {file identifier: F1, merge block identifier: B2, offset: 0, length: 1024}, and F1 is added to the reverse index of merge block B2.

[0083] In this way, the storage system can accurately know which data needs to be migrated, achieving precise "point-and-click" reclamation. This avoids any unnecessary data movement, further reduces I / O consumption, and allows the system to implement more refined and intelligent reclamation strategies (such as prioritizing the reclamation of the most fragmented blocks of valid data).

[0084] S104. After the effective file data migration is completed, update the forward index corresponding to the effective storage data, and delete the first merge block and the reverse index corresponding to the first merge block.

[0085] After the migration of valid file data is complete, the forward indexes corresponding to these valid files need to be updated to point to the new storage location (the second merge block), including information such as the valid file identifier, storage location (merge block identifier), offset, and length. Once all valid file data migration is complete, the first merge block and its corresponding reverse index are deleted, releasing the storage space occupied by the first merge block and ensuring efficient use of storage space.

[0086] Specifically, iterate through the list of valid files after migration and obtain the identifier for each file. For each file, query its storage location (offset and length) in the second merge block, update the forward index, and modify the file's storage location information to match the information in the second merge block. Mark the forward index records in the source merge block (first merge block) as invalid (e.g., set the valid flag to false). Delete the set of reverse indexes corresponding to the first merge block, releasing the memory or storage space occupied by the indexes. Delete the first merge block, releasing the storage space it occupies.

[0087] During the process of updating the forward index and deleting the first merge block, detailed log information is recorded, including operation time, file identifier, merge block identifier, etc.

[0088] This embodiment provides a space reclamation method based on inverted indexes. This method creates a forward index pointing to a merged block for each storage file, and simultaneously creates a unique corresponding inverted index for each merged block, providing a foundation for rapid file location and management. When a space reclamation event is triggered, the list of valid files is directly obtained by reading the inverted index, avoiding the time-consuming global metadata scan in traditional methods and significantly improving the efficiency of space reclamation. Valid file data is read from the first merged block based on the list of valid files and migrated to the currently available second merged block, processing only valid data and reducing unnecessary I / O operations. After the data migration is complete, the forward index is updated and the first merged block and its inverted index are deleted, releasing storage space and further optimizing the utilization of storage resources. This method, through fine-grained index management and an on-demand reclamation strategy, greatly improves space reclamation efficiency, reduces storage system resource consumption, and enhances the storage performance and scalability of the storage system.

[0089] Please see Figure 4 , Figure 4 This is a schematic diagram of the current embodiment of a space reclamation device based on reverse index provided in this application. The space reclamation device based on reverse index is used to perform the aforementioned space reclamation method based on reverse index.

[0090] like Figure 4 As shown, the space reclamation device 200 based on reverse index includes: an index building module 201, a space reclamation triggering module 202, a data migration module 203, and an index updating module 204.

[0091] The index building module 201 is used to create a forward index pointing to the merged block for each storage file in the storage space, and to create a unique corresponding reverse index for each merged block; The space reclamation triggering module 202 is used to read the reverse index of the first merged block and obtain a list of valid files when the space reclamation event of the first merged block is triggered. Data migration module 203 is used to read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the currently available second merge block in the storage space; The index update module 204 is used to update the forward index corresponding to the valid storage data after the valid file data migration is completed, and delete the first merge block and the reverse index corresponding to the first merge block.

[0092] In one embodiment, the space reclamation device 200 based on reverse indexing further includes an event triggering module, comprising: An indicator detection unit is used to detect the space reclamation monitoring indicators of the first merging block; The first event triggering unit is used to trigger a space reclamation event of the first merged block when the space reclamation monitoring index of the first merged block meets the preset space reclamation conditions.

[0093] In one embodiment, the space reclamation monitoring indicators include at least one of the following: used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score; the event triggering first unit includes: The space reclamation event triggering subunit is used to trigger the space reclamation event of the first merged block when one or more of the space reclamation monitoring indicators, such as the used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score, meet the space reclamation conditions.

[0094] In one embodiment, the space reclamation device 200 based on reverse indexing further includes an event-triggered prediction module, comprising: The historical data acquisition unit is used to acquire historical spatial monitoring data of the first merging block within a preset historical period; The file failure probability prediction unit is used to analyze the historical spatial monitoring data within the preset historical period through a pre-trained machine learning model, and predict the file failure probability of the first merged block in the next detection period. The second event trigger unit is used to trigger the space reclamation event of the first merged block when the probability of the file failure is greater than a preset probability threshold.

[0095] In one embodiment, the data migration module 203 includes: An evaluation metric calculation unit is used to traverse all currently available candidate merge blocks in the storage space and calculate at least one data migration evaluation metric corresponding to each candidate merge block. A data migration strategy determination unit is used to determine a data migration strategy for the valid file data based on the data migration requirements of the valid file data. The priority score acquisition unit is used to perform weighted calculation on at least one data migration evaluation index corresponding to each candidate merge block based on the data migration strategy, and obtain the migration priority score corresponding to each candidate merge block. A data migration unit is configured to determine at least one second merge block based on the migration priority score corresponding to each of the candidate merge blocks, so as to migrate the valid file data to the second merge block according to the data migration strategy.

[0096] In one embodiment, the space reclamation device 200 based on reverse indexing further includes a file writing module, comprising: A block selection unit is used to determine the currently available third merge block from the storage space by using a block selection algorithm when a write request for a file to be stored is received. The file writing unit is used to write the file to be stored into the third merge block, and update the inverted index corresponding to the third merge block based on the file identifier of the file to be stored, thereby completing the writing operation of the file to be stored.

[0097] In one embodiment, the space reclamation device 200 based on reverse indexing further includes a file deletion module, comprising: The block query unit is used to query the fourth merge block where the file to be deleted is located when a deletion request for a file to be deleted is received; The file deletion unit is used to mark the valid flag bit corresponding to the file to be deleted in the forward index of the fourth merge block as an invalid flag bit after the file to be deleted is successfully deleted, and to delete the file identifier corresponding to the file to be deleted in the reverse index of the fourth merge block, thereby completing the deletion operation of the file to be deleted.

[0098] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device and each module described above can be referred to the corresponding process in the aforementioned embodiment of the space reclamation method based on reverse index, and will not be repeated here.

[0099] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0100] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0101] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0102] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any space reclamation method based on reverse indexing.

[0103] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0104] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, the processor can perform any space reclamation method based on reverse indexing.

[0105] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0107] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Create a forward index pointing to the merged block for each storage file in the storage space, and create a unique corresponding reverse index for each merged block.

[0108] When the space reclamation event of the first merge block is triggered, the inverted index of the first merge block is read to obtain a list of valid files.

[0109] Read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the currently available second merge block in the storage space.

[0110] After the migration of the valid file data is completed, the forward index corresponding to the valid stored data is updated, and the first merged block and the reverse index corresponding to the first merged block are deleted.

[0111] In one embodiment, before implementing the step of reading the inverted index of the first merge block to obtain a valid file list when the space reclamation event of the first merge block is triggered, the processor is further configured to implement: Detect the space reclamation monitoring indicators of the first merged block; When the space reclamation monitoring indicators of the first merged block meet the preset space reclamation conditions, the space reclamation event of the first merged block is triggered.

[0112] In one embodiment, the space reclamation monitoring metrics include at least one of the following: used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score; when the processor triggers the space reclamation event of the first merged block when the space reclamation monitoring metrics of the first merged block meet the preset space reclamation conditions, it is configured to: When one or more of the space reclamation monitoring indicators, such as the used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score, meet the space reclamation conditions, the space reclamation event of the first merged block is triggered.

[0113] In one embodiment, before implementing the step of reading the inverted index of the first merge block to obtain a valid file list when the space reclamation event of the first merge block is triggered, the processor is further configured to implement: Acquire historical spatial monitoring data of the first merged block within a preset historical period; By using a pre-trained machine learning model, the historical spatial monitoring data within the preset historical period is analyzed to predict the probability of the first merged block becoming invalid in the next detection period. When the probability of the file becoming invalid is greater than a preset probability threshold, the space reclamation event of the first merged block is triggered.

[0114] In one embodiment, when implementing the migration of the valid file data to the second merge block currently available in the storage space, the processor is configured to: Traverse all currently available candidate merge blocks in the storage space and calculate at least one data migration evaluation metric corresponding to each candidate merge block; Based on the data migration requirements of the valid file data, determine the data migration strategy for the valid file data; Based on the data migration strategy, at least one data migration evaluation index corresponding to each candidate merge block is weighted and calculated to obtain the migration priority score corresponding to each candidate merge block. Based on the migration priority score corresponding to each of the candidate merge blocks, at least one second merge block is determined so as to migrate the valid file data to the second merge block according to the data migration strategy.

[0115] In one embodiment, after implementing the creation of a forward index pointing to a merged block for each storage file in the storage space, and the creation of a unique corresponding reverse index for each merged block, the processor is further configured to implement: Upon receiving a write request for a file to be stored, a block selection algorithm is used to determine the currently available third merge block from the storage space; The file to be stored is written into the third merge block, and the inverted index corresponding to the third merge block is updated based on the file identifier of the file to be stored, thus completing the write operation of the file to be stored.

[0116] In one embodiment, after implementing the creation of a forward index pointing to a merged block for each storage file in the storage space, and the creation of a unique corresponding reverse index for each merged block, the processor is further configured to implement: Upon receiving a deletion request for a file to be deleted, query the fourth merge block where the file to be deleted is located; After the file to be deleted is successfully deleted, the valid flag bit corresponding to the file to be deleted in the forward index of the fourth merge block is marked as an invalid flag bit, and the file identifier corresponding to the file to be deleted in the reverse index of the fourth merge block is deleted, thus completing the deletion operation of the file to be deleted.

[0117] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the space reclamation methods based on reverse indexes provided in the embodiments of this application.

[0118] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A space reclamation method based on reverse indexes, characterized in that, The method includes: Create a forward index pointing to the merged block for each storage file in the storage space, and create a unique corresponding reverse index for each merged block; When the space reclamation event of the first merge block is triggered, the reverse index of the first merge block is read to obtain a list of valid files; Read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the currently available second merge block in the storage space; After the migration of the valid file data is completed, the forward index corresponding to the valid stored data is updated, and the first merged block and the reverse index corresponding to the first merged block are deleted.

2. The space reclamation method based on reverse index according to claim 1, characterized in that, Before reading the inverted index of the first merged block to obtain a valid file list when the space reclamation event of the first merged block is triggered, the process further includes: Detect the space reclamation monitoring indicators of the first merged block; When the space reclamation monitoring indicators of the first merged block meet the preset space reclamation conditions, the space reclamation event of the first merged block is triggered.

3. The space reclamation method based on reverse index according to claim 2, characterized in that, The space reclamation monitoring indicators include at least one of the following: used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score; The step of triggering a space reclamation event for the first merged block when the space reclamation monitoring indicators of the first merged block meet preset space reclamation conditions includes: When one or more of the space reclamation monitoring indicators, such as the used space ratio, fragmentation rate, effective file density, and space reclamation GC priority score, meet the space reclamation conditions, the space reclamation event of the first merged block is triggered.

4. The space reclamation method based on reverse index according to claim 1, characterized in that, Before reading the inverted index of the first merged block to obtain a valid file list when the space reclamation event of the first merged block is triggered, the process further includes: Acquire historical spatial monitoring data of the first merged block within a preset historical period; By using a pre-trained machine learning model, the historical spatial monitoring data within the preset historical period is analyzed to predict the probability of the first merged block becoming invalid in the next detection period. When the probability of the file becoming invalid is greater than a preset probability threshold, the space reclamation event of the first merged block is triggered.

5. The space reclamation method based on reverse index according to claim 1, characterized in that, The step of migrating the valid file data to the currently available second merge block in the storage space includes: Traverse all currently available candidate merge blocks in the storage space and calculate at least one data migration evaluation metric corresponding to each candidate merge block; Based on the data migration requirements of the valid file data, determine the data migration strategy for the valid file data; Based on the data migration strategy, at least one data migration evaluation index corresponding to each candidate merge block is weighted and calculated to obtain the migration priority score corresponding to each candidate merge block. Based on the migration priority score corresponding to each of the candidate merge blocks, at least one second merge block is determined so as to migrate the valid file data to the second merge block according to the data migration strategy.

6. The space reclamation method based on reverse index according to claim 1, characterized in that, After creating a forward index pointing to the merged block for each storage file in the storage space, and creating a unique corresponding reverse index for each merged block, the method further includes: Upon receiving a write request for a file to be stored, a block selection algorithm is used to determine the currently available third merge block from the storage space; The file to be stored is written into the third merge block, and the inverted index corresponding to the third merge block is updated based on the file identifier of the file to be stored, thus completing the write operation of the file to be stored.

7. The space reclamation method based on reverse index according to claim 1, characterized in that, After creating a forward index pointing to the merged block for each storage file in the storage space, and creating a unique corresponding reverse index for each merged block, the method further includes: Upon receiving a deletion request for a file to be deleted, query the fourth merge block where the file to be deleted is located; After the file to be deleted is successfully deleted, the valid flag bit corresponding to the file to be deleted in the forward index of the fourth merge block is marked as an invalid flag bit, and the file identifier corresponding to the file to be deleted in the reverse index of the fourth merge block is deleted, thus completing the deletion operation of the file to be deleted.

8. A space reclamation device based on reverse indexing, characterized in that, The space reclamation device based on reverse index includes: An index building module is used to create a forward index pointing to a merged block for each storage file in the storage space, and to create a unique corresponding reverse index for each merged block; The space reclamation triggering module is used to read the reverse index of the first merged block and obtain a list of valid files when the space reclamation event of the first merged block is triggered. The data migration module is used to read the valid file data corresponding to the valid file list from the first merge block, and migrate the valid file data to the second merge block currently available in the storage space; The index update module is used to update the forward index corresponding to the valid storage data after the valid file data migration is completed, and to delete the first merge block and the reverse index corresponding to the first merge block.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the space reclamation method based on reverse index as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the space reclamation method based on reverse index as described in any one of claims 1 to 7.

Images