Reverse indexing method and device based on grouped bitmap, equipment and storage medium

Abstract

The invention provides a reverse indexing method and device based on a grouped bitmap, equipment and a storage medium, and relates to the technical field of computers. According to the method, each combined block in the storage space is divided into a plurality of reverse index groups, a unique index key is generated, and the effective bitmap and the file identification array are constructed by utilizing the bit information of the group groups to form the index value, so that grouped management of the reverse indexes is realized. The effective bitmap reduces the time complexity of file operation and improves the index updating and query efficiency. The combination of the file identification array and the effective bitmap not only retains the unique identification information of the file, but also optimizes the use of the storage space. When the file operation request is processed, the target reverse index group can be quickly positioned, and the corresponding operation is executed, so that the response speed is higher, the reverse index management efficiency is improved, and an efficient, flexible and reliable solution is provided for small file management of financial transaction data and medical inquiry data in a distributed storage system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for reverse indexing based on grouped bitmaps. Background Technology

[0002] In distributed storage systems, especially in the financial and healthcare industries, efficient data management and storage are crucial. The financial sector handles massive amounts of transaction records and user data, while the healthcare industry deals with sensitive information such as patient medical records and test results. This data is typically stored as small files, and efficient garbage collection (GC) is essential for freeing up storage space after employing small file merging techniques.

[0003] Existing technologies typically employ a solution of creating inverted metadata indexes for merged blocks to avoid global scans, thereby improving GC efficiency. However, this basic inverted indexing scheme reveals new and significant drawbacks in its maintenance and management, hindering further improvements in system performance, especially in financial and healthcare scenarios where data security and processing efficiency are extremely critical.

[0004] First, update operations on inverted indexes, especially deletion operations, are inefficient. If an inverted index is simply implemented as an unordered list or a flat collection, when a small file is deleted, the system needs to traverse the entire list to locate and remove the file's identifier. This operation has a time complexity of O(n), and in a merged block storing tens of thousands of small files, a single deletion operation can introduce considerable latency. In financial trading systems, this latency can lead to significant delays in transaction processing, impacting user experience and system real-time performance. In medical information systems, this latency can affect the timely updating and retrieval of medical records, thereby impacting the efficiency and quality of medical services. When the system faces frequent file deletion or update requests, the overhead of maintaining the inverted index increases dramatically, becoming a new performance bottleneck and offsetting the benefits it brings during the garbage collection (GC) phase.

[0005] Secondly, the storage space of inverted indexes themselves suffers from waste and bloat. As files are continuously deleted, a large number of invalid "holes" (i.e., space occupied by identifiers of deleted files) accumulate in the underlying inverted index. Due to the lack of an effective internal cleanup mechanism, this invalid data continuously occupies valuable metadata storage space, causing the storage capacity of metadata engines (such as TiKV) to be rapidly consumed, thereby affecting the stability and scalability of the entire cluster. In the financial and healthcare industries, this waste of storage space not only increases hardware costs but may also lead to a decline in system performance, affecting the efficiency and security of data processing.

[0006] Finally, the basic inverted index structure is coarse and cannot support fine-grained optimization strategies. The flat index structure prevents the system from understanding the distribution of data within merged blocks. The garbage collector cannot distinguish which areas have more concentrated invalid data, thus preventing the implementation of intelligent strategies such as "reclaiming the most fragmented space first," limiting further improvements in GC efficiency. In the financial and healthcare industries, this lack of fine-grained index management can lead to inefficient data processing, impacting overall system performance and data security.

[0007] Therefore, based on the existing implementation of basic inverted indexes, how to improve the application efficiency of inverted indexes to meet the stringent requirements of the financial and medical industries for data processing efficiency and security has become an urgent technical problem to be solved. Summary of the Invention

[0008] This application provides a method, apparatus, device, and storage medium for reverse indexing based on grouped bitmaps, aiming to solve the technical problems of low efficiency in reverse index update operations, waste of storage space resources, and low efficiency in reverse index application due to lack of fine-grained management in the prior art.

[0009] In a first aspect, this application provides a reverse indexing method based on grouped bitmaps, the reverse indexing method based on grouped bitmaps comprising the following steps: Divide each merged block in the storage space into at least one inverted index group; Based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group, the index key of each reverse index group is determined. Based on the group bit information corresponding to each inverted index group, construct the effective bitmap and file identifier array of each inverted index group, and obtain the index value of each inverted index group; Upon receiving a file operation request for the target file, an index update or query operation corresponding to the file operation request is executed based on the index key and index value of the inverted index group in response to the file operation request.

[0010] Secondly, this application also provides a reverse indexing device based on a grouped bitmap, the reverse indexing device based on a grouped bitmap comprising: The block partitioning module is used to divide each merged block in the storage space into at least one inverted index group; The index key determination module is used to determine the index key of each reverse index group based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group. The index value acquisition module is used to construct a valid bitmap and file identifier array for each of the reverse index groups based on the grouping bit information corresponding to each of the reverse index groups, and to obtain the index value of each of the reverse index groups. The reverse index module is used to, upon receiving a file operation request for a target file, perform an index update or query operation corresponding to the file operation request based on the index key and index value of the reverse index group, in response to the file operation request.

[0011] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the reverse indexing method based on grouped bitmaps as described above.

[0012] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the above-described reverse indexing method based on a grouped bitmap.

[0013] This application provides a method, apparatus, computer device, and storage medium for inverted indexing based on grouped bitmaps. The method divides each merged block in the storage space into multiple inverted index groups, generates unique index keys based on block and group information, and further utilizes group bit information to construct a valid bitmap and a file identifier array to form index values. This achieves grouped management of the inverted index, refining the granularity of index operations to a single inverted index group, avoiding the global scan of the entire merged block in traditional methods, and significantly reducing operational overhead. The introduction of valid bitmaps reduces the time complexity of file operations, greatly improving the efficiency of index updates and queries. Simultaneously, the combination of the file identifier array and valid bitmaps preserves the unique identification information of files while optimizing storage space utilization through the compactness of the bitmap. When processing file operation requests, the target inverted index group can be quickly located and the corresponding operation executed, resulting in faster response times. This refined grouped index management method not only improves garbage collection efficiency but also optimizes storage space utilization and reduces the accumulation of invalid data, thereby improving the overall management and application efficiency of inverted indexes. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of an application environment for a reverse indexing method based on a grouped bitmap according to an embodiment of the present invention; Figure 2A flowchart illustrating an embodiment of a reverse indexing method based on a grouped bitmap provided in this application; Figure 3 This is a schematic diagram of a grouped bitmap reverse index provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an embodiment of a reverse indexing device based on a grouped bitmap provided in this application. Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0019] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The reverse indexing method based on grouped bitmaps provided in this invention can be applied to applications such as... Figure 1In this application environment, the client communicates with the server via a network. When the server receives a file operation request from the client, it can divide each merged block in the storage space into at least one inverted index group; based on the block information of the merged block corresponding to the inverted index group and the group information of each inverted index group, it determines the index key of each inverted index group; according to the group bit information corresponding to each inverted index group, it constructs a valid bitmap and a file identifier array for each inverted index group to obtain the index value of each inverted index group; when it receives a file operation request for the target file, it performs an index update or query operation corresponding to the file operation request according to the index key and index value of the inverted index group to respond to the file operation request.

[0022] This invention addresses the technical problems of low efficiency in reverse index update operations, wasted storage space resources, and low application efficiency of reverse indexes due to a lack of fine-grained management in the management of small files of financial transaction data and medical consultation data using existing distributed storage technologies. By dividing each merged block in the storage space into multiple reverse index groups and generating unique index keys based on block and group information, and further utilizing group bit information to construct an effective bitmap and file identifier array to form index values, this invention achieves grouped management of the reverse index. This refines the granularity of index operations to individual reverse index groups, avoiding the global scan of the entire merged block in traditional methods and significantly reducing operational overhead. The introduction of the effective bitmap reduces the time complexity of file operations, greatly improving the efficiency of index updates and queries. Simultaneously, the combination of the file identifier array and the effective bitmap preserves the unique identification information of the file while optimizing storage space utilization through the compactness of the bitmap. When processing file operation requests, the target reverse index group can be quickly located and the corresponding operation executed, resulting in faster response times. This refined grouped index management approach not only improves garbage collection (GC) efficiency but also optimizes storage space utilization and reduces the accumulation of invalid data. As a result, it improves the overall management and application efficiency of inverted indexes, providing an efficient, flexible, and reliable solution for small file management in distributed storage systems.

[0023] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0024] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of a reverse indexing method based on a grouped bitmap provided in this application.

[0025] like Figure 2 As shown, the reverse indexing method based on grouped bitmaps includes steps S101 to S104.

[0026] S101. Divide each merged block in the storage space into at least one inverted index group.

[0027] In distributed storage systems, storage space is typically divided into multiple merge blocks, each a logical unit used to store a certain number of small files. The size of a merge block can be configured according to the system's storage needs and performance requirements; for example, a merge block can contain thousands of small files. This design aims to optimize storage efficiency and improve data access speed.

[0028] To further optimize the management of files within a merge block, this embodiment divides each merge block into multiple inverted index groups. An inverted index group is a sub-unit within a merge block, used to manage a fixed number of small files. For example, one inverted index group can manage 128 small files. By dividing a merge block into multiple inverted index groups, the granularity of index management can be refined from the entire merge block to each inverted index group, thereby achieving more efficient data management.

[0029] Specifically, during storage system initialization, the size of each merge block and the group size of the inverted index group are determined based on the storage system's configuration parameters. For example, the merge block size is 1MB (a unit of storage space); the inverted index group size is 128 small files. It can be understood that the size of each inverted index group and the number of files within the group are dynamically configurable to adapt to different storage needs and performance requirements.

[0030] When a new merge block is created, the system automatically divides it into multiple inverted index groups. First, based on the size of the merge block and the size of each inverted index group, the number of inverted index groups the merge block can be divided into is calculated. For example, if a merge block can store 1024 small files, and each inverted index group manages 128 small files, then the merge block can be divided into 8 inverted index groups. Then, the storage system assigns a unique identifier to each inverted index group to facilitate precise index management and operations; its format can be "RG_Merged Block ID_Group Sequence Number". For example, the identifier for the first inverted index group of merge block 10587 is "RG_10587_1".

[0031] The introduction of inverted index groups makes file operations (such as insertion, deletion, and querying) more efficient. For example, the time complexity of a file deletion operation can be optimized from the traditional O(n) to O(1) because only the bitmap within the inverted index group needs to be updated, without traversing the entire merge block. By periodically cleaning up invalid inverted index groups, metadata space can be effectively reclaimed, preventing the metadata engine from bloating due to the accumulation of invalid indexes. The structure of inverted index groups allows the system to manage the data distribution within the merge block more finely, supporting more intelligent garbage collection strategies, such as prioritizing the reclamation of regions with a high proportion of invalid data.

[0032] S102. Based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group, determine the index key of each reverse index group.

[0033] To efficiently locate and manage each inverted index group, the storage system generates a globally unique index key for each inverted index group. The index key is a unique identifier for the inverted index group, used for quickly locating and manipulating each inverted index group within the metadata management system. An efficient and unique index key design can significantly improve system performance and scalability. The index key needs to contain sufficient information to quickly distinguish between different merge blocks and different inverted index groups within the same merge block.

[0034] To achieve efficient management and rapid location, the index key can consist of three parts: a prefix, a merge block ID, and a group number. The prefix is ​​a fixed prefix used to identify that this is a key belonging to an inverted index group, such as "RG_". The merge block ID is a unique identifier for a merge block, used to distinguish different merge blocks. The group number is a sequence number identifying different inverted index groups within the same merge block, used to distinguish different groups within the same merge block. This structural design ensures the index key is globally unique while retaining sufficient information for fast parsing and manipulation.

[0035] During system initialization or merge block creation, the storage system assigns a unique merge block ID to each merge block. The merge block ID is globally unique and used to identify different merge blocks. Merge block information is typically stored in a metadata management system, including the merge block size, storage location, and total number of files within the merge block.

[0036] When a merged block is divided into multiple inverted index groups, the storage system assigns a group number to each inverted index group. The group numbers start from 0 and increment sequentially. For example, if a merged block is divided into 8 inverted index groups, the group numbers will be 0 to 7.

[0037] For each inverted index group, the storage system generates an index key based on its merge block ID and group sequence number. Specifically, it retrieves the merge block ID of the current inverted index group from the metadata management system and determines its group sequence number based on the order of the inverted index group within the merge block. The prefix "RG_", the merge block ID, and the group sequence number are concatenated into a string to form the index key.

[0038] For example, suppose the storage system creates a new merge block with merge block ID 10587. This merge block is divided into 8 inverted index groups, each managing 128 small files. The storage system will generate an index key for each inverted index group as follows: the index key for the 0th inverted index group is RG_10587_0; the index key for the 1st inverted index group is RG_10587_1; ...; the index key for the 7th inverted index group is RG_10587_7.

[0039] The generated index key will be stored in the metadata management system and managed together with the index values ​​(valid bitmap and file identifier array) of the inverted index group. The metadata management system supports fast lookup and update operations, ensuring that the index key can be used efficiently for locating and manipulating the inverted index group.

[0040] During storage system operation, when a merged block needs to expand into a new inverted index group, the storage system assigns a new group number and generates a new index key. When a merged block is deleted, the storage system cleans up the index keys and related metadata of all its inverted index groups to free up storage space. When accessing a specific inverted index group is needed, its merged block ID and group number can be quickly obtained by parsing the index key. For example, for the index key RG_10587_3, the storage system can quickly resolve the merged block ID to 10587 and the group number to 3. This allows the storage system to quickly locate the target inverted index group and perform corresponding operations (such as file writing, deletion, or querying).

[0041] This embodiment ensures the uniqueness and operability of each inverted index group within the metadata management system by efficiently generating, storing, and managing the index keys of inverted index groups. This design not only improves the data management performance of the storage system but also enhances its scalability and maintainability.

[0042] S103. Based on the grouping information corresponding to each of the reverse index groups, construct the effective bitmap and file identifier array of each of the reverse index groups, and obtain the index value of each of the reverse index groups.

[0043] The index value of each inverted index group consists of two core components: a valid bitmap and a file identifier array.

[0044] The valid bitmap is a compact sequence of bits used to identify the validity of each file slot within a group. Each bit corresponds to a file slot, with a value of 1 indicating that the slot is valid (i.e., the file exists) and a value of 0 indicating that the slot is invalid (i.e., the file has been deleted or is not in use).

[0045] The file identifier array is a fixed-length ordered array used to store the unique identifiers (such as FileIDs) of each small file within a group. Each position in the array corresponds one-to-one with a bit in the valid bitmap, allowing for quick location and manipulation of specific files through the file identifier.

[0046] Group bit information refers to the status information of file slots within each inverted index group, including which slots are occupied and which slots are empty. Using this information, the storage system can dynamically construct a valid bitmap and a file identifier array, thereby forming a complete index value.

[0047] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a grouped bitmap reverse index structure provided in an embodiment of this application. Each reverse index group includes index keys, such as RG_10587_0 and RG_10587_1, indicating that there are two reverse index groups in the merged block with merged block ID 10587, and the group numbers of the two reverse index groups are 0 and 1. Each reverse index group is 6 bits long, meaning each reverse index group manages 6 file slots. The reverse index group with index key RG_10587_0 includes six file slots from file1 to file6. According to its valid bitmap, file2, file4, file5, and file6 are currently valid, while file1 and file3 are currently invalid. Similarly, the reverse index group with index key RG_10587_1 includes six file slots from file7 to file12. According to its valid bitmap, file8, file10, file11, and file12 are currently invalid, while file7 and file9 are currently valid.

[0048] Further, based on the validity of at least one file slot managed by the reverse index group, a valid bitmap of the reverse index group is determined; based on the unique identifier of the valid file stored in each file slot, a file identifier array of the reverse index group is determined; and based on the valid bitmap and the file identifier array, an index value of the reverse index group is constructed.

[0049] When creating an inverted index group, the storage system needs to initialize a valid bitmap and a file identifier array, and allocate a valid bitmap with a length equal to the group size for each inverted index group.

[0050] Specifically, based on the system configuration, the number of file slots managed by each inverted index group is determined (e.g., 128 slots); a valid bitmap with a length equal to the group size is created, initially set to all 0 values ​​to indicate that all slots are empty. Simultaneously, the storage system allocates a fixed-length file identifier array to each inverted index group to store the unique identifiers of each small file within the group.

[0051] The valid bitmap and file identifier array are combined into an index value for an inverted index group, which is then stored in the metadata management system for subsequent indexing operations and queries. Specifically, the valid bitmap and file identifier array can be encapsulated into a structured index value object; this index value is stored in the metadata management system and associated with the index key of the inverted index group.

[0052] When a file is written to a merge block, the valid bitmap is scanned to find the first bit with a value of 0, thus determining the location of a free slot. The storage system sequentially fills the file's unique identifier into the corresponding position in the file identifier array and sets the corresponding bit in the valid bitmap to 1, indicating that the slot is already occupied.

[0053] When a file is deleted, its unique identifier is used to locate its corresponding slot in the file identifier array. The storage system sets the corresponding bit in the valid bitmap to 0, indicating that the slot is empty; the identifier in the file identifier array remains unchanged, but the file corresponding to that slot is no longer valid.

[0054] When a storage system needs to query the status of a file, it uses the file's unique identifier to find the corresponding slot in the file identifier array. Based on the bit value at the corresponding position in the valid bitmap, it determines whether the file is valid. If the bit value is 1, the file is valid; if the bit value is 0, the file is invalid.

[0055] This embodiment provides an index value composed of an effective bitmap and a file identifier array, which can efficiently represent and manage the file status within the inverted index group and support fast file operations (such as insertion, deletion, and query). The time complexity of bitmap operations is O(1), which significantly improves the efficiency of file deletion and insertion; the compact bitmap representation of file status reduces the storage space of metadata; and this structure can realize dynamic file updates and queries, adapting to high-concurrency file operation scenarios, which not only improves the efficiency of file operations but also optimizes the use of storage space.

[0056] S104. Upon receiving a file operation request for the target file, perform an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group to respond to the file operation request.

[0057] In one embodiment, file operation requests may include file write requests, file delete requests, and file query requests. File write requests are used to write new files to the storage system; file delete requests are used to delete specified files from the storage system; and file query requests are used to query the status of specified files (whether they exist, are valid, etc.). File operation requests may also include other types of operation requests, such as file data modification requests, file splitting requests, file reassembly requests, file merging requests, and file migration requests. Each operation request requires corresponding update or query operations on the index keys and index values ​​of the inverted index group to ensure data consistency and efficiency of the system.

[0058] When responding to file operation requests and executing corresponding file operations, the index key is used to quickly locate the inverted index group where the target file resides, while the index value (valid bitmap and file identifier array) is used to manage and query the status of files within the inverted index group. Specifically, through the index key, the storage system can quickly find the corresponding index value (including the valid bitmap and file identifier array) to perform the specific file operation. The valid bitmap in the index value identifies the validity of the file slot, and the file identifier array stores the unique identifier of the file. Through these two components, the storage system can efficiently perform file operations.

[0059] When the storage system receives a file operation request (such as write, delete, or query) for a target file, it locates the corresponding merge block and target file or target storage location based on the index key and index value of the inverted index group. Depending on the type of file operation request, it performs the corresponding index update or query operation and returns the result. This inverted indexing method based on grouped bitmaps significantly improves the efficiency of index operations, especially optimizing the time complexity of file deletion operations from O(n) to O(1).

[0060] Specifically, the storage system receives a file operation request from a client. The request includes the target file's unique identifier (such as FileID) and the operation type (write, delete, or query). Based on the target file's unique identifier, the storage system locates the merge block and inverted index group containing the target file using either a forward index or a metadata management system. The storage system parses the file identifier to extract the merge block ID and inverted index group number from the file's unique identifier; it then generates an index key for the inverted index group based on the merge block ID and group number, and uses this index key to look up the corresponding index value (valid bitmap and file identifier array) in the metadata management system.

[0061] In one embodiment, if the file operation request is a file write request, based on the file write request, the file identifier of the target file is obtained, and a target merge block for writing the target file is determined; based on the index values ​​of each of the inverted index groups in the target merge block, the currently free target slot in the target merge block is found; the target file is written to the target slot, and based on the file identifier of the target file, the index value of the inverted index group corresponding to the target slot is updated to complete the writing of the target file.

[0062] When the storage system receives a file write request, it sequentially fills the free positions of the file identifier array with the unique identifier of the target file using the index key and index value of the inverted index group, and sets the corresponding bit in the valid bit map to the valid state, thereby achieving an efficient file write operation.

[0063] When a new file needs to be written, the storage system receives the file write request, obtains the unique identifier of the target file and the target merge block ID used to store the target file. It locates the corresponding inverted index group based on the target merge block ID. If the current inverted index group is full, a new inverted index group is automatically allocated. Based on the index value of the inverted index group, it finds the first free position in the file identifier array and fills that position with the unique identifier of the target file; simultaneously, it sets the corresponding bit in the valid bitmap to 1, indicating that the file slot is valid. The metadata management system is then updated to record the new index value.

[0064] Specifically, in response to a file write request, the storage system can scan the valid bitmap of the target merge block, find the first bit with a value of 0 within the target merge block, and determine the location of the free slot; fill the unique identifier of the target file into the corresponding position in the file identifier array; set the corresponding bit in the valid bitmap to 1, indicating that the slot is occupied. The updated index values ​​(valid bitmap and file identifier array) are written back to the metadata management system, and a successful write result is returned to the client.

[0065] For example, suppose the target file file9 needs to be written to the inverted index group RG_10587_3. In the inverted index group RG_10587_3, the first 8 bits of the current valid bitmap are 10001010, and the first 8 positions of the file identifier array are [file1, null, null, file4, file5, null, file7, file8]. When a file write request for the target file file9 is received, an empty slot in the inverted index group RG_10587_3 is searched. The second bit of its valid bitmap is 0, indicating that the second slot is empty. The target file file9 is written to this slot, the file identifier array is updated to [file1, file9, null, file4, file5, null, file7, file8], and the valid bitmap is updated to 11001010. The new index value is then written back to the metadata management system.

[0066] In one embodiment, if a target merge block is specified in the file write request, the storage system prioritizes searching for free slots within the target merge block. Specifically, based on the target merge block ID specified in the file write request, the storage system locates the corresponding target merge block. In all inverted index groups of the target merge block, the valid bitmap of each group is scanned sequentially to find bits with a value of 0 to determine free slots. If a free slot is found, the storage system further determines whether the storage space of that slot is sufficient to completely write all file data of the target file. If the target file is large and requires multiple slots, the storage system can allocate multiple free slots consecutively to meet the file storage requirements. If there are no free slots in the target merge block, or the storage space of the free slots is insufficient to completely write all file data of the target file, the storage system can return a reminder message to the client, prompting the user to re-specify the target merge block for file writing. The reminder message may include the following: the current storage status of the target merge block (e.g., full or insufficient remaining space); suggestions for the user to specify other merge blocks or expand the storage capacity of the current merge block; and one or more recommended merge block IDs that have sufficient free slots and storage space.

[0067] If the file write request does not specify a target merge block for file writing, the storage system automatically selects a suitable merge block according to a preset strategy. The selection strategy can be based on factors such as: the current storage utilization of the merge block (preferably prioritizing merge blocks with lower utilization); the access frequency of the merge block (preferably prioritizing merge blocks with lower access frequency); and the geographical location or storage media type of the merge block (selecting the optimal merge block based on system configuration and performance requirements). In the automatically selected merge block, the storage system can scan the valid bitmap of the inverted index group to find free slots and determine if the storage space meets the requirements of the target file. If enough free slots are found, the system writes the target file to the selected merge block and updates the corresponding index values ​​(valid bitmap and file identifier array). If not enough free slots are found, the storage system can automatically expand the storage capacity of the current merge block, create a new inverted index group, and write the file to the new slots; or it can reselect another suitable merge block and repeat the above steps. Regardless of whether the file write is successful, the system returns the corresponding result information to the client. If the file is successfully written, a confirmation message is returned, including the file storage location (such as the merge block ID and inverted index group sequence number). If file writing fails, the failure reason is returned, such as "not enough free slots found" or "insufficient storage space".

[0068] By employing the above methods, the storage system can flexibly handle file write requests, ensuring the efficiency and reliability of file write operations regardless of whether the user specifies a target merge block. Simultaneously, reasonable alerts and automatic selection mechanisms enhance the user experience and overall performance of the storage system.

[0069] In one embodiment, if the file operation request is a file deletion request, the file identifier of the target file is obtained; based on the file identifier, the target inverted index group where the target file is located and the index position of the target file in the target inverted index group are queried; the bit corresponding to the index position in the valid bitmap of the target inverted index group is set to an invalid state.

[0070] If the file operation request is a file deletion request, when the storage system receives the file deletion request, it locates the target merge block and inverted index group where the target file is located by the unique identifier of the target file, and then finds its index position in the array within the group, and sets the corresponding bit in the valid bit map to an invalid state.

[0071] Specifically, when the storage system receives a file deletion request from a client, the request contains the unique identifier (FileID) of the target file. This identifier is a globally unique identifier for the file, used to quickly locate the file's position within the storage system. Using a forward index (usually a hash table or other efficient index structure), the system quickly locates the target file's merge block and inverted index group based on its unique identifier. The forward index stores the mapping between file identifiers and merge blocks and inverted index groups, making the search process extremely efficient.

[0072] After locating the target inverted index group, the specific index position of the target file within that group is further found using the file identifier array (FileIDArray). The file identifier array is an ordered array that stores unique identifiers for all files within the inverted index group.

[0073] Based on the index position of the target file in the file identifier array, find the corresponding valid bit in the bitmap. Set this bit from 1 to 0 to indicate that the file slot is invalid. This operation has a time complexity of O(1) because it directly accesses and updates the bitmap through the index, without traversing the entire array or index structure, making it extremely fast and completely avoiding traversal overhead.

[0074] The updated index values ​​(including the valid bitmap and file identifier array) are written back to the metadata management system to ensure that the system's metadata is up-to-date. The metadata management system is responsible for storing and managing the index information of all inverted index groups and supports fast read and write operations.

[0075] The system returns a confirmation message to the client indicating successful file deletion. If any exceptions are encountered during the deletion process (such as the file already being deleted or not existing), the storage system will return corresponding error messages for the client to handle.

[0076] Generally, during file deletion, exceptions may occur such as file already deleted, file not found, or index inconsistencies. In these cases, the storage system needs to handle the exceptions. Specifically, if the target file has been deleted (the corresponding bit in the valid bitmap is already 0), the system returns a "file deleted" message. If the unique identifier of the target file is not found in the forward index or file identifier array, the system returns a "file not found" error message. If the information in the forward and reverse index groups is inconsistent, the system logs the exception and returns an "index inconsistency" error message, prompting the administrator to check and repair.

[0077] In one embodiment, if the file operation request is a file query request, the file identifier of the target file is obtained; based on the file identifier, the target inverted index group in which the target file is located is queried; based on the index value of the target inverted index group, the current file position and current file status of the target file are queried to respond to the file query request, wherein the current file status includes a valid file status and an invalid file status.

[0078] If the file operation request is a file query request, the unique identifier of the target file is used to locate its inverted index group and its position within the group, and the validity of the target file is determined based on the valid bitmap.

[0079] When the storage system receives a file query request, it locates the inverted index group and its position within the group using the unique identifier of the target file, and determines whether the target file is valid based on the valid bitmap. This process achieves efficient file query operations with a time complexity of O(1).

[0080] Specifically, the storage system receives a file query request from a client, which includes the unique identifier (FileID) of the target file. This identifier is a globally unique identifier for the file, used to quickly locate the file's position within the storage system. Using a forward index (usually a hash table or other efficient index structure), the system quickly locates the target file's block and inverted index group based on its unique identifier. The forward index stores the mapping between file identifiers and the blocks and inverted index groups, making the search process extremely efficient.

[0081] After locating the target inverted index group, the specific index position of the target file within that group is further found using the FileIDArray. The FileIDArray is an ordered array that stores unique identifiers for all files within the inverted index group. Based on the target file's index position in the FileIDArray, the corresponding bit in the valid bitmap is located. The value of this bit is checked: if the bit value is 1, the target file is valid; if the bit value is 0, the target file is invalid (it has been deleted or never written to).

[0082] Based on the results of the valid bitmap check, the query results are returned to the client: if the file is valid, the message "file exists and is valid" is returned; if the file is invalid, the message "file does not exist or has been deleted" is returned.

[0083] Generally, in addition to validity queries, file metadata queries, file location information queries, and file version information queries can also be performed. Specifically, if the target file is valid, the storage system can further return the file's metadata information, such as file size, creation time, and last modification time; if the client requests a query for file location information, it can return the merge block ID and inverted index group number where the target file is located, facilitating further operations by the client; if the client requests a query for file version information, it can return the current version number of the target file or other version-related information.

[0084] Understandably, during file lookup, issues such as invalid file identifiers and inconsistent indexes may occur. If the unique identifier of the target file is not found in the forward index, the storage system can return an "invalid file identifier" error message. If the information in the forward and reverse index groups is inconsistent (e.g., the corresponding file identifier is not found in the file identifier array), the storage system records an exception log and returns an "inconsistent index" error message, prompting the administrator to check and repair.

[0085] This embodiment provides a reverse indexing method based on grouped bitmaps. This method divides each merged block in the storage space into multiple reverse index groups, generates unique index keys based on block and group information, and further utilizes group bit information to construct a valid bitmap and a file identifier array to form index values. This achieves grouped management of the reverse index, refining the granularity of index operations to a single reverse index group, avoiding the global scan of the entire merged block in traditional methods, and significantly reducing operational overhead. The introduction of valid bitmaps reduces the time complexity of file operations, greatly improving the efficiency of index updates and queries. Simultaneously, the combination of the file identifier array and valid bitmaps preserves the unique identification information of files while optimizing storage space utilization through the compactness of the bitmap. When processing file operation requests, the target reverse index group can be quickly located and the corresponding operation executed, resulting in faster response times. This refined grouped index management method not only improves garbage collection (GC) efficiency but also optimizes storage space utilization and reduces the accumulation of invalid data, thereby improving the overall management and application efficiency of reverse indexes and providing an efficient, flexible, and reliable solution for small file management in distributed storage systems.

[0086] In distributed storage systems, with frequent file creation and deletion, a large number of invalid file slots (i.e., slots with a bit value of 0) gradually accumulate in the valid bitmap of inverted index groups. These invalid slots not only occupy valuable metadata storage space but may also lead to a decrease in the performance of the metadata engine. To optimize storage space utilization and improve the overall system performance, the valid bitmap of each inverted index group can be scanned periodically to identify and clean up inverted index groups where all file slots are invalid, thereby achieving self-reclaiming and optimization of metadata space.

[0087] Furthermore, based on a preset period, the index values ​​of each of the inverted index groups are read, and the validity of each file slot in the valid bitmap corresponding to each of the inverted index groups is identified; when the validity of each file slot in the valid bitmap is invalid, the index key and index value of the inverted index group are removed to release the storage space occupied by the inverted index group.

[0088] In one embodiment, the storage system can maintain a background task, configured with a preset period (such as every hour, every 12 hours, or every day) based on the actual usage and performance requirements of the storage system, to periodically trigger the background scan task. The preset period can be set through a configuration file or management interface and supports dynamic adjustment.

[0089] The background task periodically scans all valid bitmaps of inverted index groups according to a preset cycle, identifies and deletes inverted index groups with "all zero" bitmaps, thereby realizing the self-reclamation of metadata space, preventing the metadata engine from expanding indefinitely due to the accumulation of invalid historical indexes, reducing storage costs and improving the stability of the storage system.

[0090] Specifically, the storage system can initiate a background task to scan the valid bitmaps of all inverted index groups at a preset period (e.g., hourly). The execution frequency of the background task can be configured according to the system load and storage requirements. It iterates through all inverted index groups recorded in the metadata management system. For each inverted index group, its index value (valid bitmap and file identifier array) is read, and the background task checks in turn whether all bits in its valid bitmap are 0. The valid bitmap is a compact bit sequence, where each bit represents the validity of a file slot (1 for valid, 0 for invalid). If all bits in the valid bitmap are 0, it indicates that all file slots within that inverted index group are invalid.

[0091] If all valid bitmap values ​​of a given inverted index group are 0, it indicates that all files managed by that group are invalid. This means the inverted index group no longer contains any valid files and can be safely deleted, freeing up its occupied storage space. At this point, the storage system can remove the index key and index value of the inverted index group from the metadata management system, update the metadata of the merged block, and record the number and status of remaining valid inverted index groups within the current merged block. This allows for the safe deletion of the metadata entries for the inverted index group, releasing the occupied metadata space.

[0092] Write the new index structure back to the metadata management system to ensure that the system's metadata is up-to-date. The updated metadata management system reflects information about all valid inverted index groups in the current storage system.

[0093] This embodiment can efficiently identify and remove invalid inverted index groups, release occupied storage space, thereby realizing the self-reclamation of metadata space, effectively preventing the metadata engine from expanding indefinitely due to the accumulation of invalid historical indexes, reducing storage costs and improving the stability of the storage system.

[0094] In distributed storage systems, especially when dealing with large-scale small file storage scenarios, efficient storage space management and optimized garbage collection (GC) efficiency are crucial issues. While traditional inverted index schemes improve GC efficiency to some extent, they still have significant shortcomings in index maintenance, space reclamation, and fine-grained management. For example, inverted index update operations are inefficient, especially deletion operations, and the waste and expansion of storage space all hinder further performance improvements. Furthermore, the basic inverted index structure cannot support fine-grained optimization strategies, causing the garbage collector to be unable to effectively identify and clean up regions in invalid data sets.

[0095] To further optimize storage space utilization and improve overall system performance, this technical solution proposes a space reclamation mechanism based on the proportion of invalid data. This mechanism traverses the index values ​​of inverted index groups, identifies invalid file slots in the valid bitmap, filters out inverted index groups with a high proportion of invalid data, and performs space reclamation operations on these groups, thereby releasing the occupied storage space.

[0096] Further, the index values ​​of each of the inverted index groups are traversed to identify invalid file slots in the valid bitmap corresponding to the index values, and the proportion of invalid data is obtained; inverted index groups with an invalid data proportion greater than a preset proportion threshold are selected and marked as index groups to be reclaimed; space reclamation operation is performed on the index groups to be reclaimed to release the storage space occupied by the index groups to be reclaimed.

[0097] In one embodiment, the storage system can periodically (e.g., hourly or daily) traverse the index values ​​of all inverted index groups, including a validity bitmap and a file identifier array. By analyzing the validity bitmap, the number and location of invalid file slots in each inverted index group are identified. Each bit in the validity bitmap represents the validity of a file slot (1 indicates validity, 0 indicates invalidity).

[0098] For each inverted index group, calculate the proportion of invalid data. Specifically, count the number of bits with a value of 0 in the valid bitmap (invalid file slot bits), and calculate the proportion of invalid data, i.e., the percentage of invalid file slot bits out of the total file slot bits:

[0099] Based on the statistical results, inverted index groups with an invalid data ratio greater than a preset threshold (e.g., 80%) are selected and marked as index groups to be reclaimed. The preset threshold can be dynamically adjusted according to the system's storage needs and performance requirements.

[0100] Perform space reclamation operations on inverted index groups marked as awaiting reclamation, releasing the storage space they occupy. Specifically, traverse the valid bitmap of the index groups awaiting reclamation, identify all invalid file slots, and perform a cleanup operation on each invalid file slot to release the storage space it occupies. If all file slots of a certain inverted index group are invalid (i.e., all valid bitmap values ​​are 0), delete the metadata entries for that inverted index group, releasing the storage space of the entire group.

[0101] When a few valid file slots exist in a reverse index group, directly deleting the group may result in the loss of these valid files. Therefore, these valid files can be migrated to other merge blocks, and the information (index values) of the migrated reverse index group can be updated synchronously. After the data migration is complete, the metadata entries of the original reverse index group are deleted to free up the storage space of the entire group.

[0102] Specifically, during periodic scanning of inverted index groups, the system identifies those groups with a small number of valid file slots. For example, if an inverted index group has fewer than a preset threshold of valid file slots (e.g., 10% of the total slots), it is marked as a partially valid group. A suitable merge block is selected as the target merge block, and valid files from the partially valid inverted index groups are migrated to the target merge block. Specifically, data for all valid files can be read from the original inverted index group, written to the target merge block, and new slots are allocated in the inverted index group of the target merge block. The inverted index group information of the target merge block is synchronously updated, including the valid bitmap and file identifier array, to ensure that the migrated files can be correctly indexed. After the data migration is complete, the index keys and index values ​​of the original inverted index group are removed from the metadata management system, releasing the storage space of the entire group. The metadata of the original merge block is updated, recording the number and status of the remaining valid inverted index groups within the current merge block. By identifying partially valid inverted index groups and migrating valid files to other suitable merge blocks, storage space utilization is optimized, system storage costs are reduced, and the stability of the distributed storage system is enhanced.

[0103] After completing the space reclamation operation, update the metadata management system and optimize the index structure. Specifically, update the index values ​​(valid bitmap and file identifier array) of the inverted index groups to reflect the current file status, update the metadata of the merged block, and record the number and status of the remaining valid inverted index groups in the current merged block; if an inverted index group is deleted, update the index keys and index values ​​in the metadata management system to ensure the integrity of the index structure.

[0104] This embodiment, by periodically analyzing, filtering, and cleaning up inverted index groups with a high proportion of invalid data, can not only efficiently identify and clean up invalid data and free up storage space, but also significantly improve the overall performance and storage efficiency of the system, optimize the use of storage space, reduce the storage cost of the storage system, and enhance the stability and scalability of the distributed storage system.

[0105] Please see Figure 4 , Figure 4 This is a schematic diagram of the current embodiment of a reverse indexing device based on a grouped bitmap provided in this application. The reverse indexing device based on a grouped bitmap is used to execute the aforementioned reverse indexing method based on a grouped bitmap.

[0106] like Figure 4 As shown, the reverse indexing device 200 based on grouped bitmaps includes: a block partitioning module 201, an index key determination module 202, an index value acquisition module 203, and a reverse indexing module 204.

[0107] Block partitioning module 201 is used to partition each merged block in the storage space into at least one inverted index group; The index key determination module 202 is used to determine the index key of each reverse index group based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group. The index value acquisition module 203 is used to construct a valid bitmap and file identifier array for each of the reverse index groups based on the grouping bit information corresponding to each of the reverse index groups, and to obtain the index value of each of the reverse index groups; The reverse index module 204 is used to, upon receiving a file operation request for a target file, perform an index update or query operation corresponding to the file operation request based on the index key and index value of the reverse index group, in response to the file operation request.

[0108] In one embodiment, the index value obtaining module 203 includes: A valid bitmap determination unit is used to determine the valid bitmap of the reverse index group based on the validity of at least one file slot managed by the reverse index group. The file identifier array determination unit is used to determine the file identifier array of the reverse index group based on the unique identifier of the valid file stored in each of the file slots. An index value construction unit is used to construct the index value of the inverted index group based on the valid bitmap and the file identifier array.

[0109] In one embodiment, the reverse index module 204 includes a file write operation module, comprising: The write request parsing unit is configured to, if the file operation request is a file write request, obtain the file identifier of the target file based on the file write request, and determine the target merge block for writing the target file; The idle slot lookup unit is used to look up currently idle target slots in the target merge block based on the index values ​​of each of the reverse index groups in the target merge block; The file writing unit is used to write the target file into the target slot and update the index value of the inverted index group corresponding to the target slot based on the file identifier of the target file to complete the writing of the target file.

[0110] In one embodiment, the reverse index module 204 further includes a file deletion operation module, comprising: The file identifier acquisition unit is used to acquire the file identifier of the target file if the file operation request is a file deletion request. An index position query unit is used to query, based on the file identifier, the target inverted index group in which the target file is located, and the index position of the target file in the target inverted index group; An invalid state marking unit is used to set the bit corresponding to the index position in the valid bit map of the target reverse index group to an invalid state.

[0111] In one embodiment, the reverse index module 204 further includes a file query module, comprising: The query request parsing unit is used to obtain the file identifier of the target file if the file operation request is a file query request; The reverse index group query unit is used to query the target reverse index group where the target file is located based on the file identifier; The file query unit is used to query the current file position and current file status of the target file based on the index value of the target inverted index group in response to the file query request, wherein the current file status includes a valid file status and an invalid file status.

[0112] In one embodiment, the grouped bitmap-based reverse indexing device 200 further includes a reverse index group removal module, comprising: The index value reading unit is used to read the index value of each of the reverse index groups based on a preset period, and to identify the validity of each file slot in the valid bitmap corresponding to each of the reverse index groups. The reverse index group removal unit is used to remove the index key and index value of the reverse index group when the validity of each file slot in the valid bitmap is invalid, so as to release the storage space occupied by the reverse index group.

[0113] In one embodiment, the grouped bitmap-based reverse indexing device 200 further includes a space reclamation module, comprising: An invalid data identification unit is used to traverse the index values ​​of each of the reverse index groups, identify invalid file slots in the valid bitmap corresponding to the index values, and obtain the proportion of invalid data. The index group marking unit is used to filter out reverse index groups with an invalid data ratio greater than a preset ratio threshold and mark them as index groups to be recycled. The space reclamation execution unit is used to perform space reclamation operations on the index group to be reclaimed, so as to release the storage space occupied by the index group to be reclaimed.

[0114] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device and each module described above can be referred to the corresponding process in the aforementioned embodiment of the reverse indexing method based on grouped bitmaps, and will not be repeated here.

[0115] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0116] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0117] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0118] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any reverse indexing method based on a grouped bitmap.

[0119] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0120] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to perform any reverse indexing method based on a grouped bitmap.

[0121] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0123] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Divide each merged block in the storage space into at least one inverted index group; Based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group, the index key of each reverse index group is determined. Based on the group bit information corresponding to each inverted index group, construct the effective bitmap and file identifier array of each inverted index group, and obtain the index value of each inverted index group; Upon receiving a file operation request for the target file, an index update or query operation corresponding to the file operation request is executed based on the index key and index value of the inverted index group in response to the file operation request.

[0124] In one embodiment, when the processor constructs a valid bitmap and a file identifier array for each inverted index group based on the group bit information corresponding to each inverted index group, and obtains the index value of each inverted index group, it is configured to: The valid bitmap of the reverse index group is determined based on the validity of at least one file slot managed by the reverse index group. Based on the unique identifiers of valid files stored in each of the file slots, determine the file identifier array of the reverse index group; Based on the valid bitmap and the file identifier array, construct the index value of the inverted index group.

[0125] In one embodiment, when the processor implements the step of performing an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group upon receiving a file operation request for the target file, it is configured to: If the file operation request is a file write request, based on the file write request, obtain the file identifier of the target file, and determine the target merge block to be written to the target file; Based on the index value of each of the reverse index groups in the target merge block, find the currently free target slot in the target merge block; The target file is written to the target slot, and the index value of the inverted index group corresponding to the target slot is updated based on the file identifier of the target file to complete the writing of the target file.

[0126] In one embodiment, when the processor implements the step of performing an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group upon receiving a file operation request for the target file, in response to the file operation request, it is further configured to implement: If the file operation request is a file deletion request, obtain the file identifier of the target file; Based on the file identifier, query the target inverted index group where the target file is located, and the index position of the target file in the target inverted index group; Set the bit corresponding to the index position in the valid bitmap of the target inverted index group to an invalid state.

[0127] In one embodiment, when the processor implements the step of performing an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group upon receiving a file operation request for the target file, in response to the file operation request, it is further configured to implement: If the file operation request is a file query request, obtain the file identifier of the target file; Based on the file identifier, query the target inverted index group where the target file is located; Based on the index value of the target inverted index group, query the current file position and current file status of the target file to respond to the file query request, wherein the current file status includes file valid status and file invalid status.

[0128] In one embodiment, after the processor implements the steps of constructing a valid bitmap and file identifier array for each inverted index group based on the group bit information corresponding to each inverted index group, and obtaining the index value of each inverted index group, it is further configured to implement: Based on a preset period, the index value of each of the inverted index groups is read, and the validity of each file slot in the valid bitmap corresponding to each of the inverted index groups is identified. When the validity of each file slot in the valid bitmap is invalid, the index key and index value of the reverse index group are removed to release the storage space occupied by the reverse index group.

[0129] In one embodiment, after the processor implements the steps of constructing a valid bitmap and file identifier array for each inverted index group based on the group bit information corresponding to each inverted index group, and obtaining the index value of each inverted index group, it is further configured to implement: Traverse the index values ​​of each of the inverted index groups, identify the invalid file slots in the valid bitmap corresponding to the index values, and obtain the proportion of invalid data. Filter out inverted index groups where the proportion of invalid data exceeds a preset threshold, and mark them as index groups to be recycled; Perform a space reclamation operation on the index group to be reclaimed to release the storage space occupied by the index group to be reclaimed.

[0130] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the reverse indexing methods based on grouped bitmaps provided in the embodiments of this application.

[0131] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A reverse indexing method based on grouped bitmaps, characterized in that, The method includes: Divide each merged block in the storage space into at least one inverted index group; Based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group, the index key of each reverse index group is determined. Based on the group bit information corresponding to each inverted index group, construct the effective bitmap and file identifier array of each inverted index group, and obtain the index value of each inverted index group; Upon receiving a file operation request for the target file, an index update or query operation corresponding to the file operation request is executed based on the index key and index value of the inverted index group in response to the file operation request.

2. The reverse indexing method based on grouped bitmaps according to claim 1, characterized in that, The step of constructing a valid bitmap and file identifier array for each inverted index group based on the grouping bit information corresponding to each inverted index group, and obtaining the index value of each inverted index group, includes: The valid bitmap of the reverse index group is determined based on the validity of at least one file slot managed by the reverse index group. Based on the unique identifiers of valid files stored in each of the file slots, determine the file identifier array of the reverse index group; Based on the valid bitmap and the file identifier array, construct the index value of the inverted index group.

3. The reverse indexing method based on grouped bitmaps according to claim 1, characterized in that, Upon receiving a file operation request for a target file, the step of performing an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group to respond to the file operation request includes: If the file operation request is a file write request, based on the file write request, obtain the file identifier of the target file, and determine the target merge block to be written to the target file; Based on the index value of each of the reverse index groups in the target merge block, find the currently free target slot in the target merge block; The target file is written to the target slot, and the index value of the inverted index group corresponding to the target slot is updated based on the file identifier of the target file to complete the writing of the target file.

4. The reverse indexing method based on grouped bitmaps according to claim 1, characterized in that, The step of responding to a file operation request by performing an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group upon receiving the file operation request further includes: If the file operation request is a file deletion request, obtain the file identifier of the target file; Based on the file identifier, query the target inverted index group where the target file is located, and the index position of the target file in the target inverted index group; Set the bit corresponding to the index position in the valid bitmap of the target inverted index group to an invalid state.

5. The reverse indexing method based on grouped bitmaps according to claim 1, characterized in that, The step of responding to a file operation request by performing an index update or query operation corresponding to the file operation request based on the index key and index value of the inverted index group upon receiving the file operation request further includes: If the file operation request is a file query request, obtain the file identifier of the target file; Based on the file identifier, query the target inverted index group where the target file is located; Based on the index value of the target inverted index group, query the current file position and current file status of the target file to respond to the file query request, wherein the current file status includes file valid status and file invalid status.

6. The reverse indexing method based on grouped bitmaps according to claim 1, characterized in that, After constructing the effective bitmap and file identifier array of each inverted index group based on the grouping bit information corresponding to each inverted index group, and obtaining the index value of each inverted index group, the method further includes: Based on a preset period, the index values ​​of each of the inverted index groups are read, and the validity of each file slot in the valid bitmap corresponding to each of the inverted index groups is identified. When the validity of each file slot in the valid bitmap is invalid, the index key and index value of the reverse index group are removed to release the storage space occupied by the reverse index group.

7. The reverse indexing method based on grouped bitmaps according to claim 1, characterized in that, After the processor implements the step of constructing a valid bitmap and file identifier array for each inverted index group based on the grouping bit information corresponding to each inverted index group, and obtaining the index value of each inverted index group, the processor further includes: Traverse the index values ​​of each of the inverted index groups, identify the invalid file slots in the valid bitmap corresponding to the index values, and obtain the proportion of invalid data. Filter out inverted index groups where the proportion of invalid data exceeds a preset threshold, and mark them as index groups to be recycled; Perform a space reclamation operation on the index group to be reclaimed to release the storage space occupied by the index group to be reclaimed.

8. A reverse indexing device based on a grouped bitmap, characterized in that, The inverted indexing device based on grouped bitmaps includes: The block partitioning module is used to divide each merged block in the storage space into at least one inverted index group; The index key determination module is used to determine the index key of each reverse index group based on the block information of the merged block corresponding to the reverse index group and the group information of each reverse index group. The index value acquisition module is used to construct a valid bitmap and file identifier array for each of the reverse index groups based on the grouping bit information corresponding to each of the reverse index groups, and to obtain the index value of each of the reverse index groups. The reverse index module is used to, upon receiving a file operation request for a target file, perform an index update or query operation corresponding to the file operation request based on the index key and index value of the reverse index group, in response to the file operation request.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the reverse indexing method based on a grouped bitmap as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the reverse indexing method based on a grouped bitmap as described in any one of claims 1 to 7.

Images