Garbage collection method and device of block storage system, electronic equipment and storage medium

Abstract

The invention provides a garbage collection method and device for a block storage system, electronic equipment and a storage medium, and the method comprises the steps: collecting a plurality of effective data from a to-be-collected storage unit in a garbage collection process; recombining and arranging the plurality of effective data according to logical address information to form a logical address associated data sequence; executing erasure code coding processing on the data sequence, and storing the processed data in a newly allocated erasure code storage unit to generate a new data storage layout; updating system metadata according to the data storage layout so as to map logic addresses of the plurality of effective data to new physical positions in the erasure code storage unit; and releasing the binding relationship between the to-be-recycled storage unit and the physical storage space so as to recycle the storage resources occupied by the to-be-recycled storage unit. And on the premise of ensuring the reliability and accessibility of the data, the remarkable optimization of the storage cost is finally realized.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a garbage collection method, apparatus, electronic device and storage medium for a block storage system. Background Technology

[0002] Distributed block storage systems are the cornerstone of cloud computing, providing persistent data storage capabilities for products such as cloud disks. Their underlying mechanism typically employs a multi-replica mechanism to ensure data reliability, meaning user data is replicated into multiple complete copies and stored on servers in different fault domains. Simultaneously, to improve write performance, these systems commonly use redirected write technology, sequentially appending new data to a new storage area instead of overwriting old data. The old data thus becomes garbage data awaiting reclaiming, requiring cleanup and space reclamation by the background garbage collection (GC) process. In existing technical solutions, during background garbage collection, the system directly migrates valid data collected from the storage units awaiting reclaiming to newly allocated multi-replica storage units.

[0003] However, the existing technical solutions mentioned above do not change the data storage format after the entire process is completed, and still maintain a multi-copy mode, resulting in high storage costs. Summary of the Invention

[0004] This application provides a garbage collection method, apparatus, electronic device, and storage medium for a block storage system to solve the problem of high storage costs in the prior art.

[0005] In a first aspect, this application provides a garbage collection method for a block storage system, comprising: During the waste recycling process, multiple valid data points are collected from the storage units to be recycled; The multiple valid data are reorganized and arranged according to the logical address information to form a data sequence associated with logical addresses; The data sequence is subjected to erasure coding processing, and the processed data is stored in the newly allocated erasure coding storage unit to generate a new data storage layout; The system metadata is updated according to the data storage layout to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit; Release the binding relationship between the storage unit to be reclaimed and the physical storage space in order to reclaim the storage resources occupied by the storage unit to be reclaimed.

[0006] In one possible implementation, the storage unit to be reclaimed includes multiple historical data blocks written in an append-only manner; The collection of multiple valid data from the storage units to be reclaimed includes: From each of the historical data blocks, select the valid data pages that have not been marked as obsolete; Each page of valid data selected will be considered as a valid data point.

[0007] In one possible implementation, the step of reorganizing and arranging multiple valid data according to logical address information to form a logically address-associated data sequence includes: Obtain the original logical address corresponding to each valid data from the logical address information; All valid data are sorted according to the original logical address to form a data sequence associated with the logical address.

[0008] In one possible implementation, performing erasure coding on the data sequence and storing the processed data in a newly allocated erasure coding storage unit includes: Using logical stripes of a preset size as the writing unit, the data sequence associated with the logical address is divided into at least one logical stripe unit; An erasure coding process is performed on the data of each logical stripe unit to generate multiple corresponding data segments and check segments; The data fragment is written into the data column of the erasure coding storage unit, and the verification fragment is written into the verification column of the erasure coding storage unit.

[0009] In one possible implementation, the data storage layout characterizes the correspondence between the sequential positions in the data sequence and the physical storage positions in the erasure coding storage unit; The step of updating system metadata according to the data storage layout includes: For each valid data, based on the sequential position of the valid data in the data sequence, the data storage layout is queried to obtain the new physical position of the valid data in the erasure coding storage unit; In the system metadata, the logical address of the valid data is mapped to the new physical location.

[0010] In one possible implementation, after updating the system metadata according to the data storage layout, the method further includes: Receive a read request, the read request carrying a target logical address; Query the physical storage location information corresponding to the target logical address based on the updated system metadata; If the physical storage location information indicates that the data to be read is stored in the erasure coding storage unit, the corresponding data is read from the erasure coding storage unit according to the physical storage location information.

[0011] In one possible implementation, the method further includes: Determine the amount of invalid data and the total amount of data in the storage unit to be reclaimed; The proportion of invalid data is determined based on the amount of invalid data and the total amount of data. When the proportion of invalid data exceeds a preset threshold, a garbage collection event is triggered.

[0012] Secondly, this application provides a garbage collection device for a block storage system, comprising: The collection module is used to collect multiple valid data from the storage unit to be recycled during the waste recycling process; The reorganization module is used to reorganize and arrange multiple valid data according to logical address information to form a data sequence associated with logical addresses; The processing module is used to perform erasure coding processing on the data sequence and store the processed data into the newly allocated erasure coding storage unit to generate a new data storage layout. The update module is used to update the system metadata according to the data storage layout, so as to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit; The recycling module is used to unbind the storage unit to be recycled from the physical storage space, so as to reclaim the storage resources occupied by the storage unit to be recycled.

[0013] In one possible implementation, the storage unit to be reclaimed includes multiple historical data blocks written in an append-only manner; the collection module is specifically used for: From each of the historical data blocks, select the valid data pages that have not been marked as obsolete; Each page of valid data selected will be considered as a valid data point.

[0014] In one possible implementation, the reorganization module is specifically used for: Obtain the original logical address corresponding to each valid data from the logical address information; All valid data are sorted according to the original logical address to form a data sequence associated with the logical address.

[0015] In one possible implementation, the processing module is specifically used for: Using logical stripes of a preset size as the writing unit, the data sequence associated with the logical address is divided into at least one logical stripe unit; An erasure coding process is performed on the data of each logical stripe unit to generate multiple corresponding data segments and check segments; The data fragment is written into the data column of the erasure coding storage unit, and the verification fragment is written into the verification column of the erasure coding storage unit.

[0016] In one possible implementation, the data storage layout represents the correspondence between the sequential positions in the data sequence and the physical storage positions in the erasure coding storage unit; the update module is specifically used for: For each valid data, based on the sequential position of the valid data in the data sequence, the data storage layout is queried to obtain the new physical position of the valid data in the erasure coding storage unit; In the system metadata, the logical address of the valid data is mapped to the new physical location.

[0017] In one possible implementation, the device further includes a reading module for: Receive a read request, the read request carrying a target logical address; Query the physical storage location information corresponding to the target logical address based on the updated system metadata; If the physical storage location information indicates that the data to be read is stored in the erasure coding storage unit, the corresponding data is read from the erasure coding storage unit according to the physical storage location information.

[0018] In one possible implementation, the device further includes a trigger module for: Determine the amount of invalid data and the total amount of data in the storage unit to be reclaimed; The proportion of invalid data is determined based on the amount of invalid data and the total amount of data. When the proportion of invalid data exceeds a preset threshold, a garbage collection event is triggered.

[0019] Thirdly, this application provides an apparatus comprising: a processor and a memory, the processor being configured to execute a garbage collection program of a block storage system stored in the memory, to implement the garbage collection method of the block storage system described in any one of the first aspects.

[0020] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the garbage collection method of the block storage system described in the first aspect.

[0021] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application, after collecting valid data from the storage unit to be reclaimed during garbage collection, does not simply migrate it to a new multi-copy unit as in the prior art. Instead, it reassembles and arranges the valid data according to logical address information to form a data sequence with consecutive logical addresses. Then, it performs erasure coding on this data sequence and stores the encoded data in a newly allocated erasure coding storage unit. This key processing step achieves a fundamental shift in data storage format from a multi-copy mode to a high-density erasure coding mode, directly reducing the physical storage space requirement by reducing data redundancy. Simultaneously, by updating the system metadata mapping relationship and unbinding the original storage unit, significant optimization of storage costs is achieved while ensuring data reliability and accessibility. This method, which organically integrates erasure coding with the garbage collection process, cleverly completes storage format optimization in the background, avoiding impact on foreground write performance. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0025] Figure 1 This refers to the composition and pointing relationship of sequential data blocks in the metadata of the post-garbage collection stage, after a period of reading and writing during the traditional garbage collection scheduling process. Figure 2 A flowchart illustrating an embodiment of a garbage collection method for a block storage system provided in this application; Figure 3 A schematic diagram of the structure of a 4+2 mode erasure coding storage unit provided in an embodiment of this application; Figure 4 This document defines the composition and pointer relationships of sequential data blocks in the metadata of the garbage collection scheduling process after a period of reading and writing and garbage collection. Figure 5A flowchart illustrating an embodiment of a garbage collection method for a block storage system provided in this application; Figure 6 A flowchart illustrating an embodiment of a garbage collection method for a block storage system provided in this application; Figure 7 A block diagram illustrating an embodiment of a garbage collection device for a block storage system provided in this application. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0028] To enable those skilled in the art to better understand this application, the basic architecture of the underlying distributed storage system of block storage involved in this application will first be described.

[0029] The underlying distributed storage system for block storage typically employs an address segmentation design, linearly dividing the logical address space of the user's cloud disk into multiple contiguous segment groups (typically hundreds of GB in size). Each segment group contains multiple segments (typically representing tens of GB of address space). Through this address segmentation mechanism, the underlying block storage service processes the input and output operations of the cloud disk at the segment level. Each segment is scheduled to be processed by a dedicated server, enabling multiple servers to process the input and output operations of different segments in parallel, thereby improving the overall read and write parallelism of the system.

[0030] Within the address space managed by a segment, the underlying block storage typically divides cloud disk input / output operations into fixed-size data pages for processing, with page size usually set to 4KB. For persistent data storage, data blocks are used to carry cloud disk data. These blocks employ a redirected write mechanism: when a user repeatedly writes data to the same logical offset address on the cloud disk, the underlying storage system does not overwrite the existing data. Instead, it sequentially appends the new data to the end of the current block, while marking the original data as invalid (garbage). Subsequently, through a background garbage collection process, the valid data in these blocks is migrated to newly allocated blocks, and the storage space occupied by the original blocks is released. Data blocks that support append-only semantics are called sequence blocks (sdb).

[0031] The size of an sdb is configurable, for example, it can be configured to 2GB. When an sdb is full, the system will automatically switch to a new sdb to continue writing. Each sdb contains a metadata structure that records the offset position information of each data page within the block. After a period of write operations, the composition and pointing relationship of the sdb in the segment metadata are as follows: Figure 1 As shown.

[0032] During data reading, the system traverses the database from the latest sdb. If the target data page is found in the current sdb, it is immediately returned to the client; if not found, the search continues in earlier sdbs; if the target data page is not found in any sdb, it indicates that the data page has not been written, and the system returns all zeros.

[0033] Based on the above storage architecture, this application provides a garbage collection method for a block storage system, such as... Figure 2 As shown, the method includes the following steps: Step 201: During the waste recycling process, collect multiple valid data from the storage unit to be recycled.

[0034] Storage units awaiting reclamation: These refer to historical data blocks in a distributed storage system that were written using the append-only method and contain invalid data, and whose storage space utilization has fallen below the system's set threshold.

[0035] Valid data refers to data units in the storage unit that have not been marked as logically deleted and that maintain the latest data state.

[0036] In this embodiment of the application, the storage unit to be reclaimed includes multiple historical data blocks written in an append-only manner; collecting multiple valid data from the storage unit to be reclaimed may specifically include the following steps: filtering out valid data pages that have not been marked as discarded from each of the historical data blocks; and treating each filtered valid data page as a valid data.

[0037] A valid data page is a fixed-size data unit within a data block that has not been marked as logically deleted or overwritten and still represents valid user data. The page size is typically 4KB.

[0038] This scheme involves scanning each historical data block separately, parsing the metadata information of its internal recorded data status, identifying and filtering out all valid data pages that have not been marked as obsolete, and treating each filtered valid data page as an independent valid data unit.

[0039] This solution enables the system to accurately extract all valid data from the storage units to be recycled, providing an accurate data source for subsequent data reassembly and erasure coding, ensuring the integrity and reliability of data processing during the waste recycling process, and laying a data foundation for reducing storage costs.

[0040] Step 202: Reorganize and arrange multiple valid data according to the logical address information to form a data sequence associated with logical addresses.

[0041] Logical address information refers to the offset identifier of data in the address space of the virtual block device, which is recorded in the metadata structure inside or outside the valid data unit.

[0042] Reorganization and permutation: refers to the process of reorganizing data according to a specific order.

[0043] In this embodiment of the application, step 202 may specifically include the following steps: obtaining the original logical address corresponding to each valid data from the logical address information; sorting all valid data according to the original logical address to form a data sequence associated with the logical address.

[0044] Original logical address: refers to the original offset of user data in the logical address space of the user's cloud disk, used to uniquely identify the location of the data in the cloud disk; data sequence associated with logical address: refers to a continuous set of data arranged in logical address order.

[0045] In this scheme, the system retrieves the original logical address associated with each valid data point from its associated logical address information. Subsequently, the system sorts and reorganizes all valid data in ascending order of their original logical address values ​​(or conversely, in descending order), forming a logically continuous and ordered data sequence. This sorting operation reorganizes data units that may be scattered across different data blocks in physical storage into a continuous sequence based on their logical addresses. This process prepares structured data input for subsequent erasure coding in fixed-size stripes, ensuring the logical continuity of data within the stripes. It not only improves coding efficiency but also lays a solid foundation for locality in subsequent data reading operations, making it a crucial preliminary step for achieving efficient storage space compression and reclamation.

[0046] Step 203: Perform erasure coding on the data sequence and store the processed data in the newly allocated erasure coding storage unit to generate a new data storage layout.

[0047] Erasure coding storage unit: refers to a distributed storage structure built on EC (Erasure Coding) technology, consisting of a physical stripe containing multiple data columns and multiple parity columns. Each column is carried by an sdb (Sequence Block) that supports append write operations, and the size of each column's sdb is configured to be the same. Figure 3 The diagram shown illustrates the structure of a 4+2 mode erasure coding storage unit (EC Block). Figure 3 As shown, this EC Block contains 4 data columns and 2 check columns. If the size of each column sdb is 512MB, then the storage capacity of the entire EC Block is (4+2)×512MB=3GB, and the user data it can carry is 4×512MB=2GB.

[0048] Data storage layout: used to characterize the correspondence between the sequential position in the data sequence and the physical storage location in the erasure coding storage unit.

[0049] In this embodiment of the application, step 203 may specifically include the following steps: dividing the data sequence associated with the logical address into at least one logical stripe unit using a logical stripe of a preset size as the writing unit; performing erasure coding on the data of each logical stripe unit to generate multiple corresponding data fragments and check fragments; writing the data fragments into the data column of the erasure coding storage unit, and writing the check fragments into the check column of the erasure coding storage unit.

[0050] Logical stripe: refers to the smallest data unit that undergoes erasure coding. Its size is preset according to the EC Block configuration (i.e., the default size), for example, it can be 256KB. "Data fragment" and "check fragment" refer to the data blocks and corresponding check data blocks generated after the logical stripe unit is divided by erasure coding calculation.

[0051] In this scheme, the sorted, logically consecutive data sequence is first divided into several complete logical stripe units according to a preset logical stripe size. Then, erasure coding is performed independently on each logical stripe unit. Taking a 4+2 mode EC Block as an example, the logical stripe unit is divided into four 64KB data segments, and two 64KB parity segments are generated. The logical address space of the EC Block adopts a striped mapping structure, meaning that data within a logical stripe is sequentially mapped to each data column (e.g., the first 64KB is mapped to the first data column, the second 64KB to the second data column, and so on, the fifth 64KB to the next 64KB interval of the first data column, and so on). Finally, the generated data segments and parity segments are written to the specified positions in the sdb contained in the corresponding data column and parity column of the newly allocated EC Block, according to their logical address column offsets. This process enables the efficient conversion of multi-replica data into EC encoding format. Through striped mapping and fixed-granularity writing, while ensuring data reliability, it reduces storage redundancy from multi-replica mode to EC mode, significantly improving storage space utilization and thus effectively reducing the overall storage cost of the distributed storage system.

[0052] Step 204: Update the system metadata according to the data storage layout to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit.

[0053] System metadata refers to a collection of management information that records the mapping relationship between logical addresses and physical addresses.

[0054] In this embodiment of the application, step 204 may specifically include the following steps: for each valid data, query the data storage layout according to the sequential position of the valid data in the data sequence to obtain the new physical location of the valid data in the erasure coding storage unit; and map the logical address of the valid data to the new physical location in the system metadata.

[0055] New physical location: refers to the actual physical address where valid data is located after it is written to the EC Block, including the data column / checksum column identifier, sdb block identifier, and column offset.

[0056] For each valid data point that has completed migration, the scheme queries the established EC Block data storage layout mapping table based on its sequential position in the reassembled data sequence to accurately calculate the specific physical location of the data in the EC Block. Subsequently, the original logical address of the valid data is updated and mapped to this new physical location in the system metadata.

[0057] After introducing EC Blocks, the system can migrate valid data from multiple historical sdb instances within a Segment to newly allocated EC Blocks during the background garbage collection process. This process significantly reduces data redundancy by converting data originally stored in a multi-replica format to erasure coding format, thereby effectively reducing storage costs. After garbage collection, the composition and pointer relationships of sdb instances in the Segment Meta are as follows: Figure 4 As shown.

[0058] Step 205: Release the binding relationship between the storage unit to be reclaimed and the physical storage space in order to reclaim the storage resources occupied by the storage unit to be reclaimed.

[0059] Association relationship: refers to the logical mapping relationship between the storage unit to be reclaimed and the underlying physical storage medium. This relationship is maintained and managed through system metadata.

[0060] Available status: refers to the state in which storage resources are marked by the system as being reallocatable and able to be written with new data. Storage space in this state can be included in the system's idle resource pool.

[0061] Storage resources: refers to the actual storage capacity provided by physical storage media in a distributed storage system, including but not limited to physical storage media such as hard disk space and flash memory space.

[0062] In this embodiment, the metadata management module of the operating system disconnects the mapping relationship between the storage unit to be reclaimed and the physical storage space, marking this part of the storage space as available so that it can be reallocated to new data write requests, thereby completing the storage resource reclamation and reuse process. This operation ensures that the storage system can continuously and efficiently utilize physical storage capacity and avoid waste of storage space.

[0063] The technical solution provided in this application, after collecting valid data from the storage unit to be reclaimed during garbage collection, does not simply migrate it to a new multi-copy unit as in existing technologies. Instead, it reorganizes and arranges the valid data according to logical address information, forming a data sequence with consecutive logical addresses. Then, it performs erasure coding on this data sequence and stores the encoded data in a newly allocated erasure coding storage unit. This key process achieves a fundamental shift in data storage format from a multi-copy mode to a high-density erasure coding mode, directly reducing the physical storage space requirement by reducing data redundancy. Simultaneously, by updating the system metadata mapping relationship and unbinding the original storage unit, it significantly optimizes storage costs while ensuring data reliability and accessibility. This method, which organically integrates erasure coding with the garbage collection process, cleverly optimizes the storage format in the background, avoiding impact on foreground write performance.

[0064] Figure 5 A flowchart illustrating an embodiment of a garbage collection method for a block storage system provided in this application. Figure 5 The process shown is in Figure 1 Based on the illustrated process, the following steps are included: Step 501: Receive a read request, wherein the read request carries a target logical address; Step 502: Query the physical storage location information corresponding to the target logical address based on the updated system metadata; Step 503: If the physical storage location information indicates that the data to be read is stored in the erasure coding storage unit, read the corresponding data from the erasure coding storage unit according to the physical storage location information.

[0065] For ease of understanding, steps 501-503 will be explained uniformly below: Target logical address: refers to the offset of the user data specified in the read request within the logical address space of the virtual block device.

[0066] Physical storage location information refers to the physical location identifier of the target data, which is obtained by querying system metadata. It can indicate whether the data is stored in a regular multi-replica block or an EC block. If it is stored in an EC block, the information includes details such as the data column / checksum column identifier, SDB identifier, and column offset.

[0067] In this embodiment of the application, by receiving a user's read request and parsing its target logical address, the system metadata updated after garbage collection is queried to determine the actual storage location of the data. If it is determined that the data is located in the EC Block, the data fragment is read from the precise location in the specified SDB of the corresponding column of the EC Block according to the detailed physical location information (including the column identifier, SDB identifier and offset information of the metadata record), and reassembled into the original data and returned to the requester.

[0068] Figure 5 The read process shown achieves transparent access to EC Block stored data by maintaining and querying precise metadata mappings. This ensures the consistency and retrievability of data after conversion from multi-replica to erasure coding storage format, making the optimization of storage costs completely imperceptible to front-end users, thus balancing cost and performance.

[0069] Figure 6 A flowchart illustrating an embodiment of a garbage collection method for a block storage system provided in this application. Figure 6 The process shown is in Figure 1 Based on the illustrated process, the following steps are included: Step 601: Determine the amount of invalid data and the total amount of data in the storage unit to be reclaimed; Step 602: Determine the proportion of invalid data based on the amount of invalid data and the total amount of data; Step 603: If the proportion of invalid data is greater than a preset threshold, trigger a garbage collection event.

[0070] For ease of understanding, steps 601-603 will be explained uniformly below: Invalid data volume: refers to the storage capacity occupied by data marked as discarded or logically deleted in the storage unit to be reclaimed.

[0071] Total data volume: refers to the total storage capacity currently occupied by the storage units to be reclaimed, including valid data and invalid data.

[0072] Invalid data ratio: This refers to the percentage of invalid data to the total data volume, used to quantitatively assess the space reclamation potential of storage units.

[0073] Preset threshold: This refers to the critical ratio value that the system pre-configures to determine whether to start the garbage collection process. This threshold can be adjusted according to the storage strategy, for example, set to 60%.

[0074] In this embodiment, the system scans the metadata of the storage unit to be reclaimed, counts the number of data pages marked as invalid, calculates their storage capacity, and obtains the total storage capacity currently allocated to the storage unit. Subsequently, it calculates the precise proportion of invalid data based on the amount of invalid data and the total amount of data. When the calculated proportion is greater than the system's preset threshold, it indicates that there is enough reclaimable space in the storage unit and the recycling benefit is higher than the operating cost. The system then automatically triggers a garbage collection event and starts the subsequent data migration and space reclamation process.

[0075] Figure 6 The triggering mechanism shown realizes intelligent and on-demand triggering of the garbage collection process by quantitatively evaluating the space utilization of storage units. This avoids the frequent invalid operations that may be caused by traditional fixed-period collection, significantly reduces system overhead, and ensures the timeliness and efficiency of storage space collection. It provides a key guarantee for the continuous optimization of storage resource utilization in distributed storage systems.

[0076] Figure 7 This is a block diagram illustrating an embodiment of a garbage collection device for a block storage system provided in this application. Figure 7 As shown, the device includes: Collection module 71 is used to collect multiple valid data from the storage unit to be recycled during the waste recycling process; The recombination module 72 is used to recombine and arrange multiple valid data according to logical address information to form a data sequence associated with logical addresses; Processing module 73 is used to perform erasure coding processing on the data sequence and store the processed data into the newly allocated erasure coding storage unit to generate a new data storage layout; The update module 74 is used to update the system metadata according to the data storage layout, so as to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit; The recycling module 75 is used to unbind the storage unit to be recycled from the physical storage space in order to reclaim the storage resources occupied by the storage unit to be recycled.

[0077] In one possible implementation, the storage unit to be reclaimed includes multiple historical data blocks written in an append-only manner; the collection module is specifically used for: From each of the historical data blocks, select the valid data pages that have not been marked as obsolete; Each page of valid data selected will be considered as a valid data point.

[0078] In one possible implementation, the reorganization module is specifically used for: Obtain the original logical address corresponding to each valid data from the logical address information; All valid data are sorted according to the original logical address to form a data sequence associated with the logical address.

[0079] In one possible implementation, the processing module is specifically used for: Using logical stripes of a preset size as the writing unit, the data sequence associated with the logical address is divided into at least one logical stripe unit; An erasure coding process is performed on the data of each logical stripe unit to generate multiple corresponding data segments and check segments; The data fragment is written into the data column of the erasure coding storage unit, and the verification fragment is written into the verification column of the erasure coding storage unit.

[0080] In one possible implementation, the data storage layout represents the correspondence between the sequential positions in the data sequence and the physical storage positions in the erasure coding storage unit; the update module is specifically used for: For each valid data, based on the sequential position of the valid data in the data sequence, the data storage layout is queried to obtain the new physical position of the valid data in the erasure coding storage unit; In the system metadata, the logical address of the valid data is mapped to the new physical location.

[0081] In one possible implementation, the device further includes a reading module for: Receive a read request, the read request carrying a target logical address; Query the physical storage location information corresponding to the target logical address based on the updated system metadata; If the physical storage location information indicates that the data to be read is stored in the erasure coding storage unit, the corresponding data is read from the erasure coding storage unit according to the physical storage location information.

[0082] In one possible implementation, the device further includes a trigger module for: Determine the amount of invalid data and the total amount of data in the storage unit to be reclaimed; The proportion of invalid data is determined based on the amount of invalid data and the total amount of data. When the proportion of invalid data exceeds a preset threshold, a garbage collection event is triggered.

[0083] like Figure 8As shown in the figure, this application provides a device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the garbage collection method of the block storage system provided in any of the foregoing method embodiments, including: During the waste recycling process, multiple valid data points are collected from the storage units to be recycled; The multiple valid data are reorganized and arranged according to the logical address information to form a data sequence associated with logical addresses; The data sequence is subjected to erasure coding processing, and the processed data is stored in the newly allocated erasure coding storage unit to generate a new data storage layout; The system metadata is updated according to the data storage layout to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit; Release the binding relationship between the storage unit to be reclaimed and the physical storage space in order to reclaim the storage resources occupied by the storage unit to be reclaimed.

[0084] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the garbage collection method for a block storage system as provided in any of the foregoing method embodiments.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0087] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0088] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A garbage collection method for a block storage system, characterized in that, The method includes: During the waste recycling process, multiple valid data points are collected from the storage units to be recycled; The multiple valid data are reorganized and arranged according to the logical address information to form a data sequence associated with logical addresses; The data sequence is subjected to erasure coding processing, and the processed data is stored in the newly allocated erasure coding storage unit to generate a new data storage layout; The system metadata is updated according to the data storage layout to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit; Release the binding relationship between the storage unit to be reclaimed and the physical storage space in order to reclaim the storage resources occupied by the storage unit to be reclaimed.

2. The method according to claim 1, characterized in that, The storage unit to be reclaimed includes multiple historical data blocks written in an append-only manner; The collection of multiple valid data from the storage units to be reclaimed includes: From each of the historical data blocks, select the valid data pages that have not been marked as obsolete; Each page of valid data selected will be considered as a valid data point.

3. The method according to claim 1, characterized in that, The step of reorganizing and arranging multiple valid data according to logical address information to form a logically address-associated data sequence includes: Obtain the original logical address corresponding to each valid data from the logical address information; All valid data are sorted according to the original logical address to form a data sequence associated with the logical address.

4. The method according to claim 1, characterized in that, The step of performing erasure coding on the data sequence and storing the processed data in the newly allocated erasure coding storage unit includes: Using logical stripes of a preset size as the writing unit, the data sequence associated with the logical address is divided into at least one logical stripe unit; An erasure coding process is performed on the data of each logical stripe unit to generate multiple corresponding data segments and check segments; The data fragment is written into the data column of the erasure coding storage unit, and the verification fragment is written into the verification column of the erasure coding storage unit.

5. The method according to claim 1, characterized in that, The data storage layout represents the correspondence between the sequential position in the data sequence and the physical storage position in the erasure coding storage unit; The step of updating system metadata according to the data storage layout includes: For each valid data, based on the sequential position of the valid data in the data sequence, the data storage layout is queried to obtain the new physical position of the valid data in the erasure coding storage unit; In the system metadata, the logical address of the valid data is mapped to the new physical location.

6. The method according to claim 1, characterized in that, After updating the system metadata according to the data storage layout, the method further includes: Receive a read request, the read request carrying a target logical address; Query the physical storage location information corresponding to the target logical address based on the updated system metadata; If the physical storage location information indicates that the data to be read is stored in the erasure coding storage unit, the corresponding data is read from the erasure coding storage unit according to the physical storage location information.

7. The method according to claim 1, characterized in that, The method further includes: Determine the amount of invalid data and the total amount of data in the storage unit to be reclaimed; The proportion of invalid data is determined based on the amount of invalid data and the total amount of data. When the proportion of invalid data exceeds a preset threshold, a garbage collection event is triggered.

8. A waste recycling device for a block storage system, characterized in that, The device includes: The collection module is used to collect multiple valid data from the storage unit to be recycled during the waste recycling process; The reorganization module is used to reorganize and arrange multiple valid data according to logical address information to form a data sequence associated with logical addresses; The processing module is used to perform erasure coding processing on the data sequence and store the processed data into the newly allocated erasure coding storage unit to generate a new data storage layout. The update module is used to update the system metadata according to the data storage layout, so as to map the logical addresses of multiple valid data to new physical locations in the erasure coding storage unit; The recycling module is used to unbind the storage unit to be recycled from the physical storage space, so as to reclaim the storage resources occupied by the storage unit to be recycled.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a garbage collection program of a block storage system stored in the memory to implement a garbage collection method of the block storage system according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the garbage collection method of the block storage system according to any one of claims 1-7.

Images