File storage method and device

Abstract

The invention provides a file storage method and device, and relates to the technical field of computers, in particular to the technical field of distributed data storage. A specific embodiment of the method comprises the steps of segmenting a to-be-stored file into data objects; segmenting the disk file into data blocks consistent with the data object in size; idle data blocks are applied from the disk file, and the data objects in the to-be-stored file are correspondingly written into the idle data blocks in the disk file; in response to determining that the space recovery condition is met, determining a to-be-released space from the disk file; and migrating the data in the to-be-released space to the free space in the disk file.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of distributed data storage technology. Background Technology

[0002] In a distributed storage system, the final storage medium for data is the disk. Therefore, a dedicated storage engine is needed to realize the distributed management of user data. This means ensuring the security of data storage through reasonable redundancy mechanisms, as well as completing the orderly organization and efficient scheduling of data on the disk to meet the actual needs of large-scale data storage.

[0003] The mainstream implementation in the industry is as follows: users split the original file into 4MB data objects, and the system maps them to physical collections (Volumes) for storage using a hash algorithm. Simultaneously, data redundancy protection is achieved using Volumes as the basic unit based on Erasure Coding (EC) technology. In the specific storage logic, Volumes are split into data fragments (vlets) according to a preset EC erasure ratio, and each vlet corresponds one-to-one with a 32GB disk file. Data objects are stored on the disk file through vlets. The widely used LinkedIn engine adopts this core architecture; Volumes are created on demand, and free space is reserved after data deletion for subsequent writes by the same user. Summary of the Invention

[0004] This disclosure provides a file storage method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, embodiments of this disclosure propose a file storage method, comprising: dividing a file to be stored into data objects; dividing a disk file into data blocks of the same size as the data objects; requesting free data blocks from the disk file and writing the data objects from the file to be stored into the free data blocks in the disk file accordingly; in response to determining that the space reclamation conditions are met, determining the space to be released from the disk file; and migrating the data in the space to be released to the free space in the disk file.

[0006] Secondly, embodiments of this disclosure provide a file storage device, comprising: a first segmentation module configured to segment a file to be stored into data objects; a second segmentation module configured to segment a disk file into data blocks of the same size as the data objects; a writing module configured to request free data blocks from the disk file and write the data objects from the file to be stored into the free data blocks in the disk file; a determining module configured to determine space to be released from the disk file in response to determining that space reclamation conditions are met; and a migration module configured to migrate data in the space to be released to the free space in the disk file.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] The key or essential features of the embodiments disclosed herein are not intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein: Figure 1 This is a flowchart of an embodiment of the file storage method according to the present disclosure; Figure 2 This is a flowchart of yet another embodiment of the file storage method according to the present disclosure; Figure 3 This is a flowchart of another embodiment of the file storage method according to the present disclosure; Figure 4 This is a schematic diagram of a structure of a file storage device according to an embodiment of the present disclosure; Figure 5 This is a block diagram of an electronic device used to implement the file storage method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0014] In this disclosure, the key terms involved are defined as follows: Aries: A distributed storage data foundation that provides underlying support for data storage using a distributed implementation approach; Data object (Object): A data unit written by the user, which is the smallest logical unit after the original user file is split, and its size is usually 4M; Physical Volume: A physical collection of data objects. If the kn erasure algorithm is used, it will be split into k original data fragments and nk check data fragments (i.e., vlets). Logical data fragment (vlet): The data fragment formed after the volume is split is a logical concept in the technical solution disclosed in this publication. It only records the occupancy of 32G logical space and does not form a one-to-one correspondence with a single disk file. Erasure coding (EC): A redundancy protection technology used to split stored data. kn represents k original data and the remaining nk data are check data, which can restore data when some data is lost or damaged. Disk file (Collection): Corresponds to a physical storage file of fixed size (usually 32G), which is the carrier of data blocks; Data block: A physical storage unit formed after a disk file is split. Its size is the same as the data object (usually 4M), and it is the smallest allocation unit for data storage. Backend layer: A newly added disk space management layer used to record information on occupied and free data blocks on each disk file, providing support for the allocation, migration, and reclamation of data blocks.

[0015] Figure 1 A flow 100 of an embodiment of a file storage method according to the present disclosure is shown. The file storage method includes the following steps: Step 101: Divide the file to be stored into data objects.

[0016] In this embodiment, after receiving a user's file storage request, the executing entity of the file storage method (such as a server or storage node in a distributed storage system) will divide the file to be stored into several data objects according to a preset fixed granularity. The preset fixed granularity is typically 4MB. This size setting balances data transmission efficiency with metadata management overhead and maintains consistency with the data block size after subsequent disk file splitting, laying the foundation for efficient data storage and reuse. The original file written by the user is split into standardized data objects through this step, facilitating unified distributed management and redundancy protection by the system.

[0017] Step 102: Divide the disk file into data blocks that are the same size as the data objects.

[0018] In this embodiment, the execution entity first configures the physical storage medium as a fixed-capacity disk file (i.e., a Collection), typically 32GB. To achieve precise matching between data objects and physical storage units, and thus support space reuse for different logical data shards (vlets), the execution entity divides each 32GB disk file into data blocks of the same size as the data objects, i.e., each data block is 4MB in size. Through this splitting method, the disk file is divided into several standardized physical storage units, breaking the one-to-one binding relationship between vlets and disk files in traditional technologies. This allows vlets corresponding to different volumes to share different data blocks within the same disk file, providing a structural foundation for improving disk space utilization.

[0019] Step 103: Request a free data block from the disk file and write the corresponding data object from the file to be stored into the free data block in the disk file.

[0020] In this embodiment, the execution entity requests free data blocks from any unfilled disk file based on the logical space occupancy requirements of the vlet. Since the vlet has been changed to a logical concept, only recording the occupancy of 32GB of logical space and no longer bound to a single disk file, vlets corresponding to different volumes can flexibly request free data blocks from multiple disk files. During data writing, the execution entity records the association information of each data block. This association information includes at least the disk file identifier to which the data block belongs, the offset of the data block in that disk file, the vlet identifier corresponding to the data block, and the logical position of the data block within the vlet. This association information ensures accurate location and reading of data objects, providing crucial support for subsequent data migration, recycling, and reuse. Simultaneously, disk files are created on demand; the execution entity only creates a new 32GB disk file as the carrier for subsequent data writing when all currently created disk files are full, avoiding the waste of idle disk resources.

[0021] Step 104: In response to determining that the space reclamation conditions are met, determine the space to be freed from the disk file.

[0022] In this embodiment, as user data is written and deleted, free space gradually accumulates in the disk file. This includes fine-grained holes formed by deleting some data within a data block, free data blocks formed after all data objects in a data block are deleted, and even high proportions of free space in some disk files due to the deletion of large amounts of data. To achieve efficient utilization of disk space, the execution entity monitors the space usage status of the disk file in real time. When it detects that the preset space reclamation conditions are met, it determines the space to be released from the disk file. The space reclamation conditions include two core scenarios: one is that the proportion of free data blocks in the disk file reaches a first preset proportion threshold, in which case the space to be released is the overall space corresponding to the disk file; the other is that the proportion of free data in a single data block reaches a second preset proportion threshold, in which case the space to be released is the local space corresponding to that data block. By identifying and reclaiming free space at different granularities, the problems of space waste and space misalignment in traditional technologies can be comprehensively solved.

[0023] Step 105: Migrate the data in the space to be released to the free space in the disk file.

[0024] In this embodiment, after determining the space to be released, the execution entity first verifies whether the target free space can accommodate the data in the space to be released, ensuring the security and feasibility of data migration. If the verification passes, the valid data in the space to be released is migrated to the target free space. Specifically, if the space to be released is the entire space of a disk file, the execution entity will migrate all valid data of occupied data blocks in that disk file to free data blocks in other disk files that are not full. After the migration is completed, the original disk file is deleted, releasing 32G of disk space. If the space to be released is a partial space of a data block, the execution entity will migrate the valid data in that data block to other data blocks containing free areas. After the migration is completed, the original data block is marked as free, allowing other vlets to apply for reuse. Through this data migration mechanism, centralized release and efficient reuse of free space are achieved, significantly improving the utilization rate of disk space.

[0025] In some embodiments, the disk file identifier to which the occupied data block belongs and the offset of the occupied data block in the disk file are recorded.

[0026] To ensure accurate location and retrieval of data objects in a distributed storage system, after a data object is written to a data block, detailed key location information occupying the data block is recorded, including the disk file identifier to which the data block belongs and the offset of the data block within the corresponding disk file. The disk file identifier is used to quickly locate the physical storage medium where the data block resides, while the offset is used to precisely locate the specific storage location of the data block within the disk file. Combined with the vlet identifier and the logical position of the data block within the vlet, this information constitutes a complete data object location system, enabling the system to quickly find the target data in subsequent read, migration, and recycling operations, ensuring the efficiency and accuracy of data operations.

[0027] In some embodiments, in response to determining that a disk file is full and that there are unwritten data objects in the file to be stored, a new disk file is created to continue storing the unwritten data objects.

[0028] The system monitors the data block occupancy of each disk file in real time. When it is determined that all data blocks of a disk file are occupied (i.e., the disk file is full), and there are still some data objects to be stored that have not been written, a new fixed-capacity (32GB) disk file is automatically created. The newly created disk file is also divided into 4MB data blocks, and free data blocks are requested from the new disk file to continue storing the unwritten data objects. This mechanism ensures that user files to be stored can be stored continuously and completely, avoiding storage interruptions due to a single disk file being full. At the same time, through a standardized disk file creation and splitting process, the consistency and stability of the entire distributed storage system architecture are guaranteed.

[0029] In some embodiments, in response to determining that there are data blocks in a disk file where all data objects have been deleted, the data blocks where all data objects have been deleted are marked as free.

[0030] The system continuously monitors the storage status of data objects in each data block. When it detects that all data objects in a data block have been deleted by users, the data block is immediately marked as free and returned to the corresponding disk file. Data blocks marked as free are added to the system's free resource pool for other vlets to reuse when requesting data blocks. This mechanism breaks the limitation of traditional technology that the corresponding disk file can only be released when all data objects in a vlet are deleted. It achieves fine-grained space reclamation at the data block level, enabling timely release of free space generated by data deletion and effectively avoiding the waste of idle disk space.

[0031] The file storage method provided in this disclosure standardizes data storage units by dividing the file to be stored into standardized data objects and the disk file into data blocks of corresponding sizes. By breaking the one-to-one correspondence between vlets and disk files, it allows vlets of different volumes to share free data blocks of the same disk file, laying the foundation for space reuse. Precise association information recording ensures accurate data location and operation. Multi-granularity space reclamation and data migration mechanisms enable efficient release and reuse of free space. This method effectively solves the problems of space waste and misalignment in traditional distributed storage technologies, significantly improves disk space utilization, reduces storage costs, and ensures data storage redundancy and security.

[0032] Figure 2 A flow 200 of another embodiment of the file storage method according to this disclosure is shown. The file storage method includes the following steps: Step 201: Divide the file to be stored into data objects.

[0033] Step 202: Divide the disk file into data blocks that are the same size as the data objects.

[0034] Step 203: Request a free data block from the disk file and write the corresponding data object from the file to be stored into the free data block in the disk file.

[0035] In this embodiment, the specific operations of steps 201-203 have been described. Figure 1 The steps 101-103 in the illustrated embodiments are described in detail and will not be repeated here.

[0036] Step 204: In response to determining that the conditions for reclaiming disk file space are met, determine the disk files to be released from the disk files.

[0037] In this embodiment, in response to determining that the disk file space reclamation conditions are met, the execution entity can identify disk files to be released from the disk files. With frequent data write and delete operations, a large number of free data blocks will be generated in some disk files, resulting in excessively low disk file space utilization and unnecessary occupation of physical storage resources. To release this inefficiently occupied disk space, the execution entity monitors the proportion of free data blocks in each disk file. When this proportion reaches a first preset threshold, a disk file-level space reclamation process is triggered, and disk files to be released are selected from among them.

[0038] In some embodiments, the proportion of free data blocks in a disk file is determined; if the proportion of free data blocks reaches a first preset proportion threshold, it is determined that the conditions for reclaiming disk file space are met.

[0039] The Backend layer provides real-time statistics on the space usage of each disk file. As a newly added disk space management layer, the Backend layer continuously records the total number of data blocks and the number of occupied data blocks for each disk file. The number of free data blocks for each disk file is obtained by calculating the difference between the total number of data blocks and the number of occupied data blocks; then, the ratio of free data blocks to total data blocks is calculated to obtain the free data block ratio of that disk file. The first preset ratio threshold can be flexibly configured according to the actual storage needs of the distributed storage system. For example, it can be set to 70%, meaning that when the free data block ratio of a disk file exceeds 70%, its space utilization is considered too low, meeting the conditions for disk file space reclamation and triggering the subsequent space reclamation process.

[0040] In some embodiments, the total number of data blocks and the number of occupied data blocks of the disk file are counted; the difference between the total number of data blocks and the number of occupied data blocks is calculated to obtain the number of free data blocks of the disk file; the ratio of the number of free data blocks to the total number of data blocks is calculated to obtain the free data block ratio.

[0041] The total number of data blocks is determined by the fixed capacity of the disk file and the data block size. For example, a 32GB disk file divided into 4MB data blocks has a total of 8192 data blocks (32GB, 4MB, 4MB). (1024M / 4M=8192). The number of occupied data blocks is counted in real time by the Backend layer. The Backend layer marks and updates the occupancy status of data blocks in each disk file. When a data block is requested for use, it is marked as occupied and included in the number of occupied data blocks; when a data block is released to a free state, it is promptly removed from the number of occupied data blocks. Through the above calculation method, the proportion of free data blocks in each disk file can be accurately obtained, providing an accurate basis for judging the conditions for disk file space reclamation.

[0042] In some embodiments, disk files to be released are selected from the disk files based on the number of data blocks occupied by each disk file.

[0043] When selecting disk files to be released, the core principle is to choose the disk file with the highest free space to maximize the release of disk space. Free space can be indirectly judged by the number of data blocks occupied by the disk file; the fewer the occupied data blocks, the more free data blocks the disk file has, the higher the proportion of free data blocks, and the higher the free space level. Sort all disk files by the number of occupied data blocks and select the disk file with the fewest occupied data blocks as the disk file to be released. If multiple disk files have the same number of occupied data blocks and the fewest available, one can be randomly selected as the disk file to be released, or a selection can be made based on auxiliary conditions such as the disk file's creation time and data access frequency.

[0044] Step 205: In response to determining that the number of free data blocks in other disk files besides the disk file to be released is not less than the number of occupied data blocks in the disk file to be released, the data objects in the disk file to be released are migrated to free data blocks in other disk files.

[0045] In this embodiment, to ensure the safe and complete migration of valid data in the disk file to be released, the execution entity first uses the Backend layer to count the total number of free data blocks in all other disk files besides the disk file to be released, and compares this number with the number of occupied data blocks in the disk file to be released. If the total number of free data blocks in other disk files is not less than the number of occupied data blocks in the disk file to be released, it indicates that the existing free resources can accommodate all valid data in the disk file to be released, and data migration is feasible. At this time, the execution entity will start the data migration process, migrating the data objects in the disk file to be released one by one to the free data blocks in other disk files. If the total number of free data blocks in other disk files is insufficient, the current disk file space reclamation process is paused, and it will be restarted when there are enough free data blocks later.

[0046] In some embodiments, free data blocks are requested from other disk files, and data objects in the disk file to be released are written to the free data blocks in other disk files accordingly; in response to determining that all data objects in the disk file to be released have been written to other disk files, the disk file to be released is deleted.

[0047] The data migration process is executed by the aforementioned entity based on the free data block information provided by the Backend layer. Sufficient free data blocks are selected from other partially filled disk files. A corresponding target free data block is allocated to each occupied data block in the disk file to be released. Based on recorded association information (such as the vlet identifier of the data block and its logical position within the vlet), the valid data of the data blocks in the disk file to be released is written to the target free data block one by one according to the original storage logic. During the data writing process, the association information is updated in real time to ensure that the migrated data objects can be accurately located and read. Once all data blocks in the disk file to be released have been migrated and the data integrity has been confirmed, the disk file to be released is deleted, completely releasing the 32GB of disk space it occupied. This space can be used to create new disk files or for other data blocks to apply for.

[0048] The file storage method provided in this disclosure achieves accurate monitoring and statistics of disk file space usage status through the Backend layer, triggering disk file-level space reclamation based on the proportion of free data blocks; it ensures maximum release of free space through scientific rules for filtering disk files to be released; and it guarantees the security and reliability of data migration through feasibility verification before migration and integrity confirmation after migration. This method can effectively clean up inefficiently occupied disk files caused by the deletion of large amounts of data, further improving the overall utilization of disk space and avoiding the space idle problem caused by the binding of vlets to disk files in traditional technologies.

[0049] Figure 3 A flow 300 according to another embodiment of the file storage method of this disclosure is shown. The file storage method includes the following steps: Step 301: Divide the file to be stored into data objects.

[0050] Step 302: Divide the disk file into data blocks that are the same size as the data objects.

[0051] Step 303: Request a free data block from the disk file and write the corresponding data object from the file to be stored into the free data block in the disk file.

[0052] In this embodiment, the specific operations of steps 301-303 have been described. Figure 1 The steps 101-103 in the illustrated embodiments are described in detail and will not be repeated here.

[0053] Step 304: In response to determining that the data block space reclamation conditions are met, determine the data blocks to be released from the occupied data blocks in the disk file.

[0054] In this embodiment, in addition to the free space at the disk file level, fine-grained free space (i.e., holes) may also exist within some occupied data blocks in the disk file. These holes can arise from reasons such as: only part of the data object being written to the data block (not full), or some data objects being deleted after the data block is full. To achieve more refined space reclamation, the aforementioned execution entity monitors the proportion of free data in each occupied data block. When the data block space reclamation conditions are met, it determines the data blocks to be released from the occupied data blocks. This reclamation process can be manually triggered by the user or automatically triggered by the system according to a preset strategy, suitable for scenarios requiring rapid release of disk space for other data storage.

[0055] In some embodiments, the proportion of idle data occupying a data block is determined; if the proportion of idle data reaches a second preset proportion threshold, it is determined that the data block space reclamation condition is met.

[0056] The total data capacity and actual occupied data capacity of each occupied data block are calculated. The total data capacity of a data block is a fixed value (usually 4MB), and the actual occupied data capacity is the total size of the data objects already stored in that data block. The amount of free data in the data block is obtained by calculating the difference between the total data capacity and the actual occupied data capacity; then, the ratio of the free data amount to the total data capacity is calculated to obtain the proportion of free data in the data block. A second preset threshold ratio can be flexibly configured according to actual needs. For example, it can be set to 50%, meaning that when the proportion of free data in an occupied data block exceeds 50%, its internal voids are considered too large, meeting the data block space reclamation conditions, and data migration and space release are required.

[0057] In some embodiments, a data block to be released is selected from the occupied data blocks based on the amount of occupied data in each occupied data block.

[0058] When selecting data blocks to be released, the proportion of free data is the core criterion, while the amount of occupied data (i.e., the actual occupied data capacity) can indirectly reflect the proportion of free data. With a fixed total data capacity, the smaller the occupied data volume, the larger the free data volume, and the higher the proportion of free data. The occupied data volume of all occupied data blocks is statistically analyzed and sorted, and the occupied data block with the smallest occupied data volume is selected as the data block to be released. This aims to release the most free space with the fewest data migration operations, thereby improving space reclamation efficiency.

[0059] Step 305: In response to determining that the amount of free data in other occupied data blocks besides the data block to be released is not less than the amount of occupied data in the data block to be released, the data in the data block to be released is migrated to other occupied data blocks.

[0060] In this embodiment, to ensure that the valid data in the data block to be released can be completely migrated without occupying additional data block resources, the execution entity first calculates the total amount of free data in all other occupied data blocks besides the data block to be released. If the total amount of free data is not less than the amount of occupied data in the data block to be released, it indicates that the free areas (holes) in other occupied data blocks are sufficient to accommodate the valid data in the data block to be released, and the backfilling of the internal holes in the data block is feasible. At this time, the execution entity will start the data migration process to migrate the valid data in the data block to be released to the free areas of other occupied data blocks. If the total amount of free data is insufficient, the current data block space reclamation process is paused, and it will be triggered again when other data blocks generate sufficient free data.

[0061] In some embodiments, data in the data block to be released is written to other occupied data blocks; in response to determining that all data in the data block to be released has been written to other occupied data blocks, the data block to be released is marked as idle.

[0062] First, target data blocks with sufficient free space and data volume are selected from other occupied data blocks. Then, based on the storage logic recorded in the association information, valid data from the data blocks to be released is read and written sequentially to the free space of the target data blocks. During the data writing process, the actual occupied data capacity and association information of the target data blocks are updated synchronously to ensure data integrity and accessibility. Once all valid data in the data block to be released has been successfully written to other occupied data blocks, the data block to be released is marked as free and returned to the corresponding disk file, included in the system's free data block resource pool, for other vlets to request and reuse. Through this fine-grained hole backfilling mechanism, the free space inside data blocks can be fully utilized, further improving disk space utilization and avoiding space waste caused by holes inside data blocks.

[0063] The file storage method provided in this disclosure addresses the issue of fine-grained voids within data blocks by designing a precise space reclamation mechanism. It triggers data block-level space reclamation by monitoring the proportion of idle data occupying a data block; ensures the security of data migration through feasibility verification; and achieves the cyclical reuse of data block resources through void backfilling and idle status marking within data blocks. This method complements the disk file-level space reclamation mechanism, forming a multi-layered, comprehensive space reclamation system that maximizes the disk space utilization of the distributed storage system and reduces storage costs.

[0064] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a file storage device, which is similar to... Figure 1Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0065] like Figure 4 As shown, the file storage device 400 of this embodiment may include: a first splitting module 401, a second splitting module 402, a writing module 403, a determining module 404, and a migration module 405. The first splitting module 401 is configured to split the file to be stored into data objects; the second splitting module 402 is configured to split the disk file into data blocks of the same size as the data objects; the writing module 403 is configured to request free data blocks from the disk file and write the data objects from the file to be stored into the corresponding free data blocks in the disk file; the determining module 404 is configured to determine space to be released from the disk file in response to determining that space reclamation conditions are met; and the migration module 405 is configured to migrate the data in the space to be released to the free space in the disk file.

[0066] In this embodiment, the specific processing of the first segmentation module 401, the second segmentation module 402, the writing module 403, the determining module 404, and the migration module 405 in the file storage device 400, and the resulting technical effects, can be found in reference to [reference needed]. Figure 1 The relevant descriptions of steps 101-105 in the corresponding embodiments will not be repeated here.

[0067] In some optional implementations of this embodiment, the determining module 404 is further configured to: determine the disk file to be released from the disk file in response to determining that the disk file space reclamation conditions are met; and the migration module 405 is further configured to: migrate the data objects in the disk file to be released to the free data blocks in other disk files in response to determining that the number of free data blocks in other disk files is not less than the number of occupied data blocks in the disk file to be released.

[0068] In some optional implementations of this embodiment, the determining module 404 is further configured to: determine the proportion of free data blocks in the disk file; if the proportion of free data blocks reaches a first preset proportion threshold, determine that the disk file space reclamation conditions are met.

[0069] In some optional implementations of this embodiment, the determining module 404 is further configured to: count the total number of data blocks and the number of occupied data blocks of the disk file; calculate the difference between the total number of data blocks and the number of occupied data blocks to obtain the number of free data blocks of the disk file; and calculate the ratio of the number of free data blocks to the total number of data blocks to obtain the free data block ratio.

[0070] In some optional implementations of this embodiment, the determining module 404 is further configured to: select disk files to be released from the disk files based on the number of data blocks occupied by each disk file.

[0071] In some optional implementations of this embodiment, the migration module 405 is further configured to: request free data blocks from other disk files, and write the data objects in the disk file to be released into the free data blocks in other disk files accordingly; in response to determining that all data objects in the disk file to be released have been written into other disk files, delete the disk file to be released.

[0072] In some optional implementations of this embodiment, the determining module 404 is further configured to: determine a data block to be released from the occupied data blocks in the disk file in response to determining that the data block space reclamation conditions are met; and the migration module 405 is further configured to: migrate the data in the data block to be released to other occupied data blocks in response to determining that the amount of free data in other occupied data blocks besides the data block to be released is not less than the amount of occupied data in the data block to be released.

[0073] In some optional implementations of this embodiment, the determining module 404 is further configured to: determine the proportion of idle data occupying the data block; if the proportion of idle data reaches a second preset proportion threshold, determine that the data block space reclamation condition is met.

[0074] In some optional implementations of this embodiment, the determining module 404 is further configured to: select data blocks to be released from the occupied data blocks based on the amount of occupied data in each occupied data block.

[0075] In some optional implementations of this embodiment, the migration module 405 is further configured to: write the data in the data block to be released into other occupied data blocks; and, in response to determining that all the data in the data block to be released has been written into other occupied data blocks, mark the data block to be released as idle.

[0076] In some optional implementations of this embodiment, the file storage device 400 further includes a recording module configured to record the disk file identifier to which the occupied data block belongs and the offset of the occupied data block in the disk file.

[0077] In some optional implementations of this embodiment, the file storage device 400 further includes: a creation module configured to create a new disk file and continue storing the unwritten data objects in response to determining that a disk file is full and that there are unwritten data objects in the file to be stored.

[0078] In some optional implementations of this embodiment, the file storage device 400 further includes a marking module configured to mark the data block in which all data objects have been deleted as free in response to determining that there are data blocks in the disk file in which all data objects have been deleted.

[0079] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0080] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0081] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0082] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0083] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0084] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the file storage method. For example, in some embodiments, the file storage method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the file storage method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the file storage method by any other suitable means (e.g., by means of firmware).

[0085] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0089] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0090] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0091] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0092] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A file storage method, comprising: Split the file to be stored into data objects; The disk file is divided into data blocks of the same size as the data object; Request a free data block from the disk file, and write the data object from the file to be stored into the free data block of the disk file accordingly; In response to determining that the space reclamation conditions are met, space to be released is determined from the disk file; The data in the space to be released is migrated to the free space in the disk file.

2. The method according to claim 1, wherein, The step of determining the space to be released from the disk file in response to determining that the space reclamation conditions are met includes: In response to determining that the conditions for reclaiming disk file space are met, determine the disk files to be released from the disk files; and The step of migrating the data in the space to be released to the free space in the disk file includes: In response to determining that the number of free data blocks in other disk files besides the disk file to be released is not less than the number of occupied data blocks in the disk file to be released, the data objects in the disk file to be released are migrated to the free data blocks in the other disk files.

3. The method according to claim 2, wherein, The determination that the disk file space reclamation conditions are met includes: Determine the proportion of free data blocks in the disk file; If the proportion of free data blocks reaches a first preset proportion threshold, it is determined that the conditions for reclaiming disk file space are met.

4. The method according to claim 3, wherein, Determining the proportion of free data blocks in the disk file includes: Calculate the total number of data blocks and the number of data blocks occupied by the disk file; The difference between the total number of data blocks and the number of occupied data blocks is calculated to obtain the number of free data blocks of the disk file; The ratio of the number of free data blocks to the total number of data blocks is calculated to obtain the free data block ratio.

5. The method according to claim 2, wherein, The step of determining the disk file to be released from the disk file includes: The disk file to be released is selected from the disk files based on the number of data blocks occupied by each disk file.

6. The method according to claim 2, wherein, The step of migrating the data objects in the disk file to be released to free data blocks in other disk files includes: Request free data blocks from the other disk files, and write the data objects from the disk files to be released into the free data blocks of the other disk files accordingly; In response to determining that all data objects in the disk file to be released have been written to the other disk files, the disk file to be released is deleted.

7. The method according to claim 1, wherein, The step of determining the space to be released from the disk file in response to determining that the space reclamation conditions are met includes: In response to determining that the data block space reclamation conditions are met, a data block to be released is determined from the occupied data blocks in the disk file; and The step of migrating the data in the space to be released to the free space in the disk file includes: In response to determining that the amount of free data in other occupied data blocks besides the data block to be released is not less than the amount of occupied data in the data block to be released, the data in the data block to be released is migrated to the other occupied data blocks.

8. The method according to claim 7, wherein, The determination that the data block space reclamation conditions are met includes: Determine the proportion of free data in the occupied data block; If the proportion of idle data reaches the second preset proportion threshold, it is determined that the data block space reclamation condition is met.

9. The method according to claim 7, wherein, The step of determining the data blocks to be released from the occupied data blocks in the disk file includes: Based on the amount of data occupied in each occupied data block, the data block to be released is selected from the occupied data blocks.

10. The method according to claim 7, wherein, The step of migrating the data in the data block to be released to the other occupied data blocks includes: Write the data in the data block to be released into the other occupied data blocks; In response to determining that all data in the data block to be released has been written to the other occupied data blocks, the data block to be released is marked as free.

11. The method according to any one of claims 1-10, wherein, The method further includes: Record the disk file identifier to which the occupied data block belongs and the offset of the occupied data block in the disk file.

12. The method according to any one of claims 1-10, wherein, The method further includes: In response to determining that the disk file is full and that the file to be stored contains unwritten data objects, a new disk file is created to continue storing the unwritten data objects.

13. The method according to any one of claims 1-10, wherein, The method further includes: In response to determining that there are data blocks in the disk file where all data objects have been deleted, the data blocks where all data objects have been deleted are marked as free.

14. A file storage device, comprising: The first segmentation module is configured to split the file to be stored into data objects; The second segmentation module is configured to segment the disk file into data blocks that are the same size as the data object. The write module is configured to request a free data block from the disk file and write the data object in the file to be stored into the free data block of the disk file. The determination module is configured to determine the space to be released from the disk file in response to determining that space reclamation conditions are met. The migration module is configured to migrate data in the space to be released to free space in the disk file.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

16. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images