File allocation method and device, storage medium and program product

Abstract

The embodiment of the invention provides a file distribution method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the steps of obtaining a to-be-stored file; determining an allocation unit of the to-be-stored file according to the file type of the to-be-stored file; wherein the file type is divided based on the size of the to-be-stored file, and when the size of the to-be-stored file is smaller than a preset threshold value, the file type is a small file, and the block is selected as a distribution unit of the to-be-stored file; when the size of the to-be-stored file is greater than or equal to a preset threshold value and the file type is a large file, selecting an interval as a distribution unit of the to-be-stored file; and distributing a target storage area for the to-be-stored file in the storage space according to the distribution unit of the to-be-stored file. According to the method, the distribution flexibility of file storage is improved, and the overall access performance and the system response speed of different types of files are improved.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more particularly to a file allocation method, device, storage medium, and program product. Background Technology

[0002] In large-scale distributed storage systems, file systems need to support efficient storage of both massive numbers of small files and extremely large files. The former requires high storage density and low fragmentation, while the latter relies on rapid allocation of contiguous space to ensure read and write performance. Therefore, there is an urgent need for a space allocation method that can dynamically adapt to different file types.

[0003] In current implementations, clustered file system space allocation primarily uses fixed-size allocation units. For example, block-based allocation strategies divide storage space into uniformly sized blocks, with files allocated according to the number of blocks. For metadata management, centralized or distributed index structures (such as B-trees or hash tables) are typically used to record block allocation status. These methods usually rely on predefined allocation granularity, tracking free and used space through metadata tables or bitmaps, and coordinating allocation operations through locking mechanisms in multi-node environments.

[0004] However, the above implementation method is difficult to meet the needs of different file types. Summary of the Invention

[0005] This application provides a file allocation method, device, storage medium, and program product to solve the problem of space allocation technology that takes into account different file types.

[0006] In a first aspect, this application provides a file allocation method applied to a resource allocation system, the resource allocation system including resource groups, the method comprising:

[0007] Get the file to be stored;

[0008] The allocation unit for the file to be stored is determined based on its file type. The file type is determined by the size of the file to be stored. When the size of the file to be stored is less than a preset threshold, the file type is small file, and blocks are selected as the allocation unit. When the size of the file to be stored is greater than or equal to the preset threshold, the file type is large file, and ranges are selected as the allocation unit.

[0009] Allocate a target storage area for the file to be stored in the storage space according to the allocation unit of the file to be stored.

[0010] In this embodiment, by comparing the file size with a preset threshold, an appropriate allocation unit (block or interval) is dynamically selected for small and large files, fundamentally solving the efficiency contradiction caused by the traditional single allocation granularity when dealing with mixed file sizes. This method improves storage space utilization (for small files) and enhances the storage and access performance of large files (for large files), thereby optimizing the overall resource allocation efficiency and I / O throughput of the storage system.

[0011] In one possible implementation, when the file to be stored is a large file, the index structure corresponding to the file to be stored is a three-level index structure, which includes a first index node layer, an indirect block layer, and a data layer; the first index node layer is used to store basic file information and indirect block pointers, the indirect block layer is used to indicate the data layer pointer, and the data layer is used to actually store the file.

[0012] When the file to be stored is a small file, the index structure corresponding to the file to be stored is a two-level index structure. The two-level index structure includes a second index node layer and a data layer. The second index node layer is used to store basic file information and data layer pointers.

[0013] This implementation achieves an optimal balance between metadata management complexity and file access efficiency by constructing a three-layer indirect index for large files and a two-layer direct index for small files. This method ensures the scalability of massive data storage for large files while reducing the indexing overhead for frequent access to small files, thereby improving the response performance of the storage system under mixed file loads from the metadata level.

[0014] In one possible implementation, the storage space is divided into multiple resource groups, each resource group is configured with a block bitmap and a range bitmap; each bit in the block bitmap corresponds to a block within the resource group and is used to mark the usage status of the block; each two consecutive bits in the range bitmap correspond to a range, where the first bit indicates whether the corresponding range has been allocated and the second bit indicates the file type of the file using the corresponding range.

[0015] This implementation achieves modular and efficient management of storage resources by logically partitioning the storage space and using a two-layer bitmap for state tracking. This method improves the speed of locating and allocating free space, and avoids interference between large and small file I / O by unifying file types within a range, providing a solid foundation for the efficient and stable operation of the system.

[0016] In one possible implementation, the storage space has a three-layer storage hierarchy. The first layer is the file system hierarchy, which is used to indicate the resource group identifier. The second layer is the resource group hierarchy, which is used to record the resource group's metadata, block bitmap, interval bitmap, interval, and small file interval source data. The third layer is the interval layer, which is used to actually store the data.

[0017] This implementation decouples and hierarchically organizes global indexing, resource management, and physical storage, constructing a highly modular and clearly defined storage management system. This structure simplifies the global management and local scheduling logic of storage space, enabling efficient and orderly resource location, status maintenance, and data access operations. Simultaneously, this layered design provides the infrastructure support for the horizontal scaling and maintainability of the storage system.

[0018] In one possible implementation, according to the allocation unit of the file to be stored, a target storage area is allocated in the storage space for the file to be stored, including:

[0019] If the allocation unit of the file to be stored is a block, then determine whether the remaining space of the storage area currently allocated to the small file meets the storage requirements of the file to be stored.

[0020] If the remaining space in the storage area currently allocated for small files meets the storage requirements of the file to be stored, allocate a new block for the file to be stored within the current storage area;

[0021] If the remaining space in the current storage area allocated for small files is insufficient to meet the storage requirements of the files to be stored, a new storage area for small files is allocated, and new blocks are allocated for the files to be stored in the new storage area.

[0022] This implementation uses a lightweight structure of small file interval metadata to achieve state memory and sequential guidance for small file storage within a resource group. It optimizes small file allocation from the traditional global search for free blocks to a sequential append-only model within a specified interval. This reduces lookup overhead and metadata operations during allocation, improves throughput for concurrent writes of a large number of small files, and avoids storage fragmentation through contiguous block allocation within the interval, thus meeting the requirements for high-density, high-concurrency storage of small files.

[0023] In one possible implementation, according to the allocation unit of the file to be stored, a target storage area is allocated in the storage space for the file to be stored, including:

[0024] If the allocation unit of the file to be stored is an interval, determine whether the remaining space of the storage interval currently allocated to the large file meets the storage requirements of the file to be stored.

[0025] If the remaining space in the storage area currently allocated to large files meets the storage requirements of the file to be stored, then space will be allocated for the file to be stored within the current storage area.

[0026] If the remaining space in the storage area currently allocated to large files is insufficient to meet the storage requirements of the file to be stored, a new storage area for large files is allocated. If the new storage area for large files is contiguous with the current storage area for large files, the new storage area for large files and the current storage area for large files are merged to obtain a merged area. Space is then allocated for the file to be stored based on the merged area.

[0027] In this implementation, by utilizing the metadata of large file intervals and the triggered contiguous interval merging mechanism, this embodiment optimizes the space utilization of individual intervals. Furthermore, by identifying and merging intervals with contiguous physical addresses during allocation, it effectively maintains the continuity of large file storage space. This implementation can reduce storage fragmentation generated during the dynamic expansion of large files, thereby improving the performance of sequential read / write operations.

[0028] In one possible implementation, after obtaining the file to be stored, the process also includes:

[0029] Based on the file extension, file creation attributes, and user-defined tags of the file to be stored, determine whether the file to be stored may be expanded into a large file;

[0030] If it is determined that the file to be stored may expand into a large file, then pre-allocation is performed on the file to be stored, where the pre-allocation is in units of intervals.

[0031] In this implementation, the file type identification and pre-allocation mechanism effectively avoids frequent reallocation and data migration issues that may arise due to dynamic file growth. For continuously growing large files, this approach provides ample contiguous storage space, ensuring storage continuity throughout the entire lifecycle, thereby improving the storage and access efficiency of such files and enhancing the system's ability to handle mixed dynamic loads.

[0032] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0033] The memory stores instructions that the computer executes;

[0034] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0035] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0036] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 A flowchart illustrating a file allocation method provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram illustrating a scenario representing a three-level index structure, provided in an embodiment of this application.

[0040] Figure 3 This is a schematic diagram of the three-tier storage hierarchy provided in the embodiments of this application;

[0041] Figure 4 This is a schematic diagram of the architecture of the small file hierarchical storage structure provided in the embodiments of this application;

[0042] Figure 5 A schematic diagram of the architecture of a three-level index structure for large files provided in this application embodiment;

[0043] Figure 6 This is a schematic diagram illustrating the process of differentiated space allocation provided in an embodiment of this application.

[0044] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] It should be noted that the file allocation method, device, storage medium and program product provided in this application can be used in the distributed field, or in any field other than distributed. This application does not limit the application field of the file allocation method, device, storage medium and program product.

[0047] The specific application scenarios of this application are mainly aimed at large-scale distributed storage environments that need to handle mixed file workloads and have high requirements for storage performance and resource utilization. Specific application scenarios include, but are not limited to:

[0048] Cloud computing and cloud storage platforms: In public or private cloud environments, platforms need to provide storage services to a massive number of users. Users simultaneously upload and store a large number of small files (such as website images, user documents, and configuration files) and a small number of extremely large files (such as virtual machine images, database backups, and high-definition videos). The storage system needs to efficiently manage this mixed file load with significant differences in size and type.

[0049] Big Data Analytics and Processing Clusters: In big data frameworks such as Hadoop and Spark, job processing typically involves both massive amounts of small intermediate result files (e.g., logs) and huge raw dataset files (e.g., log files exceeding terabytes in size). The system needs to handle the metadata pressure of small files and the high-throughput read / write requirements of large files.

[0050] Distributed database systems: The underlying storage of modern distributed databases (such as NoSQL databases) typically contains a large number of data block files, log files (usually small), and data backup or snapshot files (usually large), forming a typical mixed storage scenario of large and small files.

[0051] High-performance computing environments: In scenarios such as scientific research, engineering simulation, and weather forecasting, applications generate and access extremely large scientific datasets (e.g., simulation results, satellite images), which are typically large files. Simultaneously, the system also runs a massive number of job scripts, configuration files, and temporary logs, creating an I / O pattern where files of varying sizes coexist.

[0052] For the above application scenarios, current cluster file systems typically use a technology based on fixed-size allocation units to achieve space management.

[0053] One possible implementation is fixed block allocation, where the storage medium is pre-formatted into data blocks of uniform size (e.g., 4KB or 8KB), and the file occupies an integer number of blocks as needed. The system manages the free state of the blocks through a bitmap and records a list of all block addresses allocated to the file in the file metadata (e.g., inode).

[0054] Another possible implementation is interval allocation, where the system allocates data in continuous physical space intervals as the basic unit. Each interval is defined by a starting address and a length, and efficient data structures such as B-trees are typically used to organize these interval records to optimize the contiguous storage of large files.

[0055] To enhance management capabilities and support larger files, multi-level indexing mechanisms can be introduced, such as the classic three-level index structure. This involves setting direct pointers, indirect pointers, and other multi-level addressing methods in the metadata to expand the number of addressable data blocks. Alternatively, a hybrid allocation approach can be adopted, dividing the system into different regions and using a fixed-size block allocation strategy for small files and a larger fixed-block allocation strategy for large files, thus accommodating the storage needs of different file types.

[0056] However, the solution described above has the following technical problems:

[0057] The problem of poor adaptability of a single allocation unit: Existing technologies use fixed-size blocks or intervals as allocation units, which cannot simultaneously meet the storage density requirements of small files and the allocation efficiency requirements of large files. Small files are prone to space fragmentation during storage, and large files require frequent allocation operations, leading to a decline in system performance.

[0058] The problem of insufficient coordination between multi-level indexes and allocation units: The existing multi-level indexing mechanism has failed to be effectively combined with dynamic allocation units, resulting in large metadata storage overhead and low index query efficiency in large file scenarios, and a lack of efficient index simplification mechanism when storing small files.

[0059] The problem of inefficient merging of continuous intervals: In interval allocation scenarios, existing technologies lack automatic merging logic for continuous intervals, resulting in an excessive number of metadata records, increasing the complexity of index queries and updates, and reducing the overall performance of the system.

[0060] The file allocation method provided in this application aims to solve the aforementioned technical problems of the prior art. First, by dividing the storage space into multiple resource groups and configuring block bitmaps and interval bitmaps within each resource group, the storage areas for large and small files are physically isolated. Second, based on a comparison between the size of the file to be stored and a preset threshold, space allocation is dynamically selected either in blocks or in continuous intervals. Correspondingly, the system employs a hierarchical multi-level index structure to efficiently manage metadata under different allocation methods. Finally, automatic merging logic for continuous free intervals during space release is introduced, effectively reducing the number of metadata entries and lowering index maintenance overhead.

[0061] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0062] First, combine Figure 1The process of the file allocation method provided in the embodiments of this application will be described. Figure 1 This is a flowchart illustrating a file allocation method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes:

[0063] S101. Obtain the file to be stored.

[0064] The system receives files and their data from clients or applications that need to be written to the storage cluster.

[0065] S102. Determine the allocation unit of the file to be stored based on the file type of the file to be stored.

[0066] The file type is not fixed in advance, but is dynamically divided based on the size of the file to be stored by comparing it with a preset threshold (e.g., 128KB) in real time. This mechanism precisely matches the allocation needs of files of different sizes; small files require high storage density to save space, while large files need to quickly obtain large contiguous spaces to ensure I / O performance.

[0067] Specifically, when the size of a file to be stored is less than a preset threshold (128KB), the system classifies the file as a small file. For small files, the system selects blocks as the allocation unit. Here, a block is a smaller basic storage granularity (e.g., 4KB). Selecting block units allows for fine-grained management of storage space, storing a small file in one or more contiguous 4KB blocks, thereby effectively avoiding severe internal space fragmentation that might occur if excessively large interval units are used, thus improving storage space utilization.

[0068] Conversely, when the size of the file to be stored is greater than or equal to a preset threshold (128KB), the system classifies the file as a large file. For large files, the system selects a range as the allocation unit. This range is a large storage unit composed of multiple contiguous blocks; for example, a range may contain 32 contiguous 4KB blocks, totaling 128KB. Selecting a range unit allows for the provision of a large contiguous physical storage area for large files in a single allocation operation. This not only reduces the number of allocation operations and metadata management overhead in the file system but also ensures the physical continuity of file data, thus providing a performance foundation for subsequent sequential read / write operations.

[0069] S103. Allocate a target storage area for the file to be stored in the storage space according to the allocation unit of the file to be stored.

[0070] The storage space is a hierarchically managed clustered file system, logically employing a three-layer structure: "file system layer—resource group layer—data storage layer." The file system layer manages all resource groups; each resource group, as an independent storage management unit, is equipped with a block bitmap (managing the usage status of 4KB blocks) and a range bitmap (managing the allocation status and unique file types of 128KB ranges); the data storage layer actually contains all the storage blocks and storage ranges managed by each resource group.

[0071] The allocation process is as follows:

[0072] If the allocation unit is a block, the system will search for the current small file storage interval dedicated to small files within the current resource group. By querying the small file interval metadata (which records the interval's starting address and remaining space offset), one or more contiguous 4KB disk blocks will be allocated as the target storage area for the file to be stored within the remaining space. If the current interval space is insufficient, a new free interval will be allocated from the resource group and set as the small file storage interval for continued allocation. After allocation is complete, the block bitmap will be updated, and the allocated block pointers will be recorded in the file's second-level index structure (i.e., the inode directly points to the data block).

[0073] If the allocation unit is a range, the system directly allocates a complete, free 128KB range in the resource group as the target storage area for the file. During allocation, a range bitmap is used to ensure that the range is marked as exclusive to large files. Subsequently, the system creates or updates the large file range metadata (recording the starting address of consecutive ranges, the number of ranges, and the data offset of the last range), and associates the file with this physical space through a three-level index structure (inode → indirect block → large file range metadata). If subsequent writes to the large file cause insufficient space, the system will attempt to allocate a new range with contiguous addresses and trigger a contiguous range merging mechanism. That is, instead of adding new metadata records, the system updates the existing records' range count and other fields to maintain logical storage continuity.

[0074] The file allocation method provided in this embodiment dynamically selects the appropriate allocation unit (block or interval) for small and large files by comparing the file size with a preset threshold, fundamentally solving the efficiency contradiction caused by the traditional single allocation granularity when dealing with mixed file sizes. This method improves storage space utilization (for small files) and enhances the storage and access performance of large files (for large files), thereby optimizing the overall resource allocation efficiency and I / O throughput of the storage system.

[0075] Optionally, in some embodiments of this application, in order to efficiently map and manage the target storage area allocated to files, the system constructs an adaptive index structure based on the file size and type. This design reflects the principle of adapting management complexity to file size, which will be discussed below. Figure 2 The implementation of this index structure will be introduced. Figure 2 This is a schematic diagram illustrating a three-level index structure provided in an embodiment of this application.

[0076] For files identified as large (i.e., file size ≥ 128KB) to be stored, the system will establish a three-level index architecture to efficiently support the large amount of contiguous storage space they may occupy. For example... Figure 2 As shown, the three-level index structure includes:

[0077] The first inode layer, or file inode layer, serves as the core of file metadata, storing basic file information such as file size, permissions, and creation time. It also contains multiple indirect block pointers, which act as the starting point for the index.

[0078] Indirect Block Layer: This layer acts as an intermediate mapping layer for the index. Each indirect block contains a header and multiple data layer pointers. The indirect block layer receives indirect block pointers from the first inode layer and points to the specific physical storage location of the data layer through its own data layer pointers. This layer extends the addressing range of the index, allowing a single inode to manage a data range far exceeding the number of its direct pointers.

[0079] Data layer: This is the physical space where file data is actually stored, consisting of multiple allocated intervals.

[0080] Through the cascading mapping of the first index node layer → indirect block layer → data layer, large files can flexibly manage multiple consecutive data ranges through multi-layer indirect addressing, thus meeting the needs of massive data storage.

[0081] For files identified as small (i.e., file size < 128KB) to be stored, the system will use a simplified two-level index architecture to maximize access efficiency. This two-level index structure includes:

[0082] The second inode layer: also the file's inode layer, stores basic file information. However, unlike the third-level index, here the inode directly stores the data layer pointer, skipping the intermediate indirect block layer.

[0083] Data layer: This is the physical space where file data is actually stored, consisting of one or more allocated blocks.

[0084] By using a direct mapping from the second inode layer to the data layer, small files are spared the overhead of indirect addressing. Reducing the number of index levels allows the system to locate data blocks more quickly when accessing small files, improving the performance of aggregated access to a large number of small files.

[0085] By constructing a three-layer indirect index for large files and a two-layer direct index for small files, an optimal balance is achieved between metadata management complexity and file access efficiency. This method ensures the scalability of massive data storage for large files while reducing the index addressing overhead when frequently accessing small files, thereby improving the response performance of the storage system under mixed file loads from the metadata level.

[0086] Optionally, in some embodiments of this application, in order to manage storage space efficiently and granularly, the system may, for example, adopt a resource group partitioning and bitmap management mechanism. The core of this mechanism is to logically partition and mark the status of the storage space of the entire cluster file system, as specifically implemented as follows:

[0087] First, the storage space is logically partitioned. Logically, the storage space is divided into multiple independent, parallel-managed resource groups. Each resource group is the basic unit for storage allocation and management, with a fixed storage capacity. This partitioning method helps to distribute global storage management tasks across multiple local resource groups, improving the system's parallel processing capabilities and management granularity.

[0088] Secondly, an independent bitmap structure is configured for each resource group to enable accurate state tracking of storage units. Each resource group is associated with two types of bitmaps:

[0089] Block Bitmap: Each bit in this bitmap uniquely corresponds to a storage block (e.g., 4KB) within the resource group. The value of this bit is used to mark the real-time usage status of the corresponding block; for example, 1 indicates that the block has been allocated and used, and 0 indicates that the block is free and available. By scanning the block bitmap, the system can quickly locate free blocks within the resource group for the allocation of small files.

[0090] Interval Bitmap: This bitmap is used to manage larger granularity storage units, i.e., intervals (e.g., 128KB). In the interval bitmap, every two consecutive bits (i.e., a tuple) jointly describe the state of an interval:

[0091] The first bit (allocation status bit): indicates whether the interval has been allocated. For example, 1 indicates that it has been allocated, and 0 indicates that it is free.

[0092] The second bit (file type bit): indicates the type of file stored within this range after it is allocated. In other words, a range is typically used for one file type and cannot be mixed. For example, 1 indicates that the range is currently dedicated to storing large files, and 0 indicates that the range is currently dedicated to storing small files.

[0093] By logically partitioning the storage space and employing a two-layer bitmap for state tracking, modularization and efficient management of storage resources are achieved. This method improves the speed of locating and allocating free space, and avoids interference between I / O operations of large and small files by unifying file types within a range, providing a solid foundation for the efficient and stable operation of the system.

[0094] Optionally, in some embodiments of this application, in order to construct an efficient and easy-to-manage storage system, the system adopts, as follows: Figure 3 The three-layer storage hierarchy shown is as follows: Figure 3 This is a schematic diagram of the three-tier storage hierarchy provided in this application embodiment. This structure decouples the organization and management of storage space from the actual data storage from top to bottom, as detailed below:

[0095] First level: File system hierarchy

[0096] This layer serves as the management entry point for the entire cluster file system, and its core consists of the superblock and the resource group index table. It does not directly manage data but is responsible for maintaining a global view, recording and indicating the identifiers of all independent resource groups and their location information in the global storage space. By accessing this layer, the system can quickly locate any specific resource group.

[0097] Second level: Resource group level

[0098] This layer is the core operational layer for storage management, corresponding one-to-one with each independent resource group. Each resource group contains the following core components at this layer:

[0099] Resource group metadata: Records the management information of the resource group itself, such as total capacity, used space, health status, etc.

[0100] Management bitmap: includes block bitmap and range bitmap, which are used to track the allocation and usage status of each storage block and each storage range within the resource group (as described in the above embodiments).

[0101] Logical interval division: The space of a resource group is pre-divided into multiple logical intervals, each of which is one of the basic units of allocation.

[0102] Small file interval metadata: This is a dedicated management structure used to record key information about the last interval or active interval currently used exclusively for small files, such as the starting address of the interval and the offset of the currently available space, thereby enabling efficient and sequential allocation of small file storage.

[0103] Third level: Interval hierarchy

[0104] This layer is the final physical carrier of the data. Each logical interval defined in the second layer corresponds to an actual storage area consisting of multiple contiguous physical disk blocks (e.g., disk blocks 0, 1, 2…n) in this layer. The file's data content is written into these disk blocks. This layer is used for persistent data storage, and all the management logic described above is controlled by the metadata and bitmaps of the second layer.

[0105] By decoupling and hierarchically organizing global indexing, resource management, and physical storage, a highly modular and clearly defined storage management system is constructed. This structure simplifies the global management and local scheduling logic of storage space, enabling efficient and orderly resource location, status maintenance, and data access operations. Simultaneously, this layered design provides the infrastructure support for the horizontal scaling and maintainability of the storage system.

[0106] Optionally, when the allocation unit is a block, the following combination... Figure 4 This section describes the specific execution process for allocating target storage areas for small files. Figure 4 This is a schematic diagram of the small file hierarchical storage structure provided in this application embodiment. The core of this process is to rely on resource groups and perform efficient management through small file interval metadata. The overall architecture can be referenced. Figure 4 .

[0107] like Figure 4 As shown, the storage of small files follows a hierarchical architecture of file system → resource group → small file range metadata → range:

[0108] Global resource group management is performed at the file system level.

[0109] The resource group hierarchy is the core of management. Each resource group maintains its own metadata, bitmap, and small file range metadata, etc.

[0110] Small file interval metadata and interval hierarchy are data storage entities, and each small file exclusive interval consists of multiple contiguous disk blocks.

[0111] Under this architecture, the specific allocation and execution process and core mechanisms are as follows:

[0112] Location and Judgment: Once the system determines that the allocation unit for the file to be stored is a block (i.e., the file is a small file), it enters the management context of the corresponding resource group. The system first queries the metadata of the small file range within that resource group. This metadata is maintained by the resource group and is specifically used to manage the append-only storage of small files. It contains two key fields:

[0113] Starting address of the interval: Records the starting physical address of the last interval currently in use, which is dedicated to a small file.

[0114] Interval data offset: Records the end position of the data already occupied within this interval (offset ranges from 0 to the interval size, e.g., 128KB). This offset points to the beginning of the next available space.

[0115] The specific execution and allocation process can be divided into the following two cases:

[0116] Scenario 1: Sequential allocation within the current interval: The system calculates the space required to store the file. If the remaining space in the current interval (interval size - offset) indicated by the interval data offset can meet the requirements, the system directly allocates one or more contiguous disk blocks within the interval to the file, starting from that offset. After allocation, the system immediately updates the interval data offset field and synchronously updates the block bitmap within the resource group to mark the corresponding block as used, allowing subsequent small files to continue writing from this new offset.

[0117] Scenario 2: Assign a new interval and switch:

[0118] If the remaining space in the current interval is insufficient, the system will perform the following steps:

[0119] Allocate a new interval: The system allocates a brand new logical interval from the idle interval pool of this resource group. It is marked as "allocated" and "exclusive to small files" using the interval bitmap.

[0120] Update metadata and switch: The system fills the starting address of the newly allocated interval into the interval first address field and resets the interval data offset to 0. This means that the management scope has switched from the old interval that was full to the new current interval.

[0121] Allocation within the new range: Subsequently, within this newly allocated small file-specific range, the system allocates the necessary contiguous disk blocks for the file to be stored, starting from the starting position (offset 0), and synchronously updates the offset and block bitmap.

[0122] By using a lightweight structure of small file interval metadata, the system achieves state memory and sequential guidance for small file storage within a resource group. It optimizes small file allocation from the traditional global search for free blocks to a sequential append-only allocation within a specified interval. This approach reduces lookup overhead and metadata operations during allocation, improves throughput for concurrent writes of a large number of small files, and avoids storage fragmentation through contiguous block allocation within the interval, thus meeting the requirements for high-density, high-concurrency small file storage.

[0123] Optionally, when the allocation unit is an interval, the following will be combined with Figure 5 This section describes the specific execution process for allocating target storage areas for large files. Figure 5This is a schematic diagram of the architecture of a three-level index structure for large files provided in this application embodiment. The core of this process is to manage the storage space composed of multiple consecutive intervals through large file interval metadata.

[0124] like Figure 5 As shown, the storage index for large files adopts a three-layer architecture: "file metadata layer (inode) → indirect block layer → data range layer". Within the indirect block layer, each indirect block stores the metadata for the large file range, used to manage physically contiguous storage space. This metadata is controlled by the file's inode through the indirect blocks and contains three key fields:

[0125] Interval start address: Records the starting physical address of the first interval in this group of consecutive intervals.

[0126] Number of intervals: Records the total number of independent intervals contained in this group of consecutive intervals.

[0127] Interval data offset: Records the end position of the data written in the last interval (range 0-128KB), used to indicate the remaining space in the interval.

[0128] Under this architecture, the specific allocation and execution process is as follows:

[0129] Query and Judgment: Once the system determines that the allocation unit for the file to be stored is a range (i.e., the file is a large file or needs to be processed as a large file), it locates the corresponding large file range metadata through the index structure. Based on the number of ranges and the range data offset in the metadata, the system calculates the remaining space in the last storage range currently allocated to the file.

[0130] When executing the specific allocation process, there are two possible scenarios:

[0131] Scenario 1: Appending within the current last interval: If the calculated remaining space meets the storage requirements, the system directly allocates space within that interval, starting from the interval data offset. After writing, the interval data offset field is updated to reflect the new data boundary. This mechanism optimizes appending to large files, avoiding triggering a new space allocation for each append operation.

[0132] Scenario 2: Allocating and potentially merging new intervals: If the remaining space in the current last interval is insufficient, the system needs to allocate a new storage interval for it. This process includes two sub-steps:

[0133] Allocate a new interval: The system allocates a brand new, standard-sized interval from the resource group's free interval pool.

[0134] Continuity check and merging: The system checks whether the starting physical address of the newly allocated interval is exactly connected to the ending address of the last group of consecutive intervals (i.e., the addresses are consecutive).

[0135] If the addresses are contiguous, the contiguous range merging mechanism is triggered. The system does not create a new large file range metadata record for this new range; instead, it updates the existing metadata: incrementing the value of the range's number field by 1 and resetting the range data offset field to 0 (or the new write start offset). In this way, two physically contiguous ranges are logically merged and managed into a larger contiguous storage unit.

[0136] If the addresses are not contiguous, a new large file interval metadata record needs to be created for the new interval and linked to the file's index structure through an indirect block layer. In this case, the file's data may be stored in multiple non-contiguous groups of contiguous intervals.

[0137] By leveraging large file interval metadata and its triggered contiguous interval merging mechanism, this embodiment optimizes the space utilization of individual intervals. Furthermore, by identifying and merging intervals with contiguous physical addresses during allocation, it effectively maintains the continuity of large file storage space. This implementation reduces storage fragmentation generated during dynamic expansion of large files, thereby improving sequential read / write performance.

[0138] Optionally, in some embodiments of this application, in order to further optimize the storage performance of files that may grow dynamically, the system adds a pre-allocation judgment and execution step after obtaining the file to be stored. The specific steps are as follows:

[0139] Rule-based prediction: Before determining the initial allocation unit based on file size, the system will make a judgment based on a set of predefined scalable file characteristic rules. The judgment criteria mainly include:

[0140] File extensions: For example, specific extensions such as database log files (e.g., .log), virtual machine disk files (e.g., .vmdk), and video editing project files typically indicate that the file will grow dynamically.

[0141] File creation attributes: For example, a file may be explicitly marked as a "sparse file", "appendable", or a specific application type attribute when it is created.

[0142] User-defined labels: System administrators or users can add custom labels such as "large file expected" or "requires pre-allocation" to specific files or directories.

[0143] Pre-allocation: If, based on one or more of the above criteria, the system determines that the file to be stored belongs to a file type that may grow dynamically, then regardless of whether the current initial size of the file is less than a preset threshold (e.g., 128KB), the system will directly perform pre-allocation for the file in units of storage intervals. This means that even if a file is currently only a few KB, as long as it is determined to be potentially growing, the system will skip the block allocation process, directly allocate a complete interval for it, and mark the allocation unit as an interval. The corresponding index structure will also be constructed according to the three-level indexing method for large files.

[0144] By employing file type identification and pre-allocation mechanisms, frequent reallocation and data migration issues that may arise due to dynamic file growth are effectively avoided. For continuously growing large files, this approach provides ample contiguous storage space, ensuring storage continuity throughout the entire lifecycle, thereby improving the storage and access efficiency of such files and enhancing the system's ability to handle mixed dynamic loads.

[0145] Based on the above embodiments, the overall execution flow of the document allocation method of this application can be achieved through... Figure 6 The flowchart shown is described in general terms. Figure 6 This is a schematic diagram illustrating the process of differentiated space allocation provided in an embodiment of this application. The process shows the entire process from receiving an allocation request to executing a differentiated allocation strategy based on file type, and finally completing resource recording. The overall process is as follows:

[0146] Receiving allocation request: The upper-layer application initiates a file space allocation request to the file system.

[0147] File type determination and allocation strategy: The system first determines whether the current allocation is for a large file (e.g., file size greater than or equal to 128KB). This determination result will split the process into two core branches.

[0148] Large file allocation branch:

[0149] Checking and Reusing: If the file is determined to be large, the system first checks whether the remaining unused space in the previously allocated range of the file meets the current storage requirements. If it does, the remaining space in that range is reused directly without needing to request new space.

[0150] Allocate a new interval: If there is insufficient remaining space, the system will allocate a new standard interval from the resource group.

[0151] Consecutive Range Merging: After obtaining a new range, the system checks whether the starting physical address is contiguous with the address of the last range already allocated to that file. If they are contiguous, the contiguous range merging mechanism is triggered, merging two (or more) physically contiguous ranges into a larger logically contiguous unit to optimize storage layout. If they are not contiguous, they are managed as independent contiguous units.

[0152] Small file allocation branch:

[0153] Check and Allocation: If the file is determined to be small, the system checks whether the unused portion of the current small file's dedicated range meets the requirements. If it does, the required contiguous storage block is directly allocated from the starting offset of that portion.

[0154] Switch and allocate a new range: If the remaining space in the current range is insufficient, the system will first use the remaining part of the current range, then allocate a new small file-specific range from the resource group, and start allocating storage blocks for this request from the beginning in this new range.

[0155] Allocation result recording and updating: Regardless of which branch the physical space allocation is completed through, the process will ultimately perform the following unified operation:

[0156] The specific storage location (range address or block address) allocated is recorded in the file's metadata (such as inode and corresponding range metadata), establishing a mapping from logical file to physical data.

[0157] Synchronously update the bitmap structure (range bitmap or block bitmap) within the resource group, and mark the allocated space as "used" to ensure consistency in management status.

[0158] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0159] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0160] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0161] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0162] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0166] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0167] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0168] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0169] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0170] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0171] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0172] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A file allocation method, characterized in that, Applied to a resource allocation system, wherein the resource allocation system includes resource groups, the method includes: Get the file to be stored; The allocation unit of the file to be stored is determined according to the file type of the file to be stored; wherein, the file type is based on the size of the file to be stored. When the size of the file to be stored is less than a preset threshold, the file type is a small file, and a block is selected as the allocation unit of the file to be stored; when the size of the file to be stored is greater than or equal to the preset threshold, the file type is a large file, and a range is selected as the allocation unit of the file to be stored. According to the allocation unit of the file to be stored, allocate a target storage area for the file in the storage space.

2. The method according to claim 1, characterized in that, When the file to be stored is a large file, the index structure corresponding to the file to be stored is a three-level index structure, which includes a first index node layer, an indirect block layer, and a data layer. The first inode layer is used to store basic file information and indirect block pointers, the indirect block layer is used to indicate data layer pointers, and the data layer is used to actually store the file; When the file to be stored is a small file, the index structure corresponding to the file to be stored is a two-level index structure. The two-level index structure includes a second index node layer and a data layer. The second index node layer is used to store basic file information and a pointer to the data layer.

3. The method according to claim 2, characterized in that, The storage space is divided into multiple resource groups, and each resource group is configured with a block bitmap and a range bitmap. Each bit in the block bitmap corresponds to a block in the resource group and is used to mark the usage status of the block. Each two consecutive bits in the range bitmap correspond to a range, wherein the first bit indicates whether the corresponding range has been allocated and the second bit indicates the file type of the file using the corresponding range.

4. The method according to claim 3, characterized in that, The storage space has a three-layer storage hierarchy. The first layer is the file system hierarchy, which is used to indicate the resource group identifier. The second layer is the resource group hierarchy, which is used to record the resource group's metadata, block bitmap, interval bitmap, interval, and small file interval source data. The third layer is the interval layer, which is used to actually store the data.

5. The method according to any one of claims 1-4, characterized in that, The step of allocating a target storage area for the file to be stored in the storage space according to the allocation unit of the file to be stored includes: If the allocation unit of the file to be stored is a block, then determine whether the remaining space of the storage area currently allocated to the small file meets the storage requirements of the file to be stored. If the remaining space in the storage area currently allocated for small files meets the storage requirements of the file to be stored, a new block is allocated for the file to be stored within the current storage area; If the remaining space in the storage area currently allocated for small files does not meet the storage requirements of the file to be stored, a new small file storage area is allocated, and a new block is allocated for the file to be stored in the new small file storage area.

6. The method according to any one of claims 1-4, characterized in that, The step of allocating a target storage area for the file to be stored in the storage space according to the allocation unit of the file to be stored includes: If the allocation unit of the file to be stored is an interval, determine whether the remaining space of the storage interval currently allocated to the large file meets the storage requirements of the file to be stored. If the remaining space in the storage area currently allocated to large files meets the storage requirements of the file to be stored, then space is allocated to the file to be stored within the current storage area. If the remaining space in the storage area currently allocated to large files does not meet the storage requirements of the file to be stored, a new storage area for large files is allocated. If the new storage area for large files is contiguous with the current storage area for large files, the new storage area for large files and the current storage area for large files are merged to obtain a merged area, and space is allocated for the file to be stored based on the merged area.

7. The method according to any one of claims 1-4, characterized in that, After obtaining the file to be stored, the process also includes: Based on the file extension, file creation attributes, and user-defined tags of the file to be stored, determine whether the file to be stored may be expanded into a large file; If it is determined that the file to be stored may expand into a large file, then pre-allocation is performed on the file to be stored, wherein the pre-allocation is in units of intervals.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images