An anti-fragmentation method for portable flash memory devices

Abstract

The application discloses an anti-fragmentation method for a portable flash memory device and relates to the technical field of computer storage systems and operating system kernels. The method comprises the following steps: S1, in the file system initialization stage, a hybrid granularity partition model is established, and a storage space is proportionally divided into a small cluster area and a large cluster area according to functional requirements; S2, an extended allocation bitmap management structure is constructed in the memory; S3, in response to a space allocation request, an allocation strategy is selected according to the data characteristics of the request; and S4, an allocation process in the large cluster area. According to the application, the cluster space of the flash memory device is divided into the small cluster area and the large cluster area, various requests are classified and processed based on prompt information when allocating the space, and a relaxed reuse mechanism is introduced, so that the fragmentation process of the storage space is significantly delayed, and the average bandwidth of multi-threaded writing and the space utilization are significantly improved without affecting compatibility and space utilization.

Description

Technical Field

[0001] This invention belongs to the field of computer storage systems and operating system kernel technology, specifically relating to an anti-fragmentation method for portable flash memory devices. Background Technology

[0002] Portable flash memory devices, such as dashcams and drones, are widely used in embedded systems requiring high-throughput data recording. These devices typically use the exFAT file system, with clusters usually serving as the smallest unit of storage allocation. The actual operation of these devices often involves multiple complex processes, and storage fragmentation is unavoidable during the interfacing of these processes or the internal technical implementation. Taking multi-threaded writes as an example, when multiple threads concurrently request storage space, the file system typically employs a first-come, first-served competitive allocation strategy. This allocation mechanism leads to severe space fragmentation; a logically complete large file is not stored contiguously on the physical storage medium but rather consists of numerous discrete, isolated cluster fragments.

[0003] This fragmentation leads to a severe degradation in storage performance. The root cause lies in the mismatch between the request processing mechanism of the underlying block device driver layer and the physical space distribution. In an ideal, fragment-free state, the physical space is contiguous, and standard-sized write requests from the upper layer can directly pass through the underlying driver layer. For example... Figure 4 As shown, Figure 4 The blue areas represent clusters, and the green areas represent the I / O request splits corresponding to writes to clusters. Figure 4 In a fragment-free scenario, a request to write four consecutive 128KB clusters is split into four 120KB and one 32KB request. However, in the presence of fragmentation, due to the discontinuous physical addresses and the maximum transfer granularity limit of the underlying block device driver layer, the standard write request that originally matched the cluster size is forcibly split into multiple irregular sub-requests. For example... Figure 3 As shown, Figure 3 The blue areas represent clusters, and the green areas represent the I / O request splits corresponding to writes to clusters. Figure 3 This means that when fragmentation occurs, a request to write to a 128KB isolated cluster is split into 120KB and 8KB. This fragmentation-induced splitting of input / output requests significantly increases instruction overhead and bus usage, leading to a substantial decrease in write bandwidth. Summary of the Invention

[0004] To overcome the fragmentation and performance degradation problems existing in current portable devices, this invention provides an anti-fragmentation method for portable flash memory devices that can simultaneously take into account write performance, space utilization, and file system compatibility.

[0005] Includes the following steps: Step S1: During the file system initialization phase, a hybrid granularity partitioning model is established, and the storage space is divided into small cluster areas and large cluster areas according to functional needs. The small cluster areas are managed using standard cluster granularity, and the large cluster areas are managed using virtual large cluster granularity. A virtual large cluster contains multiple physically contiguous standard clusters. The partitioning ratio and the number of standard clusters contained in the virtual large cluster are configurable. Preferably, the number of standard clusters contained in the virtual large cluster is set to 8, i.e., 1MB in size; the size of the small cluster area is dynamically set according to the estimated proportion of small files.

[0006] Step S2: Construct an extended allocation bitmap management structure in memory, and maintain independent global search pointers and free cluster counters for the small cluster region and the large cluster region respectively; Furthermore, the extended allocation bitmap management structure also includes a relaxed reuse statistics mechanism for large cluster regions: The system scans the allocation bitmap of the large cluster region, and for each virtual large cluster, counts the number of free standard clusters within it and the position of the first free standard cluster; the state is divided into "fully empty state" and "partially occupied state", without distinguishing the specific occupancy mode; the free standard clusters within the virtual large clusters in the "partially occupied state" are included in the total free cluster count of the large cluster region, allowing them to be reused by subsequent allocation requests.

[0007] Step S3: In response to a space allocation request, select an allocation strategy based on the data characteristics of the request: If it is a directory entry, metadata, or data identified as small-granularity (small file) write or non-contiguous expansion request, perform allocation within the small cluster area using the global search pointer of the small cluster area; if it is a large file write or contiguous expansion request, perform allocation within the large cluster area. Furthermore, the allocation process within the small cluster region includes: prioritizing the search within the small cluster region; if the space in the small cluster region is insufficient, the system allows entry into the large cluster region, and uses a standard cluster-level search algorithm to scan and allocate the scattered free space within the virtual large cluster that is in a "partially occupied state".

[0008] Step S4: The allocation process within the large cluster area includes: firstly checking the virtual large cluster currently held by the target file; if there is a free standard cluster within the virtual large cluster, then the free standard cluster is directly allocated; if new space needs to be requested, then a completely empty virtual large cluster is searched within the large cluster area for allocation; if there is no space standard cluster within the virtual large cluster, then space needs to be searched within the small cluster area for allocation.

[0009] Furthermore, the allocation within the large cluster area also includes: when a new virtual large cluster is requested, the system uses a bitmap or search pointer at the large cluster granularity to quickly locate a virtual large cluster in a "completely empty" state. Once the virtual large cluster is allocated to the target file, the file gains priority for continuous expansion within the virtual large cluster. However, before the virtual large cluster is filled, its remaining space can be borrowed by other allocation requests when space is scarce. But if a virtual large cluster in a "completely empty" state cannot be found, space needs to be searched for and allocated within the small cluster area.

[0010] This invention addresses the performance issues caused by fragmentation by establishing a mixed-granularity partition layout and combining a flexible reuse strategy with an independent space search mechanism. It also balances space utilization and file system compatibility, maximizing the reclamation of fragmented space and solving the problem of space waste in traditional large-granularity allocation schemes. Attached Figure Description

[0011] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This refers to the fragmented space generated during multi-process writes in the exfat file system; Figure 3 This is a schematic diagram of fragmented cluster space and I / O partitioning under existing technology; Figure 4 This is a schematic diagram of the fragmentation-free cluster space and I / O partitioning under existing technology; Figure 5 This is a flowchart of the file system's new cluster allocation process following the method of this invention; Figure 6 This is a schematic diagram of the large cluster region allocation mechanism of the file system after the method of the present invention; Figure 7 This is a schematic diagram of the spatial distribution of the file system after the method of the present invention. Detailed Implementation

[0012] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0013] like Figure 1 As shown, an anti-fragmentation method for portable flash memory devices specifically includes the following three core stages: The first stage is the initialization stage. When the file system is mounted, the system first establishes a hybrid granularity partitioning model, logically dividing the physical storage space into "small cluster regions" and "large cluster regions" according to functional needs. The small cluster regions are managed using standard cluster granularity, while the large cluster regions are managed using virtual large cluster granularity. A virtual large cluster consists of multiple physically contiguous standard clusters. Simultaneously, a dual bitmap management structure is constructed in memory. In addition to loading the standard FAT allocation bitmap, an independent virtual large cluster bitmap is constructed for the large cluster regions. These are initialized separately for the small and large cluster regions, maintaining independent global search pointers and free cluster counters. This makes the small cluster regions an independent and usable space pool, laying the foundation for subsequent allocation.

[0014] The second phase is the application strategy routing phase. When the system receives a write request, it first determines the target region based on the file type, such as directory or metadata, file size, and extended attributes. If it is determined to be a small file or directory entry, the system directly enters the small cluster region and uses the global search pointer of the small cluster region to search for free standard clusters within that region; using the standard cluster granularity, such as 128KB, it searches for free space. If the allocation is successful, the bitmap is updated; if the small cluster region space is exhausted, a fallback mechanism is triggered, and the system jumps to a relaxed reuse mode. If it is determined to be a large file write, the system enters the large cluster allocation logic, and further determines whether a completely new space needs to be requested. For requests to request a new large cluster, the system first queries the dedicated large cluster bitmap to find a virtual large cluster with a "completely empty" status, such as 1MB of contiguous space, to ensure physical continuity. If a completely empty large cluster is found, it is allocated; if all empty large clusters are exhausted, the system also switches to the relaxed reuse mode. For requests to expand the current large cluster, the system first checks whether the current large cluster can be expanded. If a free standard cluster is found in the current large cluster, it is expanded; if no free standard cluster is found in the current large cluster, the system also switches to the relaxed reuse mode. When there are unused scattered free standard clusters within a virtual large cluster, they are counted as available space and can be allocated to other files when necessary.

[0015] The third stage is the relaxed reuse stage, which serves as a fallback. When the system cannot meet the ideal allocation condition of "small files in small clusters, large files in completely empty large clusters," such as when the small clusters are full or there are no completely empty large clusters, the system will not report an error but will enter the relaxed reuse mode. In this mode, if the small cluster area is full, the system scans the standard bitmap of the large cluster area, identifies those virtual large clusters in a "partially occupied" state, and allocates their unused scattered physical clusters to the current request; if the large cluster area cannot be expanded or a new large cluster can be requested, the system scans the standard bitmap of the small cluster area and allocates its unused physical clusters to the current request.

[0016] This invention first logically divides the storage space into "small cluster regions" and "large cluster regions." Unlike existing technologies, this invention maintains independent global search pointers and independent free counts for these two regions within the superblock structure in memory. This makes the small cluster region a truly independent and usable space pool, specifically designed to accommodate small files and fragmented data.

[0017] Secondly, this invention implements a "relaxed reuse" management strategy. For large cluster regions, when scanning or loading the allocation bitmap, it no longer enforces checks on whether the internal structure of the large cluster conforms to the strict "first 1, last 0" sequence. Any virtual large cluster containing free standard clusters is considered valid, and the number of free clusters within it is counted in the available space, making it easier to request and use large cluster regions when small cluster regions cannot allocate space. For small cluster regions, small cluster regions can be requested and used when large cluster regions cannot be expanded into larger clusters or when new large clusters can be requested.

[0018] In terms of allocation strategy, this invention dynamically selects paths based on file characteristics: For small files or requests that do not require contiguous access, small cluster pointers are preferentially used for allocation in the small cluster region. If the small cluster region is exhausted, access to the large cluster region is allowed, reusing the gaps within those large clusters that are in a "partially occupied" state. For large file writes, it is further determined whether the current request is for a completely new large cluster or for expanding an existing large cluster; requests that meet the application conditions are preferentially allocated in the large cluster region. If the large cluster region is exhausted, access to the small cluster region is permitted to request the use of standard clusters that are in a "free" state within the small cluster region.

[0019] Furthermore, this invention restores support for TRIM operations. Since strict large cluster continuity constraints are no longer maintained, TRIM operations can safely clean up free segments within large clusters at a standard cluster granularity, extending storage device lifespan.

[0020] Example 1: Spatial Layout Based on Partitions like Figure 7 As shown, during the file system mounting or initialization phase, the present invention divides the cluster space of the storage device into two logical regions according to a preset ratio. Figure 7 The left side shows the small cluster area, which retains the original 128KB standard cluster allocation granularity and is specifically used to store small files, directory entries, or fragmented data at the end of files. Figure 7 The right side shows the large cluster area, which introduces the concept of virtual large clusters of 1MB in size. Each virtual large cluster contains 8 physically contiguous 128KB standard clusters, which are specifically used to store large files such as videos.

[0021] This partitioning layout is implemented through independent management fields in the superblock information structure of memory. The system maintains separate free cluster counters and search pointers for smaller cluster regions, as well as for larger cluster regions. This independent management mechanism ensures that allocation requests for small files are confined to smaller cluster regions, preventing them from disrupting the physical continuity of larger cluster regions.

[0022] Example 2: New Cluster Application and Allocation Process like Figure 5 As shown, when the file system receives a request for new cluster space, the allocator first obtains the hint cluster information. The hint cluster typically points to the location of the last cluster allocated for that file.

[0023] The allocation logic mainly includes the following paths: In the first path, when the suggested cluster is located in a small cluster region, the system determines that the current request is suitable for small-granularity allocation. At this time, the system calls the global search pointer for the small cluster region to find and request a physical standard cluster within the small cluster region.

[0024] In the second approach, when the suggested cluster is located within a large cluster region, the system determines that the current request involves large file writing or continuous expansion. At this point, the system further examines the relative position of the suggested cluster within the virtual large cluster.

[0025] If the suggested cluster is the first physical cluster or an intermediate cluster of a certain virtual large cluster, for example... Figure 6 The third large cluster below represents the k-th physical cluster, and since subsequent physical clusters within this virtual large cluster are idle, the system performs an operation to expand the original large cluster. That is, it directly allocates the next consecutive physical cluster within this virtual large cluster, such as... Figure 6 As shown in the fourth large cluster below, subsequent free bits are set from 0 to 1. This process does not require rescanning the global bitmap, thus ensuring the physical continuity of the write operation.

[0026] If the virtual cluster containing the indicated cluster is full, or if this is a new large file write request (meaning the indicated cluster is the first physical cluster of the large cluster), the system will perform the operation of acquiring a new large cluster. The system uses a search pointer in the large cluster region to find a virtual large cluster that is completely empty and changes its status from "idle" to "occupied." Figure 6 As shown above, the first physical cluster of this virtual large cluster is assigned, as follows: Figure 6 As shown in the first and second large clusters below, the first digit of the new large cluster is set from 0 to 1. Example 3, Relaxed Reuse Mechanism In the above allocation process, this invention introduces a relaxed reuse mechanism in the large cluster area. This invention allows virtual large clusters to be in a partially occupied state. When scanning the bitmap, the system counts the scattered free clusters within the partially occupied large clusters. When system space is scarce or a non-forced contiguous allocation request is received, these scattered free clusters within the virtual large clusters are allowed to be allocated.

[0027] This strategy is combined as follows: Figure 7 The partition layout shown effectively avoids issues such as... Figure 2 The fragmentation phenomenon is illustrated. For large files, the system forces them to be stored contiguously in 1MB units within large clusters, thereby ensuring that underlying I / O requests are handled as follows: Figure 4 As shown, maintaining the complete 128KB transfer granularity avoids splitting into inefficient 120KB and 8KB combinations. Simultaneously, small files are isolated in small clusters or reuse large cluster holes, ensuring efficient space utilization while preventing small files from interfering with the continuous writing of large files, thus achieving a dual optimization of performance and space.

Claims

1. A method for preventing fragmentation in portable flash memory devices, comprising the following steps: Step S1: During the file system initialization phase, a hybrid granularity partitioning model is established, and the storage space is divided into small cluster areas and large cluster areas according to functional needs. The small cluster areas are managed using standard cluster granularity, and the large cluster areas are managed using virtual large cluster granularity. A virtual large cluster contains multiple physically contiguous standard clusters. Step S2: Construct an extended allocation bitmap management structure in memory, and maintain independent global search pointers and free cluster counters for the small cluster region and the large cluster region respectively; Step S3: In response to a space allocation request, select an allocation strategy based on the data characteristics of the request: If it is a directory entry, metadata, or data identified as small-granularity (small file) write or non-contiguous expansion request, allocate within a small cluster area and perform allocation within the small cluster area using the global search pointer of the small cluster area; if it is a large file write or contiguous expansion request, perform allocation within the large cluster area. Step S4: The allocation process within the large cluster area includes: first checking the virtual large cluster currently held by the target file; if there is a free standard cluster within the virtual large cluster, then directly allocating the free standard cluster. If new space needs to be requested, an empty virtual large cluster is searched within the large cluster area for allocation; if no standard space cluster exists within the virtual large cluster, space needs to be searched within the small cluster area for allocation.

2. The anti-fragmentation method for portable flash memory devices as described in claim 1, characterized in that, The extended allocation bitmap management structure also includes a relaxed reuse statistics mechanism for large cluster regions: The system scans the allocation bitmap of the large cluster region, and for each virtual large cluster, counts the number of free standard clusters within it and the position of the first free standard cluster; the state is divided into "fully empty state" and "partially occupied state", without distinguishing the specific occupancy mode; the free standard clusters within the virtual large clusters in the "partially occupied state" are included in the total free cluster count of the large cluster region, allowing them to be reused by subsequent allocation requests.

3. The anti-fragmentation method for portable flash memory devices as described in claim 1, characterized in that, The allocation process within the small cluster region in step S3 includes: prioritizing the search within the small cluster region; if the space in the small cluster region is insufficient, the system allows entry into the large cluster region, and uses a standard cluster-level search algorithm to scan and allocate the scattered free space within the virtual large cluster that is in a "partially occupied state".

4. The anti-fragmentation method for portable flash memory devices as described in claim 1, characterized in that, The allocation within the large cluster area also includes: when a new virtual large cluster is requested, the system uses a bitmap or search pointer at the large cluster granularity to quickly locate a virtual large cluster in a "completely empty" state; once the virtual large cluster is allocated to the target file, the file gains priority for continuous expansion within the virtual large cluster, but before the virtual large cluster is filled, its remaining space can be borrowed by other allocation requests when space is scarce; however, if a virtual large cluster in a "completely empty" state cannot be found, space needs to be searched for and allocated within the small cluster area.

5. The anti-fragmentation method for portable flash memory devices as described in claim 1, characterized in that, The partitioning ratio and the number of standard clusters included in the virtual large cluster in step S1 are configured as needed.

Images