A method, device, equipment and medium for cyclically covering a block file system

Abstract

This invention discloses a cyclic overwrite method, apparatus, device, and medium for a block file system, relating to the field of data file processing technology. This invention adds a cyclic overwrite management unit. When the number of available blocks falls below a preset threshold in the policy configuration parameters, this unit automatically triggers an overwrite process without any external instructions or manual confirmation; it is completed autonomously by the file system, achieving truly continuous 24 / 7 recording. During the overwrite process, data stored in one or more block nodes is deleted sequentially starting from the file header. In each overwrite process, data is continuously deleted until a complete free storage block is released. This processing method is a "snake-like" gradual release, with each overwrite targeting only the release of a whole block of space, gradually eroding the oldest files and avoiding drastic fluctuations in space utilization. Ultimately, it achieves refined cyclic overwrite based on real-time space management.

Description

Technical Field

[0001] This invention relates to the field of data file processing technology, and in particular to a method, apparatus, device and medium for cyclic overwriting of a block file system. Background Technology

[0002] In the field of embedded data storage technology, traditional file systems (fat, ext, etc.) have the following problems: excessive resource consumption, making them difficult to adapt to low-resource embedded devices (high memory consumption, high computational complexity); inability to guarantee real-time performance, making it difficult to meet the needs of hard real-time scenarios (metadata operation blocking, background task interference); insufficient reliability and data integrity (metadata is easily corrupted, power failure protection is lacking); and inability to handle high-concurrency storage of various types of data.

[0003] To address the problems existing in traditional file systems, in patent CN117742583A, researchers implemented a file system operation interface by customizing a block file system (including configuration management, file management, and block management). This allows for file creation, reading, writing, deletion, modification, and querying. However, when storage space is exhausted, the file system can only provide a "space full" status, leaving the responsibility of space management entirely to upper-layer applications or users. At this point, upper-layer applications or users need to manually intervene to reduce the space.

[0004] However, in devices that require continuous 24 / 7 operation, there is a time window between the "space full" alarm and manual intervention, which is difficult to manage manually. The latest and most critical data generated during this window will be difficult to manage in real time, which may lead to the loss of the most critical new data. Moreover, when manual intervention is needed, it can usually only be done on a whole file basis. When a large file occupies most of the space, deleting it will instantly free up a huge amount of space, and then it will take a long time to fill it up again. This causes a sharp "sawtooth" fluctuation in space utilization and coverage cycle, making it difficult to achieve fine-grained cyclic coverage based on real-time space management. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for cyclic overwriting of a block file system, which can solve the problems existing in the prior art.

[0006] This invention provides a method for cyclic overwriting of a block file system. The block file system includes a configuration management unit, a file management unit, and a block management unit. By setting up a cyclic overwriting management unit and connecting it with other units through an operation interface unit, the cyclic overwriting of files is performed by the cyclic overwriting management unit. The cyclic overwriting of files includes the following steps: By using a cyclic overwrite management unit to predefine policy configuration parameters for the block file system, when the block file system detects through the block management unit that the number of available blocks in the current data storage space is lower than a preset threshold in the policy configuration parameters, an overwrite write process is automatically triggered; wherein, the overwrite write process includes: The circular overwrite management unit traverses the block file system metadata by calling the file management unit, extracts the file with the smallest file number from the files that match the current data mode to be stored as the oldest target overwrite file, and obtains the overwrite write node information and block node linked list of the target overwrite file. Based on the overwrite node information of the target overwrite file, starting from the head of the target overwrite file, delete the data stored in one or more block nodes in the block file system in the order of the block node linked list, and call the block management unit to reclaim the released storage space. In a single overwrite process, data is gradually deleted until a complete free storage block is released, and the overwriting of the current target overwrite file is stopped. After the current target file is overwritten, the overwrite node information is updated in real time to the storage space of the block file system, so as to continuously free up storage space for new data to perform cyclic overwriting of new data.

[0007] Preferably, the policy configuration parameters of the predefined block file system include: The strategy configuration parameters include the main switch for the cyclic overwrite function g_cycle_func_en, the data mode cyclic overwrite support switch g_data_cycle_en, the cyclic overwrite trigger condition g_cycle_block_num, the starting position of the overwrite write information in the file system storage space g_cycle_start, and the size of the file system storage space occupied by the overwrite write information g_cycle_size. Among them, the data mode cycle overlay support switch supports up to 32 data modes.

[0008] Preferably, obtaining the overwrite node information of the target overwrite file includes: The node information data structure is defined in the cyclic coverage management unit, and the storage space driver interface of the storage resource module in the block file system is called to access the block file system storage space to initialize the node information and update the information in real time. The node information is a description of the overwrite file, which needs to be combined with the description of the overwrite node information on the basis of the file information description to obtain the complete description of the overwrite file; the node information includes the file partition part, file number index, file current overwrite position del_cur_pos, actual data header block number head, actual data tail block number tail, actual total number of blocks num, actual header block data size head_num, and actual tail block data size tail_num. The node information is read and parsed during power-on recovery. After determining that the file is being overwritten by the file attribute f_flag in the file information, the corresponding overwrite node information is directly obtained by using the data mode mode in the file information for further parsing of the data distribution. In the file write process, the node information is judged for overwrite conditions and enters the overwrite process. The node information is updated in real time during the overwrite process.

[0009] Preferably, the release of the storage space includes: The operation interface unit adds a loop overwrite operation interface through the loop overwrite management unit. The loop overwrite operation interface is based on the block node linked list of the target overwrite file and the overwrite write node information. Starting from the head of the target overwrite file, it deletes the data stored in one or more block nodes in sequence and calls the block management unit to reclaim the released storage space. Meanwhile, during a single write operation, deletion continues until a complete free storage block is released, at which point the overwriting of the current target file stops. If the current target file is completely deleted but the entire block is not released, the next eligible second-oldest file is extracted and overwritten until the entire space is released.

[0010] Preferably, after the overwrite process is completed, the block file system is powered on and restored to recover the overwritten files, including: During power-on recovery of the block file system, all file and block information in the block file system are read into the memory of the block file system according to the partitions within the block file system, as well as the overwrite management information is read into the memory of the block file system. In the memory of the block file system, file information is parsed by partition within the block file system, and block information is parsed by partition within the block file system. After parsing is completed, power-on recovery ends.

[0011] This invention also provides a cyclic overwrite device for a block file system, comprising: The overwrite module is used to automatically trigger the overwrite process when the block file system detects, through the block management unit, that the number of available blocks in the current data storage space is lower than a preset threshold in the policy configuration parameters, based on the predefined policy configuration parameters of the cyclic overwrite management unit. The overwrite process includes: The circular overwrite management unit traverses the block file system metadata by calling the file management unit, extracts the file with the smallest file number from the files that match the current data mode to be stored as the oldest target overwrite file, and obtains the overwrite write node information and block node linked list of the target overwrite file. Based on the overwrite node information of the target overwrite file, starting from the head of the target overwrite file, delete the data stored in one or more block nodes in the block file system in the order of the block node linked list, and call the block management unit to reclaim the released storage space. In a single overwrite process, data is gradually deleted until a complete free storage block is released, and the overwriting of the current target overwrite file is stopped. The update module is used to update the write node information to the block file system's storage space in real time after the current target file is overwritten, so as to continuously free up storage space for new data to be overwritten in a cyclic manner.

[0012] This invention also provides an electronic device, including a memory and a processor; The memory is used to store computer programs; When the processor executes a computer program stored in the memory, it implements the steps of a cyclic overwrite method for a block file system as described above.

[0013] This invention also provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of a cyclic overwrite method for a block file system as described above.

[0014] This invention provides a method, apparatus, device, and medium for cyclic overwriting of a block file system, which has the following advantages compared with the prior art: This invention adds a cyclic overwrite management unit to the block file system. This unit, through predefined block file system policy configuration parameters, automatically triggers an overwrite process when the block file system detects that the number of available blocks in the current data storage space is lower than a preset threshold in the policy configuration parameters. The entire process is completed autonomously by the file system without any external instructions or manual confirmation, ensuring that the data write channel is always open and new data can be stored at any time, achieving true 24 / 7 uninterrupted recording. In the overwrite process, the cyclic overwrite management unit extracts the oldest target overwrite file by traversing and starting from the header of the target overwrite file, sequentially deleting the data stored in one or more block nodes. In a single overwrite process, data is continuously deleted until a complete free storage block is released. This processing method is a "snake-like" gradual release, with each overwrite targeting only one whole block of space, gradually consuming the oldest file in order of the file block linked list. Before the file is completely overwritten, the remaining valid data of the overwritten file can still be accessed, thus avoiding drastic fluctuations in space utilization. Ultimately, it achieves fine-grained cyclic overwrite based on real-time space management. Attached Figure Description

[0015] Figure 1 A schematic diagram of the overall architecture of a cyclic overwrite method for a block file system provided in an embodiment of the present invention; Figure 2 A schematic diagram of the file writing process of a cyclic overwrite method for a block file system provided in an embodiment of the present invention; Figure 3 A schematic diagram of the data distribution after sequentially writing multiple files in a cyclic overwrite method for a block file system provided in an embodiment of the present invention; Figure 4 A schematic diagram of the overwrite process of a cyclic overwrite method for a block file system provided in an embodiment of the present invention; Figure 5 A schematic diagram of the data distribution after overwrite processing in a cyclic overwrite method for a block file system provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the power-on recovery process of a cyclic overwrite method for a block file system provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0017] In the field of embedded data storage technology, traditional file systems (fat, ext, etc.) suffer from the following problems: excessive resource consumption, making them difficult to adapt to low-resource embedded devices (high memory consumption, high computational complexity); inability to guarantee real-time performance, making them unsuitable for hard real-time scenarios (metadata operation blocking, background task interference); insufficient reliability and data integrity (metadata is easily corrupted, lack of power-loss protection); and inability to handle high-concurrency storage of various types of data. Therefore, at present, researchers have implemented a file system operation interface through a custom block file system (including configuration management, file management, and block management). This interface allows for file creation, reading, writing, deletion, modification, and querying, and includes the following features: rapid power-on recovery and support for abnormal power loss; low memory consumption; real-time file system updates; and multiple data interfaces, storage space driver interfaces, data storage space, and file system storage space provided through the storage resource module, enabling high-concurrency storage of various types of data. This addresses the problems of traditional file systems, and its functionality and stability have been verified in multiple scenarios and with multiple customers.

[0018] However, in practical applications, it has been found that when the storage space is full, files can only be manually deleted or the entire storage space can be formatted to free up storage space. The lack of automatic cyclic overwriting of old data fails to meet the needs of scenarios requiring cyclic overwriting. To address this issue, the file system module of this invention adds a "cyclic overwriting management unit," specifically: The circular overwrite management unit includes policy configuration, file extraction, node information management, and space release functions. ① Policy configuration function: A new circular overwrite configuration item is added to the configuration management unit in the file system module of the framework patent. It includes enabling and disabling the circular overwrite function, configuring whether the circular overwrite function is supported or not for specific data modes, configuring the trigger conditions for the circular overwrite function, and configuring the storage location of node information in the file system storage space.

[0019] The policy configuration features include the master switch for the cyclic overwrite function, g_cycle_func_en; the data mode cyclic overwrite support switch, g_data_cycle_en (supporting up to 32 data modes); the cyclic overwrite trigger condition, g_cycle_block_num (triggers overwrite after the number of available blocks in the storage space reaches a threshold); the starting position of the overwrite information in the file system storage space, g_cycle_start; and the size of the file system storage space occupied by the overwrite information, g_cycle_size.

[0020] ② File extraction function: Calls the file management unit in the file system module of the framework's patent, and extracts files that meet the overwrite strategy by traversing.

[0021] The file extraction function retrieves the file with the smallest number and matching data pattern from the file number table (that is, the oldest data file in the data pattern file).

[0022] ③ Node information management function: Define the node information data structure in the newly added cyclic overlay management unit, and call the storage space driver interface of the storage resource module to access the file system storage space to initialize node information and update information in real time.

[0023] Node information is an extension of file information, that is, a description of an overwrite file. It needs to be combined with the overwrite node information description on top of the file information description to obtain the complete description of the overwrite file. The node information includes the partition (part) of the file, the file number (index), the current overwrite position (del_cur_pos), the actual data header block number (head), the actual data tail block number (tail), the actual total number of blocks (num), the actual header block data size (head_num), and the actual tail block data size (tail_num). The node information is read and parsed during power-on recovery. After determining that it is an overwrite file by the file attribute (f_flag) in the file information, the corresponding overwrite node information is directly obtained by using the data mode (mode) in the file information for further parsing of the data distribution. The node information is used to determine the overwrite conditions and enter the overwrite process during the file write process, and the node information is updated in real time during the overwrite process.

[0024] ④ Space release function: A new loop overwrite operation interface is added to the operation interface unit in the file system module to realize loop overwrite processing in a "snake" manner (gradually eating up the overwritten file, and the remaining valid data of the overwritten file can still be accessed before it is completely overwritten), and storage space is released while preserving the maximum amount of valid data; the block management unit in the file system module of the framework patent is called to manage the released space so as to facilitate the use of new data files.

[0025] After adding a "circular overwrite management unit" to the file system module proposed by the researchers, its overall architecture is as follows: Figure 1 As shown, it mainly includes: Storage resource module: Data storage space is used for data storage, generally large container storage devices such as SSDs; File system storage space is used for file system storage, generally embedded storage devices such as eMMC; Storage space driver interface is used to control access to data storage space or file system storage space, generally provided by CPU or FPGA; Multiple data interface is used to control the inflow and outflow of data, generally provided by CPU or FPGA.

[0026] The file system module includes: a configuration management unit (containing file system configuration, initialization, power-on recovery, and unloading); a file management unit (containing file information management, file list management, and file block node linked list management); a block management unit (containing block information management, fragment linked list management, and empty block linked list management); and a circular overwrite management unit (containing overwrite write policy configuration management, overwrite write file extraction, overwrite write node information management, and overwrite write space release).

[0027] Specific experiment: Assume the storage parameters are: one data stream, 128G data storage space, 8G file system storage space, one partition, 32G block granularity of the file system, overwrite function is enabled and the overwrite trigger condition is g_cycle_block_num is 1 (that is, when the number of remaining available blocks is less than 1, the overwrite process is triggered).

[0028] like Figure 2 For the part of the file writing process involving the overwrite unit, when writing a file, the file is first created, and then the overwrite condition is judged (step S303). If the overwrite condition is not met, the allocation and execution of address space continue (steps S300~S301). Then the file system is updated in real time (step S302). As storage space data is written, the overwrite condition needs to be judged again before the address space needs to be distributed again (step S303). The space release process is executed after the condition is met (step S304). Only after the space is released can the released space be distributed to allow data to continue to be stored until the writing is terminated.

[0029] like Figure 3To distribute the data after writing three files consecutively, file 0 (10G in size, occupying [0,10G] space in block 0), file 1 (60G in size, occupying [10G,32G] in block 0, [0,32G] in block 1, and [0,6G] in block 2), and file 2 (40G in size, occupying [6G,32G] in block 2, and [0,14G] in block 3) are written sequentially. At this point, the number of available blocks in the entire storage space is 0. The next file write will trigger the overwrite condition to release space.

[0030] like Figure 4 The specific processing details of the overwrite procedure are as follows: Extract files of the same type (step S400). This step finds the oldest file with the same data type as the file to be written, generally the file with the smallest number from the file number table; Initialize overwrite node information (step S401). This step extracts the data distribution information from the found file information to initialize the overwrite node information; Perform loop deletion (step S402). According to the block node linked list of the overwritten file and the data distribution of the file, the data in the block node is deleted step by step according to the block node linked list; Determine whether to release the whole block (step S403). After deleting the data in the block of the overwritten file, the space is released. If the whole block can be released, the whole block is reclaimed (step S405). If the whole block cannot be released, the fragments are reclaimed, and subsequent blocks are deleted; Determine if the file is deleted (step S404). If the entire file is just a small fragment space, deleting the file only reclaims a fragment. Only by continuing to extract the next file that meets the condition and delete it until the whole block space is released can the overwrite process be completed; Finally, update the overwrite node information (step S406) so that the remaining valid data of the overwritten file can still be accessed by the file system after power-on recovery.

[0031] like Figure 5 To ensure proper data distribution within blocks when writing to file 3, the file system's storage rules dictate that fragmentation must be used first. Therefore, the [14G, 32G] space of block 3 is written, followed by an overwrite operation. The process begins by finding file 0 and performing a loop deletion, reclaiming only the [0, 10G] space of block 0. The process then continues by finding file 1 and performing a loop deletion, deleting the data in the first block node of file 1 to release space. After this release, block 0 becomes an empty block and is reclaimed. At this point, there is one available block, which can be used for file 3. Subsequent data for file 3 is then written to block 0.

[0032] like Figure 6The recovery of overwritten files in the file system power-on recovery process is as follows: The main purpose of file system power-on recovery is to read the file information, block information, and overwritten information from the file system storage space in sequence, and parse the recovered file data distribution and file list and other management information. Step S203 is a new process that reads the overwritten information into memory. The subsequent steps S201 and S202 will recover the data distribution of the remaining valid data of the overwritten file based on the file information of the overwritten file and the overwritten information, so as to ensure that the remaining data can be accessed.

[0033] This invention targets the release of only one entire block of space during each overwrite, gradually consuming the oldest files in order of the file block linked list. Compared to deleting a large file all at once, this method makes the storage space utilization curve smoother and more stable, avoiding large fluctuations in space utilization, ensuring the predictability of the data update cycle, and ensuring that the remaining valid data of the overwritten file (such as the latter half of a long video) remains readable and usable until it is completely overwritten. At the same time, through real-time updates and persistent storage of overwrite write node information, the system can accurately know the "current overwrite position" of each overwritten file after any power failure and restart. The file system can then correctly parse which data has been released and which data is still valid, thereby avoiding data logic chaos or the unavailability of the entire file due to unexpected interruption of the overwrite process.

[0034] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A cyclic overwrite method of a block file system including a configuration management unit, a file management unit, and a block management unit, wherein cyclic overwrite of a file is performed by a cyclic overwrite management unit by setting the cyclic overwrite management unit and connecting it to the other units through an operation interface unit, characterized in that, The circular overwriting of the file includes the following steps: ​ By using a cyclic overwrite management unit to predefine policy configuration parameters for the block file system, when the block file system detects through the block management unit that the number of available blocks in the current data storage space is lower than a preset threshold in the policy configuration parameters, an overwrite write process is automatically triggered; wherein, the overwrite write process includes: The circular overwrite management unit traverses the block file system metadata by calling the file management unit, extracts the file with the smallest file number from the files that match the current data mode to be stored as the oldest target overwrite file, and obtains the overwrite write node information and block node linked list of the target overwrite file. Based on the overwrite node information of the target overwrite file, starting from the head of the target overwrite file, delete the data stored in one or more block nodes in the block file system in the order of the block node linked list, and call the block management unit to reclaim the released storage space. In a single overwrite process, data is gradually deleted until a complete free storage block is released, and the overwriting of the current target overwrite file is stopped. After the current target file is overwritten, the overwrite node information is updated in real time to the storage space of the block file system, so as to continuously free up storage space for new data to perform cyclic overwriting of new data.

2. The cyclic overwrite method of a block file system according to claim 1, wherein, The policy configuration parameters of the predefined block file system include: The strategy configuration parameters include the main switch for the cyclic overwrite function g_cycle_func_en, the data mode cyclic overwrite support switch g_data_cycle_en, the cyclic overwrite trigger condition g_cycle_block_num, the starting position of the overwrite write information in the file system storage space g_cycle_start, and the size of the file system storage space occupied by the overwrite write information g_cycle_size. Among them, the data mode cycle overlay support switch supports up to 32 data modes.

3. The method for cyclic overwriting a block file system according to claim 1, characterized in that, The acquisition of the overwrite node information of the target overwrite file includes: The node information data structure is defined in the cyclic coverage management unit, and the storage space driver interface of the storage resource module in the block file system is called to access the block file system storage space to initialize the node information and update the information in real time. The node information is a description of the overwrite file, which needs to be combined with the description of the overwrite node information on the basis of the file information description to obtain the complete description of the overwrite file; the node information includes the file partition part, file number index, file current overwrite position del_cur_pos, actual data header block number head, actual data tail block number tail, actual total number of blocks num, actual header block data size head_num, and actual tail block data size tail_num. The node information is read and parsed during power-on recovery. After determining that the file is being overwritten by the file attribute f_flag in the file information, the corresponding overwrite node information is directly obtained by using the data mode mode in the file information for further parsing of the data distribution. In the file write process, the node information is judged for overwrite conditions and enters the overwrite process. The node information is updated in real time during the overwrite process.

4. The method for cyclic overwriting a block file system according to claim 1, characterized in that, The release of the storage space includes: The operation interface unit adds a loop overwrite operation interface through the loop overwrite management unit. The loop overwrite operation interface is based on the block node linked list of the target overwrite file and the overwrite write node information. Starting from the head of the target overwrite file, it deletes the data stored in one or more block nodes in sequence and calls the block management unit to reclaim the released storage space. Meanwhile, during a single write operation, deletion continues until a complete free storage block is released, at which point the overwriting of the current target file stops. If the current target file is completely deleted but the entire block is not released, the next eligible second-oldest file is extracted and overwritten until the entire space is released.

5. The method for cyclic overwriting a block file system according to claim 1, characterized in that, After the overwrite process is completed, the block file system is powered on and restored to recover the overwritten files, including: During power-on recovery of the block file system, all file and block information in the block file system are read into the memory of the block file system according to the partitions within the block file system, as well as the overwrite management information is read into the memory of the block file system. In the memory of the block file system, file information is parsed by partition within the block file system, and block information is parsed by partition within the block file system. After parsing is completed, power-on recovery ends.

6. A circular overwrite device for a block file system, characterized in that, include: The overwrite module is used to automatically trigger the overwrite process when the block file system detects, through the block management unit, that the number of available blocks in the current data storage space is lower than a preset threshold in the policy configuration parameters, based on the predefined policy configuration parameters of the cyclic overwrite management unit. The overwrite process includes: The circular overwrite management unit traverses the block file system metadata by calling the file management unit, extracts the file with the smallest file number from the files that match the current data mode to be stored as the oldest target overwrite file, and obtains the overwrite write node information and block node linked list of the target overwrite file. Based on the overwrite node information of the target overwrite file, starting from the head of the target overwrite file, delete the data stored in one or more block nodes in the block file system in the order of the block node linked list, and call the block management unit to reclaim the released storage space. In a single overwrite process, data is gradually deleted until a complete free storage block is released, and the overwriting of the current target overwrite file is stopped. The update module is used to update the write node information to the block file system's storage space in real time after the current target file is overwritten, so as to continuously free up storage space for new data to be overwritten in a cyclic manner.

7. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the steps of the cyclic overwrite method of a block file system as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store computer programs, which, when executed by a processor, implement the steps of a circular overwrite method for a block file system as described in any one of claims 1 to 5.

Images