A distributed file system management system and a file system extension method

Abstract

This invention discloses a distributed file system management system and a file system extension method, including a distributed file system management system and a file information acquisition system. The distributed file system management system contains distributed management blocks and metadata blocks for each file system. The distributed management blocks include file system index blocks (containing the storage index addresses of metadata for each file system), cross-file system indexes (pointing to cross-file system management addresses), and cross-file system management blocks (containing cross-file system file metadata, including the number of files and their names, distribution, allocation, etc.). The file information acquisition system collects file metadata (including attributes such as name, size, and creation time). This system is used to uniformly manage multiple homogeneous or heterogeneous file systems, supports single-file storage across multiple systems, and its core is the distributed management block, enabling parallel storage and retrieval.

Description

Technical Field

[0001] This invention belongs to the field of file system technology, specifically a distributed file system management system and a file system extension method. Background Technology

[0002] In the current field of computer technology, the file systems carried by various operating systems exhibit diverse characteristics, such as NTFS and extFat in Windows, jffs2 and yaffs2 in Linux, and tffs commonly used in embedded devices. While these file systems can meet basic storage needs when running independently, existing management methods have significant shortcomings in large systems containing multiple file systems: First, the lack of unified interfaces for heterogeneous file systems leads to a lack of unified management tools, resulting in a non-vertical management model and inefficient resource scheduling. Second, file storage is limited to a single file system and cannot be distributed across systems, greatly restricting the expansion potential of storage capacity. Third, traditional expansion methods rely on hardware expansion (such as adding disk slots or upgrading interface speeds), which is constrained by physical conditions and makes it difficult to simultaneously increase capacity and bandwidth. Fourth, existing interconnect tools (such as Samba) rely on network interfaces and cannot adapt to file systems on devices without network ports or with low-speed interfaces (such as I2C and SPI), resulting in insufficient compatibility. Fifth, the lack of parallel read / write mechanisms across file systems creates bandwidth bottlenecks during single-file storage and retrieval, failing to meet the high-speed data processing needs of large systems.

[0003] To address the aforementioned issues, this invention proposes a distributed file system management system and a corresponding extension method. By reconstructing the management architecture and storage logic, it achieves unified control, flexible expansion, and efficient collaboration among multiple file systems. Summary of the Invention

[0004] The purpose of this invention is to provide a distributed file system management system and a file system extension method to solve the problems in the prior art mentioned in the background, such as difficulty in unified management of multiple file systems (including homogeneous and heterogeneous), lack of cross-system storage capabilities, limited expansion and bandwidth bottlenecks.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A distributed file system management system includes a distributed management block, multiple file system metadata blocks, and a file information acquisition system. The distributed management block is configured to implement index management of each file system and collaborative control across file systems. The file information acquisition system is configured to collect file metadata from each file system and upload it to the file system metadata block.

[0007] According to the above technical solution, the distributed management block includes a file system index block, a cross-file system index, and a cross-file system management block; the file system index block is used to record the starting address of each file system metadata block, and the cross-file system index is used to point to the storage address of the cross-file system management block.

[0008] According to the above technical solution, the file system index block adopts a pointer addressing method, which directly locates the corresponding file system metadata block through the starting address. The file system metadata block stores file attribute information, including file name, file size and creation time.

[0009] According to the above technical solution, the cross-file system index adopts a pointer addressing method. The cross-file system management block is configured to manage the storage range of a single file in multiple file systems. It includes the total number of cross-file system files and detailed information of each cross-file system file. The detailed information includes the file name, storage distribution and allocation details in each file system.

[0010] According to the above technical solution, the file information acquisition system and the distributed management block interact with each other through a hardware interface, which includes one or more of the following: network interface, PCIe interface, I2C interface, or SPI interface.

[0011] Furthermore, the file information acquisition system is a software system designed for each independent file system. It can obtain information about each file in each file system. This software system calls the underlying file interface to obtain the metadata information of each file, such as fstat in Linux system, GetFileInformationByHandle in Windows system, and statfs in VxWorks.

[0012] According to the above technical solution, the file metadata collected by the file information collection system has the same storage structure as the file system metadata block, and both are encapsulated in a preset unified format. The unified format includes file identifier, capacity parameter and timestamp field.

[0013] According to the above technical solution, the cross-file system management block supports collaborative storage management of homogeneous and heterogeneous file systems. When a single file is distributed across multiple heterogeneous file systems, the cross-file system management block realizes the association mapping of each file fragment through detailed information.

[0014] According to the above technical solution, the cross-file system management block is configured to support parallel storage and parallel read operations. By parsing detailed information, the storage location of each file fragment is located, thereby realizing collaborative data reading and writing across multiple file systems.

[0015] A method for extending a distributed file system, comprising the following steps:

[0016] S1 collects file metadata from each file system through the file information collection system, encapsulates it in a unified format, and uploads it to the distributed file system management system.

[0017] S2, the distributed file system management system stores the received file metadata into the corresponding file system metadata block, and constructs a file system index block based on the starting address of each file system metadata block to achieve unified management and control of metadata;

[0018] S3, when a single file needs to be stored across multiple file systems, the file data is split into pieces, and each piece is stored in the target file system. At the same time, a cross-file system management block is created through a distributed management block, and its storage address is written into the cross-file system index.

[0019] S4, record the storage information of the corresponding file fragments in each file system into the cross-file system management block according to the preset format, and each fragment information corresponds to a cross-file system file record;

[0020] S5, in the process of adding a new file system to expand storage capacity, the file fragment information in the new file system is supplemented to the cross-file system management block according to a preset format to maintain the integrity of cross-file system file records.

[0021] According to the above technical solution, in step S3, the file data fragmentation process is independent of the file system type. The target file system includes homogeneous file systems and heterogeneous file systems, and each fragment is stored as an independent file in the target file system.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] This invention leverages the standardized index and unified metadata structure of distributed management blocks to achieve vertical and unified management of multiple homogeneous and heterogeneous file systems, resolving the management challenges caused by interface incompatibility. Cross-file system management blocks support distributed storage of single files across any number of file systems, theoretically overcoming the physical limitations of storage capacity and enabling unlimited expansion. It is compatible with various hardware interfaces such as network interfaces, PCIe, I2C, and SPI, adapting to various application scenarios including server-level and embedded devices. Through a parallel storage and retrieval mechanism using file sharding, data transmission bandwidth is significantly improved, effectively solving the bandwidth bottleneck of traditional storage. Sharding processing is independent of file system type, ensuring that homogeneous and heterogeneous systems can work collaboratively, significantly improving system compatibility and flexibility. Attached Figure Description

[0024] Figure 1 This is a diagram showing the overall composition of the distributed file system management system of this invention;

[0025] Figure 2This is a structural diagram of the distributed file system management system of the present invention;

[0026] Figure 3 This is a diagram of the distributed management block structure of the present invention;

[0027] Figure 4 This is a diagram of the file system index block structure of the present invention;

[0028] Figure 5 This is a diagram of the cross-file system index structure of the present invention;

[0029] Figure 6 This is a diagram of the cross-file system management block structure of the present invention;

[0030] Figure 7 This is a diagram illustrating the file information structure across file systems in this invention.

[0031] Figure 8 This is a diagram illustrating the cross-file system configuration of the present invention;

[0032] Figure 9 This is a structural diagram of the document information acquisition system of this invention;

[0033] Figure 10 This is a diagram of the file information structure of the file information acquisition system of this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1

[0036] like Figure 1 As shown, a distributed file system management system and a file system extension method are provided, including a distributed file system management system and a file information acquisition system, one located on the server side and the other located on the file system side;

[0037] like Figure 2 As shown, the distributed file system management system mainly includes a distributed management block and metadata blocks for each file system. The metadata of each file system is the metadata information of each file system file reported by the file information collection system.

[0038] like Figure 3 As shown, the distributed management block includes file system index blocks, cross-file system indexes, and cross-file system management blocks. It has two functions: one is to provide information on each file system index block, and the other is to manage cross-file systems.

[0039] like Figure 4 As shown, the file system index block is used to index the starting address of each file system information, and it uses a pointer addressing method.

[0040] like Figure 5 As shown, the cross-file system index is used to index the starting address of the cross-file system management block, and it uses a point-to-addressing method.

[0041] like Figure 6 As shown, the cross-file system management block is mainly used to manage the storage space occupied by a single file across multiple file systems, including the number of files across file systems and cross-file system file information.

[0042] like Figure 7 As shown, cross-file system file information includes file names, file distribution, and specific allocation details in each file system.

[0043] like Figure 8 As shown, when storing files, the management information of each level is filled in step by step through the cross-file system management blocks, and then the data is stored in each file system according to the distribution of the source end. When reading files, the management information of each level is read in reverse step by step through the cross-file system management blocks, and finally the location of each file is located. Then the file is read through each file system.

[0044] like Figure 9 As shown, the file information acquisition system is mainly responsible for the metadata information of files in each file system. The file metadata information of the file system includes the attribute information of all files in the file system.

[0045] like Figure 10 As shown, the file information includes the file name, file size, file creation time, etc., and needs to be organized and stored according to a fixed structure, which is basically consistent with the file system metadata block information in the distributed file system management system.

[0046] Step S1: Collect the metadata information of each file in the file system through the file information collection system and report it to the distributed file system management system.

[0047] Step S2: The distributed file system management system stores the file information of each file system in the file system metadata area and establishes the file system index block in the distributed management block. The file system metadata has a unified structure and is managed in a unified manner.

[0048] Step S3: When a single file occupies multiple file systems (regardless of homogeneity or heterogeneity), the collected data is first divided and stored in each file system. Each file system stores the data as a separate file. At the same time, the distributed management block creates a cross-file system management block and assigns the address to the cross-file system index.

[0049] Step S4: The information corresponding to the file in each file system across the cross-file system is stored in the cross-file system management block in a unified format, and each file is stored as a cross-file system file information.

[0050] Step S5: When the cross-file system file of the present invention is extended in the file system, the information of the corresponding file in each file system is stored in the cross-file system management block in a unified format, and each file is stored as a cross-file system file information.

[0051] This invention proposes a distributed file system management system and a set of file system extension methods, which redesign the collaborative work between different file systems (homogeneous or heterogeneous), improve the file management and storage management capabilities of the large system for each small system or even the module level, and at the same time, the large system can uniformly manage all subordinate file systems.

[0052] This system offers richer hardware interfaces and is not limited by interface speed. In other words, it provides the ability for embedded file systems to access the system, and can be used for management at both the server level and the board level.

[0053] Through cross-file system file management, a single file can exist in multiple file systems, enabling dynamic expansion of the file system and greatly improving storage capacity. Theoretically, the system has no limit to the scalability of the file system.

[0054] By managing files across file systems, a single file can be stored across multiple file systems, existing as an independent file in each file system. The overall management of the file is achieved through cross-file system file management. After splitting the flow at the source end, parallel storage can be achieved, which greatly improves storage bandwidth, meets the high-speed bandwidth requirements of large systems, and solves the storage bottleneck.

[0055] Example 2

[0056] This embodiment provides a method for a dynamic consistency protocol.

[0057] To address the issue of data consistency across heterogeneous file systems, a hybrid consistency protocol module is added to the cross-file system management block of the distributed management block. By fusing Paxos and ZAB protocols through shard status marking and version vectors, the atomicity and consistency of shard updates are achieved, ensuring data integrity when multiple systems are collaboratively stored.

[0058] To achieve data consistency management across heterogeneous file systems, this invention adds three sets of status fields to the detailed information of cross-file system management blocks, specifically including:

[0059] LockStatus (read-write lock status) has a value of 0 indicating unlocked, 1 indicating read lock, and 2 indicating write lock, and is used to control concurrent access permissions for shards;

[0060] Checksum is calculated using the SHA256 hash algorithm to verify the integrity of data during transmission or storage.

[0061] VersionVector, using an array format of (system ID, version number), is used to record the latest version information of the corresponding fragment in each file system. For example, [(NTFS,3),(ext4,3),(jffs2,3)] means that the three fragments of the file are all in version 3 in the NTFS, ext4, and jffs2 file systems.

[0062] During data state synchronization, when a file system initiates a fragment update request, the cross-file system management block first sets the LockStatus of that fragment to 2 (write lock) to block other concurrent modification operations. After the update operation is completed, the system automatically calculates the new Checksum value and increments the version number of the file system that initiated the update in the VersionVector by 1. Then, the cross-file system management block broadcasts the version update notification to all associated file systems that store that file fragment. After receiving confirmation feedback from all associated file systems, the LockStatus is reset to 0 (lock released), completing one state synchronization.

[0063] Regarding the triggering mechanism of the hybrid consistency protocol (Paxos and ZAB fusion), the system automatically enables the hybrid protocol when the number of fragments in a single file is greater than or equal to 2; when the number of fragments is 1, the native consistency mechanism of the corresponding file system is directly adopted. The implementation process of this hybrid protocol is as follows:

[0064] First, the proposal phase begins (based on the Paxos protocol). The cross-file system management block acts as the proposer and sends an update proposal to all file systems storing the file segment (acting as acceptors). The proposal includes information such as the file ID, segment ID, new VersionVector, and data digest. After receiving the proposal, the acceptor verifies whether the local segment's VersionVector is consistent with the old version carried in the proposal. If they are consistent, the acceptor returns a response ready to accept; otherwise, the acceptor rejects the proposal.

[0065] Next, the confirmation phase begins (based on the ZAB protocol). The proposer collects feedback from all acceptors. When the number of acceptors who agree to the proposal exceeds half of the total (i.e., k > n / 2, where n is the total number of file systems storing the shard), the atomic broadcast mechanism is triggered. The proposer sends a commit instruction to all acceptors, and the acceptors return a committed confirmation after performing the update operation. After receiving all committed confirmations, the cross-file system management block updates the global VersionVector, completing the consistency confirmation.

[0066] The handling mechanism for abnormal situations is as follows: If the number of acceptors who agree to the proposal does not reach the above threshold, the proposer will wait 100ms and re-initiate the proposal, and the maximum number of retries is 3; if a acceptor does not respond within the timeout period, the system will mark it as an "abnormal node" and temporarily exclude it from the consistency confirmation scope. After the node is confirmed to have recovered through heartbeat detection, its shard version information will be synchronized.

[0067] To quantify the threshold for consensus confirmation, let the total number of file systems storing a file fragment be n, and the number of nodes k that agree on the proposal must satisfy the formula: (For example, when n=3, k≥2; when n=4, k≥3).

[0068] When the version vectors V1=(s1,v1) and V2=(s2,v2) of two partitions conflict (i.e., s1≠s2 and v1≠v2), the timestamp priority principle is used to resolve the conflict, that is, the version with the more recent timestamp is retained first. Its mathematical expression is: Where T1 and T2 are the timestamps of the two segments, and T... new A timestamp for the final version.

[0069] Example 3

[0070] This embodiment provides a load-aware module to address the load imbalance between low-speed interfaces (I2C / SPI) and high-speed interfaces (PCIe / network). By adding a load-aware module to the file information acquisition system, and combining it with metadata information from the distributed management block, LSTM load prediction, dynamic weight allocation, and elastic circuit breaking are used to optimize read / write traffic across file systems.

[0071] To achieve intelligent load balancing across file systems, this embodiment constructs an integrated strategy that includes a load prediction model, a dynamic weight allocation mechanism, and an elastic circuit breaker mechanism, as specifically implemented below:

[0072] The load prediction model adopts an LSTM neural network architecture. Its input features are 6-dimensional monitoring data from the past hour, with a sampling frequency of once every 5 minutes, covering: LoadCPU (CPU utilization, in %), LoadMem (memory utilization, in %), IOPS (I / O operations per second), Latency (average response latency, in ms), InterfaceSpeed ​​(theoretical interface bandwidth, in Mbps, such as 10Mbps for SPI interface and 8000Mbps for PCIe interface), and ErrorRate (error rate, in ‰).

[0073] During model training and prediction, the historical data is first preprocessed using a sliding window method with a window size of 12 (covering 1 hour of data). The data is then mapped to the [0,1] interval using a normalization formula to eliminate the influence of dimensions. Subsequently, an LSTM model with 3 hidden layers is trained, with 64, 32, and 16 nodes in each hidden layer. The model output is the predicted LoadCPU and Latency values ​​for the next 2 hours. To adapt to real-time load fluctuations, the model parameters are updated every 30 minutes.

[0074] Based on the above load prediction results, this embodiment designs a dynamic weight allocation mechanism, where the read / write weight of each file system i is calculated by weighting four dimensions, as shown in the formula:

[0075]

[0076] In the formula, To predict CPU utilization normalized to the [0,1] interval, To predict response latency (in milliseconds). Indicates health status (1 corresponds to normal, 0.5 corresponds to downgrade, and 0 corresponds to abnormal). Set the interface priority (set to 1 for high-speed interfaces such as PCIe, and to 0.5 for low-speed interfaces such as SPI). During traffic allocation, read and write requests are determined based on the priority of each file system. Distribute proportionally, for example when At that time, 40% of the traffic will be allocated to System 1; for low-speed interface file systems, if its The system will automatically trigger traffic shifting to compensate for the load. The shifting ratio is calculated according to the following formula:

[0077]

[0078] in, This indicates the proportion of traffic transfer, used to compensate for the load on the low-speed interface file system.

[0079] The weight adjustment of the high-speed interface system j after the transfer is shown in the following formula:

[0080]

[0081] in, The proportion of total flow transferred from the low-speed system; This represents the sum of the initial weights of all low-speed systems; This represents the weight of high-speed system j among all high-speed systems, used to allocate the total transferred traffic to each high-speed system j according to the load capacity ratio of the high-speed systems.

[0082] To ensure system stability, this mechanism also includes a flexible circuit breaker strategy, which triggers a circuit breaker when a file system i meets any of the following conditions five consecutive times: (Error rate too high) (Delay exceeded) or (Overload).

[0083] After the circuit breaker is triggered, the file system Temporarily set to 0 to stop receiving new requests, and simultaneously initiate a shard migration process across file system management blocks, migrating the stored shards to a file system with higher weight; the system checks the recovery status of the file system every minute, when... At that time, the weight is gradually restored according to the circuit breaker recovery curve, that is, the weight value at minute t after the circuit breaker is triggered (t=1,2,...) is... Until the weights return to normal levels.

[0084] Example 4

[0085] This embodiment is a further refinement of Embodiment 1. In this embodiment, the distributed file system management system is deployed on the server side, and the file information acquisition system is deployed on three file system sides: the NTFS file system of Windows, the ext4 file system of Linux, and the jffs2 file system of the embedded device, forming a heterogeneous file system cluster. The file information acquisition system is connected to the server side through hardware interfaces, with the NTFS and ext4 file systems connected via a gigabit network interface, and the jffs2 file system connected via an SPI interface. Initially, each file information acquisition system initiates the metadata acquisition process: the acquisition system of the NTFS file system acquires the internal files "doc1.txt" (2MB) and "img1.jpg" (5MB), the acquisition system of the ext4 file system acquires "data1.bin" (10MB), and the acquisition system of the jffs2 file system acquires "log1.txt" (1MB). The acquired metadata is encapsulated in a unified format, including fields such as "filename, size, creation time, and file system identifier." After encapsulation, each acquisition system uploads the metadata to the distributed file system management system on the server side through the corresponding interface. After receiving the metadata, the server stores NTFS-related data in file system metadata block 1 (starting address 0x00001000), ext4-related data in file system metadata block 2 (starting address 0x00002000), and jffs2-related data in file system metadata block 3 (starting address 0x00003000). Simultaneously, it constructs file system index blocks, recording the starting addresses of the three metadata blocks, forming index entries: 0x00001000 (NTFS), 0x00002000 (ext4), and 0x00003000 (jffs2). When storing a 20MB file "bigfile.dat" across these three file systems, the system first partitions the file into three pieces in a 4:3:3 ratio, with sizes of 8MB, 6MB, and 6MB respectively. Subsequently, the first partition is stored in the NTFS file system and named "bigfile_part1.dat"; the second partition is stored in the ext4 file system and named "bigfile_part2.dat"; and the third partition is stored in the jffs2 file system and named "bigfile_part3.dat". Simultaneously, the distributed management block creates a cross-file system management block, allocates storage address 0x00005000, and writes this address into the cross-file system index.Next, the system records the storage information of each partition into the cross-file system management block according to a preset format, forming three records: the first record contains "bigfile.datpart1 NTFS storage path / usr / ntfs / starting address 0x00010000 size 8MB"; the second record contains "bigfile.datpart2 ext4 storage path / var / ext4 / starting address 0x00020000 size 6MB"; and the third record contains "bigfile.datpart3 jffs2 storage path / mnt / jffs2 / starting address 0x00030000 size 6MB". The total number of cross-file system files recorded in the cross-file system management block is 1. When storage needs to be expanded later, and a yaffs2 file system of a Linux system is added, the file information acquisition system collects the metadata of this system and uploads it. The server adds address 0x00004000 (yaffs2) to the file system index block. If "bigfile.dat" needs to be expanded by 4MB of data, the system adds a new fragment "bigfile_part4.dat" and stores it in the yaffs2 file system. Its information, "bigfile.datpart4yaffs2, storage path / home / yaffs2 / , starting address 0x00040000, size 4MB," is appended to the cross-file system management block in the same format. At this point, the total number of files across the file system remains 1, but the fragment record is updated to 4, thus expanding the storage. During file reading, the system locates the cross-file system management block (address 0x00005000) through the cross-file system index, parses the storage information of the 4 fragments, then locates the file system metadata block to which each fragment belongs through the file system index block. Finally, it accurately locates the physical location of each fragment in the corresponding file system through the metadata block information, thereby achieving parallel reading and significantly improving reading efficiency.

[0086] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0087] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed file system management system, characterized in that: It includes a distributed management block, multiple file system metadata blocks, and a file information acquisition system; the distributed management block is configured to implement index management of each file system and collaborative control across file systems, and the file information acquisition system is configured to collect file metadata of each file system and upload it to the file system metadata block; The distributed management block includes file system index blocks, cross-file system indexes, cross-file system management blocks, and a hybrid consistency protocol module; File system index blocks are used to record the starting addresses of each file system metadata block, while cross-file system indexes are used to point to the storage addresses of cross-file system management blocks; The hybrid consistency protocol module is used to implement consistency management of fragmented data across heterogeneous file systems. This module adds Lock Status, Checksum, and Version Vector fields to the detailed information of cross-file system management blocks, and integrates Paxos and ZAB protocols to form a hybrid consistency protocol, achieving atomicity and consistency in fragment updates. The LockStatus field has values ​​of 0 (unlocked), 1 (read locked), and 2 (write locked), used to control concurrent access permissions for fragments. The Checksum field is calculated using the SHA256 hash algorithm on the fragmented data, used to verify the integrity of data during transmission or storage. The Version Vector field uses an array format to record the latest version information of the corresponding fragment in each file system. The file information acquisition system includes a load awareness module for optimizing read / write traffic across file systems. This module comprises an LSTM load prediction model, a dynamic weight allocation mechanism, and a resilient circuit breaker mechanism. The LSTM load prediction model uses Load CPU, Load Mem, IOPS, Latency, Interface Speed, and Error Rate data sampled every 5 minutes over the past hour as input features, and outputs predicted Load CPU and Latency values ​​for the next 2 hours. The dynamic weight allocation mechanism calculates the read / write weights of each file system based on the LSTM load prediction model's predictions using a preset formula, and allocates read / write requests accordingly. The resilient circuit breaker mechanism triggers a circuit breaker when a file system malfunctions, stopping the reception of new requests and initiating a fragment migration process. Once the file system recovers, its weights are gradually restored according to the circuit breaker recovery curve.

2. A distributed file system management system according to claim 1, characterized in that: The triggering process of the hybrid consensus protocol is as follows: First, the proposal phase begins. The cross-file system management block acts as the proposer and sends an update proposal to all file systems that store file shards. The file system acts as the acceptor. The update proposal includes the file ID, shard ID, new Version Vector, and data digest information. After receiving the proposal, the acceptor verifies whether the Version Vector of the local shard is consistent with the old version carried in the proposal. If they are consistent, the acceptor returns a ready-to-accept response; otherwise, the proposal is rejected. Next, the confirmation phase begins. The proposer collects feedback from all recipients. When the number of recipients who agree to the proposal exceeds half of the total number, i.e., k>n / 2, where n is the total number of file systems storing the fragment, the atomic broadcast mechanism is triggered. The proposer sends a commit instruction to all recipients, and the recipients return a commit confirmation after performing the update operation. After receiving all committed confirmations, the cross-file system management block updates the global Version Vector to complete the consistency confirmation.

3. A distributed file system management system according to claim 2, characterized in that: The handling mechanism for abnormal situations is as follows: If the number of acceptors who agree to the proposal does not reach the threshold, the proposer will wait 100ms and re-initiate the proposal, and the maximum number of retries is 3; If a acceptor does not respond within the timeout period, the system will mark it as an abnormal node and temporarily exclude it from the consistency confirmation scope. After the node is confirmed to have recovered through heartbeat detection, its shard version information will be synchronized. To quantify the threshold for consensus confirmation, let the total number of file systems storing a file fragment be n, and the number of nodes k that agree on the proposal must satisfy the formula: ; When the version vectors V1=(s1,v1) and V2=(s2,v2) of two shards conflict, the timestamp priority principle is used to resolve the conflict, that is, the version with the more up-to-date timestamp is retained first. Its mathematical expression is: Where T1 and T2 are the timestamps of the two segments, and T... new A timestamp for the final version.

4. A distributed file system management system according to claim 3, characterized in that: The load awareness module optimizes read / write traffic across file systems as follows: First, the historical data is preprocessed using a sliding window method with a window size of 12. The data is then mapped to the [0,1] interval using a normalization formula to eliminate the influence of dimensions. Subsequently, an LSTM model with three hidden layers is trained, with 64, 32, and 16 nodes in each hidden layer. The model output is the predicted Load CPU and Latency values ​​for the next two hours. To adapt to real-time load fluctuations, the model parameters are updated every 30 minutes. Based on the above predicted values, a dynamic weight allocation mechanism is designed, where the read and write weights of each file system i are... The formula is obtained by weighting the results across four dimensions: In the formula, To predict CPU utilization normalized to the [0,1] interval, To predict response latency, Indicates health status. Prioritize interfaces; during traffic allocation, read and write requests are determined based on the file systems. Proportional allocation, for low-speed interface file systems, if its The system will automatically trigger traffic shifting to compensate for the load. The shifting ratio is calculated according to the following formula: in, This represents the traffic transfer ratio, used to compensate for the load on the low-speed interface file system; the weight adjustment of the high-speed interface system j after the transfer is shown in the following formula: in, The proportion of total flow transferred from the low-speed system; This represents the sum of the initial weights of all low-speed systems; represents the weight of high-speed system j among all high-speed systems, used to allocate the total transferred traffic to each high-speed system j according to the load capacity ratio of the high-speed systems.

5. A distributed file system management system according to claim 4, characterized in that: The file system index block uses a pointer addressing method, which directly locates the corresponding file system metadata block through the starting address. The file system metadata block stores file attribute information, including file name, file size and creation time. Cross-file system indexes use a pointer-based addressing method. The cross-file system management block is configured to manage the storage range of a single file in multiple file systems. It includes the total number of files in the cross-file system and detailed information of each file in the cross-file system. The detailed information includes the file name, storage distribution, and allocation details in each file system.

6. A distributed file system management system according to claim 1, characterized in that: The file information acquisition system and the distributed management block interact with each other through a hardware interface, which includes one or more of the following: network interface, PCIe interface, I2C interface, or SPI interface.

7. A distributed file system management system according to claim 6, characterized in that: The file metadata collected by the file information collection system has the same storage structure as the file system metadata block. Both are encapsulated in a preset unified format, which includes file identifier, capacity parameters, and timestamp fields.

8. A distributed file system management system according to claim 1, characterized in that: The cross-file system management block supports collaborative storage management of homogeneous and heterogeneous file systems. When a single file is distributed across multiple heterogeneous file systems, the cross-file system management block uses detailed information to achieve the association mapping of file fragments.

9. A distributed file system management system according to claim 8, characterized in that: The cross-file system management block is configured to support parallel storage and parallel read operations. By parsing detailed information, it locates the storage location of each file fragment, enabling collaborative data reading and writing across multiple file systems.

10. A method for extending a distributed file system, characterized in that: The method applied to the distributed file system management system according to any one of claims 1-9 includes the following steps: S1 collects file metadata from each file system through the file information collection system, encapsulates it in a unified format, and uploads it to the distributed file system management system. S2, the distributed file system management system stores the received file metadata into the corresponding file system metadata block, and constructs a file system index block based on the starting address of each file system metadata block to achieve unified management and control of metadata; S3, when a single file needs to be stored across multiple file systems, the file data is split into pieces, and each piece is stored in the target file system. At the same time, a cross-file system management block is created through a distributed management block, and its storage address is written into the cross-file system index. S4, record the storage information of the corresponding file fragments in each file system into the cross-file system management block according to the preset format, and each fragment information corresponds to a cross-file system file record; S5, in the process of adding a new file system to expand storage capacity, the file fragment information in the new file system is supplemented to the cross-file system management block according to a preset format to maintain the integrity of cross-file system file records.

Images