Fault-tolerant computer network file systems and methods

Abstract

A system and method for replacing file system processors, also known as hot swapping, is described. The system and method operate on a fault-tolerant network file system that includes a first file server that is operably connected to a network fabric and a second file server that is operably connected to the network fabric. The fault-tolerant network file system includes a first disk array that is operably coupled to the first file server and to the second file server and a second disk array that is operably coupled to the first file server and to the second file server. First file system information is loaded on the first file server. The first file system information includes a first intent log of proposed changes to first metadata. Second file system information is loaded on the second file server. The second file system information includes a second intent log of proposed changes to second metadata. The first file server has a copy of the second metadata, and the second file server maintains a copy of the first metadata, thereby allowing the first file server to access files on the second disk array in the event of a failure of the second file server.

Description

RELATED APPLICATIONS [0001] This application is a divisional application of and claims priority benefit under 35 U.S.C. § 120 from U.S. application Ser. No. 10 / 060,908 filed Jan. 29, 2002, which is hereby incorporated herein by reference in its entirety, and which claims priority benefit under 35 U.S.C. § 119(e) from each of the following U.S. Provisional Applications, each of which is hereby incorporated herein by reference in its entirety. [0002] U.S. Provisional Application No. 60 / 264671, filed Jan. 29, 2001, titled “DYNAMICALLY DISTRIBUTED FILE SYSTEM”; [0003] U.S. Provisional Application No. 60 / 264694, filed Jan. 29, 2001, titled “A DATA PATH ACCELERATOR ASIC FOR HIGH PERFORMANCE STORAGE SYSTEMS”; [0004] U.S. Provisional Application No. 60 / 264672, filed Jan. 29, 2001, titled “INTEGRATED FILE SYSTEM / PARITY DATA PROTECTION”; [0005] U.S. Provisional Application No. 60 / 264673, filed Jan. 29, 2001, titled “DISTRIBUTED PARITY DATA PROTECTION”; [0006] U.S. Provisional Application No. ...

AI Technical Summary

Benefits of technology

[0025] The present invention solves these and other problems by providing a dynamically distributed file system that accommodates current demands for high capacity, throughput, and reliability, while presenting to the users a single-file-system interface that appears to include every file in the system on a single server or drive. In this way, the file system is free to flexibly, transparently, and on-the-fly distribute and augment physical storage of the files in any manner that suits its needs, across disk drives, and across servers, and users can freely access any file without having specific knowledge of the files current physical location.

[0026] One embodiment includes a storage device and architecture which possesses features such as transparent scalability where disks of non-identical capacity can be fully-utilized without the “dead-space” restrictions associated with conventional disk arrays. In one embodiment a flexible storage space allocation system handles storing large and small file types to improve disk space utilization. In another embodiment an improved method for maintaining data integrity overcomes the single drive (or double) fault limitation of conventional systems in order to increase storage reliability while at the same time reducing maintenance and monitoring requirements.

[0027] In one embodiment, distributed parity groups (DPG) are integrated into the distributed file storage system technology. This architecture provides capabilities for optimizing the use of disk resources by moving frequently and infrequently accessed data blocks between drives so as to maximize the throughput and capacity utilization of each drive.

Problems solved by technology

Demands for increases in storage capacity that are transparent to the users or for hardware upgrades that lack conformity with existing system hardware cannot be accommodated, especially while the system is in use.

In addition, such systems commonly suffer from the problem of data fragmentation, and they lack the flexibility necessary to intelligently optimize use of their storage resources.

The manner of data distribution and data redundancy within the disk array impacts the performance and usability of the storage system and may result in substantial tradeoffs between performance, reliability, and flexibility.

RAID arrays can reconstruct information stored in a particular data stripe using the parity information, however, this configuration imposes the requirement that records span across all drives in the array resulting in a small stripe size relative to the stored record size.

The aforementioned RAID architecture suffers from a number of drawbacks that limit its flexibility and scalability for use in reliable storage systems.

One problem with existing RAID systems is that the data striping is designed to be used in conjunction with disks of the same size.

Each stripe occupies a fixed amount of disk space and the total number of stripes allowed in the RAID system is limited by the capacity of the smallest disk in the array.

Any additional space that may be present on drives having a capacity larger than the smallest drive goes unused as the RAID system lacks the ability to use the additional space.

This further presents a problem in upgrading the storage capacity of the RAID system, as all of the drives in the array must be replaced with larger capacity drives if additional storage space is desired.

Therefore, existing RAID systems are inflexible in terms of their drive composition, increasing the cost and inconvenience to maintain and upgrade the storage system.

A further problem with conventional RAID arrays resides in the rigid organization of data on the disks of the RAID array.

As previously described, this organization typically does not use available disk space in an efficient manner.

Data storage in this manner is typically inefficient as regions or gaps of disk space may go unused due to the file organization restrictions.

Furthermore, the fixed block size of the RAID array is not able to distinguish between large files, which benefit from larger block size, and smaller files, which benefit from smaller block size for more efficient storage and reduced wasted space.

Although conventional RAID configurations are characterized as being fault-tolerant, this capability is typically limited to single disk failures.

Should more than one (or two) disk fail or become inoperable within the RAID array before it can be replaced or repaired there is the potential for data loss.

This problem again arises from the rigid structure of data storage within the array that utilizes sequential data striping.

This problem is further exacerbated by the lack of ability of the RAID system to flexibly redistribute data to other disk areas to compensate for drive faults.

Thus, when one drive becomes inoperable within the array, the likelihood of data loss increases significantly until the drive is replaced resulting in increased maintenance and monitoring requirements when using conventional RAID systems.

Accordingly, such systems have a single point of failure, such as the governing process or the system administrator.

This often means that the such load balancing can be slow to react, difficult to optimize for a particular server, and difficult to scale as the available resources expand or contract.

In addition, conventional load balancing typically is limited to balancing processing and communications activity between servers only.

Images