Method and apparatus for storage and retrieval of very large databases using a direct pipe

Abstract

A method and apparatus for directly connecting very large data streams from an archive command into a backup data system using an “intelligent process.” An output stream is piped into an intelligent pipe-reading process and distributed over a set of temporary data stores built from raw storage resources. A pipe interface process supervises backup of each filled data store, while the remaining output stream continues to be piped into another available data store. The backup system completes archiving of the datastream, keeping a catalog of the datastream storage locations. To retrieve the data, the intelligent process is run in reverse as a pipe-writing process, requesting data from the backup system. Retrieved data traverses the data stores from the backup system and are pumped into the pipe-writing process for delivery to the pipe output file identified by the retrieve or import command.

Description

FIELD OF THE INVENTION [0001] This invention relates generally to storage and retrieval of large databases, and more particularly to the use of temporary storage for moving large files. BACKGROUND OF THE INVENTION [0002] As large computing enterprises continue their migration from centralized mainframes and volumes of data on direct access storage devices (DASD) and serial tape drives, the reliance upon open systems database technology has increased. The ability to quickly adapt mass storage systems to new platforms, while retaining high performance and reliability will remain key elements for system integrators. [0003] In earlier times, rooms full of DASD and tape drives were maintained by legions of storage operators who physically mounted and unmounted tapes and disk packs and moved them to and from libraries according to daily schedules and batch job instructions. Technology improvements allowed the use of self-contained “mass storage” units, using robotic arms to move archived ...

AI Technical Summary

Benefits of technology

[0015] In an alternative embodiment of the invention, many of the data-store management tasks and backup scheduling can be handled by a self-contained backup system adapted for the purpose, leaving less activity for the intelligent pipe process, thus freeing up additional resources in the host computer.

[0016] Features of the invention include a method and apparatus wherein temporary storage, communication, and archiving for output of a generic database export command pipe is managed in a manner transparent to the customer operating the host computer. The customer is relieved of the need to maintain additional large local storage devices for physical or logical backups. The customer does not have to construct and maintain specialized pipes or scripts for network backups, or for different physical tape devices or DASDs.

[0017] Implementing a pipe process rather than a specific export / import function reduces the number of proprietary restrictions and configuration difficulties that depend upon choice of database implementation. The backup and archive methodology remains independent of the database structure or selected DBMS. Similarly, there is no need to work with a different API from each DBMS vendor, nor any customization of the Unix operating system.

[0018] A plurality parallel pipes according to the invention can be implemented for faster transfers of segmented database backups, where that feature is implemented in the DBMS. Furthermore, the invention enables any output-generating Unix-like process (application save, non-Oracle backup / restore) to have its results sent to the existing backup system, thus freeing up local resources automatically. Similarly, the same invention can be operated in reverse, enabling any Unix-like input-receiving process to obtain its data directly from a mass storage backup system. The temporary disk space needed to implement the data stores can also be designed as part of a backup system rather than the host computer, freeing up computer resources.

Problems solved by technology

Back-up issues will also arise when converting from one database management system (DBMS) to another, or sharing databases.

Backing up of large volumes of data can be a time consuming and resource intensive operation.

Mass storage systems, disadvantageously may be unavailable while large back-up operations are performed.

Creating a physical backup of the entire database 24 of the storage system 14 often requires a large investment of time and resources.

The major drawbacks of physical backup include that logical structures, such as tables of data cannot be backed up.

Further, data cannot be transferred between machines of differing operating systems.

Additionally, data blocks cannot be reorganized into a more efficient layout.

However, presently DMBS companies generally do not provide the API necessary for a customer to properly handle the data stream generated by the logical backup.

The result is that many of the DBMSs generate very large backup files that have to be stored locally until they can be written to an archive device.

However, this solution would require each such machine to have huge amounts of excess storage capacity.

Extra storage devices, media libraries, and personnel also take up extra space in the facility.

When dealing with very large databases, these solutions could break down entirely, due to the operational difficulties of maintaining the necessary physical media, or open network connections.

As mentioned, a disadvantage of the existing methods is the large amount of time it takes to perform backups, during which the database may be partially or completely offline due to read / write interlocks.

Even though the logical backup process can be segmented into parallel streams by some DBMSs, the implementations may be proprietary and not necessarily adaptable for import to another DBMS.

Also, disadvantageously, a dedicated disk or dedicated tape is required.

Further, in known implementations, there is an inability to catalog multiple versions.

VLDB reorganization and restructuring are major operations for which there is no known highly efficient solution.

Current solutions do not use the data management services of an Enterprise Data Management Tool (such as EDM).

Existing backup systems and APIs do not have the ability to export or import partitions in parallel using all available tape drives.

Images