Event-based reconcile behavior for hierarchical storage management

Abstract

In an approach for automatically reconciling data in an HSM without impacting system performance, one or more file migration records are recorded in a reconcile database in response to migrating a file in a hierarchical storage system from primary storage to one or more tape drives. In response to an occurrence of a file event in the primary storage, the one or more file migration records in the reconcile database are updated. In response to receiving a command to unmount a first mounted tape on one of the one or more tape drives, a reconcile function is performed on the first mounted tape, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database.

Description

[Technical field]

[0001] The present invention relates generally to the field of data storage, and more particularly to automatically reconciling data in a Hierarchical Storage Management (HSM) without impacting system performance. [Background technology]

[0002] Magnetic tape data storage is a system for storing digital information on magnetic tape using digital recording. The tape is usually stored in cartridges or cassette tapes, but for drives used as tape backup for data storage, the tape is often wound on a reel. The "Linear Tape Open single reel cartridge" was introduced in 1997 with a capacity of 100 megabytes, and now in its eighth generation, it supports 12 terabytes in the same size cartridge. Magnetic tape drives are still used today, especially for offline data backup, due to their long archive stability and very favorable unit cost. Summary of the Invention [Means for solving the problem]

[0003] An embodiment of the present invention discloses a method, computer program product, and system for automatically reconciling data in an HSM without impacting system performance. In one embodiment, in response to migrating a file in a hierarchical storage system from primary storage to one or more tape drives, one or more file migration records are recorded in a reconcile database. In response to an occurrence of a file event in the primary storage, the one or more file migration records in the reconcile database are updated. In response to receiving a command to unmount a first mounted tape in one of the one or more tape drives, a reconcile function is performed on the first mounted tape, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a functional block diagram illustrating a distributed data processing environment in accordance with one embodiment of the present invention. [Figure 2a] FIG. 2a is a process diagram of an automatic reconciliation program according to one embodiment of the present invention. [Figure 2b] FIG. 2b is an illustration of a reconcile database according to one embodiment of the present invention. [Diagram 3] FIG. 3 is a flow chart diagram illustrating the operational steps of an automatic reconciliation program for reconciling a tape before it is unmounted in a computing device in the distributed data processing environment of FIG. 1 according to one embodiment of the present invention. [Figure 4] FIG. 4 is a flow chart diagram illustrating the operational steps of an automatic reconciliation program for reconciling tapes while the tape drives are idle on a computing device in the distributed data processing environment of FIG. 1 in accordance with one embodiment of the present invention. [Diagram 5] FIG. 5 illustrates a block diagram of components of a computing device executing an automatic reconciliation program within the distributed data processing environment of FIG. 1 in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The Data Management Applications Programming Interface (DMAPI) is an interface used by the System Management: Data Storage Management (XDSM) specification that allows a specialized interface to files in a storage system. The interface is file system independent, meaning that someone who writes a data management application that uses DMAPI can use it on any DMAPI file system. DMAPI was also originally implemented to allow files that were not in use to be stored offline, usually on tape, until they were requested. Once requested, the file was retrieved for use by the requesting user. A key feature of DMAPI is the use of events to notify data management applications of operations on files. Events are one of the foundations of DMAPI. In the event paradigm, the operating system notifies data management applications running in user space when a particular event occurs. For example, a data management application in user space can request to be notified every time a read of a region of a file occurs. When the operating system decides to read data from the region of interest, it notifies the data management application via a DMAPI event. In this way, data management applications can monitor and manage files and specific regions within those files.

[0006] Hierarchical storage management (HSM) is a data storage technology that automatically moves data between higher-cost and lower-cost storage media. The concept of HSM technology is similar to a memory cache in a computer, where the most actively used data is stored in expensive static random access memory, and less frequently accessed data is stored in slower dynamic random access memory.

[0007] HSM systems exist because fast storage devices, such as solid-state drive (SSD) arrays and hard disk drives, have a higher cost per byte stored than slower devices, such as magnetic tape drives. Ideally, all data would be available on the fast devices all the time, but this is cost-prohibitive for many enterprises. Instead, HSM systems store most of the data on the slower devices, and then copy the data to the faster disk drives when needed. In effect, the HSM turns the faster disk drives into a cache for the slower mass storage devices. The HSM system monitors how data is used and makes its best guess as to which data can be safely moved to the slower devices and which data should remain on the faster devices.

[0008] In a typical HSM scenario, frequently used data files are stored in primary storage, such as disk drives (including, for example, flash-based SSM drives), but are eventually migrated to tape if they have not been used for a period of time. When a user reuses a file stored on tape, it is automatically moved back to primary storage. The advantage is that the total amount of data stored can be much larger than the available disk storage capacity, but most users will not usually notice a slowdown since seldom used files are on tape. The operation of moving a file stored on the primary storage to tape is called migrate, and the operation of moving the file from tape back to the primary storage is called recall. As a result of the migrate operation, the file on the primary storage is replaced with a stub file that has a small amount of information.

[0009] When a stub file itself is deleted or renamed, such an event needs to be replayed on the tape where the file corresponding to the stub file is recorded. This operation is called a reconcile and involves synchronizing the metadata of the stub file with the tape. The reconcile operation is generally performed asynchronously with respect to the changes to the stub file. With current technology, the reconcile operation needs to be performed manually, which requires scanning the entire file system to retrieve all stub files and the metadata associated with the stub files (e.g., file paths and extended attributes), comparing the results of the scan with information about the files on tape, and finally performing a metadata synchronization. An embodiment of the present invention allows for the reconcile function to be automated without impacting system performance.

[0010] A typical implementation of a reconcile operation begins by first scanning the entire file system to retrieve all stub files and their associated metadata (e.g., file paths and extended attributes). Then, the results of the scan are compared with the file information on tape, and a metadata synchronization is performed. Finally, based on the results of the comparison, files on tape associated with stub files not found in the scan are deleted because the stub files have been deleted from the primary storage, and for each stub file found on the primary storage, the metadata obtained by the scan of the stub file is compared with the metadata of the corresponding file on tape, and the metadata of the corresponding file on tape is updated with the metadata of the stub file on the primary storage.

[0011] The problem with this scan-based reconcile operation is that it takes a long time to execute because the entire file system needs to be scanned for each reconcile operation. In addition, if only a tape is reconciled, it is inefficient because the entire file system needs to be scanned even if no changes have occurred to the stub files related to that tape. Even worse, if multiple tapes need to be reconciled, for example, if the remaining free space on multiple tapes is insufficient, multiple tapes need to be reconciled to increase the remaining free space. If a reconcile operation is performed on multiple tapes using the reconciliation scan method, it takes too much time because the reconcile operation scans the entire file system and the tapes need to be replaced if the file system spans multiple tapes.

[0012] An embodiment of the present invention introduces a way to perform reconciliation automatically without impacting system performance. Instead of scanning the file system, this implementation stores metadata events that need to be processed by the reconcile function in a database called the reconcile database. In one embodiment of the present invention, each tape in the HSM has its own reconcile database.

[0013] In one embodiment of this event-based reconciliation, an automatic reconciliation is processed in the following cases: (1) a tape is about to be unmounted and at least one event is found in the corresponding reconciliation database; (2) if a tape is mounted in a drive, the drive is idle for a predetermined period of time, and at least one event is found in the corresponding reconciliation database; and (3) if a tape drive does not have a tape mounted and the drive is idle for a predetermined period of time. In the latter case, the system selects the tape with the most reconciliation events in its reconciliation database. By using this automatic reconciliation operation, the need for a manual reconciliation operation should be eliminated, or if a manual reconciliation operation is still required, the execution time of the manual reconciliation operation should be significantly reduced. In addition, because the automatic reconciliation is processed only when a tape exchange is not required or when system resources are idle, this method can minimize the performance impact on the system.

[0014] An embodiment of the present invention uses an implementation of an event-based reconcile operation. In one embodiment, a basic reconcile operation updates the tape with a deletion or a rename, or a combination thereof, of a stub file on the primary storage. The deletion and rename operations are referred to as events. To achieve this operation, in one embodiment, every time a deletion or rename event occurs in a stub file, the deletion or rename event is recorded in a reconcile database, such as the reconcile database 232 of FIG. 2a, and the deletion and rename events recorded in the database are written to the corresponding tape when the reconcile function is executed. Such a reconcile function is called an event-based reconcile function.

[0015] In one embodiment of an HSM using DMAPI, these events can be monitored by associating hook functions with the events. Hook functions are a means of executing custom code before, after, or instead of existing code. In one embodiment, the hook functions can obtain the information needed for the reconcile operation (e.g., file path of the deleted stub file, and file path before and after renaming). A reconcile database is used to record information from the hooked events. In one embodiment, the reconcile database has a table for each tape. In one embodiment, each table has at least the columns illustrated in FIG. 2b.

[0016] An important aspect of embodiments of the present invention is the ability to monitor both operations on the reconcile database and operations on tape drives or mounted tapes or a combination thereof, and to automatically perform reconcile operations on tapes at the appropriate times, as described below.

[0017] FIG. 1 is a functional block diagram illustrating a distributed data processing environment, generally designated 100, suitable for operation of an automatic reconciliation program 112 in accordance with at least one embodiment of the present invention. As used herein, the term "distributed" describes a computer system that includes multiple physically distinct devices that operate together as a single computer system. FIG. 1 provides only an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one of ordinary skill in the art without departing from the scope of the invention as set forth by the claims.

[0018] Distributed data processing environment 100 includes computing device 110 and computing device 130 connected to network 120. Network 120 can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN), such as the Internet, or a combination thereof, and can include wired, wireless, or fiber optic connections. Network 120 can include one or more wired or wireless networks, or a combination thereof, capable of receiving and transmitting data, voice, or video signals, or a combination thereof, including multimedia signals, including voice, data, and video information. In general, network 120 can be any combination of connections and protocols that support communication between computing device 110, computing device 130, and other computing devices (not shown) in distributed data processing environment 100.

[0019] Computing device 110 can be a standalone computing device, an administrative server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In one embodiment, computing device 110 can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any programmable electronic device capable of communicating with other computing devices (not shown) in distributed data processing environment 100 via network 120. In another embodiment, computing device 110 can represent a server computing system utilizing multiple computers as a server system, for example in a cloud computing environment. In yet another embodiment, computing device 110 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers) that function as a single pool of seamless resources when accessed within distributed data processing environment 100.

[0020] In one embodiment, computing device 110 includes an automatic reconciliation program 112. In one embodiment, automatic reconciliation program 112 is a subprogram of a program, application, or larger program for intelligent connection placement across multiple logical ports. In alternative embodiments, automatic reconciliation program 112 may be located on any other device accessible by computing device 110 via network 120.

[0021] In one embodiment, the computing device 110 includes an information repository 114. In one embodiment, the information repository 114 may be managed by the automatic reconciliation program 112. In an alternative embodiment, the information repository 114 may be managed by the operating system of the device, alone or in conjunction with the automatic reconciliation program 112. The information repository 114 is a data repository that can store, collect, compare or combine information, or perform any combination thereof. In some embodiments, the information repository 114 is external to the computing device 110 and is accessed through a communication network, such as the network 120. In some embodiments, the information repository 114 is stored on the computing device 110. In some embodiments, the information repository 114 can reside on another computing device (not shown), provided that the information repository 114 is accessible by the computing device 110. The information repository 114 includes, but is not limited to, system configuration data, storage system (HSM) configuration data, file system data, and other data received by the automatic reconciliation program 112 .

[0022] Information repository 114 may be implemented using any volatile or non-volatile storage medium for storing information as known in the art. For example, information repository 114 may be implemented using a tape library, an optical library, one or more independent hard disk drives, multiple hard disk drives in a redundant array of independent disks (RAID), solid-state drives (SSD), or random-access memory (RAM). Similarly, information repository 114 may be implemented with any suitable storage architecture known in the art, such as a relational database, an object-oriented database, or one or more tables.

[0023] In one embodiment, the computing device 110 includes a reconcile database 116. In one embodiment, the reconcile database 116 may be managed by the automatic reconciliation program 112. In an alternative embodiment, the reconcile database 116 may be managed by the device's operating system, either alone or in conjunction with the automatic reconciliation program 112. The reconcile database 116 is a data repository that can store event information for the HSM. In some embodiments, the reconcile database 116 is external to the computing device 110 and is accessed through a communications network, such as the network 120. In some embodiments, the reconcile database 116 is stored at the computing device 110. In some embodiments, the reconcile database 116 can reside on another computing device (not shown), provided that the reconcile database 116 is accessible by the computing device 110.

[0024] The distributed data processing environment 100 includes a computing device 130. The computing device 130 may be a storage device, such as a tape drive or tape drive system, a standalone computing device incorporating a storage device, a management server incorporating a storage device, or any other storage device capable of receiving, transmitting, and processing HSM event data, and may communicate with other computing devices (not shown) in the distributed data processing environment 100 via a network 120. The storage on the computing device 130 may be implemented using any volatile or non-volatile storage medium for storing information, as known in the art. For example, the information repository 114 may be implemented using a tape library, an optical library, one or more independent hard disk drives, multiple hard disk drives in a redundant array of independent disks (RAID), a solid state drive (SSD), or random access memory (RAM). Similarly, the storage on the computing device 130 may be implemented with any suitable storage architecture known in the art, such as a relational database, an object-oriented database, or one or more tables.

[0025] FIG. 2a is a process diagram of an automatic reconciliation program 112, generally designated 200, according to one embodiment of the present invention.

[0026] Process diagram 200 includes file system (FS) 210, which is a file system for the HSM. FS 210 also includes DMAPI 220, which is an API used by the HSM as well as includes event functions used by embodiments of the present invention. DMAPI 220 includes two events required for the present invention: delete event 222 and rename event 224. As the event names suggest, delete event 222 tracks when a stub file is deleted from primary storage and rename event 224 tracks when a stub file on primary storage is renamed. DMAPI 220 also includes hook function 226. Hook function 226 is a function that connects to, or "hooks up," the delete event 222 and rename event 224 of DMAPI 220 to notify automatic reconciliation program 112 that the event has occurred, allowing automatic reconciliation program 112 to update reconciliation database 116 with file migration records of file system events. This allows automatic reconciliation program 112 to keep reconciliation database 116 up to date with the state of file transactions on the HSMs in order to perform automatic reconciliation of data on one or more HSM tape drives without impacting system performance.

[0027] Process diagram 200 includes HSM 230, which is the HSM for the system described above. HSM 230 includes reconcile database 232, described above (which is the same as reconcile database 116 of FIG. 1), and tape drive / cartridge 234, which is a physical tape drive. HSM 230 includes reconcile function 240, described above. In an embodiment of the invention, the mechanisms that determine when the reconcile operation is performed automatically are described below in FIGs. 3 and 4. These mechanisms are unmount function 236, described in detail in FIG. 3, and monitoring daemon 238, described in detail in FIG. 4.

[0028] 3 is a flow chart diagram of a workflow 300 illustrating the operational steps of the unmount function 236 of the automatic reconciliation program 112 for automatic reconciliation of data on one or more HSM tape drives without impacting system performance. In alternative embodiments, the steps of the workflow 300 may be performed by any other program in cooperation with the automatic reconciliation program 112. In one embodiment, the automatic reconciliation program 112, while monitoring the HSM 230 via the DMAPI 220, detects that a command has been issued to a tape drive on the HSM 230 to unmount a currently mounted tape. In one embodiment, the automatic reconciliation program 112 determines whether there is an entry in the reconciliation database 216 for the currently mounted tape that is about to be unmounted. In one embodiment, if the automatic reconciliation program 112 determines that there is an entry in the reconciliation database 216 for the tape being unmounted, the automatic reconciliation program 112 performs a reconciliation function 240 on the mounted tape. In one embodiment, if there is either no entry for this tape in the reconciliation database 232 or the reconciliation function 240 is completed, the automatic reconciliation program 112 unmounts the tape currently mounted in the tape drive.

[0029] It should be appreciated that embodiments of the present invention provide at least a portion of the unmount function 236 for automatic reconciliation of data on one or more HSM tape drives without impacting system performance. However, Figure 3 provides only an example of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made by one of ordinary skill in the art without departing from the scope of the present invention as set forth in the claims.

[0030] It should be understood that the process depicted in FIG. 3 represents one possible iteration of the automatic reconciliation program 112 that is repeated each time an unmount command is issued for a tape drive, such as tape drive / cartridge 234, connected to the HSM.

[0031] The automatic reconciliation program 112 detects a command to unmount a tape (step 302). In one embodiment, the automatic reconciliation program 112, while monitoring the HSM 230 via the DMAPI 220, detects that a command has been issued to a tape drive on the HSM 230 to unmount a currently mounted tape.

[0032] The automatic reconciliation program 112 determines whether there is an entry in the reconciliation database 232 (decision block 304). In one embodiment, the automatic reconciliation program 112 determines whether there is an entry in the reconciliation database 216 for the currently mounted tape that is being unmounted. In one embodiment, if the automatic reconciliation program 112 determines that there is no entry in the reconciliation database 216 for the tape that is being unmounted ("No" branch, decision block 304), the automatic reconciliation program 112 proceeds to step 308 and unmounts the currently mounted tape.

[0033] The automatic reconciliation program 112 performs the reconciliation function 240 on the mounted tape (step 306). In one embodiment, if the automatic reconciliation program 112 determines that there is an entry in the reconciliation database 216 for the tape being unmounted ("Yes" branch, decision block 304), the automatic reconciliation program 112 performs the reconciliation function 240 on the mounted tape.

[0034] The automatic reconciliation program 112 unmounts the selected tape (step 308). In one embodiment, if there was no entry for this tape in the reconciliation database 232 or the reconciliation function 240 has completed, the automatic reconciliation program 112 unmounts the tape currently mounted in the tape drive.

[0035] 4 is a flow chart diagram of a workflow 400 depicting the operational steps of the monitoring daemon 238 of the automatic reconciliation program 112 to perform automatic reconciliation of data on one or more HSM tape drives without impacting system performance. In alternative embodiments, the steps of the workflow 400 may be performed by any other program in cooperation with the automatic reconciliation program 112. In one embodiment, the automatic reconciliation program 112 selects an available tape drive that is part of the HSM 230, such as the tape drive / cartridge 234 from FIG. 2a. In one embodiment, the automatic reconciliation program 112 determines whether the selected tape drive is currently idle. In one embodiment, if the automatic reconciliation program 112 determines that the selected tape drive is currently idle, the automatic reconciliation program 112 determines whether a tape is mounted in the current drive. In one embodiment, if the automatic reconciliation program 112 determines that a tape is mounted in the current drive, the automatic reconciliation program 112 determines whether there is an entry in the reconciliation database 232 for the tape currently mounted in the tape drive. In one embodiment, if the automatic reconciliation program 112 determines that a tape is not currently mounted in the selected tape drive, the automatic reconciliation program 112 scans the tapes available for the current tape drive to determine whether any of the tapes have an entry in the reconciliation database 232. In one embodiment, the automatic reconciliation program 112 determines whether there is a tape available for the current tape drive that has an entry in the reconciliation database 232.In one embodiment, if the automatic reconciliation program 112 determines that there is a tape available in the current tape drive that has an entry in the reconciliation database 232, the automatic reconciliation program 112 selects the tape with the most entries in the reconciliation database 232. In one embodiment, the automatic reconciliation program 112 mounts the tape with the most entries in the reconciliation database 232, selected in step 414. In one embodiment, the automatic reconciliation program 112 performs the reconciliation function 240 on the tape currently mounted in the selected tape drive.

[0036] It should be appreciated that embodiments of the present invention provide at least the monitoring daemon 238 portion of automatic reconciliation of data on one or more HSM tape drives without impacting system performance. However, Figure 4 provides only an example of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made by one of ordinary skill in the art without departing from the scope of the present invention as set forth in the claims.

[0037] It should be understood that the process depicted in FIG. 4 represents one possible iteration of the automatic reconciliation program 112, which is continuously repeated.

[0038] The automatic reconciliation program 112 selects a tape drive (step 402). In one embodiment, the automatic reconciliation program 112 selects an available tape drive that is part of the HSM 230, such as the tape drive / cartridge 234 from FIG. 2a.

[0039] The automatic reconciliation program 112 determines whether the tape drive is idle (decision block 404). In one embodiment, the automatic reconciliation program 112 determines whether the selected tape drive is currently idle. If the automatic reconciliation program 112 determines that the selected tape drive is not currently idle ("No" branch, decision block 404), the automatic reconciliation program 112 returns to step 402 and selects the next tape drive in the HSM 230.

[0040] The automatic reconciliation program 112 determines whether a tape is mounted (decision block 406). In one embodiment, if the automatic reconciliation program 112 determines that the selected tape drive is currently idle ("Yes" branch, decision block 404), the automatic reconciliation program 112 determines whether a tape is mounted in the current drive. If the automatic reconciliation program 112 determines that a tape is not mounted in the current drive ("No" branch, decision block 406), the automatic reconciliation program 112 proceeds to step 410 and scans for available tapes for this tape drive.

[0041] The automatic reconciliation program 112 determines whether there is an entry in the reconciliation database 232 (decision block 408). In one embodiment, if the automatic reconciliation program 112 determines that a tape is mounted in the current drive ("Yes" branch, decision block 406), the automatic reconciliation program 112 determines whether there is an entry in the reconciliation database 232 for the tape currently mounted in the tape drive. If the automatic reconciliation program 112 determines that there is an entry in the reconciliation database 232 for the tape currently mounted in the tape drive ("Yes" branch, decision block 408), the automatic reconciliation program 112 proceeds to step 418 and performs the reconciliation function 240 for the tape currently mounted in the tape drive. If the automatic reconciliation program 112 determines that there is no entry in the reconciliation database 232 for the tape currently mounted in the tape drive (the "No" branch, decision block 408), the automatic reconciliation program 112 returns to step 402 and selects the next tape drive in the HSM 230.

[0042] The automatic reconciliation program 112 scans for available tapes (step 410). In one embodiment, if the automatic reconciliation program 112 determines that no tape is currently mounted in the selected tape drive ("No" branch, decision block 406), the automatic reconciliation program 112 scans for available tapes in the current tape drive to determine whether any of the tapes have an entry in the reconciliation database 232.

[0043] The automatic reconciliation program 112 determines whether there is a tape that has an entry in the reconciliation database 232 (decision block 412). In one embodiment, the automatic reconciliation program 112 determines whether there is a tape available for the current tape drive that has an entry in the reconciliation database 232. If the automatic reconciliation program 112 determines that there is no tape available for the current tape drive that has an entry in the reconciliation database 232 (the "no" branch, decision block 412), the automatic reconciliation program 112 ends the cycle.

[0044] The automatic reconciliation program 112 selects (step 414) the tape with the greatest number of events in the reconciliation database 232. In one embodiment, if the automatic reconciliation program 112 determines that there is a tape available for the current tape drive that has an entry in the reconciliation database 232 ("Yes" branch, decision block 412), the automatic reconciliation program 112 selects the tape with the most entries in the reconciliation database 232.

[0045] The automatic reconciliation program 112 mounts the selected tape, step 416. In one embodiment, the automatic reconciliation program 112 mounts the tape with the most entries in the reconciliation database 232 selected in step 414 into the selected tape drive.

[0046] The automatic reconciliation program 112 performs the reconciliation function 240 on the mounted tape (step 418). In one embodiment, the automatic reconciliation program 112 performs the reconciliation function 240 on the tape currently mounted in the selected tape drive. The automatic reconciliation program 112 then completes the cycle.

[0047] Figure 5 is a block diagram illustrating components of a computing device 110 suitable for an automatic reconciliation program 112 in accordance with at least one embodiment of the present invention. Figure 5 illustrates a computer 500, one or more processors 504 (including one or more computer processors), a communications fabric 502, memory 506, including, for example, random access memory (RAM) 516 and cache 518, persistent storage 508, a communications unit 512, an I / O interface 514, a display 522, and external devices 520. It should be understood that Figure 5 provides only an example of one embodiment and does not imply any limitation with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0048] As shown, computer 500 operates on a communications fabric 502 that provides communications between one or more computer processors 504, memory 506, persistent storage 508, communications units 512, and one or more I / O interfaces 514. Communications fabric 502 may be implemented with any architecture suitable for passing data or control information between processors 504 (e.g., microprocessors, communications processors, and network processors), memory 506, external devices 520, and any other hardware components in the system. For example, communications fabric 502 may be implemented with one or more buses.

[0049] Memory 506 and persistent storage 508 are computer-readable storage media. In the illustrated embodiment, memory 506 comprises RAM 516 and cache 518. In general, memory 506 may comprise any suitable volatile or non-volatile computer-readable storage medium. Cache 518 is a high-speed memory that enhances performance of one or more processors 504 by holding recently accessed data and near recently accessed data from RAM 516.

[0050] Program instructions for the automatic reconciliation program 112 may be stored in persistent storage 508, or more generally, any computer-readable storage medium, for execution by one or more of the individual computer processors 504 via one or more memories of memory 506. Persistent storage 508 may be a magnetic hard disk drive, a solid-state disk drive, a semiconductor storage device, a read only memory (ROM), an electronically erasable programmable read-only memory (EEPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0051] The media used by persistent storage 508 may also be removable. For example, a removable hard disk may be used for persistent storage 508. Other examples include optical and magnetic disks, thumb drives, and smart cards, which are inserted into a drive for transfer onto another computer readable storage medium that is part of persistent storage 508.

[0052] Communications unit 512, in these examples, provides for communication with other data processing systems or devices. In these examples, communications unit 512 comprises one or more network interface cards. Communications unit 512 may provide for communication through the use of either or both physical and wireless communications links. In the context of some embodiments of the present invention, sources of various input data may be physically remote to computer 500, such that input data may be received and outputs may be transmitted via communications unit 512 as well.

[0053] The one or more I / O interfaces 514 allow for input and output of data with other devices that may be connected to the computer 500. For example, the one or more I / O interfaces 514 may provide a connection to one or more external devices 520, such as a keyboard, a keypad, a touch screen, a microphone, a digital camera, or any other suitable input device or combination thereof. The one or more external devices 520 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, such as the automatic reconciliation program 112, may be stored on such portable computer-readable storage media and loaded onto the persistent storage 508 via the one or more I / O interfaces 514. The one or more I / O interfaces 514 also connect to a display 522.

[0054] Display 522 provides a mechanism for displaying data to a user and may be, for example, a computer monitor. Display 522 may also function as a touch screen, for example, the display of a tablet computer.

[0055] The programs described herein are identified based on the applications in which they are implemented in specific embodiments of the invention. However, it should be understood that any particular program names herein are used merely for convenience, and thus the invention should not be limited to use in any particular application identified or implied or identified and implied by such names.

[0056] The present invention may be a system, method, or computer program product, or any combination thereof. The computer program product may include one or more computer readable storage media having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0057] The computer readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0058] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to an individual computing device / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to the individual computing device / processing device for storage in a computer readable storage medium.

[0059] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, such as Smalltalk, C++, and the like, conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0060] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer readable program instructions.

[0061] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to produce a machine. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer programmable data processing apparatus or other device, or combinations thereof, to function in a particular manner, such that a computer readable storage medium having stored thereon includes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof.

[0062] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on a computer, other programmable data processing apparatus, or other device, implement the functions / operations identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0063] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the drawings. For example, two blocks shown in succession may actually be accomplished as one step performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, depending on the functions involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, as well as combinations of multiple blocks of the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special purpose hardware and computer instructions.

[0064] The description of various embodiments of the present disclosure is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for automatic data reconciliation, comprising: recording, by the one or more computer processors, one or more file migration records in a reconcile database in response to migrating a file in the hierarchical storage system from the primary storage to the one or more tape drives; updating, by the one or more computer processors, the one or more file migration records in the reconcile database in response to an occurrence of a file event in the primary storage, where the file event is selected from the group consisting of a delete operation and a rename operation; and in response to receiving a command to unmount a first mounted tape on a tape drive of the one or more tape drives, performing, by the one or more computer processors, a reconcile function on the first mounted tape on the one of the one or more tape drives, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database. said computer-implemented method comprising:

2. in response to receiving a command to unmount the first mounted tape on the one of the one or more tape drives, performing a reconcile function on the first mounted tape on the one of the one or more tape drives; receiving, by the one or more computer processors, a signal to unmount the first mounted tape in the one tape drive of the one or more tape drives; searching, by the one or more computer processors, for one or more file migration records for the first mounted tape in the reconcile database; executing, by the one or more computer processors, a reconcile function on the first mounted tape in response to the reconcile database including one or more file migration records for the first mounted tape, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database; and unmounting the first mounted tape by the one or more computer processors.

2. The computer-implemented method of claim 1, comprising:

3. selecting, by the one or more computer processors, a first tape drive of the one or more tape drives; determining, by the one or more computer processors, whether the first tape drive of the one or more tape drives is idle; determining, by the one or more computer processors, in response to determining that the first tape drive of the one or more tape drives is idle, whether a time that the first tape drive of the one or more tape drives is idle exceeds a threshold; determining, by the one or more computer processors, whether an online tape is mounted in the first tape drive of the one or more tape drives in response to determining that the amount of time that the first tape drive of the one or more tape drives is idle exceeds a threshold; determining, by the one or more computer processors, in response to determining that the online tape is mounted in the first tape drive of the one or more tape drives, whether one or more file migration records for the online tape exist in the reconcile database; and in response to determining that one or more file migration records for the online tape exist in the reconcile database, performing, by the one or more computer processors, the reconcile function on the online tape, where the reconcile function updates the online tape with the one or more file migration records in the reconcile database. The computer-implemented method of claim 1 , further comprising:

4. determining, by the one or more computer processors, in response to determining that an online tape is not mounted in the first tape drive of the one or more tape drives, if one or more file migration records exist in the reconcile database for one or more offline tapes for the first tape drive of the one or more tape drives; selecting, by the one or more computer processors, an offline tape of the one or more offline tapes in response to determining that the one or more file migration records exist in the reconcile database for the one or more offline tapes for the first tape drive of the one or more tape drives, where the one offline tape of the one or more offline tapes has a maximum number of file migration records in the reconcile database; mounting the one of the one or more offline tapes by the one or more computer processors; and performing, by the one or more computer processors, the reconcile function on the one offline tape of the one or more offline tapes. The computer-implemented method of claim 3 , further comprising:

5. 2. The computer-implemented method of claim 1, wherein migrating files in a hierarchical storage system from primary storage to one or more tape drives further comprises creating a stub file in the primary storage containing metadata of the migrated file.

6. The computer-implemented method of claim 1 , wherein the file migration record in the reconcile database includes at least an event type, a first file path, and a second file path.

7. 10. The computer-implemented method of claim 1, wherein the primary storage comprises a solid-state disk drive.

8. 1. A computer program for automatic data reconciliation, comprising: recording one or more file migration records in a reconcile database in response to migrating a file in the hierarchical storage system from the primary storage to the one or more tape drives; updating the one or more file migration records in the reconcile database in response to an occurrence of a file event in the primary storage, where the file event is selected from the group consisting of a delete operation and a rename operation; and in response to receiving a command to unmount a first mounted tape on a tape drive of the one or more tape drives, performing a reconcile function on the first mounted tape on the one tape drive of the one or more tape drives, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database. The computer program product causing one or more processors to execute the steps of a method comprising:

9. in response to receiving a command to unmount the first mounted tape on the one of the one or more tape drives, performing a reconcile function on the first mounted tape on the one of the one or more tape drives; receiving a signal to unmount the first mounted tape in the one tape drive of the one or more tape drives; searching the reconcile database for one or more file migration records for the first mounted tape; in response to the reconcile database including one or more file migration records for the first mounted tape, performing the reconcile function on the first mounted tape, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database; and Unmounting the first mounted tape.

9. The computer program of claim 8, comprising:

10. selecting a first tape drive of the one or more tape drives; determining whether the first tape drive of the one or more tape drives is idle; responsive to determining that the first tape drive of the one or more tape drives is idle, determining whether a time that the first tape drive of the one or more tape drives is idle exceeds a threshold; determining whether an online tape is mounted in the first tape drive of the one or more tape drives in response to determining that a time that the first tape drive of the one or more tape drives is idle exceeds a threshold; determining whether one or more file migration records for the online tape exist in the reconcile database in response to determining that the online tape is mounted in the first tape drive of the one or more tape drives; and responsive to determining that one or more file migration records for the online tape exist in the reconcile database, performing the reconcile function on the online tape, where the reconcile function updates the online tape with the one or more file migration records in the reconcile database. The computer program product of claim 8 , further comprising causing the processor to execute method steps including:

11. determining whether one or more file migration records exist in the reconcile database for one or more offline tapes for the first tape drive of the one or more tape drives in response to determining that an online tape is not mounted in the first tape drive of the one or more tape drives; in response to determining that the one or more file migration records exist in the reconcile database for the one or more offline tapes for the first tape drive of the one or more tape drives, selecting one offline tape of the one or more offline tapes, where the one offline tape of the one or more offline tapes has a maximum number of file migration records in the reconcile database; mounting the one offline tape of the one or more offline tapes; and performing the reconcile function on the one offline tape of the one or more offline tapes; The computer program product of claim 10 , further comprising causing the processor to execute method steps including:

12. 9. The computer program product of claim 8, wherein migrating files in a hierarchical storage system from primary storage to one or more tape drives further comprises creating a stub file in the primary storage that contains metadata of the migrated file.

13. The computer program product of claim 8 , wherein the file migration record in the reconcile database includes at least an event type, a first file path, and a second file path.

14. The computer program product of claim 8 , wherein the primary storage comprises a solid-state disk drive.

15. 1. A computer system for automatic data reconciliation, the computer system comprising: one or more computer processors; One or more computer-readable storage media; and program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors; said stored program instructions comprising: recording one or more file migration records in a reconcile database in response to migrating a file in the hierarchical storage system from the primary storage to the one or more tape drives; updating the one or more file migration records in the reconcile database in response to an occurrence of a file event in the primary storage, where the file event is selected from the group consisting of a delete operation and a rename operation; and in response to receiving a command to unmount a first mounted tape on a tape drive of the one or more tape drives, performing a reconcile function on the first mounted tape on the one tape drive of the one or more tape drives, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database. The computer system further comprising instructions for:

16. in response to receiving a command to unmount the first mounted tape on the one of the one or more tape drives, performing a reconcile function on the first mounted tape on the one of the one or more tape drives; receiving a signal to unmount the first mounted tape in the one tape drive of the one or more tape drives; searching the reconcile database for one or more file migration records for the first mounted tape; in response to the reconcile database including one or more file migration records for the first mounted tape, performing the reconcile function on the first mounted tape, where the reconcile function updates the first mounted tape with the one or more file migration records in the reconcile database; and Unmounting the first mounted tape 16. The computer system of claim 15, comprising one or more of the plurality of program instructions stored on the one or more computer readable storage media for:

17. selecting a first tape drive of the one or more tape drives; determining whether the first tape drive of the one or more tape drives is idle; responsive to determining that the first tape drive of the one or more tape drives is idle, determining whether a time that the first tape drive of the one or more tape drives is idle exceeds a threshold; responsive to determining that a time that the first tape drive of the one or more tape drives is idle exceeds a threshold, determining whether an online tape is mounted in the first tape drive of the one or more tape drives; determining whether one or more file migration records for the online tape exist in the reconcile database in response to determining that the online tape is mounted in the first tape drive of the one or more tape drives; and responsive to determining that one or more file migration records for the online tape exist in the reconcile database, performing the reconcile function on the online tape, where the reconcile function updates the online tape with the one or more file migration records in the reconcile database.

16. The computer system of claim 15, further comprising one or more of the plurality of program instructions stored on the one or more computer readable storage media for:

18. in response to determining that an online tape is not mounted in the first tape drive of the one or more tape drives, determining whether one or more file migration records exist in the reconcile database for one or more offline tapes for the first tape drive of the one or more tape drives; in response to determining that the one or more file migration records exist in the reconcile database for the one or more offline tapes for the first tape drive of the one or more tape drives, selecting an offline tape of the one or more offline tapes, where the one offline tape of the one or more offline tapes has a maximum number of file migration records in the reconcile database; mounting the one offline tape of the one or more offline tapes; and performing the reconcile function on the one offline tape of the one or more offline tapes; 20. The computer system of claim 17, further comprising one or more of the plurality of program instructions stored on the one or more computer readable storage media for:

19. 16. The computer system of claim 15, wherein migrating files in a hierarchical storage system from primary storage to one or more tape drives further comprises creating a stub file in the primary storage that contains metadata of the migrated file.

20. 16. The computer system of claim 15, wherein the file migration record in the reconcile database includes at least an event type, a first file path, and a second file path.

Images