Virtual memory technique for efficiently solving connected problems in a distributed environment

Abstract

Embodiments herein present a virtual memory technique for efficiently solving connected problems in a distributed environment. An exemplary method begins by selectively loading computing task segments into a first node, wherein the number of computing task segments is the maximum number of computing task segments that will fit within the first node. The computing task segments are enclosed within wrappers, wherein the wrappers are streamed into the first node at a maximum rate that a disk can read and write data. Further, the wrappers comprise priority wrappers that must be computed in order to process the computing task segments; and thus, the priority wrappers are loaded into the first node first. In addition, the computing task segments comprise computing task segment block collections, wherein the method creates a list of computing task segment block collection jobs that must be completed in order to process one of the computing task segments.

Description

CROSS-REFERENCE TO RELATED APPLICATION[0001]The present application is related to a new U.S. patent application, filed concurrently, to Tobin J. Lehman, Robert Glenn Deen and James Kaufman, entitled “An Efficient Super Cluster Implementation For Solving Connected Problems In A Distributed Environment”, having (IBM), assigned to the present assignee, and incorporated herein by reference.BACKGROUND OF THE INVENTION[0002]1. Field of the Invention[0003]Embodiments herein present a virtual memory technique for efficiently solving connected problems in a distributed environment.[0004]2. Description of the Related Art[0005]There is a class of connected problems that involve problem sizes that span a single machine—common examples are Finite Element Model problems that involve simulating large physical systems. When dividing a problem across multiple machines in order to speed up the overall elapsed computation time, a common technique is to use a set of machines with enough memory that the...

AI Technical Summary

Benefits of technology

The invention is a method for efficiently solving connected problems in a distributed environment. It uses a natural problem partitioning system called OptimalGrid to efficiently manage memory. The method selectively loads computing task segments into a first node, where the number of computing task segments is the maximum that can fit in memory. It then unloads the results from the first node and processes them in other nodes. This approach is faster and more efficient than using operating system virtual memory. The invention also includes a method for handling a computing task when a computing task segment is too large for a single computing node by dividing it into smaller parts and selectively loading them onto the node. Overall, the invention improves the speed and efficiency of solving connected problems in a distributed environment.

Problems solved by technology

However, when adequate compute resources are not available, one typically waits until they are available.

The only real alternative—using compute nodes with less than sufficient physical memory—is not attractive if it requires using Operating System virtual memory.

The non-clustered nature of this type of problem would result in a fairly random access pattern that would thrash the disk drive and slow down the problem computation to a crawl.

Although an application-specific solution could be used, this would not qualify as a general-purpose solution.

Second, it is much faster than the alternative of using Operating System virtual memory.

Images