System and method for correlating sequential input file size with scalable resource consumption - Patents.com

Abstract

A system for predicting resources required for a program, the system including a processor, a storage device accessible by the processor, and a sequential file prediction program that, when executed by the processor, configures the system to: access a history file to determine sizes of past sequential input files input to a customer program and sizes of past resulting sequential output files generated by the customer program from processing the sequential input files, determine a correlation between the sizes of the past sequential input files and the sizes of the resulting past sequential output files, and use the correlation to predict sizes of future sequential output files based on a current size of the current sequential input file, and use the predicted future consumption of the scalable resource to perform at least one of memory allocation or scheduling of batch jobs executed by the system.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 334,362, filed April 25, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] The present embodiment relates to techniques and processes for relating input file size to scalable resource consumption. [Background technology]

[0003] Mainframe computing is a computing platform used by the world's largest companies today. Mainframes often handle many workloads such as accounts receivable, general ledger, payroll, and a variety of applications required for specific business requirements. These workloads are commonly referred to as jobs.

[0004] Mainframe environments contain databases that are sequential files that can be accessed in any order through indexing. A sequential file is generally a file that contains records of the same structure and length. Sequential files are processed sequentially from the beginning of the file to the end. These sequential files typically reside on direct access storage devices (DASD) or disk drives. Regardless of the storage medium, sequential files are processed by customer programs that perform data operations on the records in the sequential files and store the manipulated records in an output file that is typically assigned a predetermined size that may not be optimal for the manipulated records. Traditionally, the amount of output file required for the program's output, the time the program will take, and the resources it will consume are unknown. Summary of the Invention

[0005] A system for use in predicting resources required for a program, the system comprising: a processor; a storage device accessible by the processor; and a sequential file prediction program that, when executed by the processor, configures the system to: access a history file to determine sizes of past sequential input files input to a customer program and sizes of past sequential output files resulting from processing the sequential input files by the customer program; determine a correlation between the sizes of the past sequential input files and the sizes of the resulting past sequential output files; use the correlation to predict future consumption of scalable resources including future sizes of future sequential output files based on a current size of a current sequential input file; and use the predicted future consumption of scalable resources to perform at least one of memory allocation or to determine scheduling of batch jobs executed by the system, wherein the scalable resources include at least one of processing time and memory allocation. [Brief description of the drawings]

[0006] The drawings depict one or more implementations consistent with the present concepts and are meant to be illustrative and not limiting, in which like reference numerals refer to the same or similar elements. [Figure 1] FIG. 1 is a block diagram of a process for reading and writing I / O files to DASD and tape, and creating and recording system management facility data, according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a process for allocating a predetermined output file size for storing a processed sequential file according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram of a process for predicting output file size based on input file information according to an embodiment of the present disclosure. [Figure 4]FIG. 4 is a block diagram of a process for predicting output file size based on input file information according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a block diagram of a linear regression process for predicting output file size based on input file information for all file combinations according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram of a linear regression process for predicting output file size based on input file information for a particular file, according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a block diagram of a neural network process for predicting output file size based on input file information according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram of a process for predicting output file size based on input file information in a specific use case according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a simplified functional block diagram of a computer that may be configured as a server and / or mainframe. [Figure 10] FIG. 10 is a simplified functional block diagram of a personal computer or other workstation or terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Introduction In the following detailed description, numerous specific details are set forth by way of example in order to fully understand the relevant teachings. However, it will be apparent to those skilled in the art that the present teachings may be practiced without such details. In another aspect, well-known methods, procedures, components, and / or circuits have been described in a relatively general manner, without detailed descriptions, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0008] Many software programs, including International Business Machine (IBM)® z / OS programs, operate on sequential files and perform matches on records in these files. Programs can take multiple input files and generate multiple output files. The size of the output produced by these programs is typically not known before execution, but for practical reasons, output files are not allowed to grow indefinitely (e.g., their size is defined at creation time). If the program's output exceeds the capacity allocated to an output file, the program fails and terminates with an error status.

[0009] A program (e.g., a z / OS program) may fail to complete or perform poorly (as indicated by increased processing time) if it is limited by some resource. For example, a program may not complete if there are limitations on output file size or working memory size. A CPU-intensive program may take longer to complete if other CPU-intensive programs are running. If the operating system knows the resource requirements of a program before it runs it, it can allocate sufficient resources (CPU time, I / O time, memory allocation, etc.) to the job or reschedule its execution to minimize the load on the system.

[0010] In general, a sequential file is a file that contains records of the same structure and length. Sequential files are processed sequentially from the beginning of the file to the end. These sequential files usually reside on direct access storage devices (DASD) or disk drives. Regardless of the storage medium, sequential files are usually processed by a client program by performing data manipulations on the records in the sequential file and storing the manipulated records in an output file of a given size.

[0011] In one embodiment, records in a sequential file may contain information about transaction events, such as, for example, credit card transactions, phone calls, secure purchases, etc. A customer program (e.g., a credit card company program) can process these sequential files by reading the sequential files to match the records to new records written to a sequential output file. Some examples of possible matches include pulling specific statements from each record required for a particular department within the credit card company (e.g., accounting, sales, logistics, etc.), creating records summarizing transactions by unique entity, and merging data about each transaction from two different sources (e.g., matching purchase orders with sales orders).

[0012] Typically, a customer program is designed to perform a series of steps as part of a particular batch job, each of which executes the customer program and provides a programmatic reference to a sequential file through a set of parameters (e.g., "DD statements") that describe the dataset (name, size, etc.) and specify the input and output resources required for the dataset. A single customer program can perform different functions depending on what parameters are provided to it. However, the parameters provided to the customer program are unlikely to change for a particular step within a particular job, which is defined by a job name, a step name, and optionally a procedure step name. A procedure is a set of steps that can be used across multiple jobs and can have parameters that affect its execution. The names of the DD statements are specific to the customer program, providing a layer of abstraction so that the customer program does not need to know the specific names of the sequential files or their locations. DD statements map to input or output files. The size of the output file is defined at the time of creation, and the customer program may fail to complete if the output is larger than the size of the output file. Additionally, the output size of a customer program generally correlates to the processing time (e.g., CPU load, input / output (I / O) time, combined CPU / IO time, etc.) and working memory required to process the data set records. Therefore, a means to use DD statements to predict appropriate output file sizes for optimal memory allocation (e.g., allocate optimal output file sizes) and to predict processing resources to determine system timeliness (e.g., how much memory and processing time is needed to pull, manipulate, and store account statements from sequential files) would be beneficial to customers.

[0013] In the mainframe environment of the present embodiment, a job is a collection of programs, or job steps, that instruct the processor to perform a particular function. When a job is represented as a sequence of Job Control Language (JCL) statements, each job step is defined by an EXEC statement that references a program and is supported by DD statements that represent the datasets to be processed. The DD statements may include information such as the dataset name, the dataset volume (e.g., where the dataset is stored), and the dataset size. A job contains one or more programs, or job steps, but a job step, in turn, may represent a procedure, program, and / or other process that includes one or more steps, functions, and / or other components that each perform some form of processing. Thus, a job step is a single entry in a job, but is not limited to performing a single step of processing. Jobs are often referred to as batch jobs because a collection of JCL is batched by the processor. In this mainframe environment, jobs are processed continuously. These jobs periodically reference datasets (such as sequential files) as part of their processing. Thus, at a given time, several jobs may have several datasets open and may be processing data associated with these open datasets.

[0014] As shown in FIG. 1, an IBM® mainframe, also known as the "z platform," includes a CPU / ZOS 102 running the z / OS operating system, executes customer programs for reading and writing (e.g., sequential files) jobs / tasks 108 to DASD 104 and tape 106, and utilizes a USERCATALOG system (not shown) to account for data residing among the many storage devices typically configured in the system. These user catalogs are components of the Data Facility Product (DFP) and are an integral part of the Operating System (OS). As part of the operation of the system, jobs and tasks are executed that create, modify, and delete data on the system, and the catalog records this activity in a catalog structure for referential integrity. The catalog itself is a file that is indexed so that data can be quickly located where it resides in the storage environment. ICF is an acronym that represents this function. ICF stands for Integrated Catalog Facility. All storage devices work with the concept that data is stored on volumes, either tape volumes or disk volumes (also called DASD, direct access storage devices). The difference between tape and DASD is that DASD storage is online to the system, whereas tape data is more typically offline and brought online when required by a process or task. A DASD volume may contain many sequential files, also called data sets. There are usually multiple paths to reach data on the same DASD volume, so multiple tasks can access disk sequential file data sets on the same volume simultaneously. Tapes, on the other hand, are assigned to one task at a time. Tape data can stack multiple data sets on a volume, just like DASD. However, only one task can access one single data set on a tape volume at a time. Tape data is processed linearly or sequentially.The primary relevant information for a data set is the name of the data set, the volume on which the data resides, and in the case of tape data, the file sequence on the linear tape in which the data resides. Regardless of the storage means, the customer program executes jobs / tasks on sequential input files and generates sequential output files. Additionally, the z / OS operating system executes a sequential file prediction algorithm 110 that predicts output file sizes and other consumed resources based on input file sizes. The sequential file prediction algorithm 110 is described in more detail below. The sequential file prediction algorithm 110 is distinct from a customer program in that the customer program is any program whose failures may be prevented or whose performance may be improved based on predictions made by the sequential file prediction algorithm 110. The sequential file prediction algorithm 110 may be coded in a sequential file prediction program. It is envisioned that the sequential file prediction algorithm 110 may operate in a partially or fully recursive manner.

[0015] 2 is a block diagram of a conventional example process for allocating a predetermined output file size. In steps 202 and 204, a job and step for processing a sequential file begins execution. In step 206, the system allocates an input file DD 210 that identifies the input file for processing. In step 208, the system also allocates an output file DD 212 having a predetermined size for storing the processed information. In step 214, the client program processes the input file indicated by the input file DD and stores the results in the output file indicated by the output file DD, and in step 216, the step / job ends.

[0016] Rather than allocating an output file DD of a predetermined size, which may or may not be optimal for storing the output of the customer program, it would be more useful to predict the output file size. FIG. 3 is a block diagram of an example of a process for predicting the size of an output file based on input file information in an input file DD. Similar to FIG. 2, in steps 302 and 304, a job and step for processing a sequential file begins execution. In step 306, the system allocates an input file DD 308 for processing. However, rather than allocating an output file DD of a predetermined size, in step 310, the system analyzes information from the input file DD (DD name, DD size, etc.) and predicts the optimal size of one or more output files for storing the processed information, and in steps 320 and 322, allocates the output file DD accordingly. In step 324, the customer program processes the input file indicated by the input file DD and stores the results in the output file indicated by the output file DD, and in step 326, the step / job ends. Additionally, in step 312, the system sends the predicted resource usage to an operating system (e.g., z / OS) which may then take action to manage overall system resources, such as choosing to continue the job or reschedule the job, in step 314. If in step 316 it is determined that the system has rescheduled the job, then the step / job ends in step 326. However, if in step 316 it is determined that the system has not rescheduled the job, processing continues in step 318.

[0017] Rather than running a job at a predetermined time when the current load on the system is likely to be high, it is more beneficial to predict the processing resources required to run the job and reschedule the job for a later time when the system is less loaded. If a job is initiated and is predicted to use a large amount of working memory when there is little working memory available on the system, the operating system may choose to schedule the job for a later time when more working memory is available for the job.

[0018] Analysis of input file DD information to predict the optimal size of one or more output files, as well as the consumption of other scalable resources (e.g., DD names, CPU time, I / O time, memory allocation, etc.) can be performed by a number of different methods. Among others, these methods include linear regression and neural networks that determine a prediction function based on historical file data that shows known relationships between past input file sizes and their corresponding resource requirements. These prediction methods are described below.

[0019] Predictive Solutions Overview The goal of a predictive solution is to analyze historical data of job executions to create a predictive model to calculate the amount of one or more scalable resources required by a customer program given the size of the input DDs. Some examples of scalable resources include CPU time, real-world time, size of output DDs, and amount of working memory used. Historical data for every job is stored in SMF records. From these SMF records, the system can extract the job name, procedure name, step name, customer program name, list of input and output DDs and their sizes, CPU time, execution time, number of I / O operations, etc.

[0020] Job names, procedure names, step names, and customer program names are used as identifiers to build predictive models. Models may not be tied to customer program names alone, as they may perform different actions depending on parameters, and these parameters may not be accessible from SMF records. However, parameters provided to customer programs are unlikely to change within one particular job, so the behavior of customer programs is likely to remain constant from one job run to the next.

[0021] FIG. 4 is a block diagram of an overall solution for predicting required output file size and processing resources based on input file information. Specifically, in step 402, the system searches a history file showing known past input file sizes, past resource consumption, and their resulting past output file sizes for jobs prior to running the customer program. The idea is that there is a correlation between input file size and resulting resource consumption that can be determined and used for future predictions. Customer programs may use multiple input and output files, adding another level of complexity in learning the relationship (e.g., correlation) between input and output file sizes. As with any data set, the more observations in the data set, the more accurate the conclusions can be (e.g., the more relationships that are analyzed, the more accurate the correlations can be determined).

[0022] More specifically, the system uses this historical data to train one of the predictive algorithms in steps 404-408, which may include linear regression on all input and output file sizes, linear regression on the resource of interest (e.g., size of a particular output DD or IO time), and a neural network to predict all resources used in one algorithm. Once the linear function or neural network is trained, the system can use the predictive algorithm in step 410 to accurately predict future resource consumption based on future input file sizes processed by the customer program. The development / training of linear equations and neural networks for resource consumption prediction is described in more detail in Figures 5-8.

[0023] Linear regression solution 5 is a block diagram of an example linear regression process for predicting resource consumption based on input file information and presented combinations of input and output DD statements for all possible DD combinations. In step 502, the system retrieves a history file showing the relationship between input file size and the resulting resource consumption for previous jobs executed by the customer program. In step 504, the system calculates a linear function that best fits the input file size and the resource usage retrieved from the history file.

[0024] For example, a linear regression can be performed with input file size as the explanatory variable and output file size as the predictor variable. If the hypothesis holds true for the program steps in the history file, a correlation between input and output file size is established and output size can be predicted for future files. In this example, the computer program may decrease / increase the output file size so that a rate of change variable between input and output can also be calculated and recorded. In another example, the customer program splits and joins records and has multiple inputs and outputs. This type of customer program may require multiple iterations of the linear regression described above.

[0025] The result of step 504 in Figure 5 is a linear function that predicts output file size for all possible combinations of input files vs. output files. For example, if a customer program can process up to X input files and generate up to Y output files, a linear regression is run separately for each combination (e.g., an X x Y linear regression). Once the linear regressions are calculated, the system can select a linear function given the future input / output file combinations in step 506. For example, if a future job contains two input files and four output files, a linear function for the 2 x 4 scenario is selected and used to predict output file size and other resource consumption.

[0026] Rather than generating a linear function for all possible numbers of input files versus all possible numbers of output files, the system can perform a linear regression of all input file sizes versus a particular consumable scalable resource (e.g., a particular DD name or working memory). FIG. 6 is a block diagram of an embodiment of a linear regression process for predicting output file size based on input file information for a particular file DD name. In step 602, the system retrieves a history file showing the relationship between input file size and resulting output file size for previous jobs executed by the customer program. In step 604, the system determines the DD names that may be executed by the customer program. For example, customer program A may be designed to output one of five files with each DD name. Then, in step 606, the system calculates the best-fit linear function for the input / output file combination for each DD name separately. For example, a linear function is calculated for output files for DD name 1, and another linear function is calculated for output files for DD name 2. In step 608, the system selects a linear function for a given input / output file combination for the customer program's DD name, inputs the input file sizes for the given DD name processed by the customer program into the selected linear function, calculates a predicted output file size and other resource consumption as a result of processing the DD name file, and can use the predicted output file size for memory allocation and processing time predictions, as well as job / step rescheduling if beneficial.

[0027] In linear regression, there may be more than N coefficients for N input DDs. For example, for each input DD, X, there are k transformations (e.g., X 2 , X 3 , log(X), sqrt(X), etc.) which are input to a linear equation. The total number of inputs is then k*N, where k is the number of transformations applied to each input DD. The linear equation outputs the true size of one output DD.

[0028] Nevertheless, if there is a linear relationship between the size of the input and the resources consumed, then all input DD(s n ) with some weights (w n ) and construct a linear equation that sums the values ​​to get the amount of the desired resource.

number

[0029] When a nonlinear relationship exists between the size of the input and the resources consumed, a set of transformation functions (f k ) for the size of each input to generate a set of inputs. Each input DD can be fed into a transformation function f (squared, cubed, logarithm, etc.) to provide it as another input, expanding the possibility of nonlinear relationships. Suppose k transformations are applied to n input DDs.

number

number

[0030] The weights are specific to one resource, while other resources have a different set of weights used. Once the set of weights for each desired resource is determined, the system can calculate the projected resource usage given a list of input sizes. Assuming there are n input DDs and m output DDs, the size of the output DDs can be predicted using the following operation:

number

number

[0031] The input size vector, including the other columns of weights

number

[0032] The weights in this formula can be calculated by analyzing the history file data from the SMF records. By examining the sizes of the input and output DDs for each run of the customer program, the optimal weights that best predict the size of the output DD can be calculated.

[0033] For one customer program, only one linear regression is needed per output DD. The weights (w n ), a function that can compute an output size given a list of input sizes. If there are n unique input DDs across k runs of a particular client program, the system can construct this set of linear equations for each output DD.

number

number

number

number

number

[0034] As a result,

number

number

number

[0035]

number

number

number

[0036] Gradient descent is an iterative process in which partial derivatives of the cost function C are taken with respect to each weight to create a gradient. This gradient is then used to adjust all the weights towards the minimum of the cost function.

number

number

number

[0037] Here, α is the "learning rate",

number

[0038] The gradient vector, like a derivative, describes the rate of change. This vector represents the direction and speed of the fastest increase. By subtracting the gradient from the weights, the weight vector moves in the direction of fastest decrease, toward the minimum. This is done by repeatedly lowering the gradient by a small amount and re-evaluating it at the next point to prevent overshooting the minimum.

[0039] Neural Network Solution FIG. 7 is a block diagram of an example of another solution using a neural network process to predict output file size based on input file information. In step 702, the system searches a history file showing the relationship between input file size and resulting output file size of previous jobs executed by the customer program. In step 704, the system determines the DD names and output files of the customer program. For example, customer program A may output five files with respective DD names. Then, in step 706, the system trains the neural network by the following steps: 1) input the input file size values ​​for each DD name combination into the input nodes of the neural network, 2) search the nonlinear and conditional relationships between the input size and the output size in the hidden layer, 3) output the predicted output file size for the given DD input combination, 4) compare the predicted output file size with the obtainable past output file size to determine the difference, 5) adjust the weights of the neural network based on the difference, and 6) repeat the steps until the neural network is trained. Then, in step 708, the system can use the trained neural network to: 1) Feed input file size values ​​for a given combination of DD names to the input node of a trained neural network; 2) Output a predicted output file size for the given input combination of DDs; 3) Use the predicted output file size for memory allocation and processing time predictions, as well as job / step rescheduling if beneficial.

[0040] In one neural network, the size of all output DDs can be predicted at once. The neural network is constructed to have one input layer (one set) of input nodes, one output layer of output nodes, and multiple layers of nodes between the inputs and outputs. In each layer, each node receives inputs from all the nodes in the previous layer and computes a set of linear equations for these inputs. The resulting values ​​are passed to each node in the next layer, which computes its own set of linear equations. Very complex relationships can be modeled, and even external variables such as parameters and flags passed to the client program can be coded as inputs to the network.

[0041] The input to this algorithm can take a variety of forms, for example: a) each input node can receive the actual size of one input DD, b) the size of one input DD can be split across multiple input nodes, c) each input node can be a binary number of the size of the input DD, each input node can be a binary number related to the order of magnitude of the size, each input node can be a normalized size of the input DD (scaled to be between 0 and 1), and the input and output data are then manipulated to convert between the actual size and the data the algorithm uses during processing.

[0042] Gradient descent can also be used to train an algorithm, as mentioned in the previous proposal, to find the correct weights and biases for each node. By providing the algorithm with past data, a cost function can be evaluated for each data point to assess how close the algorithm was to the desired result. By evaluating the cost function over many past data points, a gradient can be calculated to determine which weights and biases need to be changed in the network to minimize the cost function. Since the gradient can vary over the vector space of all the weights and biases, small steps are taken down the gradient and this process is repeated many times.

[0043] Using a neural network has several favorable advantages: more complex relationships between inputs and outputs can be modeled, one model can solve all the input and output DDs for a particular customer program, parameters that affect the output size can be encoded as inputs to the network and taken into account. Some parameters include input file size, logical record length and block size of the input file, record format, day of week / year of job execution, parameters passed to the customer program encoded as on-off flags or enumerated values, DD names which may be encoded as indexes, and job execution time.

[0044] Some options for neural networks include: a) a network that does not do any data normalization. In this case, each input / output node is equal to the unbounded size of the corresponding input / output DD. This is essentially a multi-level linear equation operator, where each layer of the network is a set of linear equations just like in the first example. b) The size of each input / output DD is encoded in binary, and each input / output node of the network corresponds to one specific bit of the size of one specific input DD or output DD. c) Normalize the input data using logarithmic or z-score value normalization, possibly passing the result through a normalization function such as sigmoid or tanh. Using logarithms allows for a dramatic reduction in the range of input sizes while still maintaining meaningful differences between extremely large and extremely small data sets. With typical normalization functions that do not use logarithms (such as sigmoid), the difference becomes negligible at the extremes, even if there are large amounts of difference. d) Each input / output node corresponds to a specific digit. This is a generalization of the above two methods. This is preferable in that the system can have a bucket into which it puts each DD based on how large it is, and instead of needing 40 bits (nodes) just to describe one 1TB DD, the system can encode it in 10 bits or less. Using this approach, the size of the inputs can be more clearly differentiated, resulting in a more accurate model.

[0045] Once the data has been normalized, the operation of the network is essentially the same. A neural network is conceptually divided into layers, each of which contains a collection of nodes. There is one input layer, one output layer, and multiple intermediate or "hidden" layers. Starting with the first intermediate layer, each node in a layer receives input from each node in the previous layer. Each node stores a value (

number

number

number

[0046] If the previous layer has m nodes and the current layer has n nodes, the layers can be processed by the following operations:

number

number

[0047] The above formula is the function for each node of the network, and the below formula is the formula for calculating one layer of the network. σ is an activation function that helps to limit the range of the output. The subscript x 0 and x 1 serves to distinguish the output of the previous layer from the output of the current layer. The output of each layer is provided as input to the next layer and can be passed through some activation functions to constrain the range of the node's output. Activation functions help to prevent values ​​from becoming too large and meaningless to the network or exceeding the size limits of the data type.

[0048] Stochastic gradient descent is used to train neural networks by gradually approximating optimal weights and biases to model the relationship between the inputs and outputs. By combining all weights and biases from all layers into a single vector, the gradient of the cost function with respect to all weights and biases can be calculated. Note that the partial derivatives of the cost function will be different for each layer, as downstream layers change the impact that a particular weight or bias has on the cost function.

[0049] Once the linear function is calculated, the system selects the linear function for a given input / output file combination for the DD name of the client program, inputs the input file sizes for the given DD name processed by the client program into the selected linear function, calculates a predicted output file size as a result of processing the DD name file, and uses the predicted output file size to predict memory allocation and processing time.

[0050] Use Cases FIG. 8 is a block diagram of an example process for predicting output file size based on input file information in a specific use case of a credit card company. In step 802, the system retrieves a history file of previous input file and output file size combinations previously processed by a credit card company customer program (e.g., an accounting program). In step 804, the system determines the DD name for the output file of the credit card company program (e.g., an "accounting" file) used by the customer program (e.g., a program that pulls a particular statement and reports to the credit card company's accounting department). In step 806, the system calculates a linear function (e.g., a linear function for the "accounting" file) that best fits the input / output file combination for the "accounting" DD name, or trains a neural network to predict the output accounting file size based on the input accounting file size. Then, in step 808, the system inputs the input file size processed by the customer program into the selected linear function or trained neural network, calculates the predicted output file size(s) for the accounting DD name after processing, and uses the predicted output file size for job memory allocation and processing time prediction, as well as job / step rescheduling if beneficial. In one embodiment, the system can allocate memory to store the output file based on the predicted output file size. In another embodiment, the system can predict the processing time of the input files to generate the output file. As described above, this processing time can be used to modify / schedule the job, etc.

[0051] System Hardware 9 and 10 are functional block diagrams of computer hardware platforms. FIG. 9 illustrates a network or host computer platform that may be commonly used to implement servers and / or mainframes, including, but not limited to, a tape system 902, DASD 904, user I / O 906, communication ports 908, and a system processor 910 that includes a CPU, memory, operating system, applications, batch jobs, SMF, etc. FIG. 10 depicts a computer with user interface elements that may be used to implement a personal computer or other type of workstation or terminal device, although the computer of FIG. 10 may also function as a server if appropriately programmed. In this embodiment, the computer of FIG. 10 includes a monitor 1000, keyboard 1002, mouse 1004, communication ports 1006, database 1008, user I / O 1010, RAM 1012, ROM 1014, CPU 1016, etc. The structure and general operation of devices such as those shown in FIG. 9 and 10 are believed to be self-evident from the general diagram.

[0052] For example, note that a mainframe includes a data communications interface for packet data communications and an input / output (I / O) controller. The I / O controller manages communications to various I / O elements and storage devices. The storage devices include one or more direct access storage devices (DASDs) and / or one or more tape systems. Such storage devices provide storage for data, batch processing, and jobs for managing applications. The mainframe includes communications ports, I / O controllers, and an internal communications bus that provides communication channels between one or more system processors. Each system processor includes one or more central processing units (CPUs), local memory associated with each CPU, and shared memory available to any CPU. An operating system (OS) executed by the system processor manages the various jobs and applications currently being executed and performs appropriate processing. The OS also provides system management functions (SMFs) and open exits for managing the operation of the mainframe and the various jobs and applications currently being executed. The hardware elements, operating systems, jobs, and applications of such a mainframe may be conventional. Of course, mainframe functionality may be implemented in a distributed manner on multiple similar platforms to distribute processing load and / or replicated across one or more similar platforms to provide processing redundancy. Accordingly, a replicated environment is also depicted in Figure 9. Although the details of a replicated environment are not depicted, such a replicated environment would typically include similar components as already described in connection with the primary mainframe of Figure 9.

[0053] It should be noted that a computer type user terminal equipment such as a PC also includes a data communication interface CPU, a main memory, and one or more mass storage devices for storing user data and various executable programs. The various types of user terminal equipment also include various user input / output devices. For example, a computer may include a keyboard, a cursor control / selection device such as a mouse, a trackball, a touchpad, and a display for image output. The hardware equipment, operating systems, and programming languages ​​of such user terminal equipment may be essentially conventional.

[0054] Thus, an aspect of the method for recording the start and end times of each step in a job executed in a mainframe computing environment as outlined above can be embodied in a computer program. An aspect of the program of the present technology can be considered a "product" or "article of manufacture" in the form of executable code and / or associated data typically carried or embodied in some type of machine-readable medium. A "storage" type medium includes tangible memory of a computer, processor, etc., or any or all of its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programs at any time. All or part of the software may be communicated over a global information network (e.g., the Internet) or various other telecommunications networks. Such communication may, for example, allow the software to be loaded from one computer or processor to another, such as from a management server or host computer to a mainframe platform that executes various jobs. Thus, other types of media that may carry software elements include optical, electrical and electromagnetic waves, such as those used over physical interfaces between local devices, over wired and optical fixed line networks, and over various air links. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0055] Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s), which may be used to hold data sets of sequential files or customer / forecast programs of enterprise applications. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Thus, common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, CD-ROMs, DVDs or DVD-ROMs, other optical media, punch cards paper tapes, other physical storage media with patterns of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, other memory chips or cartridges, or other media from which a computer can read program code and / or data. Many of these forms of computer readable media may be involved in transmitting one or more sequences of instructions to a processor for execution.

[0056] The terms and expressions used herein shall be understood to have the ordinary meanings given to such terms and expressions with respect to their respective corresponding areas of inquiry and study, unless otherwise specified herein. Relative terms such as first and second are used only to distinguish one entity or action from another and do not necessarily require or imply an actual relationship or sequence between such entities or actions. The terms "comprises," "comprising," "includes," "including," or other variations are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus consisting of a list of elements may include other elements not expressly listed or inherent to such process, method, article, or apparatus, rather than including only those elements. An element preceded by "a" or "an" does not, without further constraints, preclude the presence of additional identical elements in the process, method, article, or apparatus that constitutes the element.

[0057] Unless otherwise indicated, all measurements, values, ratings, positions, dimensions, sizes, and other specifications set forth herein are approximate rather than exact, and are intended to have a reasonable range consistent with the functions to which they pertain and those customary in the art to which they pertain.

[0058] While the above describes what are considered to be the best mode and / or other embodiments, it is understood that various modifications may be made thereto and that the subject matter disclosed herein may be embodied in a variety of forms and embodiments and may be applied to numerous applications, only a few of which are described herein. It is intended by the following claims to claim all such modifications and variations that fall within the true scope of the concepts.

Claims

1. 1. A system for use in predicting resources required for a program, the system comprising: A processor; a storage device accessible to the processor; a sequential file prediction program, when executed by the processor, accessing a history file to determine the sizes of past sequential input files input to the client program and the sizes of past sequential output files generated as a result of said client program processing said sequential input files; determining a correlation between the size of the historical sequential input file and the size of the resulting historical sequential output file; utilizing the correlation to predict future consumption of the scalable resource, including future sizes of future sequential output files, based on the current sizes of current sequential input files; utilizing the predicted future consumption of the scalable resource to perform at least one memory allocation or to determine scheduling of batch jobs to be executed on the system; A sequential file prediction program for configuring the system as described above; Including, The scalable resources include at least one of processing time, working memory, and input / output time. system.

2. 2. The system of claim 1, wherein the processor is further configured to determine the correlation by performing a linear regression on the sizes of the past sequential input files and the resulting consumption of the scalable resource, the linear regression generating a linear function for performing the prediction of the future consumption of the scalable resource based on a current size of the current sequential input file.

3. 3. The system of claim 2, wherein the processor is further configured to perform the linear regression as N×M linear regressions for N×M combinations of N input files and M output files, where N and M are integer values ​​ranging from 1 to a maximum number of input files and output files supported by the program.

4. 3. The system of claim 2, wherein the processor is further configured to perform, for each of the scalable resources consumed, the linear regressions, which are K×N linear regressions, for N input files and K transformations, where N is an integer value ranging from 1 to a number indicating a data definition (DD) of each input file.

5. 2. The system of claim 1, wherein the processor is further configured to determine the correlation by training a neural network by inputting the sizes of the past sequential input files, processing the sizes of the past sequential input files based on configured weights, predicting resource consumption of the scalable resource based on configured weights, calculating a difference between the predicted resource consumption and a known resulting consumption of the past scalable resource, and adjusting the configured weights to minimize the difference.

6. 6. The system of claim 5, wherein the processor is further configured to train the neural network based on at least one of the following: a size of the input file, a logical record length and a block size of the input file, a record format, a timing of job execution, parameters passed to the program from the input file, a data definition (DD) name of the input file, and a job execution time.

7. The system of claim 1 , wherein the processor is further configured to extract the history file from a system management facility (SMF) record indicative of job-related information.

8. The system of claim 1 , wherein the input and output files include transaction data for the customer.

9. 2. The system of claim 1, wherein the customer program used during processing of the past input files of the historical data is the same as the customer program used during processing of the future input files.

10. 2. The system of claim 1, wherein the processor is further configured to use at least one of the predicted future size, CPU time, and memory consumption of future sequential output files to determine at least one of a memory allocation for storing the output files and a processing time for generating the output files.

11. 1. A method for use in predicting resources required for a program, the method comprising: a processor accessing a history file to determine sizes of past sequential input files input to a client program and sizes of past sequential output files generated as a result of said client program processing said sequential input files; the processor determines a correlation between the size of the historical sequential input file and, as a result, the size of the historical sequential output file; the processor utilizing the correlation to predict future consumption of the scalable resource, including future sizes of future sequential output files, based on current sizes of current sequential input files; the processor utilizes the predicted future consumption of the scalable resource to perform at least one memory allocation or to determine the scheduling of batch jobs to be executed by the system. Including, The scalable resources include at least one of processing time, working memory, and input / output time. method.

12. and determining the correlation by performing a linear regression on the sizes of the past sequential input files and the resulting consumption of the scalable resource, the linear regression generating a linear function for performing the prediction of the future consumption of the scalable resource based on the current size of the current sequential input file. The method of claim 11.

13. and the processor further comprises: executing the linear regression as N×M linear regressions for N×M combinations of N input files and M output files, where N and M are integer values ​​from 1 to a maximum number of input files and output files supported by the program. The method of claim 12.

14. and the processor performing the linear regressions for each of N input files and the scalable resources consumed, the linear regressions being K×N linear regressions for K transformations, where N is an integer value ranging from 1 to a number indicating a data definition (DD) of each input file. The method of claim 12.

15. the processor determines the correlation by training a neural network by inputting the sizes of the past sequential input files, processing the sizes of the past sequential input files based on configured weights, predicting resource consumption of the scalable resource, calculating a difference between the predicted resource consumption and the known consumption as a result of the past scalable resource, and adjusting the configured weights to minimize the difference. The method of claim 11 further comprising:

16. the processor trains the neural network based on at least one of the input file size, the logical record length and block size of the input file, the record format, the timing of job execution, parameters passed to the program from the input file, the data definition (DD) name of the input file, and job execution time; The method of claim 15 further comprising:

17. the processor extracts the history file from a system management facility (SMF) record indicating job related information; The method of claim 11 further comprising:

18. The method of claim 11 , wherein the input and output files include transaction data for the customer.

19. 12. The method of claim 11, wherein the customer program used during processing of the past input files of the historical data is the same as the customer program used during processing of the future input files.

20. the processor uses at least one of the predicted future size of a future sequential output file, CPU time, and memory consumption to determine at least one memory allocation for storing the output file and processing time for generating the output file; The method of claim 11 further comprising:

Images