Systems and methods for associating sequential input file sizes with scalable resource consumption

Abstract

A system for predicting resources required for a program. The system comprises a processor; a storage device accessible by the processor; and a sequential file prediction program that, when executed by the processor, configures the system to access the history file to determine a size of a past sequential input file input to the customer program and a size of a resulting past sequential output file generated by the customer program processing the sequential input file, the correlation between the size of the past sequential input file and the result size of the past sequential output file is determined, and the future size of the future sequential output file is predicted based on the current size of the current sequential input file by using the correlation. And utilizing the predicted future consumption of the scalable resource to perform at least one of the memory allocations or determine a schedule of batch jobs being executed by the system.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 334,362, filed on April 25, 2022, and the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] This example relates to techniques and processes for correlating input file size with consumption of scalable resources. Background Art

[0003] Mainframe computing is the computing platform used by the world's largest companies today. Mainframes typically handle many workloads such as accounts receivable, general ledger, payroll, and various applications required for specific business needs. These workloads are often called jobs.

[0004] The mainframe environment includes a database, which may be a sequential file that can be accessed in any order through an index. A sequential file is a file that typically contains records of the same structure and length. Sequential files are processed sequentially from the beginning of the file to the end of the file. These sequential files typically reside on a direct access storage device (DASD) or a disk drive. Regardless of the storage medium, the sequential files are processed by a client program that performs data operations on the records in the sequential file and stores the operated records in an output file, which is typically allocated a predefined size that may not be optimal for the records being operated on. Traditionally, it is not known how much output file space is required for the output of a program, how long the program will take, or how many resources it will consume. Summary of the invention

[0005] A system for predicting resources required by a program. The system includes: a processor; a storage device accessible by the processor; and a sequential file prediction program, which, when executed by the processor, configures the system to: access a history file to determine the size of past sequential input files input to a client program and the size of past sequential output files generated by the client program processing the sequential input files, determine the correlation between the size of the past sequential input files and the result size of the past sequential output files, use the correlation to predict future consumption of scalable resources including future sizes of future sequential output files based on the current size of the current sequential input files, and use the predicted future consumption of the scalable resources to perform at least one of memory allocation or determine the scheduling of batch jobs being executed by the system, wherein the scalable resources include at least one of processing time and memory allocation. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings depict one or more embodiments of the inventive concept by way of example only and not limitation.In the accompanying drawings, the same reference numerals represent the same or similar elements.

[0006] Figure 1 is a block diagram of a process for reading and writing input / output files to DASD and tapes and creating and recording system management facility data according to an embodiment of the present disclosure.

[0007] Figure 2 is a block diagram of a process for allocating a predefined output file size for storing processed sequential files according to an embodiment of the present disclosure.

[0008] Figure 3 is a block diagram of a process for predicting an output file size based on input file information according to an embodiment of the present disclosure.

[0009] Figure 4 is a block diagram of a process for predicting an output file size based on input file information according to an embodiment of the present disclosure.

[0010] Figure 5 is a block diagram of a linear regression process for predicting output file sizes based on input file information of all file combinations according to an embodiment of the present disclosure.

[0011] Figure 6 is a block diagram of a linear regression process for predicting an output file size based on input file information of a specified file according to an embodiment of the present disclosure.

[0012] Figure 7 is a block diagram of a neural network process for predicting output file size based on input file information according to an embodiment of the present disclosure.

[0013] Figure 8 is a block diagram of a process for predicting output file size based on input file information in a specific use case according to an embodiment of the present disclosure.

[0014] Fig. 9 is a simplified functional block diagram of a computer that can be configured as a server and / or a mainframe.

[0015] Fig.10 It is a simplified functional block diagram of a personal computer or other workstation or terminal device. DETAILED DESCRIPTION introduction

[0016] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to those skilled in the art that the present teachings may be practiced without these details. In other cases, well-known methods, processes, components and / or circuits have been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of the present teachings.

[0017] Many software programs , including IBM z / OS programs) operate on sequential files and perform collation of the records within those files. Programs can take multiple input files and produce multiple output files. The size of the output produced by these programs is generally unknown prior to execution, but for practical reasons, output files cannot grow indefinitely (e.g., their size is defined at the time of their creation). If the output of a program exceeds the space allocated for an output file, the program fails and terminates with an error status.

[0018] If a program (e.g., az / OS program) is limited by certain resources, it may not complete or may have reduced performance (indicated by increased processing time). For example, if a program is limited in output file size or working memory size, the program may not complete. If a program is CPU intensive, it may take longer to process when other CPU intensive programs are running. If the operating system knows in advance what the resource requirements of a program are before executing it, it can allocate sufficient resources (e.g., CPU time, I / O time, memory allocation, etc.) to the job, or reschedule execution to minimize the load on the system.

[0019] A sequential file is a file that usually contains records of the same structure and length. Sequential files are processed sequentially from the beginning to the end of the file. These sequential files usually reside on a direct access storage device (DASD) or a disk drive. Regardless of the storage medium, sequential files are usually processed by a client program by performing data operations on the records in the sequential file and storing the operated records in an output file of a predetermined size.

[0020] In an example, the records in the sequential files may contain information about transaction events, such as credit card transactions, telephone calls, securities purchases, etc. A client program (e.g., a credit card company program) may process these sequential files by reading the sequential files to collate the records into new records that are written to an output sequential file. Some examples of possible collation include extracting specific transaction details from each record that is needed by a specific department within the credit card company (accounting, sales, logistics, etc.), producing records that summarize transactions for each unique entity, and merging data about each transaction from two different sources (such as matching a purchase order to a sales order).

[0021] Client programs are typically designed to be executed as part of a batch job that defines a series of steps, each of which executes the client program and provides a program reference to a sequential file via a set of parameters (e.g., "DD statements") that describe a data set (name, size, etc.), and specifies the input and output resources required for the data set. A single client program can perform different functions depending on the parameters provided. However, the parameters provided to the client program are unlikely to change for a specific step within a specific job defined by the job name, step name, and optional process step name. A process is a series of steps that can be used across multiple jobs and can have parameters that affect its execution. The name of the DD statement is specific to the client program and provides an abstraction layer so that the client program does not need to know the specific name of the sequential file or where it resides. The DD statement can be mapped to an input file or an output file. The size of the output file is defined at creation time, and if the output is larger than the size of the output file, the client program may not be completed. In addition, the size output of the client program is usually related to the processing time (e.g., CPU load, input / output (I / O) time, a combination of CPU / IO time, etc.) and the working memory required to process the data set records. Therefore, a device for using DD statements to predict appropriate output file sizes for optimal memory allocation (e.g., allocating optimal output file sizes) and for predicting processing resources for determining system timing would benefit customers (e.g., how much memory and processing time is required to extract, operate and store transaction details from sequential files).

[0022] In the example mainframe environment, a job is a collection of programs or job steps that instruct a processor to perform a specific function. When a job is represented as a sequence of job control language (JCL) statements, each job step is defined by an EXEC statement that references a program and supported by a DD statement that represents a data set to be processed. The DD statement may include information such as a data set name, a data set volume (e.g., the location where the data set is stored), and a data set size. Although a job includes one or more programs or job steps, a single job step may represent a process, a program, and / or other processing that includes one or more steps, functions, and / or other components, and the one or more steps, functions, and / or other components all perform some form of processing. Therefore, a job step is a single entry within a job, but is not limited to executing a single processing step. Jobs are also commonly referred to as batch jobs because a collection of JCLs is processed in batches by a processor. In this mainframe environment, jobs are processed continuously. These jobs periodically reference data sets (e.g., sequential files) as part of their processing. Thus, at any given point in time, a certain number of jobs may have a certain number of open data sets, and may be processing data associated with those open data sets.

[0023] like Figure 1 As shown, an IBM mainframe (also referred to as the "z platform") includes a CPU / ZOS 102 running the z / OS operating system, client programs that execute jobs / tasks 108 that read and write (e.g., sequential files) to DASD 104 and tape 106, and utilizes a USERCATALOG system (not shown) to describe data residing in multiple storage devices typically configured into the system. These user directories are components of the Data Facility product (DFP) and are an integral part of the operating system (OS). As part of the operation of the system, jobs and tasks running on the system create, modify, or delete data, and the directory records this activity in a directory structure for referential integrity. The directory itself is an index file that enables the data to be quickly located if it exists in the storage environment. ICF is an acronym that describes the facility. ICF stands for Integrated Catalog Facility. All storage works based on the concept that data is contained in a volume, which can be a tape volume or a disk volume, also known as a DASD, direct access storage device. The difference between tape and DASD is that DASD storage is online to the system, while tape data is more typically offline and is brought online when it is requested by a process or task. A DASD volume can contain multiple sequential files, also called data sets. Multiple tasks can access disk sequential file data sets on the same volume simultaneously because there are usually multiple paths to the data on the same DASD volume. On the other hand, tape is dedicated to a single task at a time. Tape data can stack multiple data sets on a volume like DASD. However, only 1 task can access 1 single data set on a tape volume at a time. Tape data is processed in a linear or sequential manner. The main relevant information for any data set is the name of the data set, the volume where the data resides, and in the case of tape data, where the data resides in the linear tape file sequence. Regardless of the storage device, the client program executes the job / task on the sequential input files and produces sequential output files. In addition, the z / OS operating system executes a sequential file prediction algorithm 110, which predicts the output file size and other consumable resources based on the input file size. The sequential file prediction algorithm 110 is described in more detail below. The sequential file prediction algorithm 110 is distinguished from a client program in that a client program is any program that can prevent failures or improve performance based on predictions made by the sequential file prediction algorithm 110. The sequential file prediction algorithm 110 can be encoded in the sequential file prediction program. It is contemplated that the sequential file prediction algorithm 110 can be expressed in a partially or fully recursive manner.

[0024] Figure 22 is a block diagram of a conventional example of a process for allocating a predefined output file size. In steps 202 and 204, the job and step begin execution to process sequential files. In step 206, the system allocates input file DD 210, which identifies the input file for processing. In step 208, the system also allocates output file DD 212, which has a predetermined size for storing processed information. In step 214, the client process the input file indicated by input file DD and store the results in the output file indicated by output file DD, and the step / job ends in step 216.

[0025] It would be beneficial to predict the output file size, rather than allocating an output file DD of a predetermined size, which may or may not be optimal for storage of the client program output. Figure 3 is a block diagram of an example of a process for predicting the size of an output file based on input file information in an input file DD. Figure 2 , in steps 302 and 304, the job and step begin execution to process sequential files. In step 306, the system allocates an input file DD 308 for processing. However, instead of allocating an output file DD of a predetermined size, step 310 analyzes information from the input file DD (DD name, DD size, etc.) and predicts the optimal size of one or more output files for storing the processed information, and allocates the output files DD accordingly in steps 320 and 322. In step 324, the client program processes the input file indicated by the input file DD and stores the results in the output file indicated by the output file DD, and the step / job ends in step 326. In addition, at step 312, the system sends the predicted resource usage to the operating system (e.g., z / OS), which allows the operating system to perform actions such as choosing to continue the job as is, or reschedule the job in step 314 to manage system-wide resources. If it is determined in step 316 that the system reschedules the job, the step / job ends in step 326. However, if it is determined in step 316 that the system has not rescheduled the job, processing continues in step 318.

[0026] Rather than executing a job at a scheduled time when the current load on the system may be high, it would be beneficial to predict the processing resources required to execute the job and reschedule the job for a later time when the system load is lower. If a job is initiated and predicted to use a large amount of working memory when the available working memory on the system is low, the operating system may choose to schedule the job at a later time when more working memory is available for the job.

[0027] The analysis of the input file DD information to predict the optimal size of one or more output files, as well as predicting the consumption of other scalable resources (e.g., DD name, CPU time, I / O time, memory allocation, etc.) can be performed by many different methods. These methods include linear regression techniques and neural network techniques, etc., which determine prediction functions based on historical file data indicating known relationships between past input file sizes and their corresponding resource requirements. These prediction techniques are described below. General Forecasting Solutions

[0028] The goal of the prediction solution is to analyze the historical data of job executions to create a prediction model that calculates the amount of one or more scalable resources required by a client program given the size of the input DD. Some examples of scalable resources include CPU time, real time, size of output DD, and amount of working memory used. The historical data of all jobs is saved in SMF records. From these SMF records, the system can extract the job name, process name, step name, client program name, list of input DDs and output DDs and their sizes, CPU time, run time, number of I / O operations, etc.

[0029] Job names, process names, step names, and client program names are used as identifiers to build the predictive model. A model may not be tied to a client program name alone, because it may perform different actions depending on the parameters, and these parameters may not be accessible from SMF records. However, the parameters provided to a client program are unlikely to change within a particular job, so the behavior of the client program is likely to remain consistent from one job execution to the next.

[0030] Figure 4 4 is a block diagram of an overall solution for predicting output file size and required processing resources based on input file information. Specifically, in step 402, the system retrieves a history file that shows known past input file sizes, past resource consumption, and their resulting past output file sizes for previous jobs that executed a client program. The idea is that the size of the input file and the resulting resource consumption have a correlation that can be determined and then used for future predictions. A client program can use multiple input and output files, which adds another level of complexity in learning the relationship (e.g., correlation) of input file size to output file size. As with any other data set, the more observations in the data set, the more accurate the conclusions (e.g., the more relationships analyzed, the more accurate the determination of the correlation).

[0031] More specifically, the system uses the historical data to train one of the prediction algorithms in steps 404-408, which may include a linear regression for all input / output file sizes, a linear regression for target resources (e.g., the size of a specific output DD or IO time), and a neural network that can predict all resources used in an algorithm. Once the linear function or neural network is trained, the system can then use the prediction algorithm in step 410 to accurately predict future resource consumption based on the future input file sizes to be processed by the client program. Figure 5-Figure 8 Developing / training linear equations and neural networks for resource consumption prediction is described in more detail in . Linear Regression Solution

[0032] Figure 5 is a block diagram of an example of a linear regression process for predicting resource consumption based on input file information and a combination of input and output DD statements presented for all possible DD combinations. In step 502, the system retrieves a history file showing the relationship between input file sizes and their resulting resource consumption for previous jobs executed by the client program. In step 504, the system calculates a linear function that best fits the input file size to the resource usage retrieved from the history file.

[0033] For example, a linear regression can be performed using the input file size as an explanatory variable and the output file size as a predictor variable. If the assumption holds for the program steps in the history file, a correlation between the input file size and the output file size can be established so that the output size of future files can be predicted. In one example, the client program can reduce / increase the output file size, so a variable of the percentage change between the input and output can also be calculated and recorded. In another example, the client program splits and merges records and has multiple inputs and outputs. This type of client program may require multiple passes through the linear regression described previously.

[0034] Figure 5 The result of step 504 in will be a linear function that predicts the output file size for all possible combinations of input files and output files. For example, if the client can process up to X input files and produce up to Y output files, a linear regression will be performed separately for each combination (e.g., X times Y linear regressions). Once the linear regressions are calculated, in step 506, the system can then select a linear function given a future input / output file combination. For example, if a future job utilization includes 2 input files and 4 output files, a linear function for a 2×4 scenario is selected and used to predict output file size and other resource consumption.

[0035] Rather than generating a linear function for all possible numbers of input files versus all possible numbers of output files, the system can perform a linear regression of all input file sizes for a particular consumable, scalable resource (eg, a particular DD name or working memory). Figure 6 6 is a block diagram of an example of a linear regression process for predicting output file size based on input file information of a specified file DD name. In step 602, the system retrieves a history file showing the relationship between the input file size and the resulting output file size of previous jobs executed by a client program. In step 604, the system determines the possible DD names to be processed by the client program. For example, client program A may be designed to output one of five files with corresponding DD names. Then, in step 606, the system calculates the linear function that best fits the input / output file combination of each DD name, respectively. For example, a linear function for the DD name 1 output file is calculated, while another linear function for the DD name 2 output file is calculated. In step 608, the system can then select a linear function for a given input / output file combination of the client program's DD name, input the input file size of the given DD name to be processed by the client program to the selected linear function, calculate the predicted output file size and the consumption of other resources as a result of processing the DD name file, and use the predicted output file size for memory allocation and processing time prediction and rescheduling of jobs / steps (if beneficial).

[0036] For linear regression, for N inputs DD, there may be more than N factors. For example, for each input DD, X, k transformations may be applied to it (e.g., X 2 , X 3 , log(X), sqrt(X), etc.), which are then fed into the linear equation. Therefore, the total number of inputs will be k*N, with k being the number of transformations applied to each input DD. The linear equation will then output the true size of an output DD.

[0037] In any case, if there is a linear relationship between the size of the input and the consumable resources, then a linear equation can be constructed that converts each input DD(s n ) multiplied by a certain weight (w n ) and add the values ​​together to get the number of desired resources: w1*s1+w2*s2+…+w n *s n +c=r

[0038] If there is a nonlinear relationship between the input size and the consumable resources, a set of transformation functions (f k) is applied to each input size to produce a set of inputs. Each input DD can also be passed through a transformation function f (such as squaring, cubing, or taking the logarithm) and provided as another input to open up the possibility of non-linear relationships. If there are k transformations applied to n inputs DD: w1f1(s1)+w2f2(s1)+…+w k f k (s1)+w k+1 f1(s2)+…+w nk-1 f k-1 (s n )+w nk f k (s n )=r The applied transformations can be summarized as follows:

[0039] The weights are specific to one resource, and other resources will use different sets of weights. Once a set of weights has been determined for each desired resource, the system can compute the predicted resource usage given a list of input sizes. Assuming there are n input DDs and m output DDs, the size of the output DDs can be predicted using the following operation:

[0040] You can also include another column of weights and add the value 1 to the input size vector to add the bias constant to the equation at the end of .

[0041] The weights for this equation can be calculated by analyzing the historical file data from the SMF logs. By looking at the size of the input and output DDs for each run of the client program, the best weights for the best predicted output DD size can be calculated.

[0042] Only one linear regression is needed for each output DD of each client program. Find the weights (w n ) will produce a function that can compute the output size given a list of input sizes. If there are n unique input DDs in k runs of a particular customer program, the system can construct this system of linear equations for each output DD: It can be written as: k represents the index of the historical execution of the client program and is used to avoid confusion with m used above, which represents the index of the unique output DD. The system can be solved using the following equation S+ =(S T *S) -1 *S T

[0043] This will result in and The minimum square difference between them. Although calculating S + will be computationally expensive, but only needs to be computed once per client. Then, for each output DD of that client, we can multiply the vector of all its history sizes by S + to obtain the weight vector. Once all vector, we can multiply the row vector by the vector of known input size to predict the size of all output DDs.

[0044] Solution Another approach is to use gradient descent.

[0045] Gradient descent is an iterative process where the partial derivative of the cost function C is taken with respect to each weight in order to create a gradient. This gradient is then used to adjust all weights in order to step towards the minimum in the cost function:

[0046] Among them, α is the "learning rate", represents the weight vector at a specific iteration of training, w t,n represents the value in the weight vector, and C i Denotes the cost function associated with a particular training observation among the k observations.

[0047] The gradient vector (like the derivative) describes the rate of change. This vector describes the direction and rate of fastest increase. By subtracting the gradient from the weights, the weight vector is moved in the direction of fastest decrease in order to move towards the minimum. This is done iteratively, taking small steps down the gradient and re-evaluating at the next point in order to prevent overshooting the minimum. Neural Network Solutions

[0048] Figure 7is a block diagram of an example of another solution for using neural network processing to predict output file size based on input file information. In step 702, the system retrieves a history file that shows the relationship between the input file size and the resulting output file size of previous jobs executed by the client program. In step 704, the system determines the DD name and output file of the user program. For example, client program A can output five files with corresponding DD names. Then, in step 706, the system trains the neural network by the following steps: 1) feeding the input file size value for each combination of dd names into the input node of the neural network, 2) searching for nonlinear and conditional relationships between input size and output size in the hidden layer, 3) outputting the predicted output file size for a given input combination of DD, 4) comparing the predicted output file size with the retrieved historical output file size to determine the difference, 5) adjusting the neural network weights based on the difference, and 6) repeating these steps until the neural network is trained. Then in step 708, the system can use the trained neural network to: 1) feed the input file size values ​​for a given combination of DD names into the input nodes of the trained neural network, 2) output the predicted output file size for the given input combination of DDs, and 3) use the predicted output file size for memory allocation and processing time prediction and rescheduling of jobs / steps (if beneficial).

[0049] A single neural network can predict the size of all output DDs at once. The neural network can be constructed as one input layer (a set) of input nodes, one output layer of output nodes, and multiple layers of nodes between the inputs and outputs. In each layer, each node receives inputs from all nodes in the previous layer and calculates a set of linear equations on these inputs. The resulting values ​​are then passed to each node in the next layer to make it calculate its own set of linear equations. Very complex relationships can be modeled, and external variables such as parameters or flags passed to the client program can also be encoded as inputs to the network, which it can then interpret.

[0050] The input to the algorithm can take many forms. For example: a) each input node can receive the actual size of an input DD, b) the size of an input DD can be split across multiple input nodes, c) each input node can be a binary number of the size of the input DD, each input node can be a binary number related to the order of magnitude of the size, and each input node can be the normalized size of the input DD (scaled between 0 and 1). The input and output data are then manipulated to convert between the actual size and the data used by the algorithm during processing.

[0051] Gradient descent can also be used to train algorithms, as mentioned in the previous proposal, to find the correct weights and biases for each node. By feeding the algorithm historical data, a cost function can be evaluated for each data point to grade how close the algorithm is to the desired outcome. By evaluating the cost function over many historical data points, a gradient can be calculated to determine which weights and biases need to be changed in the network to minimize the cost function. Since the gradient can change over the vector space of all weights and biases, small steps are taken down the gradient, and the process is repeated many times.

[0052] There are several advantages to using neural networks. They can model more complex relationships between inputs and outputs, one model can solve for all input and output DDs for a particular client program, and parameters that affect the output size can be encoded as inputs to the network, which can then account for it. Some parameters can include input file size, logical record length and block size of the input file, record format, day of week / year the job executes, parameters passed to the client program (encoded as on-off flags or enumerated values), name of the DD (which can be encoded as an index), duration of job execution.

[0053] Some options for neural networks include: a) a network without any type of data normalization. In this case, each input and output node will be equal to the unbounded size of the corresponding input / output DD. This is essentially a multi-level linear equation solver, where each layer of the network is a set of linear equations exactly like the first example, b) the size of each input and output DD will be encoded in binary, where each input / output node of the network corresponds to a specific bit of the size of a specific input or output DD, c) normalizing the input data using logarithms or z-score normalization of the values, and possibly passing the resulting values ​​through a normalization function such as sigmoid or tanh. Using the logarithm will greatly reduce the range of input sizes while maintaining meaningful differences between extremely large or small data sets. Using a typical normalization function such as sigmoid without the logarithm will result in negligible differences between values ​​at the extremes, even if they vary greatly, d) each input and output node corresponds to a specific order of magnitude. This is a generalization between the two approaches above. This has the advantage that the system can create buckets to put each DD into based on their size, and the system can encode this in 10 bits or less, rather than requiring 40 bits (nodes) just to describe a single 1TB DD. Using this approach can provide better differentiation between the sizes of the inputs, resulting in a more accurate model.

[0054] Once the data is normalized, the operation of the network is essentially the same. A neural network is conceptually divided into multiple layers, where each layer consists of a set of nodes. There is an input layer, an output layer, and several intermediate or "hidden" layers. Starting from the first intermediate layer, each node in the layer receives input from each node in the previous layer. Each node has a set of weights It multiplies the value And add bias b:

[0055] If there are m nodes in the previous layer, and n nodes in the current layer, the following operations can be used to process the layers:

[0056] The top equation is a function for each node in the network, while the bottom equation is the equation for one layer of the computational network. σ is an activation function that helps limit the range of the output. The subscripts x0 and x1 are used to distinguish the output of the previous layer from the output of the current layer. The output of each layer is provided as the input to the next layer, possibly through some kind of activation function, which is used to limit the output range of the node. The activation function helps prevent values ​​from becoming too large and becoming meaningless to the network or overflowing the limits of the data type size.

[0057] Stochastic gradient descent is used to train a neural network by gradually approximating the optimal weights and biases to model the relationship between input and output. By combining all weights and biases from all layers into a single vector, the gradient of the cost function can be calculated with respect to all weights and biases. Note that the partial derivatives of the cost function will differ from layer to layer because downstream layers will change the effect that a particular weight or bias has on the cost function.

[0058] Once the linear function is calculated, the system selects a linear function for a given input / output file combination of the client program's DD name, inputs the input file size of the given DD name to be processed by the client program to the selected linear function, calculates a predicted output file size as a result of processing the DD name file, and uses the predicted output file size for memory allocation and processing time prediction. Use Cases

[0059] Figure 8is a block diagram of an example of a process for predicting output file size based on input file information in a specific use case of a credit card company. In step 802, the system retrieves a history file of previous input and output file size combinations that were previously processed by a client program (e.g., an accounting program) of the credit card company. In step 804, the system determines the target DD name of the output file (e.g., an "accounting" file) of the credit card company program used by the client program (e.g., a program that extracts and reports certain transaction details to the credit card company's accounting department). In step 806, the system calculates a linear function that best fits the input / output file combination of the "accounting" DD name (e.g., a linear function of the "accounting" file), or trains a neural network to predict the output accounting file size based on the input accounting file size. Then, in step 808, the system inputs the input file size to be processed by the client program to the selected linear function or trained neural network, calculates the predicted output file size of the accounting DD name after processing, and uses the predicted output file size for memory allocation and processing time prediction of jobs and rescheduling of jobs / steps (if beneficial). In one example, the system can allocate memory for storing output files based on the predicted output file size. In another example, the system can predict the processing time of the input file to generate the output file. As described above, the processing time can be used to modify / schedule the job, etc. System Hardware

[0060] Fig. 9 and Fig.10 A functional block diagram illustrating a computer hardware platform is provided. Fig. 9 A network or host platform is shown that may typically be used to implement a server and / or mainframe, including but not limited to a tape system 902, a DASD 904, user input / output 906, communication ports 908, and a system processor 910 including a CPU, memory, operating system, applications, batch jobs, SMF, etc. Fig.10 A computer with user interface elements is depicted which may be used to implement a personal computer or other type of workstation or terminal device, but which, if properly programmed, Fig.10 The computer can also act as a server. In this example, Fig.10 The computer in FIG. 1 includes a monitor 1000, a keyboard 1002, a mouse 1004, a communication port 1006, a database 1008, a user I / O 1010, a RAM 1012, a ROM 1014, a CPU 1016, etc. It is believed that Fig. 9 and Fig.10 The structure and general operation of such a device shown in should be self-explanatory from the high-level illustration.

[0061] Note that the host includes, for example, a data communication interface and an input / output (I / O) controller for packet data communication. The I / O controller manages communication with various I / O elements and storage facilities. The storage facilities include one or more direct access storage devices (DASD) and / or one or more tape systems. Such storage facilities provide storage for data, jobs and applications for managing batch processing. The host includes an internal communication bus that provides a communication channel between a communication port, an I / O controller and one or more system processors. Each system processor includes one or more central processing units (CPUs) and a local memory corresponding to each CPU, as well as a shared memory that can be used for any CPU. The operating system (OS) executed by the system processor manages the various jobs and applications currently running to perform appropriate processing. The OS also provides a system management facility (SMF) and open exit points for managing the operation of the mainframe and the various jobs and applications currently running. The hardware elements, operating systems, jobs and applications of such a mainframe can be conventional in nature. Of course, the mainframe functions can be implemented in a distributed manner on multiple similar platforms to distribute the processing load, and / or replicated across one or more similar platforms to provide redundancy for processing. Therefore, Fig. 9 A replication environment is also depicted. Although the details of the replication environment are not depicted, such a replication environment generally includes Fig. 9 Similar components are described for major mainframes.

[0062] It should also be noted that a computer-type user terminal device (such as a PC) similarly includes a data communication interface CPU, a main memory, and one or more mass storage devices for storing user data and various executable programs. Various types of user terminal devices will also include various user input and output elements. For example, a computer may include a keyboard and a cursor control / selection device, such as a mouse, trackball, or touchpad; and a display for visual output. The hardware elements, operating system, and programming language of such a user terminal device may be conventional in nature.

[0063] Thus, aspects of the method outlined above for recording the time at which each step within a job executed within a mainframe computing environment begins and ends may be embodied in programming. The programmatic aspects of the technology may be considered a "product" or "article of manufacture," typically in the form of executable code and / or associated data carried or embodied on a type of machine-readable medium. "Storage" type media include any or all tangible memory of a computer, processor, etc., or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which may provide non-transitory storage for software programming at any time. All or part of the software may sometimes be made available over a global information network (e.g., ) or various other telecommunication networks. For example, such communications can enable software to be loaded from one computer or processor into another, for example, from a management server or mainframe into a mainframe platform that will perform various jobs. Thus, another type of medium that can carry software elements includes optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline networks, and through various air links. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0064] For example, non-volatile storage media include optical or magnetic disks, such as any storage device in any computer, etc., such as data sets that can be used to maintain sequential files and customer / prediction programs for enterprise applications. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card paper tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, or any other media from which a computer can read programming code and / or data. Many of these forms of computer-readable media may involve transmitting one or more sequences of one or more instructions to a processor for execution.

[0065] It should be understood that the terms and expressions used in this article have the common meaning consistent with these terms and expressions about each investigation and research field corresponding to it, unless a specific meaning has been set forth in addition in this article. Relational terms such as the first and second can be used only to distinguish an entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The term "comprises", "comprising", "includes", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including the list of elements not only include those elements, but also can include other elements that are not clearly listed or inherent to such process, method, article or equipment. In the absence of further constraints, the element behind "a" or "an" does not exclude the existence of other identical elements in the process, method, article or equipment including the element.

[0066] Unless otherwise indicated, any and all measurements, values, ratings, positions, magnitudes, sizes, and other specifications set forth in this specification (including in the appended claims) are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they are related and the customary practices in the art to which they belong.

[0067] Although the foregoing has described what is considered to be the best mode and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied to many applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A system for predicting resources required by a program, the system comprising: processor; a storage device accessible by the processor; as well as a sequential file prediction program, which, when executed by the processor, configures the system to: accessing a history file to determine the size of past sequential input files input to a client program and the size of resultant past sequential output files produced by said client program processing said sequential input files, determining a correlation between the size of the past sequential input file and the resulting size of the past sequential output file, utilizing the correlation to predict future consumption of scalable resources including future sizes of future sequential output files based on current sizes of current sequential input files, and utilizing the predicted future consumption of the scalable resource to perform at least one of memory allocation or determining a schedule of a batch job being executed by the system, The scalable resources include at least one of processing time, working memory, or input / output time.

2. The system according to claim 1, wherein: The processor is also configured to determine the correlation by performing a linear regression on the sizes of the past sequential input files and the resulting consumption of the scalable resource, the linear regression producing a linear function for performing a prediction of future consumption of the scalable resource based on the current size of the current sequential input file.

3. The system according to claim 2, wherein: The processor is further configured to perform the linear regression as N×M linear regressions for N×M combinations of N input files by M output files, where N and M are integer values ​​ranging between 1 and a maximum number of input files and output files supported by the program.

4. The system according to claim 2, wherein: The processor is further configured to perform K×N linear regressions for the N input files and to perform K transformations for each scalable resource consumed, where N is an integer value ranging from 1 to an integer value indicating the number of data definitions (DD) per input file.

5. The system according to claim 1, wherein: The processor is further configured to determine the correlation by training a neural network by inputting the sizes of the past sequential input files, processing the sizes of the past sequential input files based on set weights, predicting resource consumption of the scalable resource, calculating a difference between the predicted resource consumption and known resulting consumption of the scalable resource in the past, and adjusting the set weights to attempt to minimize the difference.

6. The system according to claim 5, wherein: The processor is further configured to train the neural network based on at least one of an input file size and a logical record length and block size of the input file, a record format, a timing of job execution, parameters passed from the input file to the program, a data definition (DD) name of the input file, and a duration of job execution.

7. The system according to claim 1, wherein: The processor is further configured to extract the history file from a system management facility (SMF) log indicating job-related information.

8. The system according to claim 1, wherein: The input file and the output file contain the customer's transaction data.

9. The system according to claim 1, wherein: The client program used during processing of past input files of historical data is the same as the client program used during processing of the future input files.

10. The system according to claim 1, wherein: The processor is further configured to use at least one of the predicted future size of the future sequential output file, CPU time, and memory consumption to determine at least one of a memory allocation for storing the output file and a processing time for generating the output file.

11. A method for predicting resources required by a program, the method comprising: accessing, by a processor, a history file to determine the size of past sequential input files input to a client program and the size of resultant past sequential output files produced by said client program processing said sequential input files; determining, by the processor, a correlation between the size of the past sequential input file and the resulting size of the past sequential output file; utilizing, by the processor, the correlation to predict future consumption of scalable resources including future sizes of future sequential output files based on current sizes of current sequential input files; as well as utilizing, by the processor, the predicted future consumption of the scalable resource to perform at least one of memory allocation or determining a schedule of a batch job being executed by the system, The scalable resources include at least one of processing time, working memory, or input / output time.

12. The method according to claim 11, further comprising: The correlation is determined by the processor by performing a linear regression on the sizes of the past sequential input files and the resulting consumption of scalable resources, the linear regression producing a linear function for performing a prediction of future consumption of the scalable resource based on the current size of the current sequential input file.

13. The method according to claim 12, further comprising: The linear regression is performed by the processor as N×M linear regressions for N×M combinations of N input files by M output files, where N and M are integer values ​​ranging between 1 and a maximum number of input files and output files supported by the program.

14. The method according to claim 12, further comprising: K x N linear regressions are performed by the processor for the N input files and K transformations are performed for each scalable resource consumed, where N is an integer value ranging from 1 to indicate the number of data definitions (DD) per input file.

15. The method according to claim 11, further comprising: The correlation is determined by a processor through the following steps: training a neural network by inputting the sizes of the past sequential input files, processing the sizes of the past sequential input files based on set weights, predicting resource consumption of the scalable resource, calculating the difference between the predicted resource consumption and the known resulting consumption of the scalable resource in the past, and adjusting the set weights to attempt to minimize the difference.

16. The method according to claim 15, further comprising: The neural network is trained by the processor based on at least one of an input file size and a logical record length and block size of the input file, a record format, a timing of job execution, parameters passed from the input file to the program, a data definition (DD) name of the input file, and a duration of job execution.

17. The method according to claim 11, further comprising: The history file is extracted by the processor from a system management facility (SMF) log indicating job-related information.

18. The method according to claim 11, wherein: The input file and the output file contain the customer's transaction data.

19. The method according to claim 1, wherein: The client program used during processing of past input files of historical data is the same as the client program used during processing of the future input files.

20. The method according to claim 11, further comprising: At least one of a predicted future size of the future sequential output file, CPU time, and memory consumption is used by the processor to determine at least one of a memory allocation for storing the output file and a processing time for generating the output file.

Images