Batch operation deduction method, related device and computer storage medium
By collecting data and performing time series forecasting on the batch processing nodes of the bank's core system, a predicted time series is generated. The batch scheduling is then optimized using a deductive model, which solves the problem of high complexity in end-of-day batch processing and improves processing efficiency and controllability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AGRICULTURAL BANK OF CHINA
- Filing Date
- 2023-03-24
- Publication Date
- 2026-05-15
AI Technical Summary
The current bank core system's end-of-day batch processing is highly complex, prone to errors, and has uncontrollable time consumption, resulting in low efficiency.
By collecting basic data from a batch of running nodes, a raw time series is generated. Then, using time series forecasting methods and inference models, the execution results and resource consumption at future time points are predicted, thereby optimizing the batch scheduling order.
It improves the efficiency of batch scheduling, reduces the error rate and time consumption of end-of-day batch processing, and enhances the controllability of processing.
Smart Images
Figure CN116307668B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a batch operation simulation method, related apparatus and computer storage medium. Background Technology
[0002] In the core operations of mainstream banks today, batch processing is an important component, generally divided into end-of-day batch processing and intraday batch processing. Intraday batch processing usually refers to document-based payouts and deductions, while the focus and difficulty of batch processing lies mainly in end-of-day batch processing. End-of-day batch processing has a very complex logic, with its main end-of-day steps being posting, accrual, and interest settlement, as well as many other steps such as account opening processing, loss reporting processing, voucher processing, agreed-upon accounting processing, report generation, data organization and provision, data cleaning, and so on.
[0003] End-of-day batch processing flowcharts typically contain dozens or even hundreds of steps. Since online transactions are relatively fewer at night, and system resources are generally more abundant, logic that is inconvenient to process during the day or requires date changes is usually moved to the end-of-day stage. In centralized systems, batch processing is much faster than single-transaction processing, and some logic requiring data processing is also placed at the end of the day, such as business status reports and bank statements. Due to factors such as large volume, product complexity, and numerous end-of-day steps, the complexity of core end-of-day processes in mainstream banks is already very high, leading to frequent errors and uncontrollable processing time. Summary of the Invention
[0004] In view of this, this application provides a batch operation simulation method, related apparatus and computer storage medium. By simulating the operation after batching, the batch operation simulation results can effectively improve the arrangement efficiency of batch scheduling sequence.
[0005] The first aspect of this application provides a batch running simulation method, including:
[0006] Periodically collect basic data from all running nodes in batches; wherein, the basic data includes at least system data, application data, and business data;
[0007] For each running node, an original time series of the running node is generated based on the basic data of the running node; wherein, the original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features and higher-order features;
[0008] Based on the original time series of the running node, a preset time series prediction method is used to generate a predicted time series of the running node; wherein, the predicted time series includes at least one future time point and the predicted time dimension feature of the future time point;
[0009] The predicted time series of the running node is input into the inference model, and the running inference result of the running node is output. The inference model is obtained by training a mathematical model with at least one training sample. The training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes. The actual running results include at least the actual execution result, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes. The running inference result of the predicted time series of the running node includes at least the inferred execution result, inferred runtime, and inferred resource consumption at each future time point in the predicted time series of the running node.
[0010] The batch of operational simulation results is generated based on the operational simulation results of the predicted time series of all the operational nodes.
[0011] Optionally, generating the original time series of each running node based on its basic data includes:
[0012] For each running node, data at the same time point are identified and merged from the basic data of the running node to obtain the time dimension features of the time point;
[0013] The original time series of the running node is generated based on all the time points and the time dimension characteristics of the time points.
[0014] Optionally, the time intervals between the time points are the same.
[0015] Optionally, after generating the predicted time series based on the original time series using a preset time series prediction method, the method further includes:
[0016] When the real time reaches a future time point in the predicted time series, the real time dimension features of the time point under the real time are obtained.
[0017] The preset time series prediction method is adjusted by utilizing the error between the real time dimension features of the real time point and the predicted time dimension features of the future time point.
[0018] Optionally, the method for constructing the inference model includes:
[0019] Construct a training sample set; wherein the training sample set includes at least one training sample; the training sample includes the time series of the running nodes of the training sample and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at time points in the time series of the running nodes of the training sample.
[0020] The time series of the training sample running nodes is input into the mathematical model, and the running simulation results corresponding to the time series of the training sample running nodes are output; wherein, the running simulation results corresponding to the time series of the training sample running nodes include at least the simulation execution results, simulation runtime, and simulation resource consumption at time points in the time series of the training sample running nodes;
[0021] Based on the error between the operational deduction results corresponding to the time series of the training sample running nodes and the actual operational results corresponding to the time series of the training sample nodes, the mathematical model is adjusted until the error between the operational deduction results corresponding to the time series of the training sample running nodes and the actual operational results corresponding to the time series of the training sample nodes satisfies a preset convergence condition. The mathematical model that satisfies the preset convergence condition is then used as the deduction model.
[0022] A second aspect of this application provides a batch operation simulation device, comprising:
[0023] The data acquisition unit is used to periodically collect basic data from all running nodes in batches; wherein, the basic data includes at least system data, application data, and business data.
[0024] The first generation unit is configured to generate an original time series for each running node based on the basic data of the running node; wherein the original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features;
[0025] The second generation unit is used to generate a predicted time series of the running node based on the original time series of the running node using a preset time series prediction method; wherein, the predicted time series includes at least one future time point and the predicted time dimension feature of the future time point.
[0026] The first input unit is used to input the predicted time series of the running node into the inference model and output the running inference result of the running node; wherein, the inference model is obtained by training a mathematical model with at least one training sample; the training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution result, actual runtime, and actual resource consumption at time points in the time series of the training sample running nodes; the running inference result of the predicted time series of the running node includes at least the inferred execution result, inferred runtime, and inferred resource consumption at future time points in the predicted time series of the running node;
[0027] The simulation result generation unit is used to generate the batch of simulation results based on the simulation results of the predicted time series of all the running nodes.
[0028] Optionally, the first generation unit includes:
[0029] The merging unit is used to identify and merge data at the same time point in the basic data of each running node to obtain the time dimension feature of the time point.
[0030] The first generation subunit is used to generate the original time series of the running node based on all the time points and the time dimension characteristics of the time points.
[0031] Optionally, the time intervals between the time points are the same.
[0032] Optionally, the batch operation simulation device further includes:
[0033] The acquisition unit is used to acquire the real time dimension features of the time point when the real time reaches the future time point in the predicted time series.
[0034] The first adjustment unit is used to adjust the preset time series prediction method by utilizing the error between the real time dimension features of the real time point and the predicted time dimension features of the future time point.
[0035] Optionally, the building blocks of the inference model include:
[0036] A training sample set construction unit is used to construct a training sample set; wherein, the training sample set includes at least one training sample; the training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at time points in the time series of the training sample running nodes.
[0037] The second input unit is used to input the time series of the training sample running nodes into the mathematical model and output the running deduction result corresponding to the time series of the training sample running nodes; wherein, the running deduction result corresponding to the time series of the training sample running nodes includes at least the deduction execution result, deduction runtime and deduction resource consumption at time points in the time series of the training sample running nodes;
[0038] The second adjustment unit is used to adjust the mathematical model based on the error between the running deduction result corresponding to the time series of the training sample running node and the actual running result corresponding to the time series of the training sample node, until the error between the running deduction result corresponding to the time series of the training sample running node and the actual running result corresponding to the time series of the training sample node satisfies a preset convergence condition, and the mathematical model that satisfies the preset convergence condition is used as the deduction model.
[0039] A third aspect of this application provides an electronic device, comprising:
[0040] One or more processors;
[0041] A storage device on which one or more programs are stored;
[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the batch execution simulation method as described in any one of the first aspects.
[0043] The fourth aspect of this application provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the batch execution simulation method as described in any one of the first aspects.
[0044] As can be seen from the above solutions, this application provides a batch operation simulation method, related apparatus, and computer storage medium. The batch operation simulation method includes: periodically collecting basic data of all running nodes in batches; wherein the basic data includes at least system data, application data, and business data; then, for each running node, generating an original time series of the running node based on the basic data of the running node; wherein the original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features; and then, based on the original time series of the running nodes, generating a predicted time series of the running nodes using a preset time series prediction method; wherein the predicted time series includes at least one future time point and the predicted time dimension of the future time point. The process involves several steps: First, the predicted time series of the running nodes is input into the simulation model, and the simulation results of the running nodes are output. The simulation model is trained using at least one training sample. The training sample includes the time series of the training sample running nodes and the actual running results corresponding to those time series. The actual running results include at least the actual execution results, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes. The simulation results of the predicted time series of the running nodes include at least the simulation execution results, simulation runtime, and simulation resource consumption at future time points in the predicted time series of the running nodes. Finally, based on the simulation results of the predicted time series of all the running nodes, the batch simulation results are generated. By simulating the running conditions after the batch, the batch simulation results can effectively improve the orchestration efficiency of the batch scheduling order. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0046] Figure 1 A detailed flowchart of a batch operation simulation method provided in this application embodiment;
[0047] Figure 2 A flowchart illustrating a batch execution simulation method provided in another embodiment of this application;
[0048] Figure 3 A flowchart illustrating a batch execution simulation method provided in another embodiment of this application;
[0049] Figure 4 A flowchart illustrating a method for constructing a deductive model, as provided in another embodiment of this application;
[0050] Figure 5 A schematic diagram of a batch operation simulation device provided for another embodiment of this application;
[0051] Figure 6 This is a schematic diagram of an electronic device for implementing a batch operation simulation method, as provided in another embodiment of this application. Detailed Implementation
[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0053] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0054] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0055] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0056] First, the technical terms used in this application will be explained:
[0057] Batch processing refers to a processing model where a computer program executes a series of tasks based on a batch of input data without human intervention. Simply put, batch processing is a mode where the input data typically contains multiple records, and the processing generally involves no human interaction. Another mainstream processing mode, online processing, differs significantly from batch processing in that online processing typically generates a processing cycle for each input record, directly returning the result to the caller. Batch processing has significant advantages in scenarios where real-time performance and interactivity requirements are not high, but the volume of data to be processed is relatively large. Common examples include bank reconciliation, batch payroll processing in online banking, and batch log backups in log systems.
[0058] Time series forecasting methods: Time series forecasting is a type of regression forecasting method, belonging to quantitative forecasting. Its basic principle is: on the one hand, it acknowledges the continuity of the development of things and uses past time series data for statistical analysis to infer the development trend of things; on the other hand, it fully considers the randomness caused by accidental factors. In order to eliminate the impact of random fluctuations, it uses historical data for statistical analysis and performs appropriate data processing to make trend predictions.
[0059] Time series forecasting methods can be used for short-term, medium-term, and long-term forecasting. Depending on the data analysis methods used, they can be further categorized as: simple time series average method, weighted time series average method, moving average method, weighted moving average method, trend forecasting method, exponential smoothing method, seasonal trend forecasting method, market life cycle forecasting method, etc.
[0060] The simple sequential average method, also known as the arithmetic mean method, uses statistical values from several historical periods as observations and calculates the arithmetic mean as the predicted value for the next period. This method is based on the assumption that "this has been the case in the past, and it will be the case in the future," equating and averaging recent and long-term data. Therefore, it is only suitable for predicting trends where changes are not significant. If things show an upward or downward trend, this method is not suitable.
[0061] The weighted chronological average method involves weighting historical data from different periods according to their relative impact on the near and long term, calculating the average value, and using this average as the forecast value for the next period.
[0062] The simple moving average method calculates the arithmetic mean of several periods and uses it as the forecast value for the next period.
[0063] The weighted moving average method is a simple moving average calculated with added weights. When determining the weights, recent observations should be given a larger weight, and long-term observations should be given a smaller weight.
[0064] While the methods described above are simple and can quickly produce forecasts, their accuracy is relatively poor because they do not consider new trends in overall socio-economic development and the influence of other factors. Therefore, the forecast results should be revised as necessary based on new circumstances.
[0065] Exponential smoothing is a method of forecasting based on historical data, using an exponentially weighted approach to calculate the next period's forecast. Essentially derived from the internal weighted moving average method, its advantage lies in its ability to calculate the next period's forecast using only the previous period's actual and predicted values, thus saving significant data and processing time, reducing data storage requirements, and offering a simple and convenient method. It is a widely used short-term forecasting method internationally.
[0066] Seasonal trend forecasting methods predict the seasonal trends of economic phenomena based on the recurring seasonal variations in indices. Different methods can be used to calculate seasonal indices, with the two most common being the quarterly (monthly) averaging method and the moving average method: a. Quarterly (monthly) averaging method: This involves averaging the annual values by quarter (or month) and dividing by the overall quarterly (or monthly) average to obtain the quarterly (monthly) index. This method can be used to analyze the seasonal variations in economic phenomena such as production, sales, raw material reserves, and projected working capital needs; b. Moving average method: This involves using moving averages to calculate proportions to determine typical seasonal indices.
[0067] Market life cycle forecasting involves analyzing and studying the market life cycle of a product. For example, to predict the sales volume of a product in its growth stage, the most common method is to plot a time series curve based on statistical data, and then extrapolate the curve to obtain future sales trends. The simplest extrapolation method is linear extrapolation, which is suitable for forecasting durable consumer goods.
[0068] This application provides a batch execution simulation method, such as... Figure 1 As shown, the specific steps include:
[0069] S101. Periodically collect basic data from all running nodes in batches.
[0070] The basic data includes at least system data, application data, and business data. System data includes, but is not limited to, CPU, memory, network, and I / O data; application data includes, but is not limited to, application logs, port availability, process availability, and process resource consumption; business data includes, but is not limited to, traffic, error rate, and latency, etc.
[0071] In the specific implementation of this application, the regular timing can be preset and modified by technical personnel or relevant authorized staff, and is not limited here. The data collection devices used can be, but are not limited to, Hadoop's Chukwa, Apache Flume, Facebook's Scribe, LinkedIn's Kafka, etc., and are also not limited here.
[0072] Taking Flume as an example, Flume is a highly available, highly reliable, distributed system for collecting, aggregating, and transmitting massive amounts of information. As an information collection tool, it is very lightweight; based on multiple Flume Agents, it can build a very complex and powerful information collection system. This application deploys Flume Agents to all running nodes in a batch, and aggregates and organizes them through a Flume host cluster to form an information collection device.
[0073] S102. For each running node, generate the original time series of the running node based on the basic data of the running node.
[0074] The original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features.
[0075] It should be noted that in the practical application of this application, determining the size of the original time series of the running nodes is particularly crucial; that is, how much information from the past should we refer to as input when predicting the future. Too little information may be insufficient, while too much information will introduce irrelevant information from earlier periods.
[0076] Therefore, this application provides for the generation of multiple feature types:
[0077] (1) Basic features are the categorical features that distinguish different sequences, such as different batch numbers, the business system to which they belong, etc. By adding these categorical features, different time series data can be put into a large table for unified training. Theoretically, the model can automatically learn the similarity between these categories, thus improving its generalization ability.
[0078] (2) Dynamic features, including but not limited to Lag features and date features, change over time. These can be further divided into two categories. One category consists of information that cannot be obtained in advance during prediction, such as the predicted value itself and unknowable information related to the predicted value, such as future memory usage and CPU usage time. For this type of information, feature construction can only be performed strictly within the historical window, generally focusing on Lag features. The other category consists of information that can be obtained in advance, such as a clear pricing plan, allowing prediction of the number of batch nodes planned to run at T+1. For this type of feature, information at the corresponding time point can be directly added, just like static features.
[0079] To better understand Lag features, we can assume the predicted horizon length is only 1 day, while the historical feature window length is 7 days. Therefore, the most basic feature we can construct is the historical value of each day over the past 7 days to predict the value for the 8th day. For the time itself, various date-derived features can also be created. For example, when making predictions at the day level, we can add features such as the day of the week, the day of the month, the month itself, and whether it's a weekday, etc.
[0080] (3) Higher-order features: Based on the information mentioned in (1) and (2), various complex derivative features can be further developed, namely higher-order features. For example, based on the Lag feature, various statistical features within a window can be developed, such as the average, maximum, minimum, and standard deviation of the past n time points.
[0081] Optionally, in another embodiment of this application, one implementation of step S102 is as follows: Figure 2 As shown, it includes:
[0082] S201. For each running node, identify and merge the data at the same time point in the basic data of the running node to obtain the time dimension features of the time point.
[0083] It should be noted that, in the specific implementation process of this application, the time intervals between time nodes can be the same or different. The time intervals can be preset and modified by technical personnel or relevant authorized personnel, etc., and are not limited here.
[0084] S202. Generate the original time series of the running nodes based on all time points and the time dimension characteristics of the time points.
[0085] S103. Based on the original time series of the running nodes, generate the predicted time series of the running nodes using a preset time series prediction method.
[0086] The predicted time series includes at least one future time point and the predicted time dimension features of the future time point.
[0087] It should be noted that the preset time series forecasting method used in this application may be, but is not limited to, the simple time series average method, the weighted time series average method, the moving average method, the weighted moving average method, the trend forecasting method, the exponential smoothing method, the seasonal trend forecasting method, the market life cycle forecasting method, etc., and is not limited here.
[0088] In the specific implementation process of this application, the predicted time dimension features of the next time point (i.e. the first future time point) of the current time point are generated by a preset time series prediction method, and then the predicted time dimension features of the next time point (i.e. the second future time point) are generated by the preset time series prediction method. Finally, a continuous predicted time series including at least one future time point and the predicted time dimension features of the future time point can be obtained.
[0089] In the specific implementation of this application, after generating the predicted time series based on the original time series using a preset time series prediction method, one implementation method involves batch running the inference method, such as... Figure 3 As shown, it may also include:
[0090] S301. When the real time reaches a future time point in the predicted time series, obtain the real time dimension features of the time point under the real time.
[0091] S302. Adjust the preset time series prediction method by utilizing the error between the real time dimension characteristics of a time point in real time and the predicted time dimension characteristics of a future time point.
[0092] In practical applications, the preset time series prediction method is continuously adjusted based on the real time dimension characteristics of newly obtained time nodes and the predicted time dimension characteristics of predicted future time nodes, so that the subsequently generated predicted time series become more and more accurate.
[0093] S104. Input the predicted time series of the running node into the inference model, and output the running inference results of the running node.
[0094] The inference model is obtained by training the mathematical model with at least one training sample; the training sample includes the time series of the running nodes of the training sample and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at the time points in the time series of the running nodes of the training sample; the inference results of the predicted time series of the running nodes include at least the inferred execution results, inferred runtime, and inferred resource consumption at future time points in the predicted time series of the running nodes.
[0095] In the process of establishing mathematical models, they can be classified into dynamic models and static models based on whether they are time-varying. Based on whether the model parameters change continuously in space, they can be classified into distributed parameter models and lumped parameter models. Based on whether the model parameters change continuously with time, they can be classified into continuous system models and discrete system models. Based on the mathematical description of the model, they can also be classified into ordinary differential equations, partial differential equations, difference equations, discrete event models, etc., without further limitation here.
[0096] After the mathematical model is established, verifying its credibility is an essential step. The methods for verifying model credibility typically involve: first, experts analyzing and evaluating the model; then, statistical analysis of the data used in the modeling; and finally, a trial run of the model, comparing the preliminary simulation results with the estimated results.
[0097] Simulation computation is the process of numerically experimenting with and solving an established model. Different models require different solution methods. For example, continuous systems are typically described using ordinary differential equations, transfer functions, or even partial differential equations. Since obtaining analytical solutions to these equations is nearly impossible, numerical methods are always employed. For instance, various numerical integration methods are mainly used for ordinary differential equations, while finite difference methods, characteristic methods, Monte Carlo methods, or finite element methods are used for partial differential equations. On the other hand, discrete event systems are usually modeled using probabilistic models. The simulation process is essentially a numerical experiment, and the parameters must conform to certain probability distributions. Different simulation methods exist for different types of discrete event systems (such as stochastic service systems, stochastic inventory systems, and stochastic network planning).
[0098] Optionally, in the specific implementation of this application, one implementation method of the deduction model construction method is as follows: Figure 4 As shown, it includes:
[0099] S401. Construct a training sample set.
[0100] The training sample set includes at least one training sample; the training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes.
[0101] S402. Input the time series of the training sample running nodes into the mathematical model, and output the running deduction results corresponding to the time series of the training sample running nodes.
[0102] The runtime simulation results corresponding to the time series of the training sample running nodes include at least the simulation execution results, simulation runtime, and simulation resource consumption at each time point in the time series of the training sample running nodes.
[0103] S403. Determine whether the error between the time series of the training sample running nodes and the actual running results of the training sample nodes meets the preset convergence conditions.
[0104] The preset convergence conditions can be pre-set and modified by technical personnel or authorized staff, and are not limited here.
[0105] It should be noted that in the specific implementation of this application, the model is not limited to using the preset convergence condition to train the model, but can also be trained using the preset number of iterations, and no limitation is made here.
[0106] Specifically, if it is determined that the error between the time series of the training sample running node and the actual running result of the training sample node meets the preset convergence condition, then step S404 is executed; if it is determined that the error between the time series of the training sample running node and the actual running result of the training sample node does not meet the preset convergence condition, then step S405 is executed.
[0107] S404. Use the mathematical model that satisfies the preset convergence conditions as the derivation model.
[0108] S405. Adjust the mathematical model based on the error between the operational deduction results corresponding to the time series of the training sample nodes and the actual operational results corresponding to the time series of the training sample nodes.
[0109] S105. Generate a batch of operational simulation results based on the predicted time series of all operational nodes.
[0110] As can be seen from the above scheme, this application provides a batch operation simulation method, including: periodically collecting basic data of all running nodes in batches; wherein, the basic data includes at least system data, application data, and business data; then, for each running node, generating the original time series of the running node based on the basic data of the running node; wherein, the original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features; then, based on the original time series of the running nodes, generating the predicted time series of the running nodes using a preset time series prediction method; wherein, the predicted time series includes at least one future time point and the predicted time dimension features of the future time point; after that, the running nodes... The predicted time series data is input into the simulation model, and the output is the simulation results of the running nodes. The simulation model is trained using at least one training sample. The training sample includes the time series of the training sample running nodes and the corresponding actual running results. The actual running results include at least the actual execution result, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes. The simulation results of the predicted time series of the running nodes include at least the simulation execution result, simulation runtime, and simulation resource consumption at future time points in the predicted time series of the running nodes. Finally, based on the simulation results of the predicted time series of all running nodes, a batch of simulation results is generated. By simulating the running conditions after batching, the batch simulation results can effectively improve the orchestration efficiency of the batch scheduling sequence.
[0111] Another embodiment of this application provides a batch operation simulation device, such as... Figure 5 As shown, it specifically includes:
[0112] The data acquisition unit 501 is used to periodically collect basic data from all running nodes in batches.
[0113] The basic data includes at least system data, application data, and business data.
[0114] The first generation unit 502 is used to generate the original time series of each running node based on the basic data of the running node.
[0115] The original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features.
[0116] Optionally, in another embodiment of this application, one implementation of the first generating unit 502 includes:
[0117] The merging unit is used to identify and merge data at the same time point in the basic data of each running node to obtain the time dimension features of that time point.
[0118] It should be noted that, in the specific implementation process of this application, the time intervals between time nodes can be the same or different. The time intervals can be preset and modified by technical personnel or relevant authorized personnel, etc., and are not limited here.
[0119] The first generation subunit is used to generate the original time series of the running nodes based on all time points and the time dimension characteristics of the time points.
[0120] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 2 As shown, it will not be elaborated further here.
[0121] The second generation unit 503 is used to generate a predicted time series of the running node based on the original time series of the running node using a preset time series prediction method.
[0122] The predicted time series includes at least one future time point and the predicted time dimension features of the future time point.
[0123] The first input unit 504 is used to input the predicted time series of the running node into the inference model and output the running inference result of the running node.
[0124] The inference model is obtained by training the mathematical model with at least one training sample; the training sample includes the time series of the running nodes of the training sample and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at the time points in the time series of the running nodes of the training sample; the inference results of the predicted time series of the running nodes include at least the inferred execution results, inferred runtime, and inferred resource consumption at future time points in the predicted time series of the running nodes.
[0125] The simulation result generation unit 505 is used to generate batch simulation results based on the simulation results of the predicted time series of all running nodes.
[0126] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.
[0127] Optionally, in another embodiment of this application, one implementation of the batch operation of the simulation device further includes:
[0128] The acquisition unit is used to acquire the real time dimension features of the time point in the real time when the real time reaches the future time point in the predicted time series.
[0129] The first adjustment unit is used to adjust the preset time series prediction method by utilizing the error between the real time dimension characteristics of a time point in real time and the predicted time dimension characteristics of a future time point.
[0130] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 3 As shown, it will not be elaborated further here.
[0131] Optionally, in another embodiment of this application, one implementation of the construction unit of the inference model includes:
[0132] Training sample set construction unit, used to construct the training sample set.
[0133] The training sample set includes at least one training sample; the training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes.
[0134] The second input unit is used to input the time series of the training sample running nodes into the mathematical model and output the running deduction results corresponding to the time series of the training sample running nodes.
[0135] The runtime simulation results corresponding to the time series of the training sample running nodes include at least the simulation execution results, simulation runtime, and simulation resource consumption at each time point in the time series of the training sample running nodes.
[0136] The second adjustment unit is used to adjust the mathematical model based on the error between the running deduction results corresponding to the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes, until the error between the running deduction results corresponding to the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes meets the preset convergence condition, and the mathematical model that meets the preset convergence condition is used as the deduction model.
[0137] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 4 As shown, it will not be elaborated further here.
[0138] As can be seen from the above scheme, this application provides a batch operation simulation device, including: a data acquisition unit 501 periodically collecting basic data of all running nodes in batches; wherein, the basic data includes at least system data, application data, and business data; then, a first generation unit 502 generates the original time series of each running node based on the basic data of the running node; wherein, the original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features; a second generation unit 503 generates the predicted time series of the running nodes based on the original time series of the running nodes using a preset time series prediction method; wherein, the predicted time series includes at least one future time point and the predicted time dimension features of the future time point; then, a first output unit 502 generates the predicted time series of the running nodes based on the original time series of the running nodes using a preset time series prediction method; wherein, the predicted time series includes at least one future time point and the predicted time dimension features of the future time point; then, a first output unit 503 generates the predicted time series of the running nodes based on the original time series of the running nodes using a preset time series prediction method; wherein, the predicted time series includes at least one future time point and the predicted time dimension features of the future time point; then, a first output unit 502 generates the predicted time series of the running nodes based on the original time series of the running nodes using a preset time series prediction method. Input unit 504 inputs the predicted time series of the running nodes into the simulation model and outputs the simulation results of the running nodes. The simulation model is trained using at least one training sample. The training sample includes the time series of the training sample running nodes and the corresponding actual running results. The actual running results include at least the actual execution results, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes. The simulation results of the predicted time series of the running nodes include at least the simulation execution results, simulation runtime, and simulation resource consumption at future time points in the predicted time series of the running nodes. Finally, simulation result generation unit 505 generates batch simulation results based on the simulation results of the predicted time series of all running nodes. By simulating the running conditions after batching, the batch simulation results can effectively improve the orchestration efficiency of the batch scheduling order.
[0139] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0140] Another embodiment of this application provides an electronic device, such as... Figure 6 As shown, it includes:
[0141] One or more processors 601.
[0142] Storage device 602, on which one or more programs are stored.
[0143] When the one or more programs are executed by the one or more processors 601, the one or more processors 601 implement the batch execution simulation method as described in any of the above embodiments.
[0144] Another embodiment of this application provides a computer storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the batch execution simulation method as described in any of the above embodiments.
[0145] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0146] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0147] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0148] Another embodiment of this application provides a computer program product, which, when executed, is used to perform the batch execution simulation method described above.
[0149] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of this application.
[0150] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
[0151] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0152] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described application concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions claimed in this application.
Claims
1. A batch operation simulation method, characterized in that, include: Periodically collect basic data from all running nodes in batches; wherein, the basic data includes at least system data, application data and business data, the system data includes CPU, memory, network and IO data, the application data includes application logs, port liveness, process liveness and process resource consumption, and the business data includes traffic, error rate and latency; For each running node, an original time series of the running node is generated based on the basic data of the running node, including: for each running node, identifying and merging data at the same time point in the basic data of the running node to obtain the time dimension feature of the time point; generating the original time series of the running node based on all the time points and the time dimension features of the time points, wherein the original time series includes at least one time point, and the time dimension feature of each time point includes at least basic features, dynamic features, and higher-order features; Based on the original time series of the running node, a preset time series prediction method is used to generate a predicted time series of the running node; wherein, the predicted time series includes at least one future time point and the predicted time dimension feature of the future time point; The predicted time series of the running node is input into the inference model, and the running inference result of the running node is output. The inference model is obtained by training a mathematical model with at least one training sample. The training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes. The actual running results include at least the actual execution result, actual runtime, and actual resource consumption at each time point in the time series of the training sample running nodes. The running inference result of the predicted time series of the running node includes at least the inferred execution result, inferred runtime, and inferred resource consumption at each future time point in the predicted time series of the running node. The batch of operational simulation results is generated based on the operational simulation results of the predicted time series of all the operational nodes.
2. The operational simulation method according to claim 1, characterized in that, The time intervals between these time points are the same.
3. The operational simulation method according to claim 1, characterized in that, After generating the predicted time series based on the original time series using a preset time series prediction method, the process further includes: When the real time reaches a future time point in the predicted time series, the real time dimension features of the time point under the real time are obtained. The preset time series prediction method is adjusted by utilizing the error between the real time dimension features of the real time point and the predicted time dimension features of the future time point.
4. The operational simulation method according to claim 1, characterized in that, The method for constructing the inference model includes: Construct a training sample set; wherein the training sample set includes at least one training sample; the training sample includes the time series of the running nodes of the training sample and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at time points in the time series of the running nodes of the training sample. The time series of the training sample running nodes is input into the mathematical model, and the running simulation results corresponding to the time series of the training sample running nodes are output; wherein, the running simulation results corresponding to the time series of the training sample running nodes include at least the simulation execution results, simulation runtime, and simulation resource consumption at time points in the time series of the training sample running nodes; Based on the error between the operational deduction results corresponding to the time series of the training sample running nodes and the actual operational results corresponding to the time series of the training sample nodes, the mathematical model is adjusted until the error between the operational deduction results corresponding to the time series of the training sample running nodes and the actual operational results corresponding to the time series of the training sample nodes satisfies a preset convergence condition. The mathematical model that satisfies the preset convergence condition is then used as the deduction model.
5. A batch operation simulation device, characterized in that, include: The data collection unit is used to periodically collect basic data from all running nodes in batches. The basic data includes at least system data, application data, and business data. The system data includes CPU, memory, network, and I / O data. The application data includes application logs, port liveness, process liveness, and process resource consumption. The business data includes traffic, error rate, and latency. The first generation unit is configured to generate an original time series for each running node based on the basic data of the running node; wherein the original time series includes at least one time point, and the time dimension features of each time point include at least basic features, dynamic features, and higher-order features; The second generation unit is used to generate a predicted time series of the running node based on the original time series of the running node using a preset time series prediction method; wherein, the predicted time series includes at least one future time point and the predicted time dimension feature of the future time point. The first input unit is used to input the predicted time series of the running node into the inference model and output the running inference result of the running node; wherein, the inference model is obtained by training a mathematical model with at least one training sample; the training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution result, actual runtime, and actual resource consumption at time points in the time series of the training sample running nodes; the running inference result of the predicted time series of the running node includes at least the inferred execution result, inferred runtime, and inferred resource consumption at future time points in the predicted time series of the running node; The simulation result generation unit is used to generate the batch of simulation results based on the simulation results of the predicted time series of all the running nodes; The first generation unit includes: The merging unit is used to identify and merge data at the same time point in the basic data of each running node to obtain the time dimension feature of the time point. The first generation subunit is used to generate the original time series of the running node based on all the time points and the time dimension characteristics of the time points.
6. The operational simulation device according to claim 5, characterized in that, The building blocks of the inference model include: A training sample set construction unit is used to construct a training sample set; wherein, the training sample set includes at least one training sample; the training sample includes the time series of the training sample running nodes and the actual running results corresponding to the time series of the training sample nodes; the actual running results include at least the actual execution results, actual runtime, and actual resource consumption at time points in the time series of the training sample running nodes. The second input unit is used to input the time series of the training sample running nodes into the mathematical model and output the running deduction result corresponding to the time series of the training sample running nodes; wherein, the running deduction result corresponding to the time series of the training sample running nodes includes at least the deduction execution result, deduction runtime and deduction resource consumption at time points in the time series of the training sample running nodes; The second adjustment unit is used to adjust the mathematical model based on the error between the running deduction result corresponding to the time series of the training sample running node and the actual running result corresponding to the time series of the training sample node, until the error between the running deduction result corresponding to the time series of the training sample running node and the actual running result corresponding to the time series of the training sample node satisfies a preset convergence condition, and the mathematical model that satisfies the preset convergence condition is used as the deduction model.
7. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the batch execution simulation method as described in any one of claims 1 to 4.
8. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the batch execution simulation method as described in any one of claims 1 to 4.