Self-adaptive optimization method and device for batch scheduling graph, electronic equipment and storage medium
By constructing a training sample set, predicting node runtime, generating a dynamic DAG scheduling graph, and calculating the critical path, the problem of low batch scheduling efficiency in commercial banking business systems is solved, and the overall efficiency of batch scheduling is improved.
Patent Information
- Application Number
- CN202511734157.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies are insufficient to effectively assess and improve the overall scheduling performance of batch scheduling in commercial banking systems, especially when calculating the time required for batch critical paths.
By constructing a training sample set based on batch historical operation log data, using a time series model to predict node runtime, generating a dynamic DAG scheduling graph, calculating the critical path of the weighted graph, and selecting the DAG graph with the shortest execution time as the optimal scheduling model.
It enables the calculation of time consumption for each scheduling arrangement from a global perspective, thereby improving the overall batch scheduling efficiency.
Smart Images
Figure CN121501461A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more specifically, to an adaptive optimization method, apparatus, electronic device, and storage medium for batch scheduling graphs. Background Technology
[0002] During operation, commercial banking systems require batch processing of business data at night or during system downtime. A typical batch consists of multiple batches, each processing a specific type of business data. Each batch comprises multiple job nodes, each completing a step within the batch. These nodes have a sequential relationship; once a node finishes execution, the next node begins. Therefore, the nodes of the system's batch operations form a directed acyclic graph (DAG). Batch execution has multiple paths from start to finish, and the longest path is the critical path, determining the longest execution time for a batch. Calculating the execution time of a batch is therefore crucial during system batch execution. Summary of the Invention
[0003] In view of this, this application provides an adaptive optimization method, apparatus, electronic device and storage medium for batch scheduling graphs, which calculates the time consumption of each scheduling arrangement from a global perspective, so as to improve the overall batch scheduling efficiency based on the time consumption.
[0004] To achieve the above objectives, the following solution is proposed:
[0005] An adaptive optimization method for batch scheduling graphs, applied to electronic devices, is used to adaptively optimize batch scheduling graphs generated based on time-series prediction, critical paths of AOE networks, and dynamic DAGs. The adaptive optimization method includes the following steps:
[0006] A training sample set is constructed based on the collected batch of historical operation log data;
[0007] The model is trained based on the training sample set to obtain a time series model, and the runtime of each node in the future is predicted based on the time series model.
[0008] A dynamic DAG scheduling graph set is generated using the strong dependency relationships of the input batch scheduling graphs.
[0009] Based on the dynamic DAG scheduling graph set and the runtime, multiple weighted graphs are generated, and the critical path of each weighted graph is calculated;
[0010] By processing the critical path, the optimal scheduling model for the batch to be optimized is obtained.
[0011] Optionally, constructing a training sample set based on the collected batch of historical operation log data includes the following steps:
[0012] The batch historical operation logs are collected from the batch scheduling management system through offline data connection. The batch historical operation log data includes some or all of the batch start time, batch end time, batch operation status, accounting date, and batch scheduling times.
[0013] The batch of historical operation data is preprocessed to obtain standardized data.
[0014] Based on the standardized data, the running time of batch nodes is calculated to obtain the running time data of each batch node on each accounting date;
[0015] The runtime data of each batch node is stored in the database to obtain the training sample set.
[0016] Optionally, the step of training the model based on the training sample set to obtain a time series model, and predicting the future runtime of each node based on the time series model, includes the following steps:
[0017] The training sample set data is learned based on the Prophet algorithm to establish a batch time series model;
[0018] The runtime is obtained by making predictions based on the time series model.
[0019] Optionally, generating a dynamic DAG scheduling graph set using the time series model includes the following steps:
[0020] Receive the strong dependency node relationships in the batch scheduling graph to be optimized, input by experts;
[0021] A set of DAG scheduling graphs is generated randomly based on the nodes in the batch scheduling graph.
[0022] Store the DAG scheduling graph set.
[0023] Optionally, the step of generating multiple weighted graphs based on the dynamic DAG scheduling graph set and the runtime, and calculating the critical path of each weighted graph, includes the following steps:
[0024] For the dynamic DAG scheduling graph set, the weighted graph is generated based on the running time;
[0025] The critical path of each weighted graph is calculated using the AOE network critical path algorithm.
[0026] Optionally, the step of using a general path algorithm to calculate the critical path of each weighted graph includes the following steps:
[0027] The critical path is obtained by calculating the critical path for each weighted graph using the AOE network critical path algorithm.
[0028] Optionally, obtaining the optimal scheduling model by processing the critical path includes the following steps:
[0029] Calculate the time taken for each of the critical paths;
[0030] The dynamic DAG scheduling graph corresponding to the critical path with the shortest execution time is selected as the optimal scheduling model.
[0031] An adaptive optimization device for batch scheduling graphs, applied to electronic devices, is used for adaptive optimization of batch scheduling graphs generated based on time-series prediction, critical paths of AOE networks, and dynamic DAGs. The adaptive optimization device includes:
[0032] The sample construction module is designed to build a training sample set based on the collected batch of historical operation log data;
[0033] The model training module is designed to train the model based on the training sample set, obtain a time series model, and predict the future runtime of each node based on the time series model.
[0034] The set generation module is designed to generate a dynamic DAG scheduling graph set by utilizing the strong dependency relationships of the input batch scheduling graphs;
[0035] The weighted graph generation module is designed to generate multiple weighted graphs based on the dynamic DAG scheduling graph set and the runtime, and to calculate the critical path of each weighted graph;
[0036] The path processing module is designed to obtain the optimal scheduling model for the batch to be optimized by processing the critical path.
[0037] An electronic device includes at least one processor and a memory connected to the processor, wherein:
[0038] The memory is used to store computer programs or instructions;
[0039] The processor is used to execute the computer program or instructions to enable the electronic device to implement the adaptive optimization method as described above.
[0040] A computer-readable storage medium is applied to an electronic device, the storage medium carrying one or more computer programs that can be executed by the electronic device to enable the electronic device to perform the adaptive optimization method as described above.
[0041] As can be seen from the above technical solution, this application discloses an adaptive optimization method, apparatus, electronic device, and storage medium for batch scheduling graphs. This method and apparatus are applied to electronic devices for adaptive optimization of batch scheduling graphs generated based on time-series prediction, critical paths in AOE networks, and dynamic DAGs. Specifically, it involves constructing a training sample set based on collected batch historical runtime log data; training a model based on the training sample set to obtain a time-series model, and predicting the future runtime of each node based on the time-series model; generating a dynamic DAG scheduling graph set using the strong dependency relationships of the entered batch scheduling graphs; generating multiple weighted graphs based on the dynamic DAG scheduling graph set and runtime, and calculating the critical path of each weighted graph; and obtaining the optimal scheduling model for the batch to be optimized through processing the critical path. This solution calculates the time consumption under each scheduling arrangement from a global perspective, thereby improving the overall batch scheduling efficiency based on the time consumption. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a flowchart illustrating an adaptive optimization method for batch scheduling graphs according to an embodiment of this application;
[0044] Figure 2 This is a flowchart illustrating an adaptive optimization method for batch scheduling graphs according to an embodiment of this application;
[0045] Figure 3 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0047] This solution involves several concepts, including time series forecasting, Directed Acyclic Graph (DAG), nodes, critical path, and Area of Effect (AOE) network. Time series forecasting is a model that predicts future data trends based on historical time series data; common models include Prophet and LSTM. A DAG is a directed acyclic graph. A batch is an operation that processes system business data in batches; a system has multiple batches, each processing different business processes. A node is a unit within a batch, processing one step within the batch; nodes have sequential relationships. The critical path is the longest path from the start node to the end node in a batch, determining the batch's runtime. A project is often divided into multiple sub-projects, called activities. In a weighted directed graph, if vertices represent events, directed edges represent activities, and the weights on the edges represent the duration of the activity, the resulting graph is simply called an Area of Effect (AOE) network.
[0048] This solution aims to address the challenge of effectively evaluating and continuously improving the performance efficiency of traditional batch scheduling by employing time-series prediction algorithms, AOE critical path calculation, and dynamic DAG generation. Its core idea is to dynamically predict the runtime of each batch node based on historical runtime data of all nodes in the batch scheduling graph. For batch nodes excluding strong dependencies, a dynamic DAG is randomly generated. For each generated DAG, a critical path is calculated using the AOE critical path algorithm, with the predicted node runtime as the weight of each edge in the graph. Finally, the final runtime of each DAG scheduling graph is calculated based on this critical path, and the DAG with the shortest runtime is selected as the current optimal batch scheduling graph model. Based on this, this application proposes the following specific embodiments.
[0049] Figure 1 This is a flowchart of an adaptive optimization method for batch scheduling graphs according to an embodiment of this application.
[0050] like Figure 1 As shown, the adaptive optimization method provided in this embodiment is applied to an electronic device for adjusting a batch scheduling graph. This electronic device can be understood as a computer, server, or cloud platform with data computing and information processing capabilities. The adaptive optimization method specifically includes the following steps:
[0051] S1. Construct a training sample set based on the collected batch of historical operation log data.
[0052] First, batch historical operation log data is collected. Specifically, batch historical operation logs are collected from the batch scheduling management system through offline data integration. This data generally includes information such as batch start time, batch end time, batch operation status, accounting date, and batch scheduling count.
[0053] Next, perform batch historical runtime data preprocessing. This involves preprocessing the batch runtime log data obtained in the previous step, cleaning up log data that does not meet the specifications, and standardizing data such as time format and runtime status.
[0054] Next, the runtime calculation for each batch node is performed. That is, the runtime for each batch node on each accounting date is calculated.
[0055] Finally, the calculated batch time consumption information is stored in the database to obtain the training sample set mentioned above.
[0056] S2. Train the model based on the training sample set to obtain a time series model, and predict the runtime of each node in the future based on the time series model.
[0057] First, the model is trained on the training sample set based on the Prohet algorithm to establish a batch time series model. Specifically, the Prohet time series is used to construct hyperparameters for the batch node time information, and the error between the regression model's prediction results and the actual values is evaluated using MSE (mean squared error), RMSE (root mean squared error), and MAE (mean absolute error). Various parameters are tried to find the optimal parameters, thus obtaining the final time series prediction model.
[0058] Then, the runtime of each node over a future period is predicted based on the time series model. That is, the time change trend of the node over the next week is predicted based on the above time series prediction model, and the average value is taken as the predicted runtime of the batch of nodes.
[0059] S3. Generate a dynamic DAG scheduling graph set using the strong dependency relationships of the entered batch scheduling graphs.
[0060] First, the system receives strong dependency relationships between nodes in the batch scheduling graph to be optimized, entered by users or experts. For each batch scheduling graph to be optimized, the person in charge of the batch, in conjunction with the batch business logic, enters the nodes in the batch scheduling graph that have strong dependencies, such as a relationship where node A must run before node B.
[0061] Then, a random generation method is used to randomly generate dynamic DAG scheduling graphs based on the nodes in the batch scheduling graph to be optimized. During generation, it is necessary to ensure that the edges in the generated dynamic DAG scheduling graph satisfy the strong dependency relationship mentioned above. Based on the node set in the batch scheduling graph, multiple dynamic DAG scheduling graphs are randomly generated using an algorithm, and dynamic DAG scheduling graphs that do not satisfy the above dependency relationship are eliminated.
[0062] Finally, the multiple dynamic DAG scheduling graphs obtained above are stored to form the dynamic DAG scheduling graph set.
[0063] S4. Generate multiple weighted graphs based on the dynamic DAG scheduling graph set and runtime, and calculate the critical path of each weighted graph.
[0064] First, for the obtained dynamic DAG scheduling graph set, a weighted graph is generated based on the runtime of each node obtained above. That is, by combining the batch execution time prediction values of each node, the random DAG graph obtained in the previous step is transformed into a weighted graph, and the weight of each edge is the execution time prediction value of the preceding batch of nodes.
[0065] Then, based on the weighted graphs described above, a general critical path algorithm is used to calculate the critical path for each weighted graph. The AOE network critical path algorithm can be used here.
[0066] S5. By processing the critical path, the optimal scheduling model for the batch to be optimized is obtained.
[0067] First, calculate the time taken for each critical path;
[0068] Then, the dynamic DAG scheduling graph with the shortest execution time is selected as the best scheduling model for the batch to be optimized.
[0069] As can be seen from the above technical solution, this embodiment provides an adaptive optimization method for batch scheduling graphs. This method is applied to electronic devices to adaptively optimize batch scheduling graphs generated based on time-series prediction, critical paths in AOE networks, and dynamic DAGs. Specifically, it involves constructing a training sample set based on collected batch historical operation log data; training a model based on the training sample set to obtain a time-series model; predicting the future runtime of each node based on the time-series model; generating a dynamic DAG scheduling graph set using the strong dependency relationships of the entered batch scheduling graphs; generating multiple weighted graphs based on the dynamic DAG scheduling graph set and runtime; calculating the critical path of each weighted graph; and obtaining the optimal scheduling model for the batch to be optimized through processing the critical path. This solution calculates the time consumption under each scheduling arrangement from a global perspective, thereby improving the overall batch scheduling efficiency based on the time consumption.
[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0071] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.
[0072] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0073] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer.
[0074] Figure 2 This is a block diagram of an adaptive optimization device for batch scheduling graphs according to an embodiment of this application.
[0075] like Figure 2As shown, the adaptive optimization device provided in this embodiment is applied to an electronic device for adjusting a batch scheduling graph. This electronic device can be understood as a computer, server, or cloud platform with data computing and information processing capabilities. Specifically, the adaptive optimization device includes a sample construction module 10, a model training module 20, an ensemble generation module 30, a weighted graph generation module 40, and a path processing module 50.
[0076] The sample construction module is used to build a training sample set based on the collected batch of historical operation log data.
[0077] First, batch historical operation log data is collected. Specifically, batch historical operation logs are collected from the batch scheduling management system through offline data integration. This data generally includes information such as batch start time, batch end time, batch operation status, accounting date, and batch scheduling count.
[0078] Next, perform batch historical runtime data preprocessing. This involves preprocessing the batch runtime log data obtained in the previous step, cleaning up log data that does not meet the specifications, and standardizing data such as time format and runtime status.
[0079] Next, the runtime calculation for each batch node is performed. That is, the runtime for each batch node on each accounting date is calculated.
[0080] Finally, the calculated batch time consumption information is stored in the database to obtain the training sample set mentioned above.
[0081] The model training module is used to train the model based on the training sample set, obtain the time series model, and predict the future runtime of each node based on the time series model.
[0082] First, the model is trained on the training sample set based on the Prohet algorithm to establish a batch time series model. Specifically, the Prohet time series is used to construct hyperparameters for the batch node time information, and the error between the regression model's prediction results and the actual values is evaluated using MSE (mean squared error), RMSE (root mean squared error), and MAE (mean absolute error). Various parameters are tried to find the optimal parameters, thus obtaining the final time series prediction model.
[0083] Then, the runtime of each node over a future period is predicted based on the time series model. That is, the time change trend of the node over the next week is predicted based on the above time series prediction model, and the average value is taken as the predicted runtime of the batch of nodes.
[0084] The set generation module is used to generate a dynamic DAG scheduling graph set by utilizing the strong dependency relationships of the input batch scheduling graphs.
[0085] First, the system receives strong dependency relationships between nodes in the batch scheduling graph to be optimized, entered by users or experts. For each batch scheduling graph to be optimized, the person in charge of the batch, in conjunction with the batch business logic, enters the nodes in the batch scheduling graph that have strong dependencies, such as a relationship where node A must run before node B.
[0086] Then, a random generation method is used to randomly generate dynamic DAG scheduling graphs based on the nodes in the batch scheduling graph to be optimized. During generation, it is necessary to ensure that the edges in the generated dynamic DAG scheduling graph satisfy the strong dependency relationship mentioned above. Based on the node set in the batch scheduling graph, multiple dynamic DAG scheduling graphs are randomly generated using an algorithm, and dynamic DAG scheduling graphs that do not satisfy the above dependency relationship are eliminated.
[0087] Finally, the multiple dynamic DAG scheduling graphs obtained above are stored to form the dynamic DAG scheduling graph set.
[0088] The weighted graph generation module is used to generate multiple weighted graphs based on a dynamic DAG scheduling graph set and runtime, and to calculate the critical path of each weighted graph.
[0089] First, for the obtained dynamic DAG scheduling graph set, a weighted graph is generated based on the runtime of each node obtained above. That is, by combining the batch execution time prediction values of each node, the random DAG graph obtained in the previous step is transformed into a weighted graph, and the weight of each edge is the execution time prediction value of the preceding batch of nodes.
[0090] Then, based on the weighted graphs described above, a general critical path algorithm is used to calculate the critical path for each weighted graph. The AOE network critical path algorithm can be used here.
[0091] The path processing module is used to obtain the optimal scheduling model for the batch to be optimized by processing the critical path.
[0092] First, calculate the time taken for each critical path;
[0093] Then, the dynamic DAG scheduling graph with the shortest execution time is selected as the best scheduling model for the batch to be optimized.
[0094] As can be seen from the above technical solution, this embodiment provides an adaptive optimization device for batch scheduling graphs. This device is applied to electronic devices and is used to adaptively optimize batch scheduling graphs generated based on time series prediction, critical paths of AOE networks, and dynamic DAGs. Specifically, it constructs a training sample set based on collected batch historical operation log data; trains a model based on the training sample set to obtain a time series model, and predicts the future runtime of each node based on the time series model; generates a set of dynamic DAG scheduling graphs using the strong dependency relationships of the input batch scheduling graphs; generates multiple weighted graphs based on the set of dynamic DAG scheduling graphs and runtime, and calculates the critical path of each weighted graph; and obtains the optimal scheduling model for the batch to be optimized by processing the critical path. This solution calculates the time consumption under each scheduling arrangement from a global perspective, thereby improving the overall batch scheduling efficiency based on the time consumption.
[0095] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".
[0096] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0097] Figure 3 This is a block diagram of an electronic device according to an embodiment of this application.
[0098] The following is for reference. Figure 3 This document illustrates a structural diagram suitable for implementing the electronic device in the embodiments of this disclosure. The terminal device in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. This electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.
[0099] The electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from an input device 306 into a random access memory (RAM) 303. The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0100] Typically, the following devices can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various devices are shown in the figures, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0101] This embodiment also provides an embodiment of a computer-readable storage medium.
[0102] The aforementioned computer-readable storage medium is applied to an electronic device and carries one or more computer programs. When these programs are executed by the electronic device, they enable the device to adaptively optimize a batch scheduling graph generated based on time-series prediction, critical paths in an AOE network, and a dynamic DAG. Specifically, this involves constructing a training sample set based on collected batch historical runtime log data; training a model based on the training sample set to obtain a time-series model; predicting the future runtime of each node based on the time-series model; generating a dynamic DAG scheduling graph set using the strong dependency relationships of the entered batch scheduling graph; generating multiple weighted graphs based on the dynamic DAG scheduling graph set and runtime; calculating the critical path in each weighted graph; and obtaining the optimal scheduling model for the batch to be optimized through processing the critical paths. This scheme calculates the time consumption for each scheduling arrangement from a global perspective, thereby improving the overall batch scheduling efficiency based on the time consumption.
[0103] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0104] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0106] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.
[0107] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0108] The technical solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. An adaptive optimization method for batch scheduling graphs, applied to electronic devices, for adaptive optimization of batch scheduling graphs generated based on time-series prediction, critical paths of AOE networks, and dynamic DAGs, characterized in that... The adaptive optimization method includes the following steps: A training sample set is constructed based on the collected batch of historical operation log data; The model is trained based on the training sample set to obtain a time series model, and the runtime of each node in the future is predicted based on the time series model. A dynamic DAG scheduling graph set is generated using the strong dependency relationships of the input batch scheduling graphs. Based on the dynamic DAG scheduling graph set and the runtime, multiple weighted graphs are generated, and the critical path of each weighted graph is calculated; By processing the critical path, the optimal scheduling model for the batch to be optimized is obtained.
2. The adaptive optimization method as described in claim 1, characterized in that, The construction of the training sample set based on the collected batch of historical operation log data includes the following steps: The batch historical operation logs are collected from the batch scheduling management system through offline data connection. The batch historical operation log data includes some or all of the batch start time, batch end time, batch operation status, accounting date, and batch scheduling times. The batch of historical operation data is preprocessed to obtain standardized data; Based on the standardized data, the running time of batch nodes is calculated to obtain the running time data of each batch node on each accounting date; The runtime data of each batch node is stored in the database to obtain the training sample set.
3. The adaptive optimization method as described in claim 1, characterized in that, The step of training the model based on the training sample set to obtain a time series model, and predicting the future runtime of each node based on the time series model, includes the following steps: The training sample set data is learned based on the Prophet algorithm to establish a batch time series model; The runtime is obtained by making predictions based on the time series model.
4. The adaptive optimization method as described in claim 1, characterized in that, The step of generating a dynamic DAG scheduling graph set using the time series model includes the following steps: Receive the strong dependency node relationships in the batch scheduling graph to be optimized, input by experts; A set of DAG scheduling graphs is generated randomly based on the nodes in the batch scheduling graph. Store the DAG scheduling graph set.
5. The adaptive optimization method as described in claim 1, characterized in that, The process of generating multiple weighted graphs based on the dynamic DAG scheduling graph set and the runtime, and calculating the critical path of each weighted graph, includes the following steps: For the dynamic DAG scheduling graph set, the weighted graph is generated based on the running time; The critical path of each weighted graph is calculated using the AOE network critical path algorithm.
6. The adaptive optimization method as described in claim 5, characterized in that, The calculation of the critical path for each weighted graph using a general path algorithm includes the following steps: The critical path is obtained by calculating the critical path for each weighted graph using the AOE network critical path algorithm.
7. The adaptive optimization method as described in claim 1, characterized in that, The process of processing the critical path to obtain the optimal scheduling model includes the following steps: Calculate the time taken for each of the critical paths; The dynamic DAG scheduling graph corresponding to the critical path with the shortest execution time is selected as the optimal scheduling model.
8. An adaptive optimization device for batch scheduling graphs, applied to electronic devices, for adaptively optimizing batch scheduling graphs generated based on time-series prediction, critical paths of AOE networks, and dynamic DAGs, characterized in that... The adaptive optimization device includes: The sample construction module is designed to build a training sample set based on the collected batch of historical operation log data; The model training module is designed to train the model based on the training sample set, obtain a time series model, and predict the future runtime of each node based on the time series model. The set generation module is designed to generate a dynamic DAG scheduling graph set by utilizing the strong dependency relationships of the input batch scheduling graphs; The weighted graph generation module is designed to generate multiple weighted graphs based on the dynamic DAG scheduling graph set and the runtime, and to calculate the critical path of each weighted graph; The path processing module is designed to obtain the optimal scheduling model for the batch to be optimized by processing the critical path.
9. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs or instructions; The processor is used to execute the computer program or instructions to enable the electronic device to implement the adaptive optimization method as described in any one of claims 1 to 7.
10. A computer-readable storage medium for use in electronic devices, characterized in that, The storage medium carries one or more computer programs that can be executed by the electronic device, thereby enabling the electronic device to implement the adaptive optimization method as described in any one of claims 1 to 7.