Intelligent scheduling method and system for data development operation

Abstract

The invention relates to the technical field of data development job scheduling, and discloses an intelligent scheduling method and system for data development jobs, and the method comprises the steps: obtaining historical resource data and historical job scheduling data of a machine, obtaining job metadata of ETL jobs, and constructing a training data set; performing feature recognition according to the training data set to obtain scheduling distribution features; inputting the scheduling distribution characteristics into a preset decision tree model for training and learning, and outputting an ETL job scheduling strategy to construct a predictor; and obtaining notification information or an execution request for triggering and scheduling the ETL job, and inputting the current resource condition of the machine and preset parameters into the predictor to obtain a prediction result of the ETL job scheduling strategy. According to the method and the device, the resource allocation efficiency of the ETL operation can be improved on the premise of ensuring normal operation of the ETL operation, so that the ETL operation can be executed in time and orderly according to the requirement of a service scene.

Description

Technical Field

[0001] The present application relates to the technical field of data development job scheduling, and in particular to a method and system for intelligent scheduling of data development jobs. Background Art

[0002] ETL (Extraction-Transformation-Loading) job refers to a series of operations that extract, transform, and load data from the original data source and then store it in the target database or data warehouse after processing. It has the advantages of ensuring data consistency, improving data quality, optimizing analysis speed, and supporting data-driven decision-making.

[0003] In the prior art, the quantification of ETL jobs is mainly achieved by monitoring the server resources, quantifying the server resources and the resources required for the ETL jobs, calculating the average memory usage of the jobs, and then judging whether the current server resources can execute the current jobs based on certain logical node processing, thereby improving the resource allocation efficiency of the jobs. However, if the scheduling of ETL jobs is calculated only from hardware resources, the ETL jobs cannot be executed in a timely or orderly manner according to the needs of the business scenarios, which greatly affects the normal progress of the ETL jobs.

[0004] With respect to the above-mentioned related technologies, the inventors have found that the existing ETL job scheduling method has the problem that the scheduling is only performed from the perspective of hardware resources, which affects the normal progress of the ETL job. Summary of the invention

[0005] In order to improve the resource allocation efficiency of ETL operations while ensuring the normal operation of ETL operations, the present application provides a method and system for intelligent scheduling of data development operations.

[0006] In a first aspect, the present application provides a method for intelligent scheduling of data development operations.

[0007] This application is achieved through the following technical solutions:

[0008] A data development job intelligent scheduling method comprises the following steps:

[0009] Obtain historical machine resource data and historical job scheduling data, and

[0010] Get the job metadata of the ETL job and build a training data set;

[0011] Perform feature recognition based on the training data set to obtain scheduling distribution features;

[0012] Input the scheduling distribution characteristics into a preset decision tree model for training and learning, and output the ETL job scheduling strategy to build a predictor;

[0013] The notification information or execution request that triggers the scheduling of the ETL job is obtained, and the current resource status of the machine and the preset parameters are input into the predictor to obtain the prediction result of the ETL job scheduling strategy.

[0014] In a preferred example, the present application can be further configured as follows: the steps of obtaining historical machine resource data and historical job scheduling data, and obtaining job metadata of ETL jobs, and constructing a training data set include:

[0015] Based on a time window or a time series, the historical machine resource data, the historical job scheduling data and the job metadata are correlated and analyzed to obtain the relationship between the job scheduling and the machine resource situation in the time dimension, thereby forming the training data set.

[0016] In a preferred example, the present application can be further configured as follows: the step of performing feature recognition according to the training data set to obtain the scheduling distribution feature includes:

[0017] Analyze the historical resource data of the machine to obtain the machine resource distribution characteristics of the current available resources, the current number of concurrent remaining executable jobs, and the number of jobs in the current waiting queue;

[0018] Analyze the historical job scheduling data to obtain the job scheduling characteristics of the priority position of the current job in the waiting queue and the waiting time of the current job in the waiting queue;

[0019] The job metadata is analyzed to obtain job metadata features including job unique ID, job running time, job scheduling time, number of CPU threads required for running, minimum and maximum memory required for running, maximum tolerable delayed execution time of the job, and job priority.

[0020] In a preferred example, the present application can be further configured as follows: the scheduling distribution feature is input into a preset decision tree model for training and learning, and the steps of outputting the ETL job scheduling strategy to construct a predictor include:

[0021] The scheduling distribution feature is divided by a heuristic method, all scheduling distribution features in the current set of each division are examined one by one, and according to a square error minimization criterion, a scheduling distribution feature that satisfies the square error minimization criterion is selected as a target split point of the current set;

[0022] Based on the mean square error minimization criterion, from the target split points of all the investigated sets, the target split point that satisfies the minimum sum of the weighted sub-nodes is selected as the optimal split point;

[0023] A decision tree model is constructed based on the optimal split point, and a target variable of the decision tree model is defined as the optimal offset scheduling time of the job.

[0024] In a preferred example, the present application may be further configured as follows: the scheduling distribution feature is input into a preset decision tree model for training and learning, and the step of outputting the ETL job scheduling strategy to construct a predictor also includes:

[0025] A preset number of bootstrap samples are extracted from the scheduling distribution characteristics by sampling with replacement;

[0026] A heuristic method is used to divide each of the self-service samples, all the scheduling distribution features in the current set of each division are examined one by one, and according to a square error minimization criterion, a scheduling distribution feature that satisfies the square error minimization criterion is selected as a target split point of the current set;

[0027] Based on the mean square error minimization criterion, from the target split points of all the investigated sets, the target split point that satisfies the minimum sum of the weighted sub-nodes is selected as the optimal split point;

[0028] A sub-decision tree model of each of the self-service samples is constructed based on the optimal split point, and the target variable of each of the sub-decision tree models is defined as the optimal offset scheduling time of the job.

[0029] In a preferred example, the present application may be further configured as follows: the scheduling distribution feature is input into a preset decision tree model for training and learning, and the step of outputting the ETL job scheduling strategy to construct a predictor also includes:

[0030] When the scheduling distribution features are respectively input into the decision tree model and each of the sub-decision tree models for training, some sample features are randomly selected from all the scheduling distribution features in the current set to determine some sample features of all sets;

[0031] Based on the Gini impurity metric, determine the best splitting node of the left rear subtree of the decision tree model and each of the sub-decision tree models from the partial sample features of all sets;

[0032] The decision tree model and each of the sub-decision tree models are updated, and summarized to obtain a random forest model, wherein the average value of the prediction results of the updated decision tree model and each of the sub-decision tree models is the prediction result of the random forest model.

[0033] In a preferred example, the present application may be further configured as follows: the scheduling distribution feature is input into a preset decision tree model for training and learning, and the step of outputting the ETL job scheduling strategy to construct a predictor also includes:

[0034] The scheduling distribution characteristics are input into the random forest model for training and learning, and the ETL job scheduling strategy is output. When the output accuracy of the random forest model meets the preset conditions, the random forest model at this time is used as the predictor.

[0035] In a preferred example, the present application may be further configured as follows: after the step of obtaining the historical resource data and historical job scheduling data of the machine, it also includes:

[0036] Removing invalid or erroneous historical resource data of the machine, and removing invalid or erroneous historical job scheduling data, to obtain initial historical resource data of the machine and initial historical job scheduling data;

[0037] The initial machine historical resource data and the initial historical job scheduling data are standardized or normalized to obtain machine historical resource data and historical job scheduling data with consistent formats and numerical units, and are updated as the machine historical resource data and the historical job scheduling data.

[0038] In a second aspect, the present application provides a data development operation intelligent scheduling system.

[0039] This application is achieved through the following technical solutions:

[0040] A data development operation intelligent scheduling system, comprising:

[0041] The machine resource collection module is used to collect the current resource status and historical resource status of the machine in real time and store them in a preset database;

[0042] The scheduling log collection module is used to collect the historical job scheduling status of the system in real time and store it in a preset database;

[0043] The metadata module is used to obtain the job metadata of the ETL job and store it in the preset database;

[0044] The scheduling feature recognition module is used to construct a training data set based on the stored historical machine resource conditions, the stored historical job scheduling conditions and the job metadata, and perform feature recognition to obtain the scheduling distribution characteristics;

[0045] A scheduling learning module, used for inputting the scheduling distribution characteristics into a preset decision tree model for training and learning, and outputting an ETL job scheduling strategy to construct a predictor;

[0046] The scheduling intelligent optimization module is used to obtain the notification information or execution request that triggers the scheduling of ETL jobs, input the current resource status of the machine and the preset parameters into the predictor, and obtain the prediction result of the ETL job scheduling strategy.

[0047] In a third aspect, the present application provides a computer device.

[0048] This application is achieved through the following technical solutions:

[0049] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned data development job intelligent scheduling methods when executing the computer program.

[0050] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0051] Obtain the historical resource data and historical job scheduling data of the machine, as well as the job metadata of the ETL job, and build a training data set to quantify the scheduling resources of the ETL job in multiple dimensions from the ETL job attributes, hardware resources and historical scheduling conditions, and comprehensively consider the balance between the normal progress of the ETL job and the resource allocation efficiency; perform feature recognition based on the training data set to obtain the scheduling distribution features for machine learning; input the scheduling distribution features into the preset decision tree model for training and learning, and output the ETL job scheduling strategy to build a predictor, which has the advantages of being compatible with high-dimensional data processing, fast model training and prediction speed, easy adjustment and optimization, explanation and verification, and convenient visualization; finally, obtain the notification information or execution request that triggers the scheduling of the ETL job, input the current resource situation of the machine and the preset parameters into the predictor, and obtain the prediction result of the ETL job scheduling strategy, which can improve the resource allocation efficiency of the ETL job while ensuring the normal progress of the ETL job, so that the ETL job can be executed in a timely and orderly manner according to the needs of the business scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of the main flow of a method for intelligent scheduling of data development jobs provided as an exemplary embodiment of the present application.

[0053] Figure 2 A structural block diagram of a data development job intelligent scheduling system provided for yet another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0054] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0057] In the existing technical solutions, the quantification of ETL jobs is mainly carried out in the hardware resource dimension of calculating the average memory usage of the job, which results in the ETL job not being executed in a timely or orderly manner according to the needs of the business scenario. Therefore, in actual application scenarios, the quantification of ETL jobs must also consider various factors, such as the business nature of the job, the timeliness of job execution, whether it needs to rely on the execution of a certain upstream job before execution, the priority order of the job, etc.

[0058] This application collects historical machine resource occupancy and historical job scheduling log data to form a training data set, extracts scheduling distribution characteristics through analysis and processing, and uses machine learning algorithms to obtain the best scheduling strategy for jobs, intelligently allocate job scheduling, and thus improve system resource utilization while minimizing impact on business timeliness and ensuring stable system operation.

[0059] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0060] Reference Figure 1 , an embodiment of the present application provides a method for intelligent scheduling of data development jobs, and the main steps of the method are described as follows.

[0061] S1: Obtain historical machine resource data and historical job scheduling data, as well as job metadata of ETL jobs, to build a training data set;

[0062] S2: Perform feature recognition according to the training data set to obtain scheduling distribution features;

[0063] S3: Input the scheduling distribution features into a preset decision tree model for training and learning, and output the ETL job scheduling strategy to build a predictor;

[0064] S4: Obtain the notification information or execution request that triggers the scheduling of the ETL job, input the current resource status of the machine and the preset parameters into the predictor, and obtain the prediction result of the ETL job scheduling strategy.

[0065] Specifically, the historical resource data of the machine and the current resource data of the machine are collected, including hardware performance indicators such as CPU usage, memory usage and usage, disk space, etc. For example, the open source software Beats+ELK (Elasticsearch-Logstash-Kibana) is used to collect and analyze resource data to obtain the historical resource data of the machine. Beats can run on the target server side, collect relevant system log information in real time and send it to Logstash. Logstash acts as an intermediate layer to perform additional processing and routing on the monitoring information. The relevant system log information can be stored in the Elasticsearch database. Elasticsearch acts as a storage and indexing layer, which is responsible for receiving parsed and structured data and providing analysis and query capabilities.

[0066] When collecting all the historical job scheduling information of the system, the execution logs of all jobs can be recorded through the data middle-office system. On the one hand, the data middle-office system actively collects and stores large quantities of ETL job scheduling logs on a regular basis. Elasticsearch can be used as a time series database to store a large amount of ETL job scheduling log data; on the other hand, the data middle-office system pushes the ETL job scheduling logs to the message middleware Kafka in real time as a log cache pipeline. The intelligent scheduling system creates a consumer instance and subscribes to the log data Topic in the log cache pipeline Kafka, further consuming log data in real time, taking data out of the Topic and writing it into the Elasticsearch database, so as to support the needs of large-scale data analysis and real-time data analysis at the same time.

[0067] In addition, the job metadata of the ETL job is obtained from the historical resource data and historical job scheduling data of the collected machine, with the help of existing ELT tools or preset API interfaces.

[0068] Next, the historical resource data of the machine, historical job scheduling data, and job metadata are summarized to obtain the original training data set. The original training data set is then one-hot encoded, and the feature vector is mapped to the Euclidean space to construct the training data set.

[0069] In one embodiment, the steps of acquiring historical machine resource data and historical job scheduling data, and acquiring job metadata of ETL jobs, and constructing a training data set include:

[0070] Based on a time window or time series, an association analysis is performed on the historical resource data of the machine, the historical job scheduling data and the job metadata, such as using the CORREL function in Excel to measure the linear relationship between two variables, or, through regression analysis, with the help of regression coefficients and R-squared values, a mathematical model between dependent variables and independent variables is established to obtain the relationship between job scheduling and machine resource conditions in the time dimension, and form the training data set to further reduce the difficulty of learning the training data set for subsequent models and reduce the learning workload of the model.

[0071] In one embodiment, the training data set can be further divided into training data and verification data. For example, 70% of the data in the training data set is used as training data and 30% is used as verification data. The training data set and the verification data set are one-hot encoded, and the feature vectors are mapped to the Euclidean space to finally construct the training data set and the verification data set. The training data set is used for learning and training the model, and the verification data set is used for verifying the prediction results of the model.

[0072] Next, feature recognition is performed based on the obtained training data set. Natural language processing technology can be used to convert the training data set into a data form that can be understood and processed by the computer. The distributed representation of machine resources, job scheduling resources, etc. in the training data set is learned based on word embedding technology to obtain the scheduling distribution characteristics.

[0073] In one embodiment, the step of performing feature recognition based on the training data set to obtain scheduling distribution features includes:

[0074] Analyze the historical resource data of the machine to obtain the machine resource distribution characteristics of the current available resources, the current number of concurrent remaining executable jobs, and the number of jobs in the current waiting queue;

[0075] Analyze the historical job scheduling data to obtain the job scheduling characteristics of the priority position of the current job in the waiting queue and the waiting time of the current job in the waiting queue;

[0076] The job metadata is analyzed to obtain job metadata features including job unique ID, job running time, job scheduling time, number of CPU threads required for running, minimum and maximum memory required for running, maximum tolerable delayed execution time of the job, and job priority.

[0077] Among them, the machine resource distribution characteristics of the machine historical resource data include the current available resources (CPU and memory), the current number of concurrent remaining runnable jobs, the number of jobs in the current waiting queue, etc. The job scheduling characteristics of the historical job scheduling data include the priority position of the current job in the waiting queue, the waiting time of the current job in the waiting queue, etc. The job metadata characteristics of the job metadata include the job unique ID, job running time, job scheduling time, the number of CPU threads required for running, the minimum and maximum memory required for running, the maximum delay execution time that the job can tolerate, and the job priority, etc.

[0078] These scheduling distribution features will be used to train the decision tree model.

[0079] In one embodiment, after the step of obtaining the historical resource data and historical job scheduling data of the machine, the method further includes:

[0080] Removing invalid or erroneous historical resource data of the machine, and removing invalid or erroneous historical job scheduling data, to obtain initial historical resource data of the machine and initial historical job scheduling data;

[0081] The initial machine historical resource data and the initial historical job scheduling data are standardized or normalized to obtain machine historical resource data and historical job scheduling data with consistent formats and numerical units, and are updated as the machine historical resource data and the historical job scheduling data.

[0082] By preprocessing the collected historical job scheduling data and machine historical resource data before feature recognition, removing invalid or erroneous log entries, and standardizing or normalizing data with inconsistent formats or inconsistent numerical units, the effectiveness of the training data set can be improved, which is conducive to increasing the amount of learning data for subsequent models and enhancing the generalization ability of the model.

[0083] Furthermore, the scheduling distribution characteristics are input into a preset decision tree model for training and learning, and the ETL job scheduling strategy is output to construct a predictor.

[0084] A CART decision tree is constructed by analyzing the selected scheduling distribution features from the training data set, and the scheduling distribution features are used to train the CART decision tree regression model. It has the advantages of compatibility with high-dimensional data processing, fast model training and prediction, easy adjustment and optimization, explanation and verification, and convenient visualization. It can be applied to the analysis and learning of complex job data developed by large amounts of data in the data center, and use machine learning models to conveniently complete the intelligent scheduling strategy allocation of ETL jobs.

[0085] Specifically, the scheduling distribution feature is divided by a heuristic method, all scheduling distribution features in the current set of each division are examined one by one, and according to the square error minimization criterion, the scheduling distribution feature that satisfies the square error minimization criterion is selected as the target split point of the current set;

[0086] Based on the mean square error minimization criterion, from the target split points of all the investigated sets, the target split point that satisfies the minimum sum of the weighted sub-nodes is selected as the optimal split point;

[0087] A decision tree model is constructed based on the optimal split point, and a target variable of the decision tree model is defined as the optimal offset scheduling time of the job.

[0088] First, define the segmentation variable j and the segmentation point s, χ j represents the target value, and divides all data values ​​into two regions based on the scheduling distribution characteristics: R1(j, s) = {x|x (j) ≤s} and R2(j,s)={x|x (j) >s}, find j and s that minimize the sum of the square errors of the two regions to be divided. The formula is as follows:

[0089]

[0090] Among them, c1 is the sample output mean of R1, c2 is the sample output mean of R2, j is the feature segmentation variable, s is the segmentation point, x i is the horizontal coordinate of the target segmentation point under investigation, y i The vertical coordinate of the target dividing point under investigation.

[0091] All possible split points are evaluated at each node, and the sum of squared errors of the two parts R1 and R2 divided by each feature value are calculated through variable j, and the split point that makes the objective function (cost function) optimal is selected. The objective function uses minimization of squared error as a metric. The split variable j and split point s that minimize the sum of weighted MSE of sub-nodes are selected from all possible splits, which are the optimal split variable and the optimal split point. Finally, based on the above metrics, a decision tree model is constructed from the feature hierarchy, and the target variable of the decision tree model is defined as the optimal offset scheduling time of the job, which ensures the synchronization of local device time and UTC time, thereby improving the accuracy of scheduling execution time and reducing latency. The target variable of the existing decision tree model is the optimal execution time of job scheduling.

[0092] In order to improve the accuracy of the decision tree model and reduce the risk of overfitting, an integrated learning is further introduced on the basis of the above decision tree to form a random forest model to construct a predictor. In one embodiment, the scheduling distribution characteristics are input into a preset decision tree model for training and learning, and the ETL job scheduling strategy is output to construct a predictor. The step also includes:

[0093] A preset number of bootstrap samples are extracted from the scheduling distribution characteristics by sampling with replacement;

[0094] A heuristic method is used to divide each of the self-service samples, all the scheduling distribution features in the current set of each division are examined one by one, and according to a square error minimization criterion, a scheduling distribution feature that satisfies the square error minimization criterion is selected as a target split point of the current set;

[0095] Based on the mean square error minimization criterion, from the target split points of all the investigated sets, the target split point that satisfies the minimum sum of the weighted sub-nodes is selected as the optimal split point;

[0096] A sub-decision tree model of each of the self-service samples is constructed based on the optimal split point, and the target variable of each of the sub-decision tree models is defined as the optimal offset scheduling time of the job.

[0097] By sampling the scheduling distribution features with replacement, a new data subset is extracted, that is, the self-service sample method is used to construct n self-service samples, and each self-service sample constructs a decision tree. When training each node of the decision tree model, some sample features are randomly selected from all sample features on the node, and among these randomly selected sample features, an optimal feature is selected to divide the left rear subtree of the decision tree. Finally, the constructed decision tree model and each sub-decision tree model together constitute a random forest model. By calculating the average prediction results of all decision tree models, that is, the average of the prediction results of the decision tree model and the prediction results of each sub-decision tree model, the final prediction result of the random forest model is obtained, and a predictor is constructed to make the prediction results of the model more accurate and applicable.

[0098] In one embodiment, the step of inputting the scheduling distribution feature into a preset decision tree model for training and learning, outputting the ETL job scheduling strategy, and constructing a predictor further includes:

[0099] When the scheduling distribution features are respectively input into the decision tree model and each of the sub-decision tree models for training, some sample features are randomly selected from all the scheduling distribution features in the current set to determine some sample features of all sets;

[0100] Based on the Gini impurity metric, determine the best splitting node of the left rear subtree of the decision tree model and each of the sub-decision tree models from the partial sample features of all sets;

[0101] The decision tree model and each of the sub-decision tree models are updated, and summarized to obtain a random forest model, wherein the average value of the prediction results of the updated decision tree model and each of the sub-decision tree models is the prediction result of the random forest model.

[0102] By using Gini impurity as a metric, an optimal feature is selected to divide the left and right subtrees of the decision tree. The lower the Gini impurity, the higher the purity of the data feature and the better the training effect of the model. The calculation formula of Gini impurity is as follows:

[0103]

[0104] Among them, p i is the proportion of the i-th type of samples in the total sample data set D, is the sum of squares of all class proportions.

[0105] The feature with the largest decrease in Gini impurity is selected as the best split node to divide the left rear subtree of the decision tree and build each sub-decision tree model.

[0106] In one embodiment, the step of inputting the scheduling distribution feature into a preset decision tree model for training and learning, outputting the ETL job scheduling strategy, and constructing a predictor further includes:

[0107] The scheduling distribution characteristics are input into the random forest model for training and learning, and the ETL job scheduling strategy is output. When the output accuracy of the random forest model meets the preset conditions, the random forest model at this time is used as the predictor.

[0108] Finally, through the preset interface, the notification information of the periodically triggered scheduled ETL job execution sent by the scheduling middleware is received, or various forms of job execution requests submitted from the system interface, API, command line, etc. are received, and the notification information or execution request that triggers the scheduled ETL job is obtained, and the notification information or execution request of the currently received ETL job is converted into the input variable of the model, and the current resource status and parameters of the machine are obtained, which are input into the predictor, and the prediction result is calculated to obtain the prediction result of the ETL job scheduling strategy, and the scheduling system is notified to adjust the corresponding scheduling strategy.

[0109] Each time a job needs to be scheduled, the current resource occupancy information and preset parameters are used as input variables and input into the trained predictor to predict the optimal offset scheduling time of the target variable job, that is, the optimal scheduling execution time of the job, and the output result of the predictor is returned to the job scheduling system to assist in the corresponding scheduling strategy adjustment. Among them, the preset parameters can be job request parameters, including job name, job type, job priority, job scheduling information, job variables, task scheduling, upstream and downstream job relationships, job resource configuration and other parameter information.

[0110] In one embodiment, after obtaining notification information or an execution request that triggers the scheduling of an ETL job, the user authority of the request may be authenticated and the validity of the request parameters may be verified to ensure the security and compliance of the request.

[0111] Reference Figure 1 In one embodiment, a method for intelligently scheduling data development operations further includes the following steps:

[0112] S5: Use the newly acquired machine resource data and job scheduling data to re-derive the decision tree model and update the predictor.

[0113] Specifically, the machine resource collection module and the job scheduling log collection module are used to continuously collect the historical logs of scheduling jobs and the data information of the historical resource usage of the system, and to construct new training data sets and verification data sets. For example, a scheduled task can be set, such as collecting the historical logs of scheduling jobs and the data information of the historical resource usage of the system at a frequency of weekly / monthly, and using the new training data set for feature recognition and model training and learning, and then the model is deduced again for reinforcement learning, and a new predictor is constructed and updated to the intelligent scheduling system.

[0114] In one embodiment, a method for intelligently scheduling data development operations further includes the following steps:

[0115] S6: Continuously monitor and analyze the scheduling of prediction results, further adjust and optimize the decision tree model and update the predictor.

[0116] With the use of intelligent scheduling systems, data development tasks and resource conditions will change over time and with adjustments to system operations. Therefore, it is necessary to continuously monitor new system conditions and adjust the model's strategy in order to continuously provide the latest prediction results that are close to the actual situation and continuously improve the model.

[0117] The performance of the predictor is evaluated by using the validation data set, and the mean square error of the predictor's prediction results is calculated using the validation data set to determine the accuracy of the predictor. The scheduling of the prediction results is continuously monitored and analyzed. According to the actual situation, it is evaluated whether to further manually adjust the algorithm model to optimize the model performance and accuracy, further adjust and optimize the decision tree model and update the predictor.

[0118] In summary, a data development job intelligent scheduling method constructs a training data set by acquiring machine historical resource data and historical job scheduling data, as well as job metadata of ETL jobs, so as to quantify the scheduling resources of ETL jobs in multiple dimensions from ETL job attributes, hardware resources and historical scheduling conditions, and comprehensively consider the balance between the normal progress of ETL jobs and resource allocation efficiency; feature recognition is performed according to the training data set to obtain scheduling distribution features for machine learning; the scheduling distribution features are input into a preset decision tree model for training and learning, and the ETL job scheduling strategy is output to construct a predictor, which has the advantages of being compatible with high-dimensional data processing, fast model training and prediction speed, easy adjustment and optimization, explanation and verification, and convenient visualization; finally, the notification information or execution request that triggers the scheduling of the ETL job is obtained, and the current resource situation of the machine and the preset parameters are input into the predictor to obtain the prediction result of the ETL job scheduling strategy, which can improve the resource allocation efficiency of the ETL job while ensuring the normal progress of the ETL job, so that the ETL job can be executed in a timely and orderly manner according to the needs of the business scenario.

[0119] In addition to conducting dimensional analysis from the perspective of cluster resource conditions, an intelligent scheduling method for data development jobs also comprehensively considers factors such as the business characteristics of ETL jobs, the operating status of upstream job dependencies, job priorities, and the maximum tolerable delayed execution time of jobs. In terms of scheduling adjustment strategies, intelligent scheduling is driven by machine learning, and scheduling strategies are dynamically adjusted through reinforcement learning. This can achieve better scheduling effects, maximize resource availability, and ensure the timeliness of data development job operations without affecting the use of data services or causing business delays, thereby improving the resource allocation efficiency of jobs.

[0120] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0121] Reference Figure 2 The embodiment of the present application also provides a data development operation intelligent scheduling system, which is applied to any one of the data development operation intelligent scheduling methods in the above embodiments. The data development operation intelligent scheduling system includes:

[0122] The machine resource collection module is used to collect the current resource status and historical resource status of the machine in real time and store them in a preset database;

[0123] The scheduling log collection module is used to collect the historical job scheduling status of the system in real time and store it in a preset database;

[0124] The metadata module is used to obtain the job metadata of the ETL job and store it in the preset database;

[0125] The scheduling feature recognition module is used to construct a training data set based on the stored historical machine resource conditions, the stored historical job scheduling conditions and the job metadata, and perform feature recognition to obtain the scheduling distribution characteristics;

[0126] A scheduling learning module, used for inputting the scheduling distribution characteristics into a preset decision tree model for training and learning, and outputting an ETL job scheduling strategy to construct a predictor;

[0127] The scheduling intelligent optimization module is used to obtain the notification information or execution request that triggers the scheduling of ETL jobs, input the current resource status of the machine and the preset parameters into the predictor, and obtain the prediction result of the ETL job scheduling strategy.

[0128] Furthermore, a data development operation intelligent scheduling system also includes:

[0129] The data preprocessing module is used to remove invalid or erroneous machine historical resource data and invalid or erroneous historical job scheduling data to obtain initial machine historical resource data and initial historical job scheduling data; standardize or normalize the initial machine historical resource data and the initial historical job scheduling data to obtain machine historical resource data and historical job scheduling data with consistent format and numerical units.

[0130] Furthermore, a data development operation intelligent scheduling system also includes:

[0131] The job execution receiving module is used to preset an interface to receive notification information or execution requests that trigger the scheduling of ETL jobs.

[0132] Furthermore, a data development operation intelligent scheduling system also includes:

[0133] The model iteration module is used to re-infer the decision tree model and update the predictor using the newly acquired machine resource data and job scheduling data.

[0134] Furthermore, a data development operation intelligent scheduling system also includes:

[0135] The feedback optimization module is used to continuously monitor and analyze the scheduling of prediction results, further adjust and optimize the decision tree model and update the predictor.

[0136] For the specific definition of a data development job intelligent scheduling system, please refer to the definition of a data development job intelligent scheduling method above, which will not be repeated here.

[0137] Each module in the above-mentioned data development operation intelligent scheduling system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0138] A method and system for intelligent scheduling of data development jobs, applied to large-scale data development jobs in the data middle platform, can reasonably combine computer resources and the operating status of ETL jobs for analysis, collect historical system resource occupancy and job historical scheduling log data to form a training data set, and analyze and process the extracted scheduling distribution characteristics, and then use the machine learning model to complete the intelligent scheduling strategy allocation of the job, obtain the best scheduling strategy for the job, and intelligently allocate job scheduling. While improving the utilization rate of system resources, it also improves the business timeliness of the job and ensures the dual stability of the system and data business.

[0139] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, any of the above-mentioned data development job intelligent scheduling methods is implemented.

[0140] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any one of the above-mentioned data development job intelligent scheduling methods is implemented.

[0141] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, any one of the above-mentioned data development job intelligent scheduling methods is implemented.

[0142] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0143] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for intelligent scheduling of data development operations, characterized in that: The following steps are included: Obtain historical machine resource data and historical job scheduling data, and Get the job metadata of the ETL job and build a training data set; Perform feature recognition based on the training data set to obtain scheduling distribution features; Input the scheduling distribution characteristics into a preset decision tree model for training and learning, and output the ETL job scheduling strategy to build a predictor; The notification information or execution request that triggers the scheduling of the ETL job is obtained, and the current resource status of the machine and the preset parameters are input into the predictor to obtain the prediction result of the ETL job scheduling strategy.

2. The intelligent scheduling method for data development operations according to claim 1, characterized in that: The steps of obtaining the historical resource data and historical job scheduling data of the machine, and obtaining the job metadata of the ETL job, and constructing the training data set include: Based on a time window or a time series, the historical machine resource data, the historical job scheduling data and the job metadata are correlated and analyzed to obtain the relationship between the job scheduling and the machine resource situation in the time dimension, thereby forming the training data set.

3. The intelligent scheduling method for data development operations according to claim 1, characterized in that: The step of performing feature recognition according to the training data set to obtain scheduling distribution features includes: Analyze the historical resource data of the machine to obtain the machine resource distribution characteristics of the current available resources, the current number of concurrent remaining executable jobs, and the number of jobs in the current waiting queue; Analyze the historical job scheduling data to obtain the job scheduling characteristics of the priority position of the current job in the waiting queue and the waiting time of the current job in the waiting queue; The job metadata is analyzed to obtain job metadata features including job unique ID, job running time, job scheduling time, number of CPU threads required for running, minimum and maximum memory required for running, maximum tolerable delayed execution time of the job, and job priority.

4. The intelligent scheduling method for data development operations according to claim 1, characterized in that: The steps of inputting the scheduling distribution features into a preset decision tree model for training and learning, and outputting the ETL job scheduling strategy to construct a predictor include: The scheduling distribution feature is divided by a heuristic method, all scheduling distribution features in the current set of each division are examined one by one, and according to a square error minimization criterion, a scheduling distribution feature that satisfies the square error minimization criterion is selected as a target split point of the current set; Based on the mean square error minimization criterion, from the target split points of all the investigated sets, the target split point that satisfies the minimum sum of the weighted sub-nodes is selected as the optimal split point; A decision tree model is constructed based on the optimal split point, and a target variable of the decision tree model is defined as the optimal offset scheduling time of the job.

5. The data development operation intelligent scheduling method according to claim 4 is characterized in that: The step of inputting the scheduling distribution feature into a preset decision tree model for training and learning, outputting the ETL job scheduling strategy, and constructing a predictor also includes: A preset number of bootstrap samples are extracted from the scheduling distribution characteristics by sampling with replacement; A heuristic method is used to divide each of the self-service samples, all the scheduling distribution features in the current set of each division are examined one by one, and according to a square error minimization criterion, a scheduling distribution feature that satisfies the square error minimization criterion is selected as a target split point of the current set; Based on the mean square error minimization criterion, from the target split points of all the investigated sets, the target split point that satisfies the minimum sum of the weighted sub-nodes is selected as the optimal split point; A sub-decision tree model of each of the self-service samples is constructed based on the optimal split point, and the target variable of each of the sub-decision tree models is defined as the optimal offset scheduling time of the job.

6. The method for intelligent scheduling of data development operations according to claim 5, characterized in that: The step of inputting the scheduling distribution feature into a preset decision tree model for training and learning, outputting the ETL job scheduling strategy, and constructing a predictor also includes: When the scheduling distribution features are respectively input into the decision tree model and each of the sub-decision tree models for training, some sample features are randomly selected from all the scheduling distribution features in the current set to determine some sample features of all sets; Based on the Gini impurity metric, determine the best splitting node of the left rear subtree of the decision tree model and each of the sub-decision tree models from the partial sample features of all sets; The decision tree model and each of the sub-decision tree models are updated, and summarized to obtain a random forest model, wherein the average value of the prediction results of the updated decision tree model and each of the sub-decision tree models is the prediction result of the random forest model.

7. The data development operation intelligent scheduling method according to claim 6 is characterized in that: The step of inputting the scheduling distribution feature into a preset decision tree model for training and learning, outputting the ETL job scheduling strategy, and constructing a predictor also includes: The scheduling distribution characteristics are input into the random forest model for training and learning, and the ETL job scheduling strategy is output. When the output accuracy of the random forest model meets the preset conditions, the random forest model at this time is used as the predictor.

8. The method for intelligent scheduling of data development operations according to any one of claims 1 to 7, characterized in that: After the step of obtaining the historical resource data and historical job scheduling data of the machine, the method further includes: Removing invalid or erroneous historical resource data of the machine, and removing invalid or erroneous historical job scheduling data, to obtain initial historical resource data of the machine and initial historical job scheduling data; The initial machine historical resource data and the initial historical job scheduling data are standardized or normalized to obtain machine historical resource data and historical job scheduling data with consistent formats and numerical units, and are updated as the machine historical resource data and the historical job scheduling data.

9. A data development operation intelligent scheduling system, characterized in that: include, The machine resource collection module is used to collect the current resource status and historical resource status of the machine in real time and store them in a preset database; The scheduling log collection module is used to collect the historical job scheduling status of the system in real time and store it in a preset database; The metadata module is used to obtain the job metadata of the ETL job and store it in the preset database; The scheduling feature recognition module is used to construct a training data set based on the stored historical machine resource conditions, the stored historical job scheduling conditions and the job metadata, and perform feature recognition to obtain the scheduling distribution characteristics; A scheduling learning module, used for inputting the scheduling distribution characteristics into a preset decision tree model for training and learning, and outputting an ETL job scheduling strategy to construct a predictor; The scheduling intelligent optimization module is used to obtain the notification information or execution request that triggers the scheduling of ETL jobs, input the current resource status of the machine and the preset parameters into the predictor, and obtain the prediction result of the ETL job scheduling strategy.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images