A method, device and equipment for processing data

Abstract

The present invention provides a data processing method, device and equipment, which relates to the field of computer information processing technology; wherein the method includes: obtaining business data of a task to be scheduled, the business data including: task type, input data volume, preset task processing complexity; determining the priority of the task to be scheduled according to the business data and the preset job order of the task to be scheduled; determining the completion time prediction value of the task to be scheduled according to the business data, the priority and a preset time prediction model, the preset time prediction model is obtained by training a historical business data set based on a preset model; determining the target scheduling strategy of the task to be scheduled according to the priority and the completion time prediction value, and scheduling according to the target scheduling strategy. The scheme provided by the present invention can improve the scheduling efficiency of the task to be scheduled, reduce resource waste, and reduce scheduling complexity.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer information processing, and in particular to a data processing method, device and equipment. Background Art

[0002] As the business complexity increases, the jobs running in the system are becoming more and more complex, and job scheduling faces many problems. For example, as the number and complexity of jobs increase, the complexity of the scheduling algorithm will also increase. When multiple jobs are running at the same time, there will be competition for resources, such as CPU, memory, disk, etc., which will cause the job to run slowly or fail, resulting in high network costs and long time consumption for cross-computer room tasks in multi-cross scenarios. Different jobs may have different priorities. Due to the increase in existing business, the scheduling algorithm is complex and cannot guarantee timely scheduling of high-priority tasks, thus affecting the stability and efficiency of the system. In complex business systems, resource allocation is often uneven. Some critical jobs may require more resources to complete, while some non-critical jobs may only require fewer resources. If resources are unevenly allocated, some jobs will not be completed on time and resources will be wasted, thus affecting the operating efficiency of the entire system. Summary of the invention

[0003] The present invention provides a data processing method, device and equipment, which dynamically processes the business data of the tasks to be scheduled and adjusts the scheduling strategy according to the processing results to improve the scheduling efficiency of the tasks to be scheduled and reduce resource waste.

[0004] In order to solve the above technical problems, an embodiment of the present invention provides a data processing method, including:

[0005] Obtaining business data of the task to be scheduled, the business data including: task type, input data volume, and preset task processing complexity;

[0006] Determine the priority of the task to be scheduled according to the business data and the preset operation order of the task to be scheduled;

[0007] Determine a completion time prediction value of the task to be scheduled according to the business data, the priority and a preset time prediction model, wherein the preset time prediction model is obtained by training a historical business data set based on a preset model;

[0008] According to the priority and the predicted completion time value, a target scheduling strategy for the task to be scheduled is determined, and scheduling is performed according to the target scheduling strategy.

[0009] Optionally, determining the priority of the task to be scheduled according to the business data and a preset operation order of the task to be scheduled includes:

[0010] Determining a predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity;

[0011] Determine a predicted value of the output data volume of the task to be scheduled according to the input data volume;

[0012] The priority of the task to be scheduled is determined according to the predicted value of resource consumption, the predicted value of output data volume and the preset operation sequence of the task to be scheduled.

[0013] Optionally, determining the predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity includes:

[0014] Encoding the task type to obtain a first encoding value corresponding to the task type;

[0015] Encoding the preset task processing complexity to obtain a second encoding value corresponding to the preset task processing complexity;

[0016] The first coding value and the second coding value are input into a trained resource consumption prediction model for prediction processing to obtain a resource consumption prediction value of the task to be scheduled; the resource consumption prediction model is obtained by training the mapping relationship between historical task types, historical task processing complexity and historical task resource consumption based on a preset neural network model.

[0017] Optionally, determining a predicted value of the output data volume of the task to be scheduled according to the input data volume includes:

[0018] The input data volume is input into a trained output data volume prediction model for prediction processing to obtain a predicted value of the output data volume of the task to be scheduled; the output data volume prediction model is obtained by training historical task types, historical task input data volumes, and historical task output data volumes based on a preset gradient boosting model.

[0019] Optionally, determining the priority of the task to be scheduled according to the predicted value of resource consumption, the predicted value of output data volume, and the preset operation order of the task to be scheduled includes:

[0020] Determining a preset operation sequence of the task to be scheduled according to the upstream and downstream dependency correlation coefficients of the task to be scheduled;

[0021] According to the preset operation sequence of the task to be scheduled, the predicted value of resource consumption and the predicted value of output data volume, the target priority of the task to be scheduled is determined in a preset priority policy library.

[0022] Optionally, training the historical business data set based on a preset model to obtain the preset time prediction model includes:

[0023] Segmenting the historical business data set to obtain multiple data blocks of the historical business data set;

[0024] Evenly distributing the plurality of data blocks on a plurality of nodes of a preset distributed computing framework;

[0025] Performing data fitting on the data block on each of the plurality of nodes by using the preset model, and determining the fitting parameters corresponding to each node by using a preset optimization algorithm;

[0026] The fitting parameters corresponding to each node are aggregated to obtain the preset time prediction model.

[0027] Optionally, determining the predicted completion time value of the task to be scheduled according to the business data, the priority and a preset time prediction model includes:

[0028] Encoding the priority to obtain a third encoding value corresponding to the priority;

[0029] The input data volume, the first coding value corresponding to the task type, the second coding value corresponding to the preset task processing complexity, and the third coding value are input into the preset time prediction model for prediction processing to obtain the completion time prediction value.

[0030] An embodiment of the present invention further provides a data processing device, comprising:

[0031] An acquisition module is used to acquire business data of the task to be scheduled, wherein the business data includes: task type, input data volume, and preset task processing complexity;

[0032] A processing module is used to determine the priority of the task to be scheduled based on the business data and the preset job order of the task to be scheduled; determine the completion time prediction value of the task to be scheduled based on the business data, the priority and a preset time prediction model, wherein the preset time prediction model is obtained by training a historical business data set based on a preset model; determine the target scheduling strategy of the task to be scheduled based on the priority and the completion time prediction value, and schedule it according to the target scheduling strategy.

[0033] An embodiment of the present invention further provides a computing device, comprising: a processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed.

[0034] An embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0035] The above solution of the present invention includes at least the following beneficial effects:

[0036] The above-mentioned scheme of the present invention obtains the business data of the task to be scheduled, and the business data includes: task type, input data volume, preset task processing complexity; determines the priority of the task to be scheduled according to the business data and the preset job order of the task to be scheduled; determines the completion time prediction value of the task to be scheduled according to the business data, the priority and the preset time prediction model, and the preset time prediction model is obtained by training the historical business data set based on the preset model; determines the target scheduling strategy of the task to be scheduled according to the priority and the completion time prediction value, and schedules it according to the target scheduling strategy, thereby improving the scheduling efficiency of the task to be scheduled and reducing resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0038] Figure 2 It is a module block diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0040] First of all, it should be explained that the method described in the embodiments of the present invention can be applied to water operation and maintenance scheduling systems and water conservancy scenarios, and the task to be scheduled can be a water supply task for different set areas at a set time point; by processing the business data corresponding to the current water supply task to be scheduled, and considering the upstream and downstream sequence relationship of the current water supply task to be scheduled, the priority and completion time prediction value of the current water supply task to be scheduled are determined, and based on the priority and completion time prediction value, the corresponding target scheduling strategy is determined, and further scheduling is performed according to the target scheduling strategy, which can ensure the scheduling efficiency of the current water supply task to be scheduled and the rationality of resource allocation, thereby improving the operating efficiency of the overall system and the water resources management effect.

[0041] like Figure 1 As shown, an embodiment of the present invention provides a data processing method, including:

[0042] Step 11, obtaining the business data of the task to be scheduled, wherein the business data includes: task type, input data volume, and preset task processing complexity;

[0043] Step 12, determining the priority of the task to be scheduled according to the business data and the preset operation order of the task to be scheduled;

[0044] Step 13, determining a completion time prediction value of the to-be-scheduled task according to the business data, the priority, and a preset time prediction model, wherein the preset time prediction model is obtained by training a historical business data set based on a preset model;

[0045] Step 14: Determine the target scheduling strategy of the task to be scheduled according to the priority and the predicted completion time value, and schedule the task according to the target scheduling strategy.

[0046] In this embodiment, after the business data is acquired, feature classification processing can be performed on the business data. The business data after feature classification processing may include: task type, corresponding input data volume, preset task processing complexity and other feature data. The feature data can reflect the characteristics of the current task to be scheduled and its demand for scheduling resources. Here, the task type can determine the consumption mode of scheduling resources, the input data volume can directly affect the time for processing the current task to be scheduled, resource requirements and output data volume after the task is completed, and the preset task processing complexity can reflect the complexity of the internal logic of the current task.

[0047] Further, based on the characteristic data corresponding to the business data, analysis and processing are performed, and according to the results of the analysis and processing, the priority and completion time prediction value of the current task to be scheduled are determined in turn, and based on the priority and the completion time prediction value, a basis is provided for determining the target scheduling strategy of the current task to be scheduled;

[0048] Here, the prediction time prediction model can be obtained by training the historical business data set based on the preset model. Preferably, the preset model can be a distributed enhanced multivariate linear regression model; the distributed enhanced multivariate linear regression model realizes the prediction of completion time and has the advantages of processing large-scale data, high efficiency, scalability, flexibility, reliability and economy. The distributed enhanced multivariate linear regression model can accelerate the training of the preset time model and the calculation of the predicted task time. At the same time, it can process large-scale data sets to achieve the accuracy and stability of the completion time prediction of the scheduled tasks, thereby providing support for data operation and maintenance guarantee and subsequent data processing strategies, and ensuring the efficiency of task scheduling and processing;

[0049] According to the priority and the predicted value of the completion time, the target scheduling strategy of the task to be scheduled is determined, and scheduling is performed according to the target scheduling strategy. The tasks to be scheduled can be reasonably allocated to different processors or servers to avoid overloading of certain processors or servers, thereby improving the performance of the entire scheduling system; according to the priority of the task to be scheduled, the actual execution order of the task is reasonably arranged to meet the user's real-time requirements and ensure the processing efficiency of the task to be scheduled.

[0050] In an optional embodiment of the present invention, the above step 12 may include:

[0051] Step 121, determining a predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity;

[0052] Step 122, determining a predicted value of the output data volume of the task to be scheduled according to the input data volume;

[0053] Step 123, determining the priority of the task to be scheduled according to the predicted value of resource consumption, the predicted value of output data volume and the preset operation sequence of the task to be scheduled.

[0054] In this embodiment, the task type can determine the consumption mode of the scheduling resources, and the preset task processing complexity can reflect the complexity of the internal logic of the current task; through the task type and the preset task complexity, the resource consumption prediction value required for the current scheduling task can be determined, which provides a basis for the subsequent determination of the priority of the current task to be scheduled and the allocation of system resources, ensures the rationality of resource allocation, and reduces resource waste; the resource consumption mentioned here can be the network resources of the entire scheduling system, such as bandwidth, memory, etc.; through the resource consumption prediction value, the allocation of network resources can be optimized, the bandwidth utilization rate can be improved, and a more stable and smooth network service can be guaranteed, thereby improving the user experience;

[0055] The input data volume can directly affect the time and resource requirements for processing the current task to be scheduled, as well as the output data volume after the task is completed. By analyzing the input data volume of the current task to be scheduled, the output data volume forecast value of the current task to be scheduled is determined;

[0056] Further, based on the parameter data volume prediction value and the resource consumption prediction value, the priority of the current task to be scheduled can be determined to ensure priority execution of key tasks and resource allocation, thereby improving the overall operational efficiency and management effect of the system.

[0057] In an optional embodiment of the present invention, the above step 121 may include:

[0058] Step 1211, encoding the task type to obtain a first encoding value corresponding to the task type;

[0059] Step 1212, encoding the preset task processing complexity to obtain a second encoding value corresponding to the preset task processing complexity;

[0060] Step 1213, input the first coded value and the second coded value into a trained resource consumption prediction model for prediction processing to obtain the resource consumption prediction value of the task to be scheduled; the resource consumption prediction model is obtained by training the mapping relationship between historical task types, historical task processing complexity and historical task resource consumption based on a preset neural network model.

[0061] Here, the task type and the preset task processing complexity are encoded respectively to obtain the corresponding first encoding value and second encoding value, so as to realize the conversion of the categorical variable (task type, preset task processing complexity) to the numerical variable, thereby facilitating the subsequent prediction of resource consumption; here, preferably, the task type and the preset task processing complexity can be encoded by one-hot encoding to obtain the encoding values ​​corresponding to different categorical variables;

[0062] Furthermore, the first coding value and the second coding value are input into a trained resource consumption prediction model for prediction processing, thereby obtaining a resource consumption prediction value of the task to be scheduled; the resource consumption prediction model can be obtained by training a historical business data set based on the preset neural network model, and the specific training process of the resource consumption prediction model is as follows:

[0063] Step 31, obtaining a historical business data set: cleaning and preprocessing the data in the historical business data set (including missing value processing, normalization, etc.), and performing feature classification processing on the preprocessed historical business data to obtain historical task types, historical task processing complexity, and historical task resource consumption;

[0064] Step 32, encoding the historical task type and the historical task processing complexity to obtain corresponding encoding values, and dividing the corresponding encoding values ​​and the historical task resource consumption into a training data set and a verification data set;

[0065] Step 33, inputting the training data set into a preset neural network for training, wherein the preset neural network model may be a convolutional neural network model, and performing iterative training processing through a gradient back propagation algorithm to obtain the resource consumption prediction model;

[0066] Step 331, inputting the training data set into the convolution layer of the preset neural network model, performing convolution pooling processing on the data in the training data set according to the convolution kernels of preset size and number, to obtain a first feature data set;

[0067] Step 332: input the first feature data set into the fully connected layer of the preset convolutional neural network model, perform connection processing according to the preset residual value, and obtain the resource consumption prediction model and training optimization result;

[0068] Step 34, model evaluation and tuning: Use the validation set data to adjust model parameters, evaluate model performance, and perform model tuning as needed;

[0069] In the process of iterative training of the training data set by the preset neural network model, each iteration, the parameters of the preset loss function and the preset dynamic learning rate adjustment function of the preset neural network model are updated and adjusted by the gradient back propagation algorithm to obtain a resource consumption prediction model after an iterative update;

[0070] Use the validation data set to evaluate the predictive performance of the model. Here, the cross-validation algorithm can be used to evaluate the generalization ability of the model on different data sets, reduce the risk of overfitting, and obtain more stable performance evaluation results through multiple validations. By adjusting the internal parameters of the model (such as learning rate, regularization coefficient, etc.), find the optimal configuration so that the model performs best on the validation set, thereby improving the accuracy of the model.

[0071] By analyzing and training historical business data, a resource consumption prediction model is obtained to provide support for the priority prediction of subsequent tasks to be scheduled, thereby improving the efficiency of task scheduling; based on the predicted value of resource consumption, the resource allocation of the scheduling system can be optimized and resource waste can be reduced.

[0072] In an optional embodiment of the present invention, the above step 122 may include:

[0073] Step 1221, input the input data volume into the trained output data volume prediction model for prediction processing to obtain the output data volume prediction value of the task to be scheduled; the output data volume prediction model is obtained by training the historical task type, historical task input data volume and historical task output data volume based on a preset gradient boosting model.

[0074] When the task type of the task to be scheduled is determined, the execution environment corresponding to the scheduling or execution of the task should be determined, and the input data volume and the corresponding execution environment are input into the trained output data volume prediction model for prediction processing, thereby obtaining the output data volume prediction value of the task to be scheduled;

[0075] Here, the specific process of training the historical task type, the historical task input data volume, and the historical task output data volume based on the preset gradient boosting model to obtain the output data volume prediction model is as follows:

[0076] Step 41, obtaining a historical business data set: cleaning and preprocessing the data in the historical business data set (including missing value processing, normalization, etc.), and performing feature classification processing on the preprocessed historical business data to obtain historical task types, historical input data volumes, execution environments corresponding to historical tasks, and corresponding historical output data volumes;

[0077] Step 42, encoding the historical task type to obtain a corresponding encoding value, and dividing the corresponding encoding value, the amount of historical input data, the execution environment corresponding to the historical task, and the amount of historical output data into a training data set and a verification data set;

[0078] Step 43, model training: input the coding value corresponding to the historical task type in the training data set, the amount of historical input data, the execution environment corresponding to the historical task, and the corresponding amount of historical output data into at least one weak learner of the preset gradient boosting model to train the first CART tree, and then use this tree to predict the training set to obtain the predicted value of each training sample. Since there is a deviation between the predicted value and the true value of the verification sample, the residual can be obtained by subtracting the two; next, the second CART tree is trained. At this time, the true value of the verification sample obtained by the initial historical data division is no longer used, but the residual of the training of the first CART tree is used as the true value of the verification sample at this time; after the training of the two trees is completed, the residual of each sample can be obtained again, and then the third tree can be further trained, and so on; the total number of CART trees can be specified manually, or the training can be stopped by monitoring preset indicators (such as the error on the verification data set). When predicting a new sample, each tree will have an output value, and these output values ​​are added to obtain the final predicted value of the sample;

[0079] The specific training process is:

[0080] Step 431, inputting the data in the training data set into the input layer of the gradient boosting model for feature extraction processing to obtain corresponding training feature data;

[0081] Step 432, inputting the training feature data into at least one weak learner of the preset network model, and adjusting model parameters, which may include tree depth, learning rate, and number of iterations;

[0082] Step 433, iterative processing is performed using the data in the training feature data set, and a new decision tree is generated in each iteration. The specific steps of each iteration process include:

[0083] Step 4331, before each iteration starts, calculate the first-order derivative and the second-order derivative of the loss function at each training sample point;

[0084] Step 4332, generating a new decision tree by a greedy strategy, and calculating the prediction value corresponding to each leaf node by the parameter value corresponding to the leaf node;

[0085] Step 4333, adding the newly generated decision tree to the training model;

[0086] Step 434, determine whether the loss function reaches the minimum, if so, execute step 435, if not, execute step 433;

[0087] Step 435, input the data in the verification data set into the training model, calculate the evaluation index, and determine whether the evaluation index meets the required value. If so, save the output data volume prediction model; if not, execute step 432;

[0088] Step 44, model evaluation and verification: Use the coding value corresponding to the historical task type in the verification data set, the historical input data volume, the execution environment corresponding to the historical task, and the corresponding historical output data volume to evaluate the prediction performance of the model to ensure that the model has a high prediction accuracy;

[0089] In the process of iteratively calculating the training feature data set in the preset gradient boosting model, the network will calculate the gradient of the batch loss relative to the weight and update the weight accordingly; after multiple rounds of calculation, the network loss value will become small enough; the weights are trained through the gradient back propagation algorithm to enhance the performance of the model's self-learning, and the parameters of the output data volume prediction model are adjusted through ten-fold cross validation to further improve the accuracy of the model's prediction processing;

[0090] Preferably, the average relative error MAPE can be used as an evaluation index to evaluate the prediction accuracy of the output data volume prediction model; when the value of the average relative error MAPE is greater than a preset value, it means that the error of the prediction model is large and needs to be retrained;

[0091] The mean relative error MAPE can be calculated by the following formula: ; Where n represents the number of samples, which is a positive integer. Indicates the actual value of the output data volume of the i-th sample; Represents the predicted value of the output data volume of the i-th sample.

[0092] Through the above steps, an output data volume prediction model that shows the correlation between input data volume and output data volume is constructed and obtained, and then according to the input data volume of the current task to be scheduled, its corresponding output data volume prediction value is determined to provide support for task scheduling and resource allocation; at the same time, combined with the resource consumption prediction value of the current task to be scheduled determined by the resource consumption prediction model, task priority matching can be achieved, thereby improving the work efficiency of scheduling tasks, optimizing resource allocation, reducing costs, improving decision-making quality and enhancing competitiveness.

[0093] In an optional embodiment of the present invention, the above step 123 may include:

[0094] Step 1231, determining a preset operation sequence of the task to be scheduled according to the upstream and downstream dependency correlation coefficients of the task to be scheduled;

[0095] Step 1232, determining the target priority of the task to be scheduled in a preset priority policy library according to the preset operation sequence of the task to be scheduled, the predicted value of resource consumption, and the predicted value of output data volume.

[0096] In this embodiment, for the tasks to be scheduled, the preset operation order of the tasks to be scheduled can be determined according to the upstream and downstream dependency relationship of the current tasks to be scheduled;

[0097] Furthermore, based on the preset job sequence, the predicted value of resource consumption (the resource usage of the current system may also need to be considered), and the predicted value of the output data volume, a comprehensive match is performed in a pre-established preset priority policy library containing multiple priority matching rules to determine the target priority of the current task to be scheduled; matching and determining the target priority relies on a predefined series of rules, which can be determined based on different attributes of the task. The whole process is intuitive and easy to implement, and is suitable for scheduling environments with limited resources or high real-time requirements; preferably, the target priority strategy can also be obtained by searching for the best solution through an optimization algorithm, such as a multi-objective optimization algorithm to find the optimal solution (this algorithm is a prior art and will not be described here).

[0098] The priority of the current task to be scheduled is determined by the resource consumption prediction value determined by the resource consumption prediction model and the output data volume prediction value determined by the output data volume prediction model, so as to optimize the execution order and resource allocation of the tasks to be scheduled and improve the overall efficiency and performance of the system.

[0099] In an optional embodiment of the present invention, the preset time prediction model is obtained by training the historical business data set based on the preset model, including:

[0100] Step 21, segmenting the historical business data set to obtain multiple data blocks of the historical business data set;

[0101] Step 22, evenly distributing the plurality of data blocks on a plurality of nodes of a preset distributed computing framework;

[0102] Step 23, performing data fitting on the data block on each of the plurality of nodes by using the preset model, and determining the fitting parameters corresponding to each node by using a preset optimization algorithm;

[0103] Step 24, aggregating the fitting parameters corresponding to each node to obtain the preset time prediction model.

[0104] In this embodiment, the preset model may be a distributed enhanced multivariate linear regression model, and the preset distributed computing framework may be an Apache Spark distributed computing framework;

[0105] Before the historical business data in the historical business data set is segmented, the data in the historical business data set can be cleaned and preprocessed (including missing value processing, normalization, etc.), and the preprocessed historical business data is subjected to feature classification processing to obtain the historical task type, historical task processing complexity, historical task resource consumption, and historical task completion time; the segmentation processing here is to divide the preprocessed historical business data into historical business data corresponding to a single historical task, and after the division, encode the historical task type and historical task complexity corresponding to each historical task to obtain the corresponding first encoding value and second encoding value;

[0106] Furthermore, the historical business data corresponding to each encoded historical task is clustered to obtain multiple clusters (here, the k-means clustering algorithm can be used for processing), and the multiple clusters are evenly distributed on multiple nodes of the ApacheSpark distributed computing framework, and model training is performed on the nodes; here, historical tasks with similar historical business data in clustering processing time are divided together to facilitate the accuracy of model training and ensure the accuracy of subsequent completion time prediction;

[0107] For each node of the Apache Spark distributed computing framework, a multivariate linear regression model can be used for data fitting training; the multivariate linear regression model can be expressed as:

[0108] T=β1×B1+β2×B2+β3×Resource+β4×T completion +⋯+ε

[0109] Where T represents the predicted completion time; B1 represents the first coded value corresponding to the historical task type; B2 represents the second coded value corresponding to the historical task processing complexity; Resource represents the historical task resource consumption; T completion represents the historical task completion time, β i is the regression coefficient (model parameter), i=1, 2, 3, 4, ..., n (n is a positive integer), and ε represents the error term; here, the fitting parameters (regression coefficient β i );

[0110] For each node, the corresponding fitting parameters (regression coefficient β i ), perform aggregation processing to obtain the model parameters of the preset time prediction model ; It can be calculated by the following formula:

[0111] ;

[0112] Furthermore, for the corresponding type of business data on each node, the fitting parameters (regression coefficient β i ), you can set a weight value, which indicates the representativeness of the current type of business data (it has a greater weight in the final model) to adjust the model parameters, specifically:

[0113] ;

[0114] in, Indicates the set weight value; represents the finalized model parameters;

[0115] According to the above finally determined model parameters, the preset time prediction model is obtained.

[0116] In an optional embodiment of the present invention, the above step 13 may include:

[0117] Step 131, encoding the priority to obtain a third encoding value corresponding to the priority;

[0118] Step 132, input the input data volume, the first coding value corresponding to the task type, the second coding value corresponding to the preset task processing complexity, and the third coding value into the preset time prediction model for prediction processing to obtain the time prediction value.

[0119] In this embodiment, the priority is encoded to obtain a corresponding third encoded value to achieve conversion from a categorical variable to a numerical variable, so as to facilitate subsequent prediction of the completion time; preferably, the priority can be encoded by one-hot encoding;

[0120] Furthermore, the third coding value, the input data volume, the first coding value corresponding to the task type, and the second coding value corresponding to the preset task processing complexity are input into the preset time prediction model for prediction, thereby obtaining the completion time prediction value; the task completion time is predicted by the time prediction model obtained by training the distributed enhanced multivariate linear regression model, which can improve the accuracy and stability of the prediction, thereby providing support for data operation and maintenance assurance and subsequent data processing strategies, thereby better guiding actual production and decision-making.

[0121] In an optional embodiment of the present invention, the above step 14 may include:

[0122] Step 141, determining the first scheduling order of the tasks to be scheduled according to the target priority;

[0123] Step 142, determining a second scheduling order of the tasks to be scheduled according to the predicted completion time values;

[0124] Step 143: determine the target scheduling strategy according to the first scheduling order and the second scheduling order, and perform scheduling according to the target scheduling strategy.

[0125] In this embodiment, the first scheduling order considers the priority ranking of the tasks to be scheduled, and the second scheduling order considers the completion time relationship between the tasks to be scheduled; the target scheduling strategy may be the specific scheduling order of the current tasks to be scheduled; when there are multiple tasks to be scheduled in the system, the specifics are as follows:

[0126] Task A: water quality testing, target priority P=3 (high), predicted completion time is 2 hours;

[0127] Task B: Filtering, target priority P=2 (medium), predicted completion time is 4 hours;

[0128] Task C: Disinfection operation, target priority P=1 (low), completion time prediction value is 1 hour;

[0129] Task D: Emergency repair, target priority P=4 (highest), predicted completion time is 3 hours;

[0130] First, determine the first scheduling order: sort the tasks according to the target priority. Here, task D has the highest priority, so the first scheduling order of task D is 1. And so on: the first scheduling order of task A is 2, the first scheduling order of task B is 3, and the first scheduling order of task C is 4.

[0131] Further, the second scheduling order is determined: according to the first scheduling order and the completion time prediction value corresponding to each task, if the absolute value of the difference between the completion time prediction values ​​of two adjacent tasks is greater than or equal to a preset threshold, the first scheduling order corresponding to the two adjacent tasks is respectively confirmed as the corresponding second scheduling order; if the absolute value of the difference between the completion time prediction values ​​of two adjacent tasks is less than a preset threshold, the second scheduling order corresponding to the two adjacent tasks can be determined as the same scheduling order; here, the size of the preset threshold can be set according to the actual situation; if the preset threshold is set to 1, the absolute value of the difference between the completion time prediction values ​​of task D and task A is 1, the absolute value of the difference between the completion time prediction values ​​of task A and task B is 2, and the absolute value of the difference between the completion time prediction values ​​of task B and task C is 3, then, the second scheduling order of task A and task D is 1, the second scheduling order of task B is 2, and the second scheduling order of task C is 3. Further, the target scheduling strategy is: while scheduling and executing task D, task A can be started in parallel; when one of task D or A is scheduled and executed, task B is started to be scheduled and executed; when task B is scheduled and executed, task C is started to be scheduled and executed;

[0132] The above-mentioned embodiment of the present invention obtains the business data of the task to be scheduled, and the business data includes: task type, input data volume, preset task processing complexity; determines the priority of the task to be scheduled according to the business data and the preset job order of the task to be scheduled; determines the completion time prediction value of the task to be scheduled according to the business data, the priority and the preset time prediction model, and the preset time prediction model is obtained by training the historical business data set based on the preset model; determines the target scheduling strategy of the task to be scheduled according to the priority and the completion time prediction value, and schedules it according to the target scheduling strategy, and improves the scheduling efficiency of the task to be scheduled and reduces resource waste through dynamic monitoring and adjustment.

[0133] like Figure 2 As shown, an embodiment of the present invention further provides a data processing device 20, comprising:

[0134] The acquisition module 21 is used to acquire the business data of the task to be scheduled, and the business data includes: task type, input data volume, and preset task processing complexity;

[0135] The processing module 22 is used to determine the priority of the task to be scheduled according to the business data and the preset job order of the task to be scheduled; determine the completion time prediction value of the task to be scheduled according to the business data, the priority and the preset time prediction model, and the preset time prediction model is obtained by training the historical business data set based on the preset model; determine the target scheduling strategy of the task to be scheduled according to the priority and the completion time prediction value, and schedule it according to the target scheduling strategy.

[0136] Optionally, the processing module 22 is used to determine the priority of the task to be scheduled according to the business data and the preset operation order of the task to be scheduled, including:

[0137] Determining a predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity;

[0138] Determine a predicted value of the output data volume of the task to be scheduled according to the input data volume;

[0139] The priority of the task to be scheduled is determined according to the predicted value of resource consumption, the predicted value of output data volume and the preset operation sequence of the task to be scheduled.

[0140] Optionally, the processing module 22 is used to determine the predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity, including:

[0141] Encoding the task type to obtain a first encoding value corresponding to the task type;

[0142] Encoding the preset task processing complexity to obtain a second encoding value corresponding to the preset task processing complexity;

[0143] The first coding value and the second coding value are input into a trained resource consumption prediction model for prediction processing to obtain a resource consumption prediction value of the task to be scheduled; the resource consumption prediction model is obtained by training the mapping relationship between historical task types, historical task processing complexity and historical task resource consumption based on a preset neural network model.

[0144] Optionally, the processing module 22 is used to determine the output data volume prediction value of the task to be scheduled according to the input data volume, including:

[0145] The input data volume is input into a trained output data volume prediction model for prediction processing to obtain a predicted value of the output data volume of the task to be scheduled; the output data volume prediction model is obtained by training historical task types, historical task input data volumes, and historical task output data volumes based on a preset gradient boosting model.

[0146] Optionally, the processing module 22 is used to determine the priority of the task to be scheduled according to the predicted value of resource consumption, the predicted value of output data volume and the preset operation sequence of the task to be scheduled, including:

[0147] Determining a preset operation sequence of the task to be scheduled according to the upstream and downstream dependency correlation coefficients of the task to be scheduled;

[0148] According to the preset operation sequence of the task to be scheduled, the predicted value of resource consumption and the predicted value of output data volume, the target priority of the task to be scheduled is determined in a preset priority policy library.

[0149] Optionally, training the historical business data set based on a preset model to obtain the preset time prediction model includes:

[0150] Segmenting the historical business data set to obtain multiple data blocks of the historical business data set;

[0151] Evenly distributing the plurality of data blocks on a plurality of nodes of a preset distributed computing framework;

[0152] Performing data fitting on the data block on each of the plurality of nodes by using the preset model, and determining the fitting parameters corresponding to each node by using a preset optimization algorithm;

[0153] The fitting parameters corresponding to each node are aggregated to obtain the preset time prediction model.

[0154] Optionally, the processing module 22 is used to determine the predicted completion time value of the task to be scheduled according to the business data, the priority and a preset time prediction model, including:

[0155] Encoding the priority to obtain a third encoding value corresponding to the priority;

[0156] The input data volume, the first coding value corresponding to the task type, the second coding value corresponding to the preset task processing complexity, and the third coding value are input into the preset time prediction model for prediction processing to obtain the completion time prediction value.

[0157] It should be noted that the device is a device corresponding to the data processing method described above, and all implementation methods in the above method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.

[0158] The embodiment of the present invention further provides a computing device, including: a processor, a memory storing a computer program, and when the computer program is executed by the processor, the method described in the above embodiment is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0159] In an embodiment of the present invention, a computer-readable storage medium is further provided, which stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the above embodiment. All implementations in the above method embodiment are applicable to this embodiment, and can also achieve the same technical effect.

[0160] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0162] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0165] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0166] In addition, it should be pointed out that in the apparatus and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. Moreover, the steps of performing the above series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it is understandable that all or any steps or components of the method and apparatus of the present invention can be implemented in hardware, firmware, software or a combination thereof in any computing device (including processors, storage media, etc.) or a network of computing devices, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.

[0167] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing device. The computing device can be a well-known general device. Therefore, the purpose of the present invention can also be achieved by simply providing a program product containing a program code that implements the method or device. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order. Some steps can be performed in parallel or independently of each other.

[0168] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtaining business data of the task to be scheduled, the business data including: task type, input data volume, and preset task processing complexity; Determine the priority of the task to be scheduled according to the business data and the preset operation order of the task to be scheduled; Determine a completion time prediction value of the task to be scheduled according to the business data, the priority and a preset time prediction model, wherein the preset time prediction model is obtained by training a historical business data set based on a preset model; Determine the target scheduling strategy of the task to be scheduled according to the priority and the predicted completion time value, and schedule the task according to the target scheduling strategy; Wherein, determining the priority of the task to be scheduled according to the business data and the preset operation order of the task to be scheduled includes: Determining a predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity; Determine a predicted value of the output data volume of the task to be scheduled according to the input data volume; Determine the priority of the task to be scheduled according to the predicted value of resource consumption, the predicted value of output data volume and the preset operation order of the task to be scheduled; Wherein, determining the output data volume prediction value of the task to be scheduled according to the input data volume includes: Inputting the input data volume into a trained output data volume prediction model for prediction processing to obtain a predicted value of the output data volume of the task to be scheduled; the output data volume prediction model is obtained by training the historical task type, the historical task input data volume, and the historical task output data volume based on a preset gradient boosting model; The output data volume prediction model is obtained by training the historical task type, the historical task input data volume, and the historical task output data volume based on the preset gradient boosting model, including: Obtaining historical business data sets: Cleaning and preprocessing the data in the historical business data sets, and performing feature classification processing on the preprocessed historical business data to obtain historical business data sets. Historical task types, historical input data volume, execution environments corresponding to historical tasks, and corresponding historical output data volume; Encode the historical task type to obtain a corresponding encoding value, and divide the corresponding encoding value, the amount of historical input data, the execution environment corresponding to the historical task, and the corresponding amount of historical output data into a training data set and a verification data set; Model training: Input the encoding value corresponding to the historical task type in the training data set, the historical input data volume, the execution environment corresponding to the historical task, and the corresponding historical output data volume into at least one weak learner of the preset gradient boosting model to train the first CART tree, and then use this tree to predict the training set to obtain the predicted value of each training sample, and subtract the predicted value from the true value of the verification sample to obtain the residual; next, when training the second CART tree, use the residual of training the first CART tree as the true value of the verification sample at this time; after the training of the two trees is completed, the residual of each sample is obtained again, and then the third tree is further trained, and so on; when predicting a new sample, each tree will have an output value, and the output values ​​are added to obtain the final predicted value of the sample; Model evaluation and verification: Use the encoding values ​​corresponding to the historical task types in the verification data set, the amount of historical input data, the execution environment corresponding to the historical tasks, and the corresponding amount of historical output data to evaluate the model's predictive performance and ensure that the model has high predictive accuracy.

2. The data processing method according to claim 1, characterized in that: Determining a predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity includes: Encoding the task type to obtain a first encoding value corresponding to the task type; Encoding the preset task processing complexity to obtain a second encoding value corresponding to the preset task processing complexity; The first coding value and the second coding value are input into a trained resource consumption prediction model for prediction processing to obtain a resource consumption prediction value of the task to be scheduled; the resource consumption prediction model is obtained by training the mapping relationship between historical task types, historical task processing complexity and historical task resource consumption based on a preset neural network model.

3. The data processing method according to claim 1, characterized in that: Determining the priority of the task to be scheduled according to the predicted value of resource consumption, the predicted value of output data volume, and the preset operation order of the task to be scheduled includes: Determining a preset operation sequence of the task to be scheduled according to the upstream and downstream dependency correlation coefficients of the task to be scheduled; According to the preset operation sequence of the task to be scheduled, the predicted value of resource consumption and the predicted value of output data volume, the target priority of the task to be scheduled is determined in a preset priority policy library.

4. The data processing method according to claim 1, characterized in that: The preset time prediction model is obtained by training the historical business data set based on the preset model, including: Segmenting the historical business data set to obtain multiple data blocks of the historical business data set; Evenly distributing the plurality of data blocks on a plurality of nodes of a preset distributed computing framework; Performing data fitting on the data block on each of the plurality of nodes by using the preset model, and determining the fitting parameters corresponding to each node by using a preset optimization algorithm; The fitting parameters corresponding to each node are aggregated to obtain the preset time prediction model.

5. The data processing method according to claim 2, characterized in that: Determining the predicted completion time value of the task to be scheduled according to the business data, the priority and a preset time prediction model includes: Encoding the priority to obtain a third encoding value corresponding to the priority; The input data volume, the first coding value corresponding to the task type, the second coding value corresponding to the preset task processing complexity, and the third coding value are input into the preset time prediction model for prediction processing to obtain the completion time prediction value.

6. A data processing device, characterized in that: include: An acquisition module is used to acquire business data of the task to be scheduled, wherein the business data includes: task type, input data volume, and preset task processing complexity; A processing module, configured to determine the priority of the task to be scheduled according to the business data and the preset operation sequence of the task to be scheduled; determine the completion time prediction value of the task to be scheduled according to the business data, the priority and a preset time prediction model, wherein the preset time prediction model is obtained by training a historical business data set based on a preset model; determine the target scheduling strategy of the task to be scheduled according to the priority and the completion time prediction value, and schedule the task according to the target scheduling strategy; Wherein, determining the priority of the task to be scheduled according to the business data and the preset operation order of the task to be scheduled includes: Determining a predicted value of resource consumption of the task to be scheduled according to the task type and the preset task processing complexity; Determine a predicted value of the output data volume of the task to be scheduled according to the input data volume; Determine the priority of the task to be scheduled according to the predicted value of resource consumption, the predicted value of output data volume and the preset operation order of the task to be scheduled; Wherein, determining the output data volume prediction value of the task to be scheduled according to the input data volume includes: Inputting the input data volume into a trained output data volume prediction model for prediction processing to obtain a predicted value of the output data volume of the task to be scheduled; the output data volume prediction model is obtained by training the historical task type, the historical task input data volume, and the historical task output data volume based on a preset gradient boosting model; The output data volume prediction model is obtained by training the historical task type, the historical task input data volume, and the historical task output data volume based on the preset gradient boosting model, including: Obtaining historical business data sets: Cleaning and preprocessing the data in the historical business data sets, and performing feature classification processing on the preprocessed historical business data to obtain historical business data sets. Historical task types, historical input data volume, execution environments corresponding to historical tasks, and corresponding historical output data volume; Encode the historical task type to obtain a corresponding encoding value, and divide the corresponding encoding value, the amount of historical input data, the execution environment corresponding to the historical task, and the corresponding amount of historical output data into a training data set and a verification data set; Model training: Input the encoding value corresponding to the historical task type in the training data set, the historical input data volume, the execution environment corresponding to the historical task, and the corresponding historical output data volume into at least one weak learner of the preset gradient boosting model to train the first CART tree, and then use this tree to predict the training set to obtain the predicted value of each training sample, and subtract the predicted value from the true value of the verification sample to obtain the residual; next, when training the second CART tree, use the residual of training the first CART tree as the true value of the verification sample at this time; after the training of the two trees is completed, the residual of each sample is obtained again, and then the third tree is further trained, and so on; when predicting a new sample, each tree will have an output value, and the output values ​​are added to obtain the final predicted value of the sample; Model evaluation and verification: Use the encoding values ​​corresponding to the historical task types in the verification data set, the amount of historical input data, the execution environment corresponding to the historical tasks, and the corresponding amount of historical output data to evaluate the model's predictive performance and ensure that the model has high predictive accuracy.

7. A computing device, characterized in that include: A processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 5 is performed.

8. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images