Task scheduling method of node cluster and related equipment

Abstract

The task scheduling method of the node cluster and the related equipment provided by the embodiments of the present application, the node cluster comprising a master node and a plurality of sub-nodes, the method comprising: firstly, obtaining a plurality of processing tasks, inputting each processing task into an execution time prediction model one by one to obtain the predicted execution time of the processing task; then, determining the initial target task of each sub-node based on the predicted execution time of each processing task; finally, generating the first task scheduling information of each sub-node based on the initial target task, and sending the first task scheduling information to the master node and the sub-nodes, so that the sub-nodes obtain the task data of the initial target task from the master node according to the first task scheduling information, and in the task execution process of the sub-nodes, the subsequent target task is allocated to the sub-nodes according to the execution state and the predicted execution time of the sub-nodes for task execution, thereby effectively improving the overall computing efficiency of the node cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a task scheduling method for a node cluster and related equipment. BACKGROUND

[0002] With the rapid growth of modern computing needs, large-scale clusters have been built more and more perfectly, and with the continuous improvement of hardware level, computing on large-scale cluster tasks has become a very important task. During this period, how to reasonably schedule tasks so that the idle time of computing resources is minimized has become a key problem. On nodes with low computing scheduling efficiency, the computing effect of the same hardware specification will be much lower than that of high efficiency. Therefore, an efficient task scheduling method is crucial to the field of big data.

[0003] In related technologies, a Kubernetes (K8S) based cloud service cluster provides a unified scheduling management method. This method builds a K8S based node cluster, connects a large number of cloud servers, and uses Docker container technology to implement algorithm deployment and distribution. This mode realizes load balancing by using the service mechanism of K8S, dynamically allocates computing tasks according to real-time resource occupancy, and thus effectively schedules and manages computing tasks. However, the K8S based scheduling management method has the following problems. When executing large-scale computing tasks, the number of task responses of big data computing is relatively small, and the operation time of each operation exceeds the default configuration time of many K8S clusters. In addition, the task distribution is usually based on the real-time load of the worker node, but since the load is generally low at the beginning of the computing task, it cannot reflect the overall load of the entire task. In this state, the K8S based scheduling management method will allocate many tasks to the same computing node in the task allocation stage, causing a large number of other computing nodes to be idle, resulting in low overall computing efficiency of the node cluster. SUMMARY

[0004] The embodiments of the present application provide a task scheduling method for a node cluster and related equipment, which can improve the overall computing efficiency of the node cluster.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application proposes a task scheduling method for a node cluster, the node cluster comprising a master node and a plurality of sub-nodes, the method comprising:

[0006] obtaining a plurality of processing tasks, inputting each processing task into an execution time prediction model one by one to obtain the predicted execution time of the processing task;

[0007] determining an initial target task for each sub-node based on the predicted execution time of each processing task;

[0008] generate first task scheduling information of each of the sub-nodes based on the initial target task, and send the first task scheduling information to the master node and the sub-nodes, so that the sub-nodes acquire task data of the initial target task from the master node according to the first task scheduling information;

[0009] In a task execution process of the sub-node, a subsequent target task is assigned to the sub-node for task execution according to an execution state of the sub-node and the predicted execution time.

[0010] In some embodiments, the initial target task of each of the sub-nodes is determined based on the predicted execution time of each of the processing tasks, including:

[0011] At least one of the processing tasks whose sum of predicted execution times is within a preset execution time range is selected as the initial target task of the sub-node from the plurality of processing tasks.

[0012] In some embodiments, the first task scheduling information of each of the sub-nodes is generated based on the initial target task, including:

[0013] Execution order information of the initial target task is generated according to time size order based on the predicted execution time of the initial target task.

[0014] A storage address of task data of the initial target task in the master node is acquired.

[0015] An initial predicted total execution time is obtained based on a sum of predicted execution times of the initial target task of the sub-node, and a task scheduling time is obtained based on a product of the initial predicted total execution time and a task scheduling factor, the task scheduling time being used to represent a time node of planning a next execution task for the sub-node.

[0016] The first task scheduling information is generated based on a task identifier of the initial target task, the execution order information, the storage address, and the task scheduling time.

[0017] In some embodiments, the execution state includes a real-time execution time when the sub-node executes the initial target task, and the subsequent target task is assigned to the sub-node for task execution according to the execution state of the sub-node and the predicted execution time, including:

[0018] A task state of the initial target task in the plurality of processing tasks is set to a task execution state.

[0019] acquire the real-time execution time, and select, according to the real-time execution time and the task scheduling time, a processing task in the plurality of processing tasks in a task pending execution state as a post-target task of the child node;

[0020] generate second task scheduling information based on the post-target task, and send the second task scheduling information to the master node and the child node, so that the child node acquires task data of the post-target task from the master node according to the second task scheduling information to perform a task.

[0021] In some embodiments, the selecting, according to the real-time execution time and the task scheduling time, a processing task in the plurality of processing tasks in a task pending execution state as a post-target task of the child node comprises:

[0022] When the real-time execution time exceeds the task scheduling time, selecting, from the plurality of processing tasks, at least one processing task in a task pending execution state and having a sum of predicted execution times within a preset execution time range and a same task type as an alternative processing task group;

[0023] selecting, based on a task type of the initial target task, one alternative processing task group from the plurality of alternative task groups as a target task group, and obtaining the post-target task based on the target task group.

[0024] In some embodiments, the task type comprises a task information type, a parameter configuration type, and a geographic data type, and the selecting, based on the task type of the initial target task, one alternative processing task group from the plurality of alternative task groups as a target task group, and obtaining the post-target task based on the target task group comprises:

[0025] acquiring a target task information type, a target parameter configuration type, and a target geographic data type of the initial target task;

[0026] selecting, from the alternative task groups, an alternative task group having a task information type matching the target task information type, a parameter configuration type matching the target parameter configuration type, and a geographic data type matching the target geographic data type as the target task group;

[0027] obtaining the post-target task based on all processing tasks in the target task group.

[0028] In some embodiments, the generating second task scheduling information based on the post-target task comprises:

[0029] determine repetition information type data between the initial target task and the subsequent target task based on the target task information type and a task information type of the subsequent target task;

[0030] determine repetition parameter data between the initial target task and the subsequent target task based on the target parameter configuration type and a parameter configuration type of the subsequent target task;

[0031] determine repetition geographic data between the initial target task and the subsequent target task based on the target geographic data type and geographic data type of the subsequent target task;

[0032] generate repetition data identification based on the repetition information type data, the repetition parameter data and the repetition geographic data, and generate the second task scheduling information based on the subsequent target task and the repetition data identification, the repetition data identification being used to indicate that the sub-node does not need to acquire the repetition information type data, the repetition parameter data and the repetition geographic data when performing task data acquisition.

[0033] In some embodiments, after the first task scheduling information is generated based on all the initial target tasks and the first task scheduling information is sent to the master node and the sub-nodes, the method further comprises:

[0034] generate a task termination instruction when the real-time execution time exceeds a preset abnormal time, the preset abnormal time being greater than the predicted execution time of the initial target task;

[0035] send the task termination instruction to the sub-nodes, so that the sub-nodes terminate execution of the initial target task;

[0036] set a task state of the initial target task to a task to be executed state.

[0037] To achieve the above object, a second aspect of the embodiment of the present application proposes a task scheduling device of a node cluster, the node cluster comprising a master node and a plurality of sub-nodes, the device comprising:

[0038] an acquisition module, configured to acquire a plurality of processing tasks, input each of the processing tasks into an execution time prediction model one by one, and obtain a predicted execution time of the processing task;

[0039] a task determination module, configured to determine an initial target task of each of the sub-nodes based on the predicted execution time of each of the processing tasks;

[0040] an initial scheduling module, configured to generate first task scheduling information of each of the sub-nodes based on the initial target task, and send the first task scheduling information to the master node and the sub-nodes, so that the sub-nodes acquire task data of the initial target task from the master node according to the first task scheduling information;

[0041] a subsequent scheduling module, configured to allocate subsequent target tasks for the sub-nodes to perform tasks according to execution states of the sub-nodes and the predicted execution time during task execution of the sub-nodes.

[0042] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the task scheduling method of the node cluster when executing the computer program.

[0043] To achieve the above object, a fourth aspect of the embodiments of the present application provides a storage medium, which is a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the task scheduling method of the node cluster as described in the first aspect.

[0044] The task scheduling method of the node cluster and the related device provided by the embodiments of the present application, the node cluster comprises a master node and a plurality of sub-nodes, and the method comprises the following steps: firstly, a plurality of processing tasks are acquired, each processing task is input into an execution time prediction model one by one to obtain a predicted execution time of the processing task; then, an initial target task of each sub-node is determined based on the predicted execution time of each processing task; finally, first task scheduling information of each sub-node is generated based on the initial target task, and the first task scheduling information is sent to the master node and the sub-nodes, so that the sub-nodes acquire task data of the initial target task from the master node according to the first task scheduling information, and in the process of task execution of the sub-nodes, subsequent target tasks are allocated to the sub-nodes to perform tasks according to execution states of the sub-nodes and the predicted execution time. Before batch processing tasks are executed, the execution time prediction model generated in advance is used to predict the execution time required by each processing task, the predicted execution time of each processing task is used to select a suitable initial target task for each sub-node in the node cluster, and the first task scheduling information corresponding to the initial target task is generated, so that the sub-nodes acquire the task data corresponding to the initial target task from the master node according to the first task scheduling information to perform the task; and in the subsequent process, the task execution scheduling of each sub-node can be performed in real time according to the predicted execution time of each processing task, so as to reduce the situation that part of the sub-nodes are busy and part of the sub-nodes are idle, thereby effectively improving the overall computing efficiency of the node cluster.

[0045] Other features and advantages of the present application will be set forth in the following specification, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a structural schematic diagram of a node cluster provided by an embodiment of the present application.

[0047] Figure 2 is a flowchart of a task scheduling method of a node cluster provided by another embodiment of the present application.

[0048] Figure 3 is a construction flowchart of an execution time prediction model provided by another embodiment of the present application.

[0049] Figure 4 is a flowchart of step 202 in Figure 2

[0050] Figure 5 is a flowchart of step 203 in Figure 2

[0051] Figure 6 is a flowchart of a post-task scheduling of a node cluster provided by another embodiment of the present application.

[0052] Figure 7 is a flowchart of a node task process management provided by another embodiment of the present application.

[0053] Figure 8 is a task scheduling flowchart of a sub-node provided by another embodiment of the present application.

[0054] Figure 9 is a flowchart of step 602 in Figure 6

[0055] is a flowchart of step 902 in Figure 10 Figure 9 is a flowchart of step 603 in

[0056] Figure 11 Figure 6 is a flowchart of step 603 in

[0057] Figure 12 is a flowchart of a task error correction of a node cluster provided by another embodiment of the present application.

[0058] Figure 13 is a flowchart of a task error correction of a node cluster provided by another embodiment of the present application.

[0059] ​​​​Figure 14 This is a schematic diagram of the task scheduling process of a node cluster provided in another embodiment of this application.

[0060] Figure 15 This is a schematic diagram of the structure of a task scheduling device for a node cluster provided in an embodiment of this application.

[0061] Figure 16 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] First, let's analyze some of the terms used in this application:

[0066] Kubernetes (K8s) is a highly scalable, automated container management platform that runs in the cloud and can automatically deploy, scale, and manage containerized applications. It provides a unified and flexible infrastructure layer that enables developers to efficiently deploy, run, and maintain distributed applications in the cloud environment.

[0067] OBS (Object Storage Service) is a high-performance, highly available, secure and reliable cloud-based object storage service. It provides massive data storage and management capabilities and is suitable for various application scenarios, such as backup and archiving, big data analysis, and content distribution. Users can easily store and retrieve any type of data through standard RESTful APIs or client tools.

[0068] Feature scaling refers to the process of transforming different features' data into a similar range to avoid certain features from having an unfair impact on the model due to their large numerical range. Common methods of feature scaling include standardization and normalization. Standardization refers to transforming feature values to have a mean of 0 and a standard deviation of 1. Normalization refers to scaling feature values to the range [0, 1].

[0069] In the field of machine learning, ensemble learning methods are highly regarded for their excellent predictive performance and robustness. Ensemble learning improves the accuracy and stability of the overall model by combining multiple base models. There are two main types of ensemble learning methods: bagging and boosting.

[0070] Bagging creates multiple sub-datasets by resampling (with replacement) the training data, and trains an independent base model on each sub-dataset. The prediction results of these base models are then averaged (for regression tasks) or voted (for classification tasks) to obtain the final prediction result. The representative algorithm of bagging is the random forest.

[0071] Boosting trains a series of base models step by step, with each model trying to correct the mistakes of the previous model, gradually improving the predictive performance of the model. Common boosting algorithms include AdaBoost and Gradient Boosting Machine (GBM).

[0072] Random forest, as a typical representative of bagging, combines the advantages of bagging and decision trees. It constructs multiple base models by randomly sampling multiple sub-datasets from the original training data and training a decision tree on each sub-dataset. When splitting the nodes of each tree, random forest also randomly selects a subset of features to find the best split point. This random feature selection reduces the correlation between decision trees, thereby improving the diversity and stability of the ensemble model.

[0073] Random forest achieves high-precision prediction by integrating the prediction results of multiple decision trees. For classification tasks, random forest uses majority voting to determine the final prediction class; for regression tasks, it calculates the average of all tree prediction results to determine the final prediction value. This makes random forest not only have high prediction accuracy, but also effectively prevent overfitting, with high robustness.

[0074] With the rapid growth of modern computing needs, large-scale clusters have been built more and more perfect, with the continuous improvement of hardware level, computing on large-scale cluster tasks has become a very important task. During this period, how to reasonably schedule tasks so that the idle time of computing resources is minimized has become a key problem. The computing effect of the same hardware specification of the algorithm on the node with low computing scheduling efficiency will be much lower than that of the high efficiency. Therefore, an efficient task scheduling method is crucial to the field of big data.

[0075] In the related art, the Kubernetes (K8S) based cloud service cluster provides a unified scheduling management method. The method connects a large number of cloud servers by constructing a K8S based node cluster, and uses the Docker container technology to realize the deployment and allocation of algorithms. This mode realizes load balancing by using the service mechanism of K8S, dynamically allocates computing tasks according to real-time resource occupancy, and effectively schedules and manages computing tasks. However, the K8S based scheduling management method has the following problems. When executing large-scale computing tasks, the number of task responses of big data computing is relatively small, and the operation time of each operation exceeds the default configuration time of many K8S clusters. In addition, the task allocation is usually based on the real-time load of the worker node, but since the load is generally low at the beginning of the computing task, it cannot reflect the overall load of the entire task. In this state, the K8S based scheduling management method will allocate many tasks to the same computing node in the task allocation stage, causing a large number of computing nodes to be idle, resulting in low overall computing efficiency of the node cluster.

[0076] In order to improve the overall computing efficiency of the node cluster, the execution time prediction model is used to predict the execution time of each processing task before the batch processing task is executed, and the predicted execution time of each processing task is used to select a suitable initial target task for each sub-node in the node cluster, and the first task scheduling information corresponding to the initial target task is generated, so that the sub-node can sequentially obtain the task data corresponding to the initial target task from the master node according to the first task scheduling information. In the subsequent process, the task execution scheduling of each sub-node can be performed in real time according to the predicted execution time of each processing task, so as to reduce the situation that part of the sub-nodes are busy and part of the sub-nodes are idle, thereby effectively improving the overall computing efficiency of the node cluster.

[0077] In order to better describe the task scheduling method of the node cluster provided by the present application, the node cluster applied to the task scheduling method of the node cluster will be described first. Referring to Figure 1 is a structural schematic diagram of a node cluster provided by an embodiment of the present application. As shown in Figure 1As shown, the node cluster includes at least one master node, multiple child nodes, and model nodes. The master node manages the multiple child nodes, executes batch processing tasks, stores these batches, and configures network plugins. Child nodes include a Miniconda image for executing processing tasks, a cloud disk for caching task data, and a corresponding task list. Model nodes build and train execution time prediction models; they also use the trained models to predict the execution time of processing tasks and allocate tasks.

[0078] Furthermore, based on Kubernetes, master node data storage can be achieved through OBS cloud storage. OBS cloud storage is used for storing task data corresponding to processing tasks, task information, and sample data for model training.

[0079] Based on the above-described node cluster, the task scheduling method and related devices for the node cluster provided in this application embodiment will be further described below. The task scheduling method for the node cluster provided in this application embodiment can be applied to model nodes in the node cluster, or to a controller connected to the node cluster.

[0080] The task scheduling method for the node cluster in the embodiments of this application will be described in detail below. (Refer to...) Figure 2 This is an optional flowchart of the task scheduling method for a node cluster provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, steps 201 to 204. It is also understood that this embodiment... Figure 2 The order of steps 201 to 204 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0081] Step 201: Obtain multiple processing tasks, and input each processing task into the execution time prediction model one by one to obtain the predicted execution time of the processing task.

[0082] Step 201 is described in detail below.

[0083] In some embodiments, when the node cluster responds to the execution of the plurality of processing tasks of the batch, the task information of the plurality of processing tasks is first acquired by the node cluster, and the task data of the processing tasks is stored by the master node. Next, the processing tasks need to be sequentially allocated and scheduled to the plurality of sub-nodes that can be used for task processing. In order to further improve the overall computing efficiency of the node cluster, in the present embodiment, the predicted execution time of each processing task is obtained by first inputting each processing task into the pre-constructed execution time prediction model for prediction. Thus, the predicted execution time of each processing task is used to subsequently allocate and schedule the plurality of processing tasks to the corresponding sub-nodes for execution, thereby improving the overall computing efficiency of the node cluster.

[0084] In some embodiments, taking the node cluster with 2 master nodes, 200 sub-nodes, and 1 model node as an example, and there are 10000 processing tasks in the batch processing tasks to be executed. Each sub-node is installed with a container service, so that the processing tasks can be processed. It can be understood that the management mode of the cluster is not limited to the node cluster of the present example, and the number of nodes can vary from hundreds to tens of thousands depending on the resources and the number of calculations required. In addition, each two nodes (including master nodes, sub-nodes and model nodes) can communicate with each other, and each sub-node is equipped with a cloud hard disk for configuring, reading and transmitting task data. In addition, each sub-node is equipped with a suitable usage image required by the calculation program for processing the processing task, such as using Miniconda4.1.2 as the base image, which contains the required scientific calculation library and tool to ensure the consistency and dependency management of the calculation environment. On the basis of the base image, GDAL spams, scikit-learn and other program packages are added to read and process high-resolution image data. In addition, the node cluster is configured with a cloud storage file service system to ensure that each node can connect to the same cloud service location to pull configuration files. For example, there are three node clusters in total, a total of 600 sub-nodes, and 6 master nodes, and each 2 master nodes manage 200 sub-nodes. According to the total number of nodes and the name of the pod, the total number of available resources is collected to generate a node list. In the present embodiment, we calculate remote sensing image data, which is divided into 2000-2023 data for a total of 23 years according to the year, latitude and longitude, and each calculation task time is not uniform, and the calculation load is also not uniform.

[0085] After the calculation tile is assigned, the predicted completion time of each task is analyzed by regression analysis through the model in step two. In this example, the Sentinel-2 remote sensing data is used. The specific steps are as follows: traversing the name of the corresponding data folder of each task, extracting the feature value of the folder name, inputting the extracted feature value into the model for regression operation to obtain the predicted execution time, and recording the time data of all processing tasks together and putting them into the task pool waiting for calculation.

[0086] In addition, a task queue is created according to the task list waiting for calculation. In the initial state, the model is used to process the task time, and more tasks (4 tasks in this example) are uniformly assigned to each subnode. At the same time, the model node calculates the predicted task settlement time of each subnode and puts them into the task queue control service for control. In this embodiment, the total amount of processing tasks is 70000, and the number of subnodes available for calculation is 600. According to the type of cluster, it can be expanded, and each 200 subnodes is a node cluster. Each node cluster can undertake about 20000 tasks in each round. The related configuration file of 70000 tasks is generated in advance through a script and stored in the task pool of the obs file management system. At the same time, the running task list and the completed task list are created. For this process, the numbers in this embodiment are only for understanding. In actual situation, the number of subnodes of each node cluster can be from dozens to tens of thousands, and the number of processing tasks can be from hundreds to millions.

[0087] In some embodiments, in order to further improve the prediction accuracy of the predicted execution time of the processing task, a suitable execution time prediction model needs to be constructed and trained. How to obtain the execution time prediction model will be described in detail below.

[0088] Referring to Figure 3 is a construction flowchart of an execution time prediction model provided by an embodiment of the present application. As shown in Figure 3 In this embodiment, the random forest algorithm is selected as the method for constructing the execution time prediction model. The specific operation steps are as follows (combined with the data processing process):

[0089] 1) Data preparation and feature extraction, first, the parts of the tile number TileID are extracted from the data set, including ZoneNumber, LatitudeBand, EastingBand and NorthingBand, and these features are added to the data frame. Among them, a certain scale of data is used as the initial training set, and 200000 pieces of data of the operation time of the sentinal2 task are recorded in this embodiment, and the data set used by each group of tasks is a tile, and the folder of the tile is represented by a series of data, such as S2A_MSIL2A_20190210T031851_N0211_R118_T50UNV_20190210T053429; among them, 2019 represents the year of the data, and we use the data of the same year in each operation, and there is no special difference in the calculation amount of the same information in different years. Therefore, another valuable feature value T50UNV is adopted, which represents the geographic location information of the data information, and is specifically described as follows: T represents the UTM grid partition (usually omitted in naming, which does not affect the conversion). 50 represents the UTM area number (zonenumber). UNV represents the UTM grid square identifier, which can be converted into geographic location information. U represents the latitude band in the UTM band, and these letters from C to X (excluding I and O) represent the latitude band from south to north, and each letter represents an 8-degree latitude range. N represents the grid column (easting) from west to east. V represents the grid row (northing) from south to north. It can be seen that in the features of this example, the feature contents represented by 50 and U are much more important than N and V. At the same time, these two features represent independent geographic location information and should not be disturbed by numerical values. Therefore, the following is a series of processing of the feature value.

[0090] 2) Feature Processing, to handle the problem of different weights in features, a technique called "One-Hot Encoding" can be used to handle categorical variables. In the example of this implementation, the first two digits of the tile's ZoneNumber can be equivalently viewed as the latitude and longitude bands, which represent the approximate location of the geographic information, but should not be treated as numerical or alphabetical information, so One-Hot Encoding is applied to these two features. That is, One-Hot Encoding is applied to ZoneNumber and LatitudeBand to convert categorical features into binary vectors; for example, ZoneNumber: numerical values 1-60 are One-Hot encoded with range (1, 61); LatitudeBand: the specific values CDEFGHJKLMNPQRSTUVWX are One-Hot encoded with their string list values. At the same time, Principal Component Analysis and Feature Amplifier are used to increase the weight of these two important features. And EastingBand and NorthingBand are standardized (StandardScaler) to convert numerical features into a distribution with a mean of 0 and a standard deviation of 1.

[0091] 3) Model Pipeline, create a pipeline containing the preprocessing steps and the Random Forest Regression model. This ensures that the same preprocessing steps are applied to the data during training and prediction.

[0092] 4) Dataset Split, split the dataset obtained in 1) into training and test sets for subsequent model training and evaluation.

[0093] 5) Model Training, use the training set data to train the Random Forest Regression model. In this process, the specific steps of the Random Forest include: Random Sampling (Bagging): Randomly sample multiple samples (with replacement) from the training data to form multiple sub-sample sets. Build multiple decision trees: For each sub-sample set, build a decision tree. Each tree selects a subset of features at random to find the best split point during construction. Integrated decision tree prediction results: In the final prediction, the Random Forest averages the prediction results of multiple decision trees (regression task) or votes (classification task). After training, save the model for task time prediction.

[0094] 6) Iterative training, in the continuous running of the task, these data can be recursively used for model training. Each time new data comes, the model can be updated to keep it up to date and optimal. In this embodiment, after recording 100,000 tasks each time, the above steps 1-5 are recursively called once based on the original model, and the model is continuously updated to make the prediction more accurate. After the above steps, a sentineal2 computing task execution time prediction model can be obtained according to the tile number, which is used for task time prediction and scheduling allocation in subsequent steps.

[0095] Step 202: Determine the initial target task of each sub-node based on the predicted execution time of each processing task.

[0096] The following describes step 202 in detail.

[0097] In some embodiments, after obtaining the predicted execution time of each processing task by using the execution time prediction model, the initial target task of each sub-node for processing the batch of processing tasks will be allocated based on the predicted execution time of each processing task. The following will be further described.

[0098] Reference Figure 4 Determine the initial target task of each sub-node based on the predicted execution time of each processing task, including the following step 401.

[0099] Step 402: Select at least one processing task whose sum of predicted execution time is within the preset execution time range from the plurality of processing tasks as the initial target task of the sub-node.

[0100] The following describes step 401.

[0101] In some embodiments, in order to further improve the rationality of the initial target task allocated to each sub-node, it is necessary to first obtain the processing memory corresponding to each processing task (including the storage memory corresponding to the task data and the running memory corresponding to the execution of the task data), and obtain the node available memory corresponding to each sub-node (including the node storage memory available to the sub-node and the node running memory). In addition, a suitable preset execution time range is also set for each sub-node, such as one hour to five hours, to ensure that the predicted execution time of the processing task allocated to each sub-node can be maintained within this preset execution time range when the node cluster executes the batch of processing tasks, so as to ensure the execution efficiency of the node cluster.

[0102] Next, for each sub-node, at least one processing task in the batch of processing tasks is selected as the initial target task of the sub-node, where the sum of the predicted execution times of the at least one processing task is within the preset execution time range, and the sum of the processing memories of the at least one processing task is less than the node available memory of the sub-node.

[0103] In addition, if the task types performed by different sub-nodes are different, in addition to the step 402, a processing task corresponding to the task type of the sub-node needs to be selected as the initial target task of the sub-node.

[0104] Through the step 401, the processing memory of the processing task and the node available memory of the sub-node are used as the matching parameters, and the preset execution time range and the predicted execution time of each processing task are used as another matching parameter, so that the appropriate initial target task is selected for each sub-node as the execution task at the start of the execution of the batch of processing tasks by each sub-node, to ensure the execution reliability of the execution of the batch of processing tasks in the node cluster.

[0105] In some embodiments, when the task allocation is performed in each round, the task queue allocates the processing task that is predicted to require a longer time by the model first, to ensure that each sub-node initially calculates a task queue that requires a larger amount of calculation. In this way, the higher load (longer remaining calculation time) of the sub-node can be ensured for a long period of time.

[0106] Step 203: generating first task scheduling information of each sub-node based on the initial target task, and sending the first task scheduling information to the master node and the sub-nodes, so that the sub-nodes obtain the task data of the initial target task from the master node according to the first task scheduling information.

[0107] The step 203 is described in detail below.

[0108] In some embodiments, after determining the initial target task of each sub-node, the first task scheduling information corresponding to each sub-node is generated based on the initial target task of the sub-node, so as to facilitate the sending of the first task scheduling information to the master node and the corresponding sub-node, so that the sub-node can obtain the task data corresponding to the initial target task from the master node according to the first task scheduling information for task processing. How to generate the first task scheduling information of each sub-node based on the target task will be further described below.

[0109] Referring to Figure 5 The first task scheduling information of each sub-node is generated based on the initial target task, including the following steps 501 to 504.

[0110] Step 501: generating execution order information of the initial target task in the order of time size based on the predicted execution time of the initial target task.

[0111] Step 502: Obtain the storage address of the task data of the initial target task in the master node.

[0112] Step 503: Obtain the initial predicted total execution time based on the sum of the predicted execution times of the initial target tasks of the sub-nodes, and obtain the task scheduling time based on the product of the initial predicted total execution time and the task scheduling factor.

[0113] Step 504: Generate the first task scheduling information based on the task identification, execution order information, storage address, and task scheduling time of the initial target task.

[0114] The steps 501 to 504 are described in detail as follows.

[0115] In some embodiments, there are multiple processing tasks in the initial target tasks of the sub-nodes, in which case, for each sub-node, the execution order information of the multiple processing tasks in the initial target tasks is generated in descending order of the predicted execution times of the processing tasks.

[0116] Next, the storage addresses of the task data of the multiple processing tasks in the initial target tasks corresponding to the sub-nodes in the master node are obtained, and the initial predicted total execution time is obtained based on the sum of the predicted execution times of the initial target tasks of the sub-nodes, and then the task scheduling time corresponding to the time node at which the next execution task can be planned for the sub-node is obtained based on the product of the initial predicted total execution time and the task scheduling factor. Finally, the first task scheduling information is generated based on the task identification, execution order information, storage address, and task scheduling time of the initial target task.

[0117] Through the above steps 501 to 504, the multiple processing tasks in the initial target tasks are executed in order according to the time size of the predicted execution times of the processing tasks, and the execution of the initial target tasks by the sub-nodes can be effectively and neatly identified by combining the storage address of the initial target task and the first task scheduling information generated by the task scheduling time obtained from the initial predicted total execution time, thereby effectively improving the reliability and work efficiency of the node cluster in processing tasks.

[0118] After the child node obtains the first task scheduling information, the child node obtains the host name of the master node through a program based on the first task scheduling information, and obtains the kind of the computing task, the detailed location of the task configuration file corresponding to the task data of the initial target task, the task state and other information through the configuration file by locating the configuration file based on the host name in the cloud storage. Then, according to the configuration file, the task pulling of the child node (that is, obtaining the task data corresponding to the initial target task from the master node) is performed.

[0119] Step 204: In the task execution process of the child node, the subsequent target task is assigned to the child node for task execution according to the execution state and the predicted execution time of the child node.

[0120] Step 204 will be described in detail below.

[0121] In some implementations, the task execution scheduling of each child node can be performed in real time according to the predicted execution time of each processing task in the subsequent process, reducing the situation that part of the child nodes are busy and part of the child nodes are idle, thereby effectively improving the overall computing efficiency of the node cluster. How to assign the subsequent target task to the child node for task execution according to the execution state and the predicted execution time of the child node will be further described below.

[0122] Referring to Figure 6 According to the execution state and the predicted execution time of the child node, the subsequent target task is assigned to the child node for task execution, and further includes steps 601 to 603.

[0123] Step 601: setting the task state of the initial target task in the plurality of processing tasks to a task execution state.

[0124] Step 601 will be described in detail below.

[0125] In some embodiments, the execution state of the child node includes the real-time execution time when the child node executes the initial target task. When each child node obtains the initial target task and performs task processing, the task state of the initial target task in the plurality of processing tasks is updated and set to a task execution state. Then, during the task processing of each child node on the initial target, a judgment is made every 5 minutes, and when the remaining running time of the current task node task reaches the task scheduling time (such as 30 minutes), two tasks are actively applied from the task queue, and the length of the remaining total time is calculated through the model to ensure that the remaining total time is higher than the set threshold (the threshold is two hours in this embodiment, and if it is lower than the threshold, the task will be continuously obtained). In addition to the model judgment, it is also judged through the identification bit whether the computing task is running. If the task has been completed when the judgment is made, the next processing task is immediately obtained.

[0126] Referring toFigure 7 This is a flowchart illustrating a node task process management method provided in an embodiment of this application. Figure 7 As shown, task process management is implemented for each child node. Process management plays a crucial role in parallel processing, its primary responsibility being the effective management of child nodes within the node cluster. This task is implemented using Python middleware, ensuring that each child node in the cluster executes its tasks in an orderly manner. When a child node starts its main process, the process first retrieves the processing task information assigned by the model node to run on that child node from the main node. This process is essentially a dynamic resource allocation strategy, maximizing the utilization of resources within the node cluster without waste. Detailed processing task information for each child node is managed through a configuration file, which contains detailed task information such as task type, parameter configuration, and computational resource requirements. To achieve efficient management of child nodes within the node cluster, this embodiment also incorporates a real-time update mechanism based on status files. During the execution of processing tasks by each child node, its status is written to the status file in real time, including the child node's ID, task execution progress, and resource usage. In this way, we can read the status file to understand the running status of each child node in real time, so as to take timely action when problems occur. At the same time, we can manage files in real time through the OBS file system, with all child nodes connected to the same configuration file and code file management content, achieving unified scheduling.

[0127] Step 602: Obtain the real-time execution time of the child node when it executes the initial target task, and select the processing task with the task status of pending execution from multiple processing tasks as the subsequent target task of the child node based on the real-time execution time and the task scheduling time.

[0128] Step 602 will be described in detail below.

[0129] In some embodiments, based on Figure 7 The node task process management shown requires that, during the execution of the target task by the child node, the real-time execution time of the initial target task by the child node needs to be obtained in real time or periodically. Then, based on the real-time execution time and task scheduling time, the processing task with the task status of pending execution is selected from multiple processing tasks as the subsequent target task of the child node. This allows the child node to seamlessly connect to execute the next target task after completing the initial target task, thereby avoiding the child node from being idle and effectively improving the work efficiency of the node cluster in task processing.

[0130] Reference Figure 8This is a schematic diagram of a task scheduling process for a child node provided in an embodiment of this application. Figure 8 As shown, a Python middleware is used to monitor the task status of each child node during the execution of its processing tasks. When the task process switching scheduling is met, the task data for the new processing task is retrieved from the cloud container. After completing the next target task, hot code replacement is used to switch to the next target task for execution, ensuring the continuous execution of the child node's tasks. In each child node's image pod, the host runs the same main process, polling for new tasks. Based on the task configuration details or a detailed task list, each computation task (tile) starts a separate child process to complete the computation. In this embodiment, the computation is for remote sensing-related data. The computation tasks are separated by time and space, and each computation task retrieves the corresponding basic data. To achieve efficient management of child processes within the Pod, a process is designed where the status is written to a status file in real time, including the child node ID, the execution progress of the processing task, resource usage, and other information. In this way, we can read the status file to understand the running status of each child node in real time, so as to take appropriate action in time when problems occur. In addition, once each processing task is completed, a message indicating that the task is completed will be sent out. This message will be recorded in the list of completed tasks, with different status records for each task, regardless of whether it succeeds or fails.

[0131] The process of selecting subsequent target tasks by child nodes will be further described below.

[0132] Reference Figure 9 Based on the real-time execution time and task scheduling time, the processing task with the task status of pending execution is selected from multiple processing tasks as the subsequent target task of the child node, including the following steps 901 to 902.

[0133] Step 901: When the real-time execution time exceeds the task scheduling time, select at least one processing task from multiple processing tasks whose task status is a task pending execution state, whose predicted execution time sum is within the preset execution time range, and whose task type is the same as the task pending execution state, as a candidate processing task group.

[0134] Step 902: Based on the task type of the initial target task, select one alternative processing task group that matches the task type from multiple alternative task groups as the target task group, and obtain the subsequent target tasks based on the target task group.

[0135] Steps 901 to 902 are described in detail below.

[0136] In some embodiments, when it is determined that the real-time execution time of the sub-node performing the initial target task exceeds the task scheduling time, a plurality of processing tasks with a task state of a task to be executed, a sum of predicted execution times within a preset execution time range, and the same task type are selected as a candidate processing task group. Next, based on the task type of the initial target task corresponding to the sub-node, a candidate processing task group matching the task type is selected from the plurality of candidate task groups as a target task group corresponding to the processing task to be executed subsequently by the sub-node. Furthermore, based on all processing tasks in the target task group, a subsequent target task corresponding to the sub-node is obtained. The task type includes a task information type, a parameter configuration type, and a geographic data type. It can be understood that the task information type refers to various attributes or data types used to describe and define a processing task; the parameter configuration type refers to various setting options used in software development, system configuration, or algorithm design corresponding to the processing task; and the geographic data type refers to a data structure used to represent the features of the earth's surface.

[0137] The process of selecting a target task group based on the task type of the initial target task will be further described below.

[0138] Referring to Figure 10 , based on the task type of the initial target task, a candidate processing task group matching the task type is selected from the plurality of candidate task groups as a target task group, and a subsequent target task is obtained based on the target task group, including the following steps 1001 to 1003.

[0139] Step 1001: Obtain the target task information type, the target parameter configuration type, and the target geographic data type of the initial target task.

[0140] Step 1002: Select a candidate task group matching the target task information type in the task information type, matching the target parameter configuration type in the parameter configuration type, and matching the target geographic data type in the geographic data type from the candidate task group as the target task group.

[0141] Step 1003: Obtain a subsequent target task based on all processing tasks in the target task group.

[0142] The steps 1001 to 1003 are described in detail below.

[0143] In some embodiments, the target task information type, the target parameter configuration type and the target geographic data type of the target task are first acquired. Next, a candidate task group is selected from the candidate task groups, which has a task information type matching the target task information type, a parameter configuration type matching the target parameter configuration type, and a geographic data type matching the target geographic data type, as the target task group, so that a processing task more similar to the initial target task executed by the sub-node is selected as the subsequent target task executed by the sub-node subsequently, so that when the task data of the subsequent target task is acquired subsequently, the repeated task data between the subsequent target task and the initial target task does not need to be acquired, thereby improving the acquisition efficiency of the sub-node when acquiring the task data.

[0144] Step 603: generating second task scheduling information based on the subsequent target task, and sending the second task scheduling information to the master node and the sub-node, so that the sub-node acquires the task data of the subsequent target task from the master node according to the second task scheduling information for task execution.

[0145] The step 603 is described in detail below.

[0146] In some embodiments, after the subsequent target task of the sub-node is determined, the second task scheduling information is generated based on the subsequent target task, similar to the process of generating the first task scheduling information based on the initial target task in the step 203 described above, and the second task scheduling information is sent to the master node and the sub-node, so that the sub-node acquires the task data of the subsequent target task from the master node according to the second task scheduling information for task execution.

[0147] In addition, since the selected subsequent target task is more similar to the initial target task, when the second task scheduling information is generated, the repeated data identifier between the subsequent target task and the initial target task is also generated, and the repeated data identifier is stored in the second task scheduling information, which is described in detail as follows.

[0148] Referring to Figure 11 The second task scheduling information is generated based on the subsequent target task, including the following steps 1101 to 1104.

[0149] Step 1101: determining the repeated information type data between the initial target task and the subsequent target task based on the target task information type and the task information type of the subsequent target task.

[0150] Step 1102: determining the repeated parameter data between the initial target task and the subsequent target task based on the target parameter configuration type and the parameter configuration type of the subsequent target task.

[0151] Step 1103: determining the repeated geographic data between the initial target task and the subsequent target task based on the target geographic data type and the geographic data type of the subsequent target task.

[0152] Step 1104: generating the repeated data identifier based on the repeated information type data, the repeated parameter data and the repeated geographic data, and generating the second task scheduling information based on the subsequent target task and the repeated data identifier.

[0153] The steps 1101 to 1104 will be described in detail below.

[0154] In some embodiments, after determining the subsequent target task of the sub-node, the repeated information type data between the initial target task and the subsequent target task is determined based on the target task information type and the task information type of the subsequent target task; the repeated parameter data between the initial target task and the subsequent target task is determined based on the target parameter configuration type and the parameter configuration type of the subsequent target task; and the repeated geographic data between the initial target task and the subsequent target task is determined based on the target geographic data type and the geographic data type of the subsequent target task; finally, the repeated data identifier indicating that the sub-node does not need to acquire the repeated information type data, the repeated parameter data and the repeated geographic data when performing the task data acquisition is generated based on the repeated information type data, the repeated parameter data and the repeated geographic data, and the second task scheduling information is generated based on the subsequent target task and the repeated data identifier.

[0155] Through the steps 901 to 902, the steps 1001 to 1003 and the steps 1101 to 1104 described above, when the real-time execution time of the sub-node processing the target initial task reaches the task scheduling time, the sum of the predicted execution times of the processing tasks within the preset execution time range is used to select the processing tasks to generate a plurality of alternative processing task groups ensuring that the sub-node can work continuously and stably; then, the task type (including the task information type, the parameter configuration type and the geographic data type) of the initial target task of the sub-node is matched with the plurality of alternative processing task groups, so as to select an alternative processing task group more similar to the initial target task as the subsequent target task of the sub-node; and the repeated data identifier generated based on the repeated data between the subsequent target task and the initial target task is used, so that when the sub-node acquires the task data of the subsequent target task from the master node, the repeated task data between the subsequent target task and the initial target task does not need to be acquired, thereby improving the acquisition efficiency of the sub-node when acquiring the task data, and effectively improving the work efficiency of the node cluster when performing the task processing.

[0156] In some embodiments, after the child node uses the second task scheduling information to process and execute the task data of the subsequent target task, the node cluster will repeat the above actions until all processing tasks in this batch have been completed. However, during the execution of a processing task by a certain child node, a processing exception may occur, causing the real-time execution of a certain processing task to take too long. In this case, appropriate error correction is required. The error correction process will be further described below.

[0157] Reference Figure 12 After generating first task scheduling information based on all initial target tasks and sending the first task scheduling information to the master node and child nodes, the task scheduling method of the node cluster also includes the following steps 1201 to 1203.

[0158] Step 1201: When the real-time execution time exceeds the preset exception time, generate a task termination instruction. The preset exception time is greater than the predicted execution time of the initial target task.

[0159] Step 1202: Send the task termination command to the child node so that the child node terminates the execution of the initial target task.

[0160] Step 1203: Set the task status of the initial target task to the task pending execution status.

[0161] Steps 1201 to 1203 are described in detail below.

[0162] Reference Figure 13 This is a schematic diagram of a task error correction process provided in an embodiment of this application. Figure 13 As shown, the computation time (i.e., real-time execution time) of each processing task on all child nodes is monitored in real time. When the real-time execution time of a processing task on a certain child node exceeds a preset abnormal time (e.g., 12 hours), a task termination command is generated. The preset abnormal time is greater than the predicted execution time of each processing task. The task termination command is then sent to the child node, causing it to kill the processing task (i.e., terminate its execution). The task status of the processing task is then set to a pending execution state, so that the processing task can be rescheduled to another child node for reprocessing, thereby effectively improving the reliability of task processing in the node cluster.

[0163] It's important to note that killing a task and starting a new task are two completely independent processes. The task list that is re-fetched is the one uniformly allocated by the task scheduling system, not the previously killed task. This approach ensures that the machine cannot perform calculations when repeatedly fetching the same task list due to its own limitations.

[0164] In addition, due to the large number of word nodes in each node cluster, the communication between nodes requires time, and there is a time difference between the start of the task and the recording of the task state. The time length of scheduling and state query should have fixed rules and not appear contradictory. In this embodiment, a certain time difference is used. Specifically, after the sub-node is allocated to the corresponding processing task by the model node for 6 minutes, the execution of the processing task is started. And feedback that the processing task has started running. When the processing task is about to be completed and the model node allocates a new processing task for the sub-node, at least 6.5 minutes should be interval before the new model allocation is performed. Otherwise, if the new task allocation is performed before the task is started, the task end will be prompted in advance, causing the task progress to be chaotic.

[0165] Referring to Figure 14 is a flowchart of the task scheduling process of the node cluster provided by the embodiment of the application. The task scheduling system described by the task scheduling method of the node cluster provided by the application realizes fine scheduling of large-scale computing clusters by combining the advanced technologies of k8s cluster management and OBS parallel file system. Through the active task pulling mechanism and the monitoring of the daemon process, the system ensures the optimal configuration of resources and the efficient execution of computing tasks, and can maintain stability even in the face of fluctuations in task volume. When encountering a computing bottleneck, the system can quickly redistribute tasks to avoid single-point delay affecting the overall progress. Such design not only greatly improves the computing efficiency, but also optimizes the resource utilization while ensuring stable operation, and exhibits a powerful and adaptive computing resource management system that can effectively solve the efficiency and resource allocation problems that cannot be solved by traditional methods.

[0166] The method for task scheduling of a node cluster and the related device provided by the embodiments of the present application, the node cluster comprising a master node and a plurality of sub-nodes, the method comprising: firstly, obtaining a plurality of processing tasks, inputting each processing task into an execution time prediction model one by one to obtain the predicted execution time of the processing task; then, obtaining the processing memory of each processing task and the node available memory of each sub-node; selecting at least one processing task from the plurality of processing tasks, the sum of the predicted execution time of the at least one processing task being within a preset execution time range and the sum of the processing memory being less than the node available memory, as the initial target task of the sub-node; generating the execution order information of the initial target task according to the predicted execution time of the initial target task in the order of time size, obtaining the storage address of the task data of the initial target task in the master node, obtaining the initial predicted total execution time based on the sum of the predicted execution time of the initial target task of the sub-node, and obtaining the task scheduling time based on the product of the initial predicted total execution time and a task scheduling factor, the task scheduling time being used to represent the time node of planning the next execution task to the sub-node, generating the first task scheduling information based on the task identifier, the execution order information, the storage address and the task scheduling time of the initial target task, and sending the first task scheduling information to the master node and the sub-node, so that the sub-node obtains the task data of the initial target task from the master node according to the first task scheduling information; then, setting the task state of the initial target task to a task execution state and obtaining the real-time execution time when the sub-node executes the initial target task.When the real-time execution time exceeds the task scheduling time, a task state is selected from the multiple processes as a task to-be-executed state, at least one processing task with a predicted execution time sum within a preset execution time range and a same task type is selected as a candidate processing task group, target task information type, target parameter configuration type, and target geographic data type of an initial target task are obtained, a target task group is selected from the candidate task group, the target task group has a task information type matching the target task information type, a parameter configuration type matching the target parameter configuration type, and geographic data type matching the target geographic data type, a subsequent target task is obtained based on all processing tasks in the target task group, repeated information type data between the initial target task and the subsequent target task is determined based on the target task information type and a task information type of the subsequent target task, repeated parameter data between the initial target task and the subsequent target task is determined based on the target parameter configuration type and a parameter configuration type of the subsequent target task, repeated geographic data between the initial target task and the subsequent target task is determined based on the target geographic data type and geographic data type of the subsequent target task, a repeated data identifier is generated based on the repeated information type data, the repeated parameter data, and the repeated geographic data, a second task scheduling information is generated based on the subsequent target task and the repeated data identifier, the repeated data identifier is used to indicate that a sub-node does not need to obtain the repeated information type data, the repeated parameter data, and the repeated geographic data when performing task data acquisition, and the second task scheduling information is sent to a master node and the sub-node, so that the sub-node obtains task data of the subsequent target task from the master node to perform a task according to the second task scheduling information.

[0167] The embodiments of the present application predict the execution time required by each processing task before the batch processing task is executed, using the pre-generated execution time prediction model, use the processing memory of the processing task and the node available memory of the child node as the matching parameters, and use the preset execution time range and the predicted execution time of each processing task as another matching parameter, so as to select a suitable initial target task for each child node as the execution task at the start time of each child node executing the batch processing task, to ensure the execution reliability of the batch processing task in the node cluster; and use the time size of the predicted execution time of the plurality of processing tasks in the initial target task to perform execution sorting on the plurality of processing tasks in the initial target task, and in combination with the storage address of the initial target task and the first task scheduling information generated by the initial predicted total execution time, the execution of the initial target task by the child node can be effectively and neatly identified, thereby effectively improving the reliability and work efficiency of the node cluster in processing tasks; and in the subsequent process, the task execution scheduling of each child node can be performed in real time according to the predicted execution time of each processing task, to reduce the situation that part of the child nodes are busy and part of the child nodes are idle; in addition, when the real-time execution time of the child node processing the target initial task reaches the task scheduling time, the processing tasks whose sum of the predicted execution time of the plurality of processing tasks is within the preset execution time range are selected by using the predicted execution time to generate a plurality of backup processing task groups that ensure that the child node can work continuously and stably; next, the task type (including task information type, parameter configuration type and geographic data type) of the initial target task of the child node is matched with the plurality of backup processing task groups, so as to select a backup processing task group more similar to the initial target task as a subsequent target task of the child node; and the repeated data identifier generated by the repeated data between the subsequent target task and the initial target task is used, so that when the child node obtains the task data of the subsequent target task from the master node, the repeated task data between the subsequent target task and the initial target task does not need to be obtained, thereby improving the acquisition efficiency of the child node in obtaining the task data, and further effectively improving the overall work efficiency of the node cluster in processing tasks.

[0168] The embodiments of the present application also provide a node cluster task scheduling device, which can implement the above-mentioned node cluster task scheduling method, and refer to Figure 15 The device 1500 comprises:

[0169] The acquisition module 1510 is configured to acquire a plurality of processing tasks, input each processing task into an execution time prediction model one by one, and obtain the predicted execution time of the processing task.

[0170] The task determination module 1520 is configured to determine the initial target task of each child node based on the predicted execution time of each processing task.

[0171] The initial scheduling module 1530 is configured to generate first task scheduling information of each sub-node based on the initial target task, and send the first task scheduling information to the master node and the sub-node, so that the sub-node acquires task data of the initial target task from the master node according to the first task scheduling information;

[0172] The subsequent scheduling module 1540 is configured to allocate a subsequent target task for the sub-node to perform a task according to an execution state and a predicted execution time of the sub-node during a task execution process of the sub-node.

[0173] In some embodiments, the task determination module 1520 is further configured to:

[0174] select at least one processing task with a sum of predicted execution times within a preset execution time range from a plurality of processing tasks as the initial target task of the sub-node.

[0175] In some embodiments, the initial scheduling module 1530 is further configured to:

[0176] generate execution order information of the initial target task according to a time size order based on a predicted execution time of the initial target task;

[0177] obtain a storage address of the task data of the initial target task in the master node;

[0178] obtain an initial predicted total execution time based on a sum of predicted execution times of the initial target task of the sub-node, and obtain a task scheduling time based on a product of the initial predicted total execution time and a task scheduling factor, the task scheduling time being used to represent a time node for planning a next execution task for the sub-node;

[0179] generate the first task scheduling information based on a task identifier, the execution order information, the storage address and the task scheduling time of the initial target task.

[0180] In some embodiments, the subsequent scheduling module 1540 is further configured to:

[0181] set a task state of the initial target task as a task execution state;

[0182] obtain a real-time execution time, and select a processing task with a task state of a task to-be-executed state from a plurality of processing tasks as a subsequent target task of the sub-node according to the real-time execution time and the task scheduling time;

[0183] generate second task scheduling information based on the subsequent target task, and send the second task scheduling information to the master node and the sub-node, so that the sub-node acquires task data of the subsequent target task from the master node according to the second task scheduling information to perform a task.

[0184] In some embodiments, the post-scheduling module 1540 is further configured to:

[0185] When the real-time execution time exceeds the task scheduling time, a task state is selected from the plurality of processes as a task to-be-executed state, and a sum of predicted execution times is within a preset execution time range, and at least one processing task of the same task type is selected as an alternative processing task group;

[0186] Based on the task type of the initial target task, an alternative processing task group matching the task type is selected from the plurality of alternative task groups as a target task group, and a post-target task is obtained based on the target task group.

[0187] In some embodiments, the post-scheduling module 1540 is further configured to:

[0188] Obtain a target task information type, a target parameter configuration type, and a target geographic data type of the initial target task;

[0189] Select an alternative task group from the alternative task groups, which matches the target task information type in the task information type, matches the target parameter configuration type in the parameter configuration type, and matches the target geographic data type in the geographic data type, as the target task group;

[0190] Based on all processing tasks in the target task group, obtain a post-target task.

[0191] In some embodiments, the post-scheduling module 1540 is further configured to:

[0192] Based on the target task information type and the task information type of the post-target task, determine a repeated information type data between the initial target task and the post-target task;

[0193] Based on the target parameter configuration type and the parameter configuration type of the post-target task, determine a repeated parameter data between the initial target task and the post-target task;

[0194] Based on the target geographic data type and the geographic data type of the post-target task, determine a repeated geographic data between the initial target task and the post-target task;

[0195] Based on the repeated information type data, the repeated parameter data, and the repeated geographic data, generate a repeated data identifier, and based on the post-target task and the repeated data identifier, generate second task scheduling information, the repeated data identifier being used to indicate that a sub-node does not need to obtain the repeated information type data, the repeated parameter data, and the repeated geographic data when performing task data acquisition.

[0196] In some embodiments, the post-scheduling module 1540 is further configured to:

[0197] When the real-time execution time exceeds the preset abnormal time, which is greater than the predicted execution time of the initial target task, a task termination instruction is generated;

[0198] The task termination instruction is sent to the child node, so that the child node terminates the execution of the initial target task;

[0199] The task state of the initial target task is set to a task to be executed state.

[0200] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment are basically the same as the specific implementation of the task scheduling device of the node cluster and the specific implementation of the task scheduling method of the node cluster described above. Here, it will not be repeated.

[0201] In the embodiments of the present application, the task scheduling device of the node cluster predicts the execution time required by each processing task by using the pre-generated execution time prediction model before the batch of processing tasks are executed, uses the processing memory of the processing task and the node available memory of the sub-node as the matching parameters, and uses the preset execution time range and the predicted execution time of each processing task as another matching parameter, so as to select a suitable initial target task for each sub-node as the execution task at the start time of each sub-node executing the batch of processing tasks, to ensure the execution reliability of the batch of processing tasks in the node cluster; and the predicted execution time of the plurality of processing tasks in the initial target task is used to perform execution sorting on the plurality of processing tasks in the initial target task, and the storage address of the initial target task and the first task scheduling information generated by the initial predicted total execution time are combined, so that the execution of the initial target task by the sub-node can be effectively and neatly identified, thereby effectively improving the reliability and work efficiency of the node cluster in processing tasks; and in the subsequent process, the task execution scheduling of each sub-node can be performed in real time according to the predicted execution time of each processing task, to reduce the situation that part of the sub-nodes are busy and part of the sub-nodes are idle; in addition, when the real-time execution time of the sub-node processing the target initial task reaches the task scheduling time, the processing tasks whose sum of the predicted execution time of the plurality of processing tasks is within the preset execution time range are selected by using the predicted execution time to generate a plurality of backup processing task groups that ensure that the sub-node can work continuously and stably; next, the task type (including the task information type, the parameter configuration type and the geographic data type) of the initial target task of the sub-node is matched with the plurality of backup processing task groups, so as to select a backup processing task group that is more similar to the initial target task as a subsequent target task of the sub-node; and the repeated data identifier generated by the repeated data between the subsequent target task and the initial target task is used, so that when the sub-node obtains the task data of the subsequent target task from the master node, the repeated task data between the subsequent target task and the initial target task does not need to be obtained, thereby improving the acquisition efficiency of the sub-node in obtaining the task data, and further effectively improving the overall work efficiency of the node cluster in processing tasks.

[0202] The embodiments of the present application also provide an electronic device, comprising:

[0203] at least one memory;

[0204] at least one processor;

[0205] at least one program;

[0206] The program is stored in the memory, and the processor executes the at least one program to implement the task scheduling method of the node cluster described above. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, and the like.

[0207] Please refer to Figure 16 , Figure 16 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device includes:

[0208] The processor 1601 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0209] The memory 1602 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 1602 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1602 and are called and executed by the processor 1601 to implement the task scheduling method of the node cluster of the embodiments of the present application.

[0210] The input / output interface 1603 is used to realize information input and output.

[0211] The communication interface 1604 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0212] The bus 1605 transmits information between various components (such as the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604) of the device.

[0213] The processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604 are connected to each other through the bus 1605 for communication connection within the device.

[0214] The embodiment of the present application further provides a storage medium, which is a computer readable storage medium, and stores a computer program. The computer program is executed by a processor to implement the task scheduling method of the node cluster.

[0215] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0216] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0217] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0218] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0219] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0220] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0221] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0222] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment of the present application.

[0224] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0225] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0226] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for task scheduling of a node cluster, characterized in that, The node cluster comprises a master node and a plurality of sub-nodes, and the method comprises: obtaining a plurality of processing tasks, inputting each processing task into an execution time prediction model one by one to obtain a predicted execution time of the processing task; determining an initial target task of each sub-node based on the predicted execution time of each processing task; generating first task scheduling information of each sub-node based on the initial target task, and sending the first task scheduling information to the master node and the sub-nodes, so that the sub-nodes obtain task data of the initial target task from the master node according to the first task scheduling information; allocating a subsequent target task for the sub-nodes for task execution according to the execution state of the sub-nodes and the predicted execution time during the task execution of the sub-nodes; the execution state comprises a real-time execution time when the sub-nodes execute the initial target task, and the allocation of the subsequent target task for the sub-nodes for task execution according to the execution state of the sub-nodes and the predicted execution time comprises: setting the task state of the initial target task to a task execution state; obtaining the real-time execution time, and selecting a processing task with a task state of a task to-be-executed state from the plurality of processing tasks as the subsequent target task of the sub-nodes according to the real-time execution time and a task scheduling time; generating second task scheduling information based on the subsequent target task, and sending the second task scheduling information to the master node and the sub-nodes, so that the sub-nodes obtain task data of the subsequent target task from the master node for task execution according to the second task scheduling information; the selection of the processing task with the task state of the task to-be-executed state from the plurality of processing tasks as the subsequent target task of the sub-nodes according to the real-time execution time and the task scheduling time comprises: when the real-time execution time exceeds the task scheduling time, selecting at least one processing task with the task state of the task to-be-executed state and the same task type from the plurality of processing tasks as a candidate processing task group, the sum of the predicted execution times of which is within a preset execution time range; selecting a candidate processing task group matching the task type from the plurality of candidate processing task groups as a target task group based on the task type of the initial target task, and obtaining the subsequent target task based on the target task group, the task type comprising a task information type, a parameter configuration type and a geographic data type.

2. The method of claim 1, wherein, the determination of the initial target task of each sub-node based on the predicted execution time of each processing task comprises: selecting at least one processing task with the sum of the predicted execution times within a preset execution time range from the plurality of processing tasks as the initial target task of the sub-node.

3. The method of claim 1, wherein, the generation of the first task scheduling information of each sub-node based on the initial target task comprises: generating execution order information of the initial target task in time size order based on the predicted execution time of the initial target task; obtain a storage address of task data of the initial target task in the master node; obtain an initial predicted total execution time based on a sum of predicted execution times of the initial target task of the sub-nodes, and obtain a task scheduling time based on a product of the initial predicted total execution time and a task scheduling factor, the task scheduling time being used to represent a time node of planning a next execution task for the sub-nodes; generate the first task scheduling information based on a task identifier of the initial target task, the execution sequence information, the storage address and the task scheduling time.

4. The method of claim 1, wherein, The selecting, based on the task type of the initial target task, one target task group from the multiple candidate processing task groups that match the task type as the target task group, and obtaining the subsequent target task based on the target task group, includes: obtaining a target task information type, a target parameter configuration type and a target geographic data type of the initial target task; selecting, from the candidate processing task groups, a candidate processing task group that matches the target task information type in a task information type, matches the target parameter configuration type in a parameter configuration type, and matches the target geographic data type in a geographic data type, as the target task group; obtaining the subsequent target task based on all processing tasks in the target task group.

5. The method of claim 4, wherein, The generating, based on the subsequent target task, the second task scheduling information, includes: determining, based on the target task information type and a task information type of the subsequent target task, a repeated information type data between the initial target task and the subsequent target task; determining, based on the target parameter configuration type and a parameter configuration type of the subsequent target task, a repeated parameter data between the initial target task and the subsequent target task; determining, based on the target geographic data type and a geographic data type of the subsequent target task, a repeated geographic data between the initial target task and the subsequent target task; generating a repeated data identifier based on the repeated information type data, the repeated parameter data and the repeated geographic data, and generating the second task scheduling information based on the subsequent target task and the repeated data identifier, the repeated data identifier being used to indicate that the sub-nodes do not need to obtain the repeated information type data, the repeated parameter data and the repeated geographic data when performing task data acquisition.

6. The method of claim 1, wherein, The method further includes, after the generating, based on the initial target task, the first task scheduling information of each sub-node and sending the first task scheduling information to the master node and the sub-nodes: generating a task termination instruction when the real-time execution time exceeds a preset abnormal time, the preset abnormal time being greater than the predicted execution time of the initial target task; sending the task termination instruction to the sub-nodes, so that the sub-nodes terminate execution of the initial target task; setting a task state of the initial target task to a task to-be-executed state.

7. A task scheduling apparatus of a node cluster, characterized by comprising: The node cluster includes a master node and multiple sub-nodes, and the apparatus includes: An acquisition module is configured to acquire a plurality of processing tasks, input each processing task into an execution time prediction model one by one, and obtain a predicted execution time of the processing task. A task determination module is configured to determine an initial target task of each sub-node based on the predicted execution time of each processing task. An initial scheduling module is configured to generate first task scheduling information of each sub-node based on the initial target task, and send the first task scheduling information to the master node and the sub-node, so that the sub-node acquires task data of the initial target task from the master node according to the first task scheduling information. A subsequent scheduling module is configured to allocate a subsequent target task for the sub-node to perform a task during a task execution process of the sub-node according to an execution state of the sub-node and the predicted execution time. The execution state includes a real-time execution time when the sub-node executes the initial target task, and the allocation of the subsequent target task for the sub-node to perform a task according to the execution state of the sub-node and the predicted execution time includes: setting a task state of the initial target task to a task execution state; acquiring the real-time execution time, and selecting a processing task with a task state of a task to be executed from the plurality of processing tasks as the subsequent target task of the sub-node according to the real-time execution time and a task scheduling time; generating second task scheduling information based on the subsequent target task, and sending the second task scheduling information to the master node and the sub-node, so that the sub-node acquires task data of the subsequent target task from the master node according to the second task scheduling information to perform a task; the selection of the processing task with the task state of the task to be executed from the plurality of processing tasks as the subsequent target task of the sub-node according to the real-time execution time and the task scheduling time includes: when the real-time execution time exceeds the task scheduling time, selecting at least one processing task with the task state of the task to be executed and the same task type from the plurality of processing tasks as a candidate processing task group, the sum of the predicted execution times of the candidate processing task group being within a preset execution time range; selecting a candidate processing task group matching the task type from the plurality of candidate processing task groups as a target task group based on the task type of the initial target task, the task type including a task information type, a parameter configuration type, and a geographic data type, and obtaining the subsequent target task based on the target task group.

8. An electronic device, comprising: A memory and a processor are included, the memory stores a computer program, and the processor implements the task scheduling method of the node cluster in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the task scheduling method of the node cluster in any one of claims 1 to 6.

Images