Task scheduling method of node cluster and related equipment

Abstract

According to the task scheduling method of the node cluster and the related equipment provided by the embodiment of the invention, the node cluster comprises a main node and a plurality of sub-nodes, and the method comprises the following steps: firstly, obtaining a plurality of processing tasks, and inputting each processing task into an execution time prediction model one by one to obtain predicted execution time of the processing tasks; then, based on the predicted execution time of each processing task, determining an initial target task of each child node; and finally, first task scheduling information of each sub-node is generated based on the initial target task, and the first task scheduling information is sent to the main node and the sub-nodes, so that the sub-nodes obtain task data of the initial target task from the main node according to the first task scheduling information, and in the task execution process of the sub-nodes, the task data of the initial target task are obtained. And according to the execution state and the predicted execution time of the sub-node, distributing a subsequent target task to the sub-node for task execution, thereby effectively improving the overall calculation efficiency of the node cluster.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task scheduling method and related equipment for a node cluster. Background Art

[0002] With the rapid growth of modern computing needs, large-scale clusters have been built more and more perfectly. With the continuous improvement of hardware level, computing on large-scale cluster tasks has become a very important task. During this period, how to reasonably schedule tasks to minimize the idle time of computing resources has become a key issue. On nodes with low computing scheduling efficiency, the computing effect reflected by the computing power of the same hardware specifications will be far lower than that of high efficiency. Therefore, efficient task scheduling methods are crucial in the field of big data.

[0003] In the related art, a cloud service cluster based on Kubernetes (K8S) provides a unified scheduling management method, which builds a node cluster based on K8S, connects a large number of cloud servers, and uses Docker container technology to implement algorithm deployment and allocation. This model uses the service mechanism of K8S to achieve load balancing, dynamically allocates computing tasks according to real-time resource occupancy, and effectively schedules and manages computing tasks. However, when executing tasks for large-scale computing tasks, the scheduling management method based on K8S has a relatively small number of task responses for big data computing, and the duration of each operation exceeds the default configuration time of many K8S clusters; and the task allocation is usually based on the real-time load of the working node, but because the load at the beginning of the computing task is generally low, it cannot reflect the overall load of the entire task. In this state, the scheduling management method based on K8S will allocate many tasks to the same computing node in the task allocation stage, causing a large number of other computing nodes to be vacant, resulting in low overall computing efficiency of the node cluster. Summary of the invention

[0004] The embodiments of the present application provide a task scheduling method and related devices for a node cluster, which can improve the overall computing efficiency of the node cluster.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a task scheduling method for a node cluster, wherein the node cluster includes a master node and multiple child nodes, and the method includes:

[0006] Acquire multiple processing tasks, input each of the processing tasks into the execution time prediction model one by one, and obtain the predicted execution time of the processing task;

[0007] Determining an initial target task for each of the child nodes based on the predicted execution time of each of the processing tasks;

[0008] Generate first task scheduling information for each of the child nodes based on the initial target task, and send the first task scheduling information to the master node and the child node, so that the child node obtains task data of the initial target task from the master node according to the first task scheduling information;

[0009] During the task execution process of the subnode, a subsequent target task is allocated to the subnode for task execution according to the execution state of the subnode and the predicted execution time.

[0010] In some embodiments, determining the initial target task of each of the child nodes based on the predicted execution time of each of the processing tasks includes:

[0011] At least one of the processing tasks whose sum of predicted execution time is within a preset execution time range is selected from the multiple processing tasks as the initial target task of the child node.

[0012] In some embodiments, generating the first task scheduling information of each of the subnodes based on the initial target task includes:

[0013] Based on the predicted execution time of the initial target task, generating execution sequence information of the initial target task in order of time size;

[0014] Obtaining the storage address of the task data of the initial target task in the master node;

[0015] Based on the sum of the predicted execution times of the initial target tasks of the child nodes, an initial predicted total execution time is obtained, and based on the product of the initial predicted total execution time and the task scheduling factor, a task scheduling time is obtained, and the task scheduling time is used to represent the time node for planning the next execution task to the child node;

[0016] The first task scheduling information is generated based on the task identifier of the initial target task, the execution order information, the storage address, and the task scheduling time.

[0017] In some embodiments, the execution state includes the real-time execution time when the child node executes the initial target task, and the assigning a subsequent target task to the child node for task execution according to the execution state of the child node and the predicted execution time includes:

[0018] Setting the task status of the initial target task among the multiple processing tasks to a task execution status;

[0019] Acquire the real-time execution time, and select, according to the real-time execution time and the task scheduling time, a processing task whose task status is a task to be executed state from the multiple processing tasks as a subsequent target task of the child node;

[0020] Generate second task scheduling information based on the subsequent target task, and send the second task scheduling information to the master node and the child node, so that the child node obtains the task data of the subsequent target task from the master node according to the second task scheduling information to perform task execution.

[0021] In some embodiments, selecting, according to the real-time execution time and the task scheduling time, a processing task whose task status is a task to be executed state from the multiple processing tasks as a subsequent target task of the child node includes:

[0022] When the real-time execution time exceeds the task scheduling time, at least one processing task with a task status of being in a task-to-be-executed state and a sum of the predicted execution time within a preset execution time range and with the same task type is selected from the multiple processing tasks as an alternative processing task group;

[0023] Based on the task type of the initial target task, an alternative processing task group is selected from the plurality of alternative task groups as a target task group, and the subsequent target task is obtained based on the target task group.

[0024] In some embodiments, the task type includes a task information type, a parameter configuration type, and a geographic data type, the task type based on the initial target task, the task type based on the initial target task, selecting an alternative processing task group matching the task type from a plurality of alternative task groups as the target task group, and obtaining the subsequent target task based on the target task group, including:

[0025] Obtaining a target task information type, a target parameter configuration type, and a target geographic data type of the initial target task;

[0026] Selecting from the candidate task groups a candidate task group whose task information type matches the target task information type, whose parameter configuration type matches the target parameter configuration type, and whose geographic data type matches the target geographic data type as the target task group;

[0027] Based on all processing tasks in the target task group, the subsequent target task is obtained.

[0028] In some embodiments, generating the second task scheduling information based on the subsequent target task includes:

[0029] Based on the target task information type and the task information type of the subsequent target task, determining repeated information type data between the initial target task and the subsequent target task;

[0030] Determining repeated parameter data between the initial target task and the subsequent target task based on the target parameter configuration type and the parameter configuration type of the subsequent target task;

[0031] Determining duplicate geographic data between the initial target task and the subsequent target task based on the target geographic data type and the geographic data type of the subsequent target task;

[0032] A duplicate data identifier is generated based on the duplicate information type data, the duplicate parameter data and the duplicate geographic data, and the second task scheduling information is generated based on the subsequent target task and the duplicate data identifier, wherein the duplicate data identifier is used to indicate that the child node does not need to obtain the duplicate information type data, the duplicate parameter data and the duplicate geographic data when obtaining task data.

[0033] In some embodiments, after generating the first task scheduling information based on all the initial target tasks and sending the first task scheduling information to the master node and the child node, the method further includes:

[0034] When the real-time execution time exceeds a preset abnormal time, generating a task termination instruction, the preset abnormal time being greater than the predicted execution time of the initial target task;

[0035] Sending the task termination instruction to the child node so that the child node terminates execution of the initial target task;

[0036] The task status of the initial target task is set to a task pending execution status.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a task scheduling device for a node cluster, wherein the node cluster includes a master node and a plurality of child nodes, and the device includes:

[0038] An acquisition module, used for acquiring a plurality of processing tasks, inputting each of the processing tasks into the execution time prediction model one by one, and obtaining the predicted execution time of the processing tasks;

[0039] A task determination module, used to determine an initial target task of each of the subnodes based on the predicted execution time of each of the processing tasks;

[0040] An initial scheduling module, used to generate first task scheduling information of each of the child nodes based on the initial target task, and send the first task scheduling information to the master node and the child node, so that the child node obtains the task data of the initial target task from the master node according to the first task scheduling information;

[0041] The subsequent scheduling module allocates a subsequent target task to the subnode for task execution according to the execution status of the subnode and the predicted execution time during the task execution of the subnode.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the task scheduling method of the node cluster as described in the first aspect.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the task scheduling method of the node cluster described in the first aspect above.

[0044] The task scheduling method and related equipment of a node cluster proposed in an embodiment of the present application, the node cluster includes a main node and multiple child nodes, and the method includes: first, obtaining multiple processing tasks, inputting each processing task into an execution time prediction model one by one, and obtaining the predicted execution time of the processing task; then, based on the predicted execution time of each processing task, determining the initial target task of each child node; finally, generating first task scheduling information for each child node based on the initial target task, and sending the first task scheduling information to the main node and the child node, so that the child node obtains the task data of the initial target task from the main node according to the first task scheduling information, and, during the task execution process of the child node, assigning a subsequent target task to the child node for task execution according to the execution status and predicted execution time of the child node. Before the execution of batch processing tasks, the embodiment of the present application uses a pre-generated execution time prediction model to predict the execution time required for each processing task, and uses the predicted execution time of each processing task to select a suitable initial target task for each child node in the node cluster, and generates first task scheduling information corresponding to the initial target task, so that the child nodes can orderly obtain the task data corresponding to the initial target task from the main node according to the first task scheduling information for execution; and in the subsequent process, the task execution of each child node can be scheduled in real time according to the predicted execution time of each processing task, thereby reducing the situation where some child nodes are busy and some child nodes are idle, thereby effectively improving the overall computing efficiency of the node cluster.

[0045] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the structure of a node cluster provided in one embodiment of the present application.

[0047] Figure 2 This is a flowchart of a task scheduling method for a node cluster provided by another embodiment of the present application.

[0048] Figure 3 This is a flowchart for constructing an execution time prediction model provided by another embodiment of the present application.

[0049] Figure 4 yes Figure 2 Flow chart of step 202 in FIG.

[0050] Figure 5 yes Figure 2 Flow chart of step 203 in FIG.

[0051] Figure 6 This is a flowchart of subsequent task scheduling of a node cluster provided by another embodiment of the present application.

[0052] Figure 7 It is a flowchart of node task process management provided by another embodiment of the present application.

[0053] Figure 8 This is a schematic diagram of the task scheduling process of a sub-node provided in another embodiment of the present application.

[0054] Fig. 9 yes Figure 6 Flowchart of step 602 in FIG.

[0055] Fig.10 yes Fig. 9 Flowchart of step 902 in FIG.

[0056] Fig.11 yes Figure 6 Flowchart of step 603 in FIG.

[0057] Fig.12 This is a flowchart of task error correction of a node cluster provided by another embodiment of the present application.

[0058] Fig.13 It is a flowchart of a processing task error correction provided by another embodiment of the present application.

[0059] Fig.14 It is a flowchart of task scheduling processing of a node cluster provided by another embodiment of the present application.

[0060] Fig.15 It is a structural diagram of a task scheduling device for a node cluster provided in one embodiment of the present application.

[0061] Fig.16 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] First, some nouns involved in this application are analyzed:

[0066] The Kubernetes (K8s) cloud service cluster is a highly scalable and automated container management platform that runs in the cloud and can automatically deploy, expand and manage containerized applications. It provides a unified and flexible infrastructure layer that enables developers to efficiently deploy, run and maintain distributed applications in a cloud environment.

[0067] OBS (Object Storage Service) is a high-performance, highly available, secure and reliable cloud object storage service. It can provide storage and management capabilities for massive data and is suitable for a variety of application scenarios, such as backup and archiving, big data analysis, content distribution, etc. Users can easily store and retrieve any type of data through standard RESTful APIs or client tools.

[0068] Feature scaling refers to converting data of different features into a similar range to avoid unfair influence of some features on the model due to their large numerical range. Common feature scaling methods include standardization and normalization. Standardization refers to converting feature values ​​into a distribution with a mean of 0 and a standard deviation of 1. Normalization refers to scaling feature values ​​to the range of [0, 1].

[0069] In the field of machine learning, ensemble learning methods have attracted much attention due to their excellent predictive performance and robustness. Ensemble learning improves the accuracy and stability of the overall model by combining multiple base models. There are two main types of ensemble learning methods: bagging and boosting.

[0070] The bagging method is to create multiple sub-datasets by resampling the training data (with replacement sampling), and each sub-dataset trains an independent base model. Subsequently, the prediction results of these base models are averaged (for regression tasks) or voted (for classification tasks) to obtain the final prediction results. The representative algorithm of the bagging method is Random Forest.

[0071] The boosting method gradually improves the prediction performance of the model by gradually training a series of base models, each model trying to correct the errors of the previous model. Common boosting algorithms include AdaBoost and Gradient Boosting Machine (GBM).

[0072] As a typical representative of the bagging method, random forest combines the advantages of bagging and decision trees. It constructs multiple base models by randomly extracting multiple sub-datasets from the original training data and training a decision tree on each sub-dataset. When each tree node splits, random forest also randomly selects a feature subset to find the best split point. This random feature selection reduces the correlation between decision trees, thereby improving the diversity and stability of the ensemble model.

[0073] Random forest achieves high-precision prediction by integrating the prediction results of multiple decision trees. For classification tasks, random forest uses majority voting to determine the final prediction category; for regression tasks, it calculates the average of all tree prediction results to determine the final prediction value. This makes random forest not only have high prediction accuracy, but also effectively prevent overfitting and have high robustness.

[0074] With the rapid growth of modern computing needs, large-scale clusters have been built more and more perfectly. With the continuous improvement of hardware level, computing on large-scale cluster tasks has become a very important task. During this period, how to reasonably schedule tasks to minimize the idle time of computing resources has become a key issue. On nodes with low computing scheduling efficiency, the computing effect reflected by the computing power of the same hardware specifications will be far lower than that of high efficiency. Therefore, efficient task scheduling methods are crucial in the field of big data.

[0075] In the related art, a cloud service cluster based on Kubernetes (K8S) provides a unified scheduling management method, which builds a node cluster based on K8S, connects a large number of cloud servers, and uses Docker container technology to implement algorithm deployment and allocation. This model uses the service mechanism of K8S to achieve load balancing, dynamically allocates computing tasks according to real-time resource occupancy, and effectively schedules and manages computing tasks. However, when executing tasks for large-scale computing tasks, the scheduling management method based on K8S has a relatively small number of task responses for big data computing, and the duration of each operation exceeds the default configuration time of many K8S clusters; and the task allocation is usually based on the real-time load of the working node, but because the load at the beginning of the computing task is generally low, it cannot reflect the overall load of the entire task. In this state, the scheduling management method based on K8S will allocate many tasks to the same computing node in the task allocation stage, causing a large number of other computing nodes to be vacant, resulting in low overall computing efficiency of the node cluster.

[0076] In order to improve the overall computing efficiency of the node cluster, the embodiment of the present application uses a pre-generated execution time prediction model to predict the execution time required for each processing task before the execution of a batch of processing tasks, and uses the predicted execution time of each processing task to select a suitable initial target task for each child node in the node cluster, and generates first task scheduling information corresponding to the initial target task, so that the child nodes can orderly obtain the task data corresponding to the initial target task from the main node according to the first task scheduling information for execution; and in the subsequent process, the task execution of each child node can be scheduled in real time according to the predicted execution time of each processing task, thereby reducing the situation where some child nodes are busy and some child nodes are idle, thereby effectively improving the overall computing efficiency of the node cluster.

[0077] In order to better describe the task scheduling method of the node cluster provided by the present application, the node cluster applied to the task scheduling method of the node cluster is first described below. Figure 1 , is a schematic diagram of a node cluster structure provided in an embodiment of the present application. Figure 1As shown in , the node cluster includes at least one master node, multiple child nodes and a model node. The master node is used to perform cluster management of multiple child nodes in the node cluster, respond to the execution of batch processing tasks, store the batch processing tasks, and perform corresponding network plug-in settings. The child nodes include a miniconda image for executing processing tasks, a cloud hard disk for caching task data of processing tasks, and a corresponding task list. The model node is used to build an execution time prediction model and perform model training on the execution time prediction model; and use the trained execution time prediction model to predict the execution time of processing tasks and perform task allocation for processing tasks.

[0078] In addition, based on K8s, the data storage of the master node can be realized through OBS cloud storage. OBS cloud storage is used to store task data corresponding to the processing task, task information storage, and sample data for model training.

[0079] In conjunction with the above node cluster, the task scheduling method and related devices of the node cluster provided in the embodiment of the present application will be further described below. The task scheduling method of the node cluster provided in the embodiment of the present application can be applied to the model node in the node cluster, or the controller connected to the node cluster.

[0080] The following will describe in detail the task scheduling method of the node cluster in the embodiment of the present application. Figure 2 , which is an optional flowchart of the task scheduling method for a node cluster provided in an embodiment of the present application, Figure 2 The method may include but is not limited to steps 201 to 204. It can also be understood that this embodiment Figure 2 The order of step 201 to step 204 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0081] Step 201: Acquire multiple processing tasks, input each processing task into the execution time prediction model one by one, and obtain the predicted execution time of the processing task.

[0082] The following is a detailed description of step 201.

[0083] In some embodiments, when a node cluster responds to the execution of multiple processing tasks in a batch, the node cluster first obtains the task information of the multiple processing tasks, and the master node stores the task data of these processing tasks. Next, these processing tasks need to be orderly assigned and scheduled to multiple sub-nodes that can be used for task processing. In order to further improve the overall computing efficiency of the node cluster, in this embodiment, each processing task needs to be input into a pre-built execution time prediction model for prediction to obtain the predicted execution time of each processing task. This facilitates the subsequent use of the predicted execution time of each processing task to appropriately assign and schedule multiple processing tasks to the corresponding sub-nodes for execution, so as to improve the overall computing efficiency of the node cluster.

[0084] In some embodiments, a node cluster is provided with 2 master nodes, 200 child nodes, and 1 model node, and there are 10,000 processing tasks in the batch processing tasks to be executed. Each child node is installed with a container service so that task data can be processed for the processing tasks. It is understandable that the management method of the cluster is not limited to the node cluster of this example, and the number of nodes can range from hundreds to tens of thousands depending on the resources and the number of calculations required. In addition, each two nodes (including master nodes, child nodes, and model nodes) can communicate with each other, and each child node is equipped with a cloud hard disk for configuring, reading, and transmitting task data. In addition, each child node is equipped with a suitable image for use by a computing program for data processing of processing tasks, such as Miniconda4.1.2 as the basic image, which contains the required scientific computing libraries and tools to ensure the consistency and dependency management of the computing environment. GDAL spams, scikit-learn and other packages are added on the basis of the basic image to read, write and process high-resolution image data. In addition, the node cluster configures the cloud storage file service system to ensure that each node can connect to the same cloud service location to pull the configuration file. For example, there are three node clusters, a total of 600 sub-nodes and 6 master nodes, and every 2 master nodes manage 200 sub-nodes. According to the total number of nodes and the name of the pod, the total number of resources available for calculation is collected to generate a node list. In this embodiment, we calculate remote sensing image data, which is divided into 23 years of data from 2000 to 2023 according to the year and longitude and latitude. Each group has more than 70,000 tiles (computing nodes), and the time of each computing task is uneven, and the computing load is also uneven.

[0085] After allocating the computing tiles, the model in step 2 is used to make a budget and perform regression analysis on the estimated completion time of each task. In this example, the remote sensing data of sentineal2 is used. The specific steps are to traverse the name of the data folder corresponding to each task, extract the eigenvalues ​​for the folder name, input the extracted eigenvalues ​​into the model for regression operation, and obtain the predicted and predicted execution time. At the same time, the time data corresponding to all processing tasks are recorded and put into the task pool waiting for calculation.

[0086] In addition, a task queue is created based on the task list waiting for calculation. In the initial state, the model performs task time calculations, and each child node is evenly assigned more tasks (4 tasks in this example). At the same time, the estimated task settlement time of each child node is calculated by the model node, and all of them are placed in the task queue control service for control. In this embodiment, the total amount of processing tasks is 70,000, and the number of child nodes available for calculation is 600. According to the cluster type, it can be expanded, and every 200 child nodes form a group of node clusters, and each group of node clusters can undertake about 20,000 tasks per round. Generate the relevant configuration files of 70,000 tasks in advance through the script, store them in the task pool of the obs file management system, and at the same time, create a list of running tasks and a list of completed tasks. For this process, the numbers cited in this embodiment are only for ease of understanding. In actual situations, the number of child nodes in each node cluster can range from tens to tens of thousands, and the number of processing tasks can range from hundreds to millions.

[0087] In some embodiments, in order to further improve the prediction accuracy of the predicted execution time of the processing task, it is necessary to first construct and train a suitable execution time prediction model. The following will describe in detail how to obtain the execution time prediction model.

[0088] Reference Figure 3 , is a flowchart of building an execution time prediction model provided by an embodiment of the present application. Figure 3 As shown, in this embodiment, the random forest algorithm is selected as a method for building an execution time prediction model. The specific operation steps are as follows (explained in conjunction with the data processing process):

[0089] 1) Data preparation and feature extraction, first, extract the various parts of the tile number TileID from the data set, including ZoneNumber, LatitudeBand, EastingBand and NorthingBand, and add these features to the data frame. Among them, a certain scale of data is used as the initial training set. In this embodiment, 200,000 data of the time spent on calculating sentinal2 tasks are recorded, and the data set used by each group of tasks is a tile, and the folder of the tile is represented by a series of data, such as S2A_MSIL2A_20190210T031851_N0211_R118_T50UNV_20190210T053429; Among them, 2019 represents the year of the data. We use the data of the same year every time we calculate, and the same information in different years has no special difference in the amount of calculation. So take another valuable feature value T50UNV, which represents the geographical location information of this data information, specifically described as: T represents the UTM grid partition (usually omitted in the naming, does not affect the conversion). 50 represents the UTM zone number (zonenumber). UNV represents the UTM grid square identifier, which can be converted into geographic location information. U represents the latitude band within the UTM band. The letters from C to X (excluding I and O) represent the latitude band from south to north, and each letter represents a latitude range of 8 degrees. N represents the grid column from west to east (easting). V represents the grid row from south to north (northing). It can be seen that in the features of this example, the feature content weights represented by 50 and U are much greater than those of N and V. At the same time, these two features represent independent geographic location information and should not be interfered by numerical values, so a series of processing is performed on this feature value below.

[0090] 2) Feature processing. In order to handle the problem of different weights in the features, a technique called "One-Hot Encoding" can be used to process categorical variables. In the example of this embodiment, the first two digits of the tile number can be equivalently regarded as the area of ​​longitude and latitude. These two data represent the approximate location of the geographic information, but should not be used as numerical or letter information, so these two features are encoded with one-hot encoding. That is, ZoneNumber and LatitudeBand are encoded with one-hot encoding (One-Hot Encoding) to convert the categorical features into binary vectors; such as ZoneNumber: the value is 1-60 and the range is (1,61); LatitudeBand: the specific value is CDEFGHJKLMNPQRSTUVWX and its string list value is used as its one-hot encoding. At the same time, principal component analysis and feature amplifier are used to increase the weights of these two important features. In addition, EastingBand and NorthingBand are standardized (StandardScaler) to convert the numerical features into a distribution with a mean of 0 and a standard deviation of 1.

[0091] 3) Model pipeline, create a pipeline that includes the preprocessing steps and the random forest regression model. This ensures that the data goes through the same preprocessing steps during training and prediction.

[0092] 4) Dataset splitting: Split the dataset obtained in 1) into a training set and a test set for subsequent model training and evaluation.

[0093] 5) Model training: Use the training set data to train the random forest regression model. In this process, the specific steps of random forest include: Random sampling (Bagging): Randomly extract multiple samples from the training data (with replacement sampling) to form multiple sub-sample sets. Construct multiple decision trees: For each sub-sample set, build a decision tree. During the construction process of each tree, the best split point is found by randomly selecting a feature subset. The prediction results of the integrated decision tree: In the final prediction, the random forest averages the prediction results of multiple decision trees (regression task) or votes (classification task). After the training is completed, save the model for task time prediction.

[0094] 6) Iterative training, during the continuous operation of the task, these data can be recursively used for model training. Every time new data arrives, the model can be updated to keep it up to date and optimal. In this embodiment, after recording 100,000 tasks each time, the above steps 1-5 will be recursively called once according to the original model, and the model will be continuously updated to make the prediction more accurate. After the above steps, a prediction model for the execution time of the sentineal2 computing task can be obtained based on the tile number, which is used for task time prediction and scheduling allocation in subsequent steps.

[0095] Step 202: Based on the predicted execution time of each processing task, determine the initial target task of each child node.

[0096] The following is a detailed description of step 202.

[0097] In some embodiments, after the predicted execution time of each processing task is obtained using the execution time prediction model, the initial target task for processing the batch of processing tasks is then assigned to each subnode based on the predicted execution time of each processing task, as will be further described below.

[0098] Reference Figure 4 , based on the predicted execution time of each processing task, determining the initial target task of each child node, including the following steps 401.

[0099] Step 402: Select at least one processing task whose sum of predicted execution time is within a preset execution time range from a plurality of processing tasks as an initial target task of the child node.

[0100] The following is step 401.

[0101] In some embodiments, in order to further improve the rationality of the initial target task assigned to each child node, it is necessary to first obtain the processing memory corresponding to each processing task (including the storage memory corresponding to the task data and the running memory corresponding to the execution of the task data), and obtain the node available memory corresponding to each child node (including the node storage memory and node running memory available to the child node). In addition, a suitable preset execution time range is set for each child node in advance, such as one hour to five hours, to ensure that when the node cluster executes the batch of processing tasks, the predicted execution time corresponding to the processing task assigned to each child node can be maintained within this preset execution time range to ensure the execution efficiency of the node cluster.

[0102] Next, for each child node, at least one processing task whose sum of predicted execution time is within the preset execution time range and whose sum of processing memory is less than the node available memory of the child node is selected from multiple processing tasks in the batch as the initial target task of the child node.

[0103] Furthermore, if different sub-nodes execute different types of tasks, in addition to the above step 402, it is also necessary to select a processing task corresponding to the task type of the sub-node as the initial target task of the sub-node.

[0104] Through the above step 401, the processing memory of the processing task and the node available memory of the child node are used as matching parameters, and the preset execution time range and the predicted execution time of each processing task are used as another matching parameter, so as to select a suitable initial target task for each child node as the execution task when each child node executes the batch processing task at the beginning, so as to ensure the execution reliability of the batch processing task in the node cluster.

[0105] In some embodiments, in each round of task allocation, the task queue will first allocate the processing tasks that the model estimates will take longer to process, ensuring that each child node will initially calculate the task queue with a larger computational demand. This ensures that the child node has a higher load (the remaining computation time is longer) for a long period of time.

[0106] Step 203: Generate first task scheduling information for each child node based on the initial target task, and send the first task scheduling information to the master node and the child node, so that the child node obtains task data of the initial target task from the master node according to the first task scheduling information.

[0107] Step 203 is described in detail below.

[0108] In some embodiments, after determining the initial target task of each child node, the first task scheduling information corresponding to each child node is generated based on the initial target task corresponding to each child node, so as to send the first task scheduling information to the main node and the corresponding child node, so that the child node can obtain the task data corresponding to the initial target task from the main node according to the first task scheduling information for task processing. How to generate the first task scheduling information of each child node based on the target task will be further described below.

[0109] Reference Figure 5 , generating the first task scheduling information of each subnode based on the initial target task, including the following steps 501 to 504.

[0110] Step 501: Based on the predicted execution time of the initial target task, the execution sequence information of the initial target task is generated in order of time.

[0111] Step 502: Obtain the storage address of the task data of the initial target task in the master node.

[0112] Step 503: Based on the sum of the predicted execution times of the initial target tasks of the child nodes, the initial predicted total execution time is obtained, and based on the product of the initial predicted total execution time and the task scheduling factor, the task scheduling time is obtained.

[0113] Step 504: Generate first task scheduling information based on the task identifier, execution order information, storage address and task scheduling time of the initial target task.

[0114] Steps 501 to 504 are described in detail below.

[0115] In some embodiments, there is a situation where there are multiple processing tasks in the initial target task of a subnode. In this case, for the subnode, based on the predicted execution time of each processing task in the initial target task, execution order information of the multiple processing tasks in the initial target task is generated from large to small in order of time size.

[0116] Next, the storage address of the task data of multiple processing tasks in the initial target task corresponding to the child node in the main node is obtained, and the initial predicted total execution time is obtained based on the sum of the predicted execution time of the initial target task of the child node, and then the task scheduling time corresponding to the time node indicating that the next execution task can be scheduled for the child node is obtained based on the product of the initial predicted total execution time and the task scheduling factor. Finally, the first task scheduling information is generated based on the task identifier, execution order information, storage address and task scheduling time of the initial target task.

[0117] Through the above steps 501 to 504, the multiple processing tasks in the initial target task are sorted for execution using the predicted execution time of the multiple processing tasks in the initial target task, and combined with the storage address of the initial target task and the first task scheduling information generated by the task scheduling time obtained from the initial predicted total execution time, the execution status of the sub-node for the initial target task can be effectively and neatly identified, thereby effectively improving the reliability and work efficiency of the node cluster in task processing.

[0118] After the child node obtains the first task scheduling information, the child node obtains the host name of the master node through the program based on the first task scheduling information, and locates the configuration file based on the host name in the cloud storage, obtains the type of computing task, the detailed location of the task configuration file corresponding to the task data of the initial target task, the task status and other information through the configuration file. Then, according to the configuration file, the child node pulls the task (that is, obtains the task data corresponding to the initial target task from the master node).

[0119] Step 204: during the task execution process of the subnode, a subsequent target task is assigned to the subnode for task execution according to the execution status and predicted execution time of the subnode.

[0120] Step 204 is described in detail below.

[0121] In some implementations, in the subsequent process, the task execution scheduling of each sub-node can be performed in real time according to the predicted execution time of each processing task, reducing the situation where some sub-nodes are busy and some sub-nodes are idle, thereby effectively improving the overall computing efficiency of the node cluster. The following will further describe how to assign subsequent target tasks to sub-nodes for task execution according to the execution status and predicted execution time of the sub-nodes.

[0122] Reference Figure 6 , assigning a subsequent target task to the subnode for task execution according to the execution status and predicted execution time of the subnode, also includes the following steps 601 to 603.

[0123] Step 601: setting the task state of an initial target task among multiple processing tasks as a task execution state.

[0124] Step 601 is described in detail below.

[0125] In certain embodiments, the execution state of a child node includes the real-time execution time when the child node performs the initial target task. When each child node obtains the initial target task and performs task processing, the task state update of the initial target task in a plurality of processing tasks is set to the task execution state. Then in each child node, as in the process of performing task processing, a judgment will be performed every 5 minutes. When the remaining running time of the current task node task is judged to reach the task scheduling time (such as 30 minutes), two tasks will be actively found in the task queue for application. The length of the remaining total time will be calculated by the model at the same time, ensuring that the remaining total time is higher than the threshold value of setting (the threshold value is two hours in this embodiment, if it is lower than the threshold value, the task will continue to be obtained). In addition to the model judgment, it will also be judged by the identification position whether the computing task is running. If the task has been completed during the judgment, the acquisition of the next processing task will be performed immediately.

[0126] Reference Figure 7 , is a flowchart of a node task process management provided by an embodiment of the present application. Figure 7 As shown, task process management is performed for each child node, wherein process management occupies an important part in the parallel processing program, and its main responsibility is to effectively manage the child nodes within the node cluster. This part of the task is implemented through Python middleware, which can ensure that each child node in the node cluster can execute its own tasks in an orderly manner. When the child node starts the main process, the process will first obtain the processing task information that needs to be run on this child node from the main node and is allocated by the model node. This process is essentially a dynamic resource allocation strategy that can maximize the utilization of resources within the node cluster without causing waste of resources. The processing task details of each child node are managed through a configuration file, which contains detailed task information such as the type of processing task, parameter configuration, and computing resource requirements. In order to achieve efficient management of child nodes within the node cluster, a set of real-time update mechanisms based on status files are also designed in this embodiment. In the process of each child node executing the processing task, its status will be written to the status file in real time, including information such as the ID of the child node, the execution progress of the task, and the usage of resources. In this way, we can read the status file to understand the operation status of each sub-node in real time, so that we can handle the problem in time. At the same time, the OBS file system is used to manage files in real time, and all sub-nodes are connected to the same configuration file and code file management content to achieve unified scheduling.

[0127] Step 602: Obtain the real-time execution time when the child node executes the initial target task, and select a processing task with a task status of task to be executed from multiple processing tasks as the subsequent target task of the child node based on the real-time execution time and the task scheduling time.

[0128] Step 602 is described in detail below.

[0129] In some embodiments, based on Figure 7 The node task process management shown in the figure needs to obtain the real-time execution time of the child node when executing the initial target task in real time or periodically during the process of the child node executing the target task, and then select the processing task with the task status of the task to be executed from multiple processing tasks as the subsequent target task of the child node according to the real-time execution time and the task scheduling time, so that the child node can seamlessly execute the next target task after executing the initial target task, thereby avoiding the idle situation of the child node, and effectively improving the work efficiency of the node cluster in task processing.

[0130] Reference Figure 8, is a schematic diagram of a task scheduling process of a sub-node provided in an embodiment of the present application. Figure 8 As shown in , the task status of each child node during the execution of the processing task is detected through the python middleware. When the task process switching scheduling is met, the task data of the new processing task is obtained from the cloud container, and after the next target task is executed, the code is hot-replaced to switch to the next target task for execution to ensure the continuous execution of the child node's task. In the mirror pod of each child node, the host runs the same main process, polls and queries new tasks, and according to the task configuration details, or a detailed task list, each computing task (tile) starts a separate child process to complete the calculation. In this embodiment, the calculation is remote sensing related data, and the computing tasks are separated by time and space, and each computing task retrieves the corresponding basic data. In order to achieve efficient management of the child process within the Pod, a process is designed here, and its status will be written to the status file in real time, including the ID of the child node, the execution progress of the processing task, the usage of resources and other information. In this way, we can understand the running status of each child node in real time by reading the status file, so that we can handle the corresponding problems in time when problems occur. In addition, after each processing task is completed, the task completion information will be sent out and recorded in the completed task list. Regardless of whether the task is successful or failed, there will be different status records.

[0131] The process of selecting subsequent target tasks for a child node will be further described below.

[0132] Reference Fig. 9 , according to the real-time execution time and the task scheduling time, selecting a processing task whose task status is a task to be executed state from multiple processing tasks as a subsequent target task of the child node, including the following steps 901 to 902.

[0133] Step 901: When the real-time execution time exceeds the task scheduling time, select from multiple processing tasks a task state that is a task to be executed, and whose sum of predicted execution time is within the preset execution time range and at least one processing task of the same task type as an alternative processing task group.

[0134] Step 902: Based on the task type of the initial target task, select an alternative processing task group that matches the task type from multiple alternative task groups as the target task group, and obtain a subsequent target task based on the target task group.

[0135] Steps 901 to 902 are described in detail below.

[0136] In some embodiments, when it is determined that the real-time execution time of the subnode to execute the initial target task exceeds the task scheduling time, multiple processing tasks with the task state being the task to be executed state and the sum of the predicted execution time being within the preset execution time range and the same task type are selected from multiple processing tasks as alternative processing task groups; next, based on the task type of the initial target task corresponding to the subnode, an alternative processing task group matching the task type is selected from multiple alternative task groups as the target task group corresponding to the processing task subsequently executed by the subnode. And, based on all processing tasks in the target task group, the subsequent target task corresponding to the subnode is obtained. Among them, the task type includes task information type, parameter configuration type and geographic data type. It can be understood that the task information type refers to various attributes or data types used to describe and define a processing task; the parameter configuration type refers to various setting options used in software development, system configuration or algorithm design corresponding to the processing task; the geographic data type refers to a data structure used to represent the surface features of the earth.

[0137] The following further describes the process of how to select a target task group based on the task type of the initial target task.

[0138] Reference Fig.10 , based on the task type of the initial target task, an alternative processing task group matching the task type is selected from multiple alternative task groups as the target task group, and a subsequent target task is obtained based on the target task group, including the following steps 1001 to 1003.

[0139] Step 1001: Obtain the target task information type, target parameter configuration type, and target geographic data type of the initial target task.

[0140] Step 1002: Select from the candidate task groups a candidate task group whose task information type matches the target task information type, whose parameter configuration type matches the target parameter configuration type, and whose geographic data type matches the target geographic data type as the target task group.

[0141] Step 1003: Based on all processing tasks in the target task group, obtain subsequent target tasks.

[0142] Steps 1001 to 1003 are described in detail below.

[0143] In some embodiments, the target task information type, target parameter configuration type, and target geographic data type of the target task are first obtained. Next, an alternative task group whose task information type matches the target task information type, whose parameter configuration type matches the target parameter configuration type, and whose geographic data type matches the target geographic data type is selected from the alternative task group as the target task group, thereby selecting a processing task that is more similar to the initial target task performed by the child node as the subsequent target task to be subsequently performed by the child node, so that when subsequently obtaining the task data of the subsequent target task, it is not necessary to obtain the repeated task data between the subsequent target task and the initial target task, thereby improving the acquisition efficiency of the child node when acquiring task data.

[0144] Step 603: Generate second task scheduling information based on the subsequent target task, and send the second task scheduling information to the master node and the child node, so that the child node obtains the task data of the subsequent target task from the master node according to the second task scheduling information to perform task execution.

[0145] The following is a detailed description of step 603.

[0146] In some embodiments, after determining the subsequent target task of the child node, a process similar to the process of generating the first task scheduling information based on the initial target task in step 203 above is used to generate second task scheduling information based on the subsequent target task, and the second task scheduling information is sent to the main node and the child node, so that the child node obtains the task data of the subsequent target task from the main node according to the second task scheduling information to perform task execution.

[0147] In addition, since the selected subsequent target task is more similar to the initial target task, when generating the second task scheduling information, a duplicate data identifier between the subsequent target task and the initial target task will also be generated, and the duplicate data identifier will be stored in the second task scheduling information, as described below.

[0148] Reference Fig.11 , generating second task scheduling information based on the subsequent target task, including the following steps 1101 to 1104.

[0149] Step 1101: Based on the target task information type and the task information type of the subsequent target task, determine the repeated information type data between the initial target task and the subsequent target task.

[0150] Step 1102: Based on the target parameter configuration type and the parameter configuration type of the subsequent target task, determine the repeated parameter data between the initial target task and the subsequent target task.

[0151] Step 1103: Based on the target geographic data type and the geographic data type of the subsequent target task, determine the duplicate geographic data between the initial target task and the subsequent target task.

[0152] Step 1104: Generate a repeated data identifier based on the repeated information type data, the repeated parameter data, and the repeated geographic data, and generate second task scheduling information based on the subsequent target task and the repeated data identifier.

[0153] Steps 1101 to 1104 are described in detail below.

[0154] In some embodiments, after determining the subsequent target task of the child node, the repeated information type data between the initial target task and the subsequent target task is determined based on the target task information type and the task information type of the subsequent target task; and, based on the target parameter configuration type and the parameter configuration type of the subsequent target task, the repeated parameter data between the initial target task and the subsequent target task is determined; and, based on the target geographic data type and the geographic data type of the subsequent target task, the repeated geographic data between the initial target task and the subsequent target task is determined; finally, based on the repeated information type data, repeated parameter data and repeated geographic data, a repeated data identifier is generated to indicate that the child node does not need to obtain repeated information type data, repeated parameter data and repeated geographic data when acquiring task data, and second task scheduling information is generated based on the subsequent target task and the repeated data identifier.

[0155] Through the above steps 901 to 902, 1001 to 1003, and 1101 to 1104, when the real-time execution time of the target initial task of the child node reaches the task scheduling time, the predicted execution time is used to select a processing task whose sum of the predicted execution time of multiple processing tasks is within the preset execution time range to generate multiple alternative processing task groups that ensure that the child node can perform continuous and stable work; next, the task type (including task information type, parameter configuration type, and geographic data type) of the initial target task of the child node is matched with multiple alternative processing task groups, so as to select an alternative processing task group that is more similar to the initial target task as the subsequent target task of the child node; at the same time, the repeated data identifier generated by the repeated data between the subsequent target task and the initial target task is used, so that when the child node obtains the task data of the subsequent target task from the main node, it is not necessary to obtain the repeated task data between the subsequent target task and the initial target task, thereby improving the acquisition efficiency of the child node when acquiring task data, and then effectively improving the work efficiency of task processing in the node cluster.

[0156] In some embodiments, after the child node uses the second task scheduling information to perform the task data of the subsequent target task and executes it, the node cluster will repeat the above actions until all the processing tasks of the batch are executed. However, during the execution of the processing task by a certain child node, a processing anomaly may occur, causing the real-time execution of a certain processing task to take too long. For this situation, appropriate error correction is required. The error correction process will be further described below.

[0157] Reference Fig.12 After generating the first task scheduling information based on all the initial target tasks and sending the first task scheduling information to the master node and the child nodes, the task scheduling method of the node cluster also includes the following steps 1201 to 1203.

[0158] Step 1201: When the real-time execution time exceeds the preset abnormal time, a task termination instruction is generated, and the preset abnormal time is greater than the predicted execution time of the initial target task.

[0159] Step 1202: Send a task termination instruction to the child node, so that the child node terminates execution of the initial target task.

[0160] Step 1203: Set the task status of the initial target task to a task pending execution status.

[0161] Steps 1201 to 1203 are described in detail below.

[0162] Reference Fig.13 , is a flowchart of a processing task error correction provided by an embodiment of the present application. Fig.13 As shown in , the calculation time (i.e., real-time execution time) of each processing task executed by all child nodes is detected in real time. When the real-time execution time of a child node for a processing task exceeds the preset abnormal time (e.g., 12 hours), a task termination instruction will be generated, wherein the preset abnormal time is greater than the predicted execution time of each processing task. The task termination instruction is then sent to the child node so that the child node kills the processing task (i.e., terminates the execution of the processing task), and then sets the task status of the processing task to the task pending execution status, so as to facilitate the subsequent scheduling of the processing task to another child node for reprocessing, thereby effectively improving the reliability of the node cluster in task processing.

[0163] It should be noted that killing a task and starting a new task are two completely independent processes. The tasks that are re-pulled are the tasks uniformly assigned by the task scheduling system, not the tasks that were originally killed. This approach ensures that the same task list cannot be calculated repeatedly due to machine problems.

[0164] In addition, since there are many child nodes in each node cluster, it takes time for nodes to communicate with each other. There is a difference between the time when the task is started and the time when the task status is recorded. Note that there should be fixed rules for the length of time for scheduling and status query, and there should be no contradictions. In this embodiment, a certain time difference is used. Specifically, the execution of the processing task is started 6 minutes after the child node is assigned to the corresponding processing task through the model node. And feedback is given that the processing task has started to run. When the processing task is about to be completed and the model node assigns a new processing task to the child node, at least 6.5 minutes should be left before a new model assignment is made. Otherwise, a new task assignment is made before the task is started, which will prompt the task to end in advance, causing confusion in the task progress.

[0165] Reference Fig.14 , is a flowchart of the task scheduling process of the node cluster provided in the embodiment of the present application. The task scheduling system described in the task scheduling method of the node cluster provided in the present application realizes the fine scheduling of large-scale computing clusters by integrating the advanced technologies of k8s cluster management and OBS parallel file system. Through the active task pulling mechanism and the monitoring of the daemon process, the system ensures the optimal allocation of resources and the efficient execution of computing tasks, and maintains stability even in the face of fluctuations in the amount of tasks. When encountering a computing bottleneck, the system can quickly reallocate tasks to avoid single-point delays affecting the overall progress. Such a design not only greatly improves computing efficiency, but also optimizes resource utilization while ensuring stable operation, demonstrating a powerful and adaptive computing resource management system that can effectively address efficiency and resource allocation issues that traditional methods cannot solve.

[0166] The task scheduling method and related equipment of the node cluster proposed in the embodiment of the present application, the node cluster includes a main node and multiple child nodes, the method includes: first, obtaining multiple processing tasks, inputting each processing task into the execution time prediction model one by one, and obtaining the predicted execution time of the processing task; then, obtaining the processing memory of each processing task, and obtaining the node available memory of each child node; selecting at least one processing task whose sum of predicted execution time is within the preset execution time range and whose sum of processing memory is less than the node available memory from the multiple processing tasks as the initial target task of the child node, generating the execution order information of the initial target task in the order of time based on the predicted execution time of the initial target task, and obtaining the task data of the initial target task in the main node; The storage address in the point, based on the sum of the predicted execution time of the initial target task of the child node, obtains the initial predicted total execution time, and based on the product of the initial predicted total execution time and the task scheduling factor, obtains the task scheduling time, the task scheduling time is used to characterize the time node for planning the next execution task to the child node, based on the task identifier, execution sequence information, storage address and task scheduling time of the initial target task, generate the first task scheduling information, and send the first task scheduling information to the main node and the child node, so that the child node obtains the task data of the initial target task from the main node according to the first task scheduling information; then, the task state of the initial target task is set to the task execution state, and the real-time execution time when the child node executes the initial target task is obtained;And when the real-time execution time exceeds the task scheduling time, select a task status as a task to be executed from multiple processes, and the sum of the predicted execution time is within the preset execution time range, and at least one processing task with the same task type is used as an alternative processing task group, obtain the target task information type, target parameter configuration type and target geographic data type of the initial target task, select an alternative task group from the alternative task group whose task information type matches the target task information type, whose parameter configuration type matches the target parameter configuration type, and whose geographic data type matches the target geographic data type as the target task group, based on all processing tasks in the target task group, obtain a subsequent target task, and based on the target task information type and the task information type of the subsequent target task, determine the number of repeated information types between the initial target task and the subsequent target task. According to the target parameter configuration type and the parameter configuration type of the subsequent target task, the repeated parameter data between the initial target task and the subsequent target task is determined, and the repeated geographic data between the initial target task and the subsequent target task is determined based on the target geographic data type and the geographic data type of the subsequent target task, and a repeated data identifier is generated based on the repeated information type data, repeated parameter data and repeated geographic data, and the second task scheduling information is generated based on the subsequent target task and the repeated data identifier, and the repeated data identifier is used to indicate that the child node does not need to obtain repeated information type data, repeated parameter data and repeated geographic data when acquiring task data, and the second task scheduling information is sent to the main node and the child node, so that the child node acquires the task data of the subsequent target task from the main node according to the second task scheduling information to perform task execution.;

[0167] In an embodiment of the present application, before the execution of a batch of processing tasks, a pre-generated execution time prediction model is used to predict the execution time required for each processing task, and the processing memory of the processing task and the node available memory of the child node are used as matching parameters, and a preset execution time range and the predicted execution time of each processing task are used as another matching parameter, so as to select a suitable initial target task for each child node as the execution task when each child node executes the batch processing task at the beginning, so as to ensure the execution reliability of the batch processing task in the node cluster; and, by using the time size of the predicted execution time of multiple processing tasks in the initial target task, the execution of multiple processing tasks in the initial target task is sorted, and combined with the storage address of the initial target task and the first task scheduling information generated by the task scheduling time obtained by the initial predicted total execution time, the execution status of the child node for the initial target task can be effectively and neatly identified, thereby effectively improving the reliability and work efficiency of the node cluster in task processing; and in the subsequent process, each processing task can be sorted in real time according to the predicted execution time. The task execution scheduling of the sub-nodes can reduce the situation where some sub-nodes are busy and some sub-nodes are idle. In addition, when the real-time execution time of the sub-node processing target initial task reaches the task scheduling time, the predicted execution time is used to select the processing tasks whose predicted execution time sum of multiple processing tasks is within the preset execution time range to generate multiple alternative processing task groups to ensure that the sub-node can perform continuous and stable work. Next, the task type (including task information type, parameter configuration type and geographic data type) of the sub-node's initial target task is matched with multiple alternative processing task groups, so as to select an alternative processing task group that is more similar to the initial target task as the subsequent target task of the sub-node. At the same time, the repeated data identifier generated by the repeated data between the subsequent target task and the initial target task is used, so that when the sub-node obtains the task data of the subsequent target task from the main node, it is not necessary to obtain the repeated task data between the subsequent target task and the initial target task, thereby improving the acquisition efficiency of the sub-node when obtaining task data, and then effectively improving the overall work efficiency of task processing in the node cluster.

[0168] The present application also provides a task scheduling device for a node cluster, which can implement the task scheduling method for the node cluster. Fig.15 , the device 1500 comprises:

[0169] An acquisition module 1510 is used to acquire multiple processing tasks, input each processing task into the execution time prediction model one by one, and obtain the predicted execution time of the processing task;

[0170] A task determination module 1520, for determining an initial target task for each child node based on the predicted execution time of each processing task;

[0171] The initial scheduling module 1530 is used to generate first task scheduling information of each sub-node based on the initial target task, and send the first task scheduling information to the main node and the sub-node, so that the sub-node obtains the task data of the initial target task from the main node according to the first task scheduling information;

[0172] The subsequent scheduling module 1540 allocates subsequent target tasks to the subnodes for task execution according to the execution status and predicted execution time of the subnodes during the task execution of the subnodes.

[0173] In some embodiments, the task determination module 1520 is further configured to:

[0174] At least one processing task whose sum of predicted execution time is within a preset execution time range is selected from the multiple processing tasks as the initial target task of the child node.

[0175] In some embodiments, the initial scheduling module 1530 is further configured to:

[0176] Based on the predicted execution time of the initial target task, the execution order information of the initial target task is generated in order of time;

[0177] Get the storage address of the task data of the initial target task in the master node;

[0178] The sum of the predicted execution times of the initial target tasks of the child nodes is used to obtain the initial predicted total execution time, and the task scheduling time is obtained based on the product of the initial predicted total execution time and the task scheduling factor. The task scheduling time is used to represent the time node for planning the next execution task to the child node;

[0179] First task scheduling information is generated based on the task identifier, execution order information, storage address and task scheduling time of the initial target task.

[0180] In some embodiments, the post-order scheduling module 1540 is further configured to:

[0181] Set the task status of the initial target task to the task execution status;

[0182] The real-time execution time is obtained, and according to the real-time execution time and the task scheduling time, a processing task whose task status is a task to be executed state is selected from multiple processing tasks as a subsequent target task of the child node;

[0183] The second task scheduling information is generated based on the subsequent target task, and the second task scheduling information is sent to the main node and the child node, so that the child node obtains the task data of the subsequent target task from the main node according to the second task scheduling information to perform task execution.

[0184] In some embodiments, the post-order scheduling module 1540 is further configured to:

[0185] When the real-time execution time exceeds the task scheduling time, at least one processing task with a task status of task to be executed and a sum of predicted execution times within a preset execution time range and with the same task type is selected from multiple processing tasks as a candidate processing task group;

[0186] Based on the task type of the initial target task, an alternative processing task group matching the task type is selected from multiple alternative task groups as the target task group, and a subsequent target task is obtained based on the target task group.

[0187] In some embodiments, the post-order scheduling module 1540 is further configured to:

[0188] Obtain the target task information type, target parameter configuration type, and target geographic data type of the initial target task;

[0189] Selecting, from the candidate task groups, a candidate task group whose task information type matches the target task information type, whose parameter configuration type matches the target parameter configuration type, and whose geographic data type matches the target geographic data type as the target task group;

[0190] Based on all processing tasks in the target task group, the subsequent target task is obtained.

[0191] In some embodiments, the post-order scheduling module 1540 is further configured to:

[0192] Based on the target task information type and the task information type of the subsequent target task, determining the repeated information type data between the initial target task and the subsequent target task;

[0193] Determine the repeated parameter data between the initial target task and the subsequent target task based on the target parameter configuration type and the parameter configuration type of the subsequent target task;

[0194] Based on the target geographic data type and the geographic data type of the subsequent target task, determining the duplicate geographic data between the initial target task and the subsequent target task;

[0195] A duplicate data identifier is generated based on duplicate information type data, duplicate parameter data and duplicate geographic data, and second task scheduling information is generated based on the subsequent target task and the duplicate data identifier. The duplicate data identifier is used to indicate that the child node does not need to obtain duplicate information type data, duplicate parameter data and duplicate geographic data when obtaining task data.

[0196] In some embodiments, the post-order scheduling module 1540 is further configured to:

[0197] When the real-time execution time exceeds the preset abnormal time, a task termination instruction is generated, and the preset abnormal time is greater than the predicted execution time of the initial target task;

[0198] Sending a task termination instruction to the child node so that the child node terminates the execution of the initial target task;

[0199] Set the task status of the initial target task to the task pending execution status.

[0200] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, the specific implementation method of the task scheduling device of the node cluster is basically the same as the specific implementation method of the task scheduling method of the node cluster, and will not be repeated here.

[0201] In an embodiment of the present application, before the execution of a batch of processing tasks, the task scheduling device of the node cluster uses a pre-generated execution time prediction model to predict the execution time required for each processing task, uses the processing memory of the processing task and the node available memory of the child node as matching parameters, and uses a preset execution time range and the predicted execution time of each processing task as another matching parameter, so as to select a suitable initial target task for each child node as the execution task when each child node starts to execute the batch processing task, so as to ensure the execution reliability of the batch processing task in the node cluster; and, using the time size of the predicted execution time of multiple processing tasks in the initial target task, the execution of multiple processing tasks in the initial target task is sorted, and combined with the storage address of the initial target task and the first task scheduling information generated by the task scheduling time obtained by the initial predicted total execution time, the execution status of the child node for the initial target task can be effectively and neatly identified, thereby effectively improving the reliability and work efficiency of the node cluster in task processing; and in the subsequent process, the predicted execution time of each processing task can be determined in real time. The task execution of each sub-node is scheduled in time to reduce the situation where some sub-nodes are busy and some sub-nodes are idle; in addition, when the real-time execution time of the sub-node processing target initial task reaches the task scheduling time, the predicted execution time is used to select the processing tasks whose sum of the predicted execution times of multiple processing tasks is within the preset execution time range to generate multiple alternative processing task groups that ensure that the sub-node can perform continuous and stable work; next, the task type (including task information type, parameter configuration type and geographic data type) of the initial target task of the sub-node is matched with multiple alternative processing task groups, so as to select an alternative processing task group that is more similar to the initial target task as the subsequent target task of the sub-node; at the same time, the repeated data identifier generated by the repeated data between the subsequent target task and the initial target task is used, so that when the sub-node obtains the task data of the subsequent target task from the main node, it is not necessary to obtain the repeated task data between the subsequent target task and the initial target task, thereby improving the acquisition efficiency of the sub-node when obtaining task data, thereby effectively improving the overall work efficiency of task processing in the node cluster.

[0202] The present application also provides an electronic device, including:

[0203] at least one memory;

[0204] at least one processor;

[0205] at least one program;

[0206] The program is stored in the memory, and the processor executes the at least one program to implement the task scheduling method of the node cluster implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0207] See also Fig.16 , Fig.16 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0208] The processor 1601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0209] The memory 1602 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1602 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1602, and the task scheduling method of the node cluster in the embodiment of this application is called and executed by the processor 1601;

[0210] Input / output interface 1603, used to implement information input and output;

[0211] The communication interface 1604 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0212] A bus 1605 that transmits information between various components of the device (e.g., the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604);

[0213] The processor 1601 , the memory 1602 , the input / output interface 1603 and the communication interface 1604 are connected to each other in communication within the device via the bus 1605 .

[0214] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the task scheduling method of the node cluster is implemented.

[0215] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0216] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0217] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0218] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0220] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0222] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0226] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A task scheduling method for a node cluster, characterized in that: The node cluster includes a master node and multiple child nodes, and the method includes: Acquire multiple processing tasks, input each of the processing tasks into the execution time prediction model one by one, and obtain the predicted execution time of the processing task; Determining an initial target task for each of the child nodes based on the predicted execution time of each of the processing tasks; Generate first task scheduling information for each of the child nodes based on the initial target task, and send the first task scheduling information to the master node and the child node, so that the child node obtains task data of the initial target task from the master node according to the first task scheduling information; During the task execution process of the subnode, a subsequent target task is allocated to the subnode for task execution according to the execution state of the subnode and the predicted execution time.

2. The task scheduling method for a node cluster according to claim 1, characterized in that: The determining of the initial target task of each of the child nodes based on the predicted execution time of each of the processing tasks includes: At least one of the processing tasks whose sum of predicted execution time is within a preset execution time range is selected from the multiple processing tasks as the initial target task of the child node.

3. The task scheduling method for a node cluster according to claim 1, characterized in that: The generating the first task scheduling information of each of the subnodes based on the initial target task includes: Based on the predicted execution time of the initial target task, generating execution sequence information of the initial target task in order of time size; Obtaining the storage address of the task data of the initial target task in the master node; Based on the sum of the predicted execution times of the initial target tasks of the subnodes, an initial predicted total execution time is obtained, and based on the product of the initial predicted total execution time and the task scheduling factor, a task scheduling time is obtained, and the task scheduling time is used to represent the time node for planning the next execution task to the subnode; The first task scheduling information is generated based on the task identifier of the initial target task, the execution order information, the storage address, and the task scheduling time.

4. The task scheduling method for a node cluster according to claim 3, characterized in that: The execution state includes the real-time execution time when the child node executes the initial target task, and the assigning of a subsequent target task to the child node for task execution according to the execution state of the child node and the predicted execution time includes: Setting the task status of the initial target task to a task execution status; Acquire the real-time execution time, and select, according to the real-time execution time and the task scheduling time, a processing task whose task status is a task to be executed state from the plurality of processing tasks as a subsequent target task of the child node; Generate second task scheduling information based on the subsequent target task, and send the second task scheduling information to the master node and the child node, so that the child node obtains the task data of the subsequent target task from the master node according to the second task scheduling information to perform task execution.

5. The task scheduling method for a node cluster according to claim 4, characterized in that: The selecting, according to the real-time execution time and the task scheduling time, a processing task whose task status is a task to be executed state from the plurality of processing tasks as a subsequent target task of the child node comprises: When the real-time execution time exceeds the task scheduling time, at least one processing task with a task status of being in a task-to-be-executed state and a sum of the predicted execution time being within a preset execution time range and having the same task type is selected from the multiple processing tasks as an alternative processing task group; Based on the task type of the initial target task, an alternative processing task group matching the task type is selected from the plurality of alternative task groups as the target task group, and the subsequent target task is obtained based on the target task group.

6. The task scheduling method for a node cluster according to claim 5, characterized in that: The task type includes a task information type, a parameter configuration type, and a geographic data type. The task type based on the initial target task, selecting an alternative processing task group matching the task type from a plurality of alternative task groups as a target task group, and obtaining the subsequent target task based on the target task group, includes: Obtaining a target task information type, a target parameter configuration type, and a target geographic data type of the initial target task; Selecting from the candidate task groups a candidate task group whose task information type matches the target task information type, whose parameter configuration type matches the target parameter configuration type, and whose geographic data type matches the target geographic data type as the target task group; Based on all processing tasks in the target task group, the subsequent target task is obtained.

7. The task scheduling method for a node cluster according to claim 6, characterized in that: The generating the second task scheduling information based on the subsequent target task includes: Based on the target task information type and the task information type of the subsequent target task, determining repeated information type data between the initial target task and the subsequent target task; Determining repeated parameter data between the initial target task and the subsequent target task based on the target parameter configuration type and the parameter configuration type of the subsequent target task; Determining duplicate geographic data between the initial target task and the subsequent target task based on the target geographic data type and the geographic data type of the subsequent target task; A duplicate data identifier is generated based on the duplicate information type data, the duplicate parameter data and the duplicate geographic data, and the second task scheduling information is generated based on the subsequent target task and the duplicate data identifier, wherein the duplicate data identifier is used to indicate that the child node does not need to obtain the duplicate information type data, the duplicate parameter data and the duplicate geographic data when obtaining task data.

8. The task scheduling method for a node cluster according to claim 4, characterized in that: After generating the first task scheduling information based on all the initial target tasks and sending the first task scheduling information to the master node and the child node, the method further includes: When the real-time execution time exceeds a preset abnormal time, generating a task termination instruction, the preset abnormal time being greater than the predicted execution time of the initial target task; Sending the task termination instruction to the child node so that the child node terminates execution of the initial target task; The task status of the initial target task is set to a task pending execution status.

9. A task scheduling device for a node cluster, characterized in that: The node cluster includes a master node and a plurality of sub-nodes, and the device includes: An acquisition module, used for acquiring a plurality of processing tasks, inputting each of the processing tasks into the execution time prediction model one by one, and obtaining the predicted execution time of the processing tasks; A task determination module, used to determine an initial target task of each of the subnodes based on the predicted execution time of each of the processing tasks; An initial scheduling module, used to generate first task scheduling information of each of the child nodes based on the initial target task, and send the first task scheduling information to the master node and the child node, so that the child node obtains the task data of the initial target task from the master node according to the first task scheduling information; The subsequent scheduling module allocates a subsequent target task to the subnode for task execution according to the execution status of the subnode and the predicted execution time during the task execution of the subnode.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the task scheduling method for a node cluster as claimed in any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task scheduling method for a node cluster described in any one of claims 1 to 8 is implemented.

Images