Task scheduling method and device, storage medium and processor

Abstract

The invention discloses a task scheduling method and device, a storage medium and a processor. The method comprises the steps of generating task resource configuration information of a target task based on the similarity between the target task and a historical task and task resource configuration information of the historical task; determining a to-be-matched node set corresponding to the target task from the cluster; according to the resource information of the to-be-matched node and the task resource configuration information of the target task, obtaining a resource change rate before and after the to-be-matched node runs the target task; according to the resource change rate before and after the to-be-matched node runs the target task and the task type feature data of the target type in the to-be-matched node, obtaining a prediction score of the to-be-matched node running the target task; determining a target node from the to-be-matched nodes based on the predicted scores of the target tasks operated by the plurality of to-be-matched nodes; and scheduling the target task to the target node. According to the method, the scheduling requirements of various types of tasks and nodes are met, and the task running efficiency of the data sandbox is improved.

Description

Technical Field

[0001] The present application relates to the technical field of task scheduling, and in particular to a task scheduling method, device, storage medium and processor. Background Art

[0002] Data Sandbox is a data management tool that provides an isolated environment for safely exploring and analyzing data. Data Sandbox usually contains a copy of the data so that users can freely experiment without affecting the data in the production environment. A cluster is a group of servers connected by a fast network that work together to complete specific tasks. In a data sandbox environment, a cluster can provide higher processing power and storage capacity to support large-scale data analysis and processing. A cluster generally contains multiple nodes, each of which can be a separate computer or server. In a cluster, each node has its own processor, memory, and storage device and can run tasks independently. In a data sandbox cluster, nodes can be classified into different roles to meet different needs.

[0003] Under the influence of the rapid development of technologies such as cloud computing, big data, and artificial intelligence, enterprises often need to handle multiple types of tasks at the same time. Taking data processing as an example, these tasks can be classified into: data prediction, data analysis, data modeling, data query, etc. These different types of tasks often have different requirements for resource dimensions such as computing resources, storage resources, and network bandwidth. Running tasks in a data sandbox means that it is necessary to configure resources for the tasks and schedule the tasks to nodes. There are currently a variety of resource allocation and task scheduling strategies, such as priority-based scheduling strategies, resource demand-based scheduling strategies, and prediction model-based scheduling strategies. The existing solutions use the above-mentioned common scheduling strategies when allocating tasks to each node, but such strategies often cannot meet the scheduling requirements of multiple types of tasks and nodes, thereby affecting the task operation efficiency of the data sandbox. Summary of the invention

[0004] Based on the above problems, the present application provides a task scheduling method, device, storage medium and processor, the purpose of which is to meet the scheduling needs of various types of tasks and nodes and improve the task operation efficiency of the data sandbox.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, the present application provides a task scheduling method, comprising:

[0007] Based on the similarity between the target task and the historical tasks, and the task resource configuration information of the historical tasks, generating the task resource configuration information of the target task;

[0008] Determine a set of nodes to be matched corresponding to the target task from the cluster; the set of nodes to be matched includes multiple nodes to be matched;

[0009] Obtaining a resource change rate of the node to be matched before and after running the target task according to the resource information of the node to be matched and the task resource configuration information of the target task;

[0010] Obtaining a predicted score of the node to be matched running the target task according to the resource change rate before and after the node to be matched runs the target task, and the task type feature data of the node to be matched corresponding to the target type; the target type is the task type of the target task;

[0011] Determine a target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched each running the target task;

[0012] The target task is dispatched to the target node.

[0013] Optionally, in the above method, generating the task resource configuration information of the target task based on the similarity between the target task and the historical task, and the task resource configuration information of the historical task, includes:

[0014] Based on the similarity with the target task, a plurality of pre-selected tasks are determined from the plurality of historical tasks; the pre-selected tasks are historical tasks whose similarity with the target task is higher than a similarity threshold;

[0015] Determine a reference task from the plurality of pre-selected tasks based on at least one of the respective running time, resource occupancy, and task resource configuration information of the plurality of pre-selected tasks;

[0016] The task resource configuration information of the target task is generated by using the task resource configuration information of the reference task.

[0017] Optionally, the method as described above, based on the respective running times of the plurality of pre-selected tasks, determining a reference task from the plurality of pre-selected tasks comprises:

[0018] Using a task with the shortest running time among the plurality of pre-selected tasks as a reference task;

[0019] Determining a reference task from the plurality of pre-selected tasks based on the resource occupation of each of the plurality of pre-selected tasks includes:

[0020] taking a task that occupies the least resources among the plurality of pre-selected tasks as a reference task;

[0021] Determining a reference task from the plurality of pre-selected tasks based on the task resource configuration information of each of the plurality of pre-selected tasks includes:

[0022] According to the task resource configuration information of each of the plurality of pre-selected tasks, a group of task resource configuration information with the highest overlap rate is determined, and any pre-selected task corresponding to the group of task resource configuration information with the highest overlap rate is used as a reference task.

[0023] Optionally, in the above method, the similarity between the target task and the historical task is obtained as follows:

[0024] Obtain a task comparison reference feature set of the target task and a task comparison reference feature set of the historical task; wherein the task comparison reference feature set of the target task includes data features of the target task, task pre-operation features of the target task, and features of multiple resource dimensions of the target task within a task pre-operation time period; the task comparison reference feature set of the historical task includes data features of the historical task, task operation features of the historical task, and features of multiple resource dimensions of the historical task within a task operation time period;

[0025] Based on the differences in corresponding elements between the task comparison reference feature set of the target task and the task comparison reference feature set of the historical tasks, a similarity index value between the target task and the historical tasks is obtained; the similarity index value is negatively correlated with the similarity.

[0026] Optionally, in the method described above, the task type characteristic data includes a plurality of resource correlation weights corresponding one-to-one to the plurality of resource dimensions; obtaining a predicted score of the node to be matched running the target task according to the resource change rate before and after the node to be matched runs the target task, and the task type characteristic data of the node to be matched corresponding to the target type, comprises:

[0027] The resource load change rate of the node to be matched is calculated according to the resource occupancy index data of the node to be matched and the resource change rate before and after the node to be matched runs the target task; the resource occupancy index data is calculated according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the node to be matched in the cluster in the multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to the multiple resource dimensions;

[0028] Performing normalization processing on the task type feature data of the to-be-matched node corresponding to the target type to obtain normalized task type feature data;

[0029] Calculate the task type feature matching degree of the to-be-matched node corresponding to the target type according to the resource occupancy index data and the normalized task type feature data;

[0030] The predicted score of the to-be-matched node running the target task is calculated based on the resource load change rate and the task type feature matching degree.

[0031] Optionally, in the above method, determining the target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched each running the target task includes:

[0032] A node with the highest prediction score for running the target task is determined from the multiple nodes to be matched as the target node.

[0033] Optionally, in the method as described above, the resource information of the node to be matched includes the total amount of resources corresponding to the node to be matched in multiple resource dimensions, and the amount of resources occupied by the node to be matched in the multiple resource dimensions; the task resource configuration information of the target task includes the resource configuration values ​​of the multiple resource dimensions;

[0034] The obtaining, according to the resource information of the node to be matched and the task resource configuration information of the target task, the resource change rate before and after the node to be matched runs the target task comprises:

[0035] Performing a division operation on the occupied resource amount and the total resource amount in the resource information of the node to be matched according to the resource dimension, to obtain the resource occupation rates respectively corresponding to the multiple resource dimensions before the node to be matched runs the target task;

[0036] According to the resource dimension, sum the resource occupied amount in the resource information of the node to be matched and the resource configuration value in the task resource configuration information of the target task, and divide the sum by the total amount of resources of the corresponding resource dimension in the resource information of the node to be matched, to obtain the resource occupation rates corresponding to the multiple resource dimensions after the node to be matched runs the target task;

[0037] The difference in resource occupancy rate of each resource dimension after the node to be matched runs the target task and before running the target task is divided by the resource occupancy rate of the corresponding resource dimension before running the target task to obtain the resource change rates corresponding to the multiple resource dimensions before and after the node to be matched runs the target task.

[0038] Optionally, in the method described above, determining the set of nodes to be matched corresponding to the target task from the cluster includes:

[0039] A node whose node type matches the target type is determined from the cluster, and the node whose node type matches the target type is added to a to-be-matched node set corresponding to the target task.

[0040] Optionally, in the above method, the classification method of the nodes in the cluster includes:

[0041] According to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions, obtaining resource occupation index data of the target node to be classified; the resource occupation index data includes multiple resource occupation weights corresponding to the multiple resource dimensions one by one;

[0042] According to the task running data of the target node to be classified for multiple task types, multiple groups of task type feature data of the target node to be classified are obtained; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions;

[0043] The resource occupation weight corresponding to each resource dimension in the resource occupation index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types;

[0044] The node category of the target node to be classified is configured according to the numerical values ​​of the adaptability between the target node to be classified and the multiple task types.

[0045] Optionally, in the method as described above, the cluster includes nodes with multiple different identities in the blockchain network; the nodes with multiple different identities include at least: an initiator node and a participant node; the initiator node and the participant node reach a consensus on the same task.

[0046] In a second aspect, the present application provides a task scheduling device, comprising:

[0047] A resource configuration generation module, used to generate task resource configuration information of the target task based on the similarity between the target task and the historical task, and the task resource configuration information of the historical task;

[0048] A first node determination module is used to determine a set of nodes to be matched corresponding to the target task from a cluster; the set of nodes to be matched includes a plurality of nodes to be matched;

[0049] A resource change calculation module, used to obtain the resource change rate of the to-be-matched node before and after running the target task according to the resource information of the to-be-matched node and the task resource configuration information of the target task;

[0050] A node prediction module, used to obtain a prediction score of the node to be matched running the target task according to the resource change rate before and after the node to be matched runs the target task, and the task type feature data of the node to be matched corresponding to the target type; the target type is the task type of the target task;

[0051] A second node determination module is used to determine a target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched respectively running the target task;

[0052] A scheduling module is used to schedule the target task to the target node.

[0053] In a third aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the task scheduling method described in any of the above embodiments.

[0054] In a fourth aspect, the present application provides a processor, which is used to run a program, wherein the program executes the task scheduling method described in any one of the above embodiments when running.

[0055] The method of the present application generates the task resource configuration information of the target task based on the similarity between the target task and the historical task, as well as the task resource configuration information of the historical task, so as to match the most suitable task resource configuration information for the target task. Then determine the set of nodes to be matched corresponding to the target task from the cluster; obtain the resource change rate before and after the node to be matched runs the target task according to the resource information of the node to be matched and the task resource configuration information of the target task; obtain the predicted score of the node to be matched running the target task according to the resource change rate before and after the node to be matched runs the target task and the task type characteristic data of the target type in the node to be matched, and then evaluate the adaptability between the node to be matched and the target task to improve the accuracy of the target node selection. Then, based on the predicted scores of the target tasks of multiple nodes to be matched, determine the target node from the nodes to be matched; schedule the target task to the target node to ensure that the target task can be scheduled to the node that is most suitable for executing the target task. The method of the present application meets the scheduling requirements of various types of tasks and nodes, and improves the task operation efficiency of the data sandbox. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A flowchart of an embodiment of a task scheduling method provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of another embodiment of a task scheduling method provided in an embodiment of the present application;

[0059] Figure 3 A flowchart of another embodiment of a task scheduling method provided in an embodiment of the present application;

[0060] Figure 4 A flowchart of another embodiment of a task scheduling method provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of another embodiment of a task scheduling method provided in an embodiment of the present application;

[0062] Figure 6 A schematic flow chart of an embodiment of a cluster node classification method in a task scheduling method provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of a blockchain-based container cluster node provided in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of the structure of an embodiment of a task scheduling device provided in an embodiment of the present application;

[0065] Fig. 9 A schematic diagram of another embodiment of a task scheduling device provided in an embodiment of the present application;

[0066] Fig.10 A structural diagram of another embodiment of a task scheduling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] As described above, there are currently a variety of resource allocation and task scheduling strategies, such as priority-based scheduling strategies, resource demand-based scheduling strategies, prediction model-based scheduling strategies, etc. Existing solutions use the above common scheduling strategies to allocate tasks to each node.

[0068] After research, the inventors proposed a task scheduling method, device, storage medium and processor to solve the technical problem that the commonly used scheduling strategies in the prior art often cannot meet the scheduling requirements of various types of tasks and nodes, thereby affecting the task operation efficiency of the data sandbox.

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] See also Figure 1 , this figure is a flow chart of an embodiment of a task scheduling method provided in an embodiment of the present application.

[0071] S101: Generate task resource configuration information of the target task based on the similarity between the target task and the historical tasks, and the task resource configuration information of the historical tasks.

[0072] In this embodiment, the similarity between the target task and the historical task is calculated based on the characteristics, data characteristics and operation data of multiple resource dimensions in the corresponding time period of the historical task running process and the target task pre-running process, and the sizes of the similarity values ​​corresponding to the multiple historical tasks are compared to determine a plurality of pre-selected tasks; and based on the running time, resource occupancy and task resource configuration information of each of these pre-selected tasks, a reference task corresponding to the target task is determined from the plurality of pre-selected tasks, and the task resource configuration information corresponding to the reference task is used as the task resource configuration information of the target task.

[0073] It can be understood that when determining the reference task based on the running time, resource occupancy and task resource configuration information of multiple pre-selected tasks, any one of the running time, resource occupancy and task resource configuration information can be selected to determine the reference task; or two or more of the information can be selected for comprehensive judgment based on the user's needs and preferences to determine the reference task that meets the needs.

[0074] S102: Determine a set of nodes to be matched corresponding to the target task from the cluster; the set of nodes to be matched includes a plurality of nodes to be matched.

[0075] In this embodiment, there is a one-to-one correspondence between node types and task types; for example, computing nodes correspond to computing tasks; prediction nodes correspond to prediction tasks; and analysis nodes correspond to analysis tasks. According to the node types of the nodes in the cluster and the task types corresponding to the target tasks, multiple nodes to be matched corresponding to the target tasks are determined to obtain a set of nodes to be matched.

[0076] S103: Obtain the resource change rate of the node to be matched before and after the node to be matched runs the target task according to the resource information of the node to be matched and the task resource configuration information of the target task.

[0077] In this embodiment, for example, the resource information of the node to be matched includes the total amount of resources corresponding to the node to be matched in multiple resource dimensions and the occupied amount of resources of the node to be matched in multiple resource dimensions. The task resource configuration information of the target task is the task resource configuration information corresponding to the reference task obtained in S101. Among them, the multiple resource dimensions may include but are not limited to disk storage capacity, disk throughput, CPU computing power, GPU computing power, and network bandwidth.

[0078] According to the same resource dimension, the resource change rate of the node to be matched before and after running the target task is obtained through calculation processing based on the resource information of the node to be matched and the task resource configuration information of the target task. Specifically, the resource change rate of the node to be matched before and after running the target task includes the resource change rates corresponding to multiple resource dimensions.

[0079] S104: Obtain a predicted score for the target task being run by the node to be matched according to the resource change rate before and after the node to be matched runs the target task, and the task type feature data corresponding to the target type of the node to be matched.

[0080] The target type is the task type of the target task.

[0081] In this embodiment, for example, the resource change rate before and after the i-th matching node runs the target task is represented by R i Indicates; Taking the target type as the computing task type as an example, the task type feature data of the i-th node to be matched corresponding to the computing task type is represented by Cor i c Indicated by; where the resource change rate before and after the matching node runs the target task is represented by R i Contains resource change rates corresponding to multiple resource dimensions, and task type feature data Cor corresponding to the computing task type of the node to be matched i c It contains the correlation weights corresponding to multiple resource dimensions.

[0082] According to the resource change rate before and after the node to be matched runs the target task and the task type characteristic data of the node to be matched corresponding to the target type, the predicted score S(i) of the i-th node to be matched running the target task is obtained. Among them, if the predicted score is larger, it means that the node to be matched is more suitable for executing the target task, and the execution efficiency of the target task is also higher. Correspondingly, if the predicted score is smaller, it means that the execution efficiency of the node to be matched to execute the target task is also lower.

[0083] S105: Determine a target node from the multiple nodes to be matched based on the predicted scores of the target tasks respectively executed by the multiple nodes to be matched.

[0084] In this embodiment, according to the predicted scores of the target tasks of the multiple nodes to be matched, the node to be matched with the highest predicted score is selected as the target task. At this time, the execution efficiency of the target task executed by the node to be matched is the highest, and the stability of the cluster is also the highest.

[0085] S106: Schedule the target task to the target node.

[0086] In this embodiment, the target task is scheduled to the determined best matching target node for subsequent execution of the target task.

[0087] In this embodiment, the method of the present application generates the task resource configuration information of the target task based on the similarity between the target task and the historical task, as well as the task resource configuration information of the historical task, so as to match the most suitable task resource configuration information for the target task. Then determine the set of nodes to be matched corresponding to the target task from the cluster; obtain the resource change rate before and after the target task is run by the node to be matched according to the resource information of the node to be matched and the task resource configuration information of the target task; obtain the predicted score of the target task run by the node to be matched according to the resource change rate before and after the target task is run by the node to be matched and the task type characteristic data of the target type in the node to be matched, and then evaluate the adaptability between the node to be matched and the target task to improve the accuracy of the target node selection. Then, based on the predicted scores of the target tasks run by multiple nodes to be matched, determine the target node from the nodes to be matched; schedule the target task to the target node to ensure that the target task can be scheduled to the node that is most suitable for executing the target task. The method of the present application meets the scheduling requirements of various types of tasks and nodes, and improves the task operation efficiency of the data sandbox.

[0088] See also Figure 2 , which is a flowchart of another embodiment of a task scheduling method provided in an embodiment of the present application. Figure 1 ,exist Figure 2 Demonstrates a specific implementation method for generating task resource configuration information for target tasks, specifically combining Figure 2 Based on the above embodiment, a specific implementation of "generating the task resource configuration information of the target task based on the similarity between the target task and the historical task, and the task resource configuration information of the historical task" in S101 includes:

[0089] S1011: Determine a plurality of pre-selected tasks from a plurality of historical tasks based on similarities with the target task.

[0090] Among them, the pre-selected tasks are historical tasks whose similarity with the target task is higher than the similarity threshold.

[0091] In this embodiment, for example, based on the similarity between each historical task and the target task, multiple historical tasks having similarities higher than a similarity threshold are determined from the multiple historical tasks as pre-selected tasks.

[0092] S1012: Determine a reference task from the plurality of pre-selected tasks based on at least one of the running time, resource occupancy, and task resource configuration information of the plurality of pre-selected tasks.

[0093] In this embodiment, for example, based on obtaining one or more of the running time corresponding to each of the multiple pre-selected tasks, the resource occupancy during the running process, and the task resource configuration information as a selection method, a pre-selected task that is closest to the target task is selected from the multiple pre-selected tasks as a reference task corresponding to the target task.

[0094] S1013: Generate task resource configuration information of the target task using the task resource configuration information of the reference task.

[0095] In this embodiment, the task resource configuration information of the reference task corresponding to the target task is used as the task resource configuration information of the target task, so as to be used for the subsequent operation of the target task, thereby determining the target node and scheduling the target task to the target node.

[0096] Combination Figure 2 As shown, after executing S1011-S1013, the previous embodiment may be referred to and S102-S106 may be continued to realize the scheduling of the target task.

[0097] In this embodiment, multiple pre-selected tasks are determined from multiple historical tasks based on the similarity with the target task; a reference task is determined from the multiple pre-selected tasks based on at least one of the running time, resource occupancy, and task resource configuration information of each of the multiple pre-selected tasks; and the task resource configuration information of the target task is generated using the task resource configuration information of the reference task. Compared with the prior art, the method of the present application can screen out the historical tasks that are closest to the current task as pre-selected tasks by calculating the similarity with the target task, thereby ensuring that subsequent resource configuration is closer to actual needs. At the same time, further considering actual factors such as running time and resource occupancy in the pre-selected tasks, and selecting the most appropriate reference task helps to avoid resource waste and over-configuration, and improves the accuracy of resource configuration.

[0098] Further, based on the above embodiment, in S1012, determining a reference task from the plurality of pre-selected tasks based on the respective running times of the plurality of pre-selected tasks includes:

[0099] The task with the shortest running time among multiple pre-selected tasks is used as the reference task.

[0100] In this embodiment, if the reference task is selected based on the running efficiency of the pre-selected task, a pre-selected task with the shortest running time is determined from a plurality of pre-selected tasks according to the task running time as the reference task.

[0101] In S1012, based on the resource occupation of each of the plurality of pre-selected tasks, determining a reference task from the plurality of pre-selected tasks includes:

[0102] The task that occupies the least resources among multiple pre-selected tasks is used as the reference task.

[0103] In this embodiment, if the reference task is selected based on the resource occupation size, then according to the resource occupations of the plurality of pre-selected tasks, a pre-selected task with the least resource occupation is determined as the reference task.

[0104] In S1012, based on the task resource configuration information of each of the plurality of pre-selected tasks, determining a reference task from the plurality of pre-selected tasks includes:

[0105] According to the task resource configuration information of each of the plurality of pre-selected tasks, a group of task resource configuration information with the highest overlap rate is determined, and any pre-selected task corresponding to the group of task resource configuration information with the highest overlap rate is used as a reference task.

[0106] In this embodiment, if the reference task is selected based on the commonly used task resource configuration, then based on the task resource configuration information of each of the multiple pre-selected tasks obtained, a group of task resource configuration information with the highest task resource configuration overlap rate is determined, and any one of the multiple pre-selected tasks corresponding to the group of task resource configuration information is used as the reference task.

[0107] In this embodiment, a task with the shortest running time among multiple pre-selected tasks is used as a reference task; and / or, a task with the least resource usage among multiple pre-selected tasks is used as a reference task; and / or, based on the task resource configuration information of each of the multiple pre-selected tasks, a group of task resource configuration information with the highest overlap rate is determined, and any pre-selected task corresponding to the group of task resource configuration information with the highest overlap rate is used as a reference task. Compared with the prior art, the method of the present application can be considered according to the user's own needs, by selecting the task with the least resource usage, the shortest running time, and the highest overlap rate of task resource configuration information as the reference task, to more comprehensively evaluate the performance and resource utilization efficiency of each pre-selected task, achieve the selection of the closest reference task, ensure that the target task is more reasonable in resource allocation, and avoid resource waste or insufficient configuration.

[0108] See also Figure 3 , which is a flowchart of another embodiment of a task scheduling method provided in an embodiment of the present application. Figure 1 Before S101, it is necessary to obtain the similarity between the target task and the historical task. Figure 3 Based on the above embodiment, a specific implementation method of obtaining the similarity between the target task and the historical task includes:

[0109] S301: Acquire a task comparison reference feature set of a target task and a task comparison reference feature set of historical tasks.

[0110] Among them, the task comparison reference feature set of the target task includes the data features of the target task, the task pre-operation features of the target task, and the features of multiple resource dimensions of the target task within the task pre-operation time period; the task comparison reference feature set of the historical task includes the data features of the historical task, the task operation features of the historical task, and the features of multiple resource dimensions of the historical task within the task operation time period.

[0111] In this embodiment, in the acquired task comparison reference feature set of the target task, the data features of the target task include but are not limited to one or more of the number of data rows and the data size; the task pre-operation features of the target task include but are not limited to one or more of the number of iterations and the time of each iteration; the characteristics of multiple resource dimensions of the target task within the task pre-operation time period include but are not limited to one or more of the characteristics of the resource dimension of disk storage capacity, the characteristics of the resource dimension of disk throughput, the characteristics of the resource dimension of CPU computing power, the characteristics of the resource dimension of GPU computing power, and the characteristics of the resource dimension of network bandwidth.

[0112] Accordingly, the task comparison reference features in the task comparison reference feature set of the historical tasks are consistent with the task comparison reference feature dimensions of the target task.

[0113] S302: Obtain a similarity index value between the target task and the historical tasks based on differences in corresponding elements between the task comparison reference feature set of the target task and the task comparison reference feature set of the historical tasks.

[0114] In this embodiment, for example, the following formula is used to calculate the difference between the corresponding elements of the task comparison reference feature set of the target task and the task comparison reference feature set of the historical task:

[0115]

[0116] Among them, x i represents the i-th comparison reference feature in the task comparison reference feature set X of the target task; y i represents the corresponding ith comparison reference feature in the task comparison reference feature set Y of the historical task; k represents the number of reference features; and E is the similarity index value between the target task and the historical task obtained by calculation.

[0117] It can be understood that the similarity index value is negatively correlated with the similarity. When the similarity index value E between the target task and the historical task is smaller, it means that the similarity between the target task and the historical task is higher; correspondingly, when the similarity index value E between the target task and the historical task is larger, it means that the similarity between the target task and the historical task is lower.

[0118] In this embodiment, by obtaining the task comparison reference feature set of the target task and the task comparison reference feature set of the historical task; based on the difference of the corresponding elements between the task comparison reference feature set of the target task and the task comparison reference feature set of the historical task, the similarity index value between the target task and the historical task is obtained. Compared with the prior art, the method of the present application can more accurately judge the degree of association between the historical task and the target task by directly comparing the similarity of the target task and the historical task in key features, thereby improving the accuracy of determining the reference task.

[0119] See also Figure 4 , which is a flow chart of another embodiment of a task scheduling method provided by an embodiment of the present application. Based on the above embodiment, the task type feature data includes multiple resource correlation weights corresponding to multiple resource dimensions. Then, a specific implementation method of "obtaining the predicted score of the target task of the node to be matched according to the resource change rate before and after the node to be matched runs the target task, and the task type feature data of the node to be matched corresponding to the target type" in S104 includes:

[0120] S1041: Calculate the resource load change rate of the node to be matched according to the resource occupancy index data of the node to be matched and the resource change rate of the node to be matched before and after the node to be matched runs the target task.

[0121] The resource occupancy index data is calculated based on the characteristics of the cluster in multiple resource dimensions and the characteristics of the nodes to be matched in the cluster in multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to the multiple resource dimensions.

[0122] In this embodiment, the resource occupancy index data of the i-th node to be matched can be expressed as W i ; The resource change rate before and after the i-th matching node runs the target task can be expressed as R i ; Use the following formula to calculate the resource load change rate L of the node to be matched i :

[0123]

[0124] The coefficient is the inverse of the number of resource dimensions of the nodes to be matched in the cluster. In this example, the number of resource dimensions is 5. In another example, if the number of resource dimensions of the nodes to be matched in the cluster is 4, then the resource load change rate L i The corresponding calculation formula is as follows:

[0125]

[0126] Among them, the smaller the resource load change rate of the node to be matched is, the higher and more stable the utilization rate of the node resources in the cluster by the target task before and after the node to be matched is run.

[0127] S1042: performing normalization processing on the task type feature data of the to-be-matched node corresponding to the target type to obtain normalized task type feature data.

[0128] In this embodiment, for example, the task type feature data corresponding to the target type of the i-th node to be matched includes multiple resource correlation weights corresponding to multiple resource dimensions. If the multiple resource dimensions include disk storage capacity S, disk throughput T, CPU computing power C, GPU computing power G, and network bandwidth B, a total of five resource dimensions, then the task type is the task type feature data Cor corresponding to the computing task. i c It can be expressed as:

[0129]

[0130] Among them, for example, Indicates the resource relevance weight of the i-th node to be matched when it is facing the computing task type and the resource dimension is the disk storage capacity S; Indicates the resource relevance weight of the i-th node to be matched when the resource dimension is disk throughput T for computing task type.

[0131] Task type feature data Cor for computing tasks i c , normalized under multiple resource dimensions, the normalized task type feature data U i It can be expressed as:

[0132]

[0133] Among them, for example, Indicates the resource relevance weight of the i-th node to be matched for the computing task type under the resource dimension of disk storage capacity S The corresponding normalized resource relevance weight.

[0134] S1043: Calculate the task type feature matching degree of the to-be-matched node corresponding to the target type according to the resource occupancy index data and the normalized task type feature data.

[0135] In this embodiment, for example, according to the resource occupancy index data W of the i-th node to be matched i And the normalized task type feature data U i, the following formula is used to calculate the task type feature matching degree T of the i-th matching node corresponding to the target type i :

[0136]

[0137] The greater the matching degree of the task type feature of the to-be-matched node corresponding to the target type, the higher the running efficiency of the target task in the to-be-matched node.

[0138] S1044: Calculate the predicted score of the target task to be run by the node to be matched according to the resource load change rate and the task type feature matching degree.

[0139] In this embodiment, for example, according to the resource load change rate L of the i-th node to be matched i and the task type feature matching degree T corresponding to the target type i , calculate the predicted score S(i) of the target task to be run by the matching node. Specifically, the task type feature matching degree T corresponding to the target type i It is positively correlated with the predicted score S(i); while the resource load change rate L of the node to be matched i Negatively correlated with the predicted score S(i). For example, in this embodiment, the predicted score S(i) can be calculated using the following formula:

[0140] S(i)=T i / L i

[0141] As another example, the prediction score S(i) can be calculated using the following formula:

[0142] S(i)=T i +1 / L i

[0143] Among them, when the resource load change rate L of the node to be matched i When unchanged, if the task type feature matching degree T corresponding to the target type i The larger the value, the higher the prediction score S(i); if the task type feature matching degree T corresponding to the target type is i The smaller the prediction score S(i), the smaller the prediction score S(i). i If the resource load change rate L of the node to be matched is unchanged, i The smaller it is, the larger the predicted score S(i) is; if the resource load change rate L of the node to be matched i The larger it is, the smaller the prediction score S(i) is.

[0144] Combination Figure 4As shown, before executing S1041-S1044, you can refer to the previous embodiment and execute S101-S103 to realize the resource change rate before and after the node to be matched runs the target task; and after executing S1041-S1044, you can refer to the previous embodiment and continue to execute S105-S106 to realize the scheduling of the target task.

[0145] In this embodiment, the resource load change rate of the node to be matched is calculated based on the resource occupancy index data of the node to be matched and the resource change rate before and after the node to be matched runs the target task; the task type feature data corresponding to the target type of the node to be matched is normalized to obtain the normalized task type feature data; the task type feature matching degree of the node to be matched corresponding to the target type is calculated based on the resource occupancy index data and the normalized task type feature data; the predicted score of the target task of the node to be matched is calculated based on the resource load change rate and the task type feature matching degree. Compared with the prior art, the method of the present application can accurately evaluate the resource state change of the node before and after running the target task by calculating the resource occupancy index data and the resource load change rate of the node to be matched; and finally obtain the predicted score of the target task of the node to be matched by calculating the task type feature matching degree, thereby improving the accuracy of matching the target task with the node to be matched.

[0146] Further, based on the above embodiment, in order to ensure that a more suitable target node is matched for the target task, a specific implementation of "determining the target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched each running the target task" in S105 includes:

[0147] The node with the highest prediction score for running the target task is determined from multiple nodes to be matched as the target node.

[0148] In this embodiment, according to the prediction scores corresponding to the target tasks of the multiple nodes to be matched, a node to be matched with the highest prediction score is selected as the target node to schedule the target task to the target node.

[0149] In this embodiment, the node with the highest predicted score for running the target task is determined from multiple nodes to be matched as the target node. Compared with the prior art, the method of the present application selects the node with the highest predicted score as the target node, which improves the stability of the cluster and the efficiency of task operation.

[0150] See also Figure 5, which is a flow chart of another embodiment of a task scheduling method provided by an embodiment of the present application. Based on the above embodiment, the resource information of the node to be matched includes the total amount of resources corresponding to the node to be matched in multiple resource dimensions, and the resource occupied by the node to be matched in multiple resource dimensions; the task resource configuration information of the target task includes the resource configuration values ​​of multiple resource dimensions.

[0151] At this time, a specific implementation of "obtaining the resource change rate of the to-be-matched node before and after running the target task according to the resource information of the to-be-matched node and the task resource configuration information of the target task" in S103 includes:

[0152] S1031: Perform a division operation on the occupied resource amount and the total resource amount in the resource information of the to-be-matched node according to the resource dimension, and obtain the resource occupation rates corresponding to the multiple resource dimensions before the to-be-matched node runs the target task.

[0153] In this embodiment, the total amount of resources corresponding to the to-be-matched node in multiple resource dimensions in the resource information of the to-be-matched node is expressed as [S i ,T i ,C i ,G i ,B i ]; the resource usage of the node to be matched in multiple resource dimensions is expressed as [S a ,T a ,C a ,G a ,B a The task resource configuration information of the target task includes the resource configuration values ​​of multiple resource dimensions and can be expressed as [S t ,T t ,C t ,G t ,B t ]. Among them, S means that the resource dimension is disk storage; T means that the resource dimension is disk throughput; C means that the resource dimension is CPU computing power; G means that the resource dimension is GPU computing power; B means that the resource dimension is network bandwidth.

[0154] In each resource dimension, the resource occupied by the node to be matched in the resource dimension is divided by the corresponding total resource amount. The resource occupancy rates corresponding to multiple resource dimensions before the node to be matched runs the target task can be expressed by the following expression:

[0155]

[0156] S1032: Sum the resource usage in the resource information of the node to be matched and the resource configuration value in the task resource configuration information of the target task according to the resource dimension, and divide the sum by the total amount of resources in the corresponding resource dimension in the resource information of the node to be matched, to obtain the resource usage rates corresponding to the multiple resource dimensions after the node to be matched runs the target task.

[0157] In this embodiment, the resource occupied amount [S a ,T a ,C a ,G a ,B a ] and the resource configuration value [S t ,T t ,C t ,G t ,B t ] and we get [S a +S t ,T a +T t ,C a +C t ,G a +G t ,B a +B t Then, according to the resource dimension, the obtained sum value is compared with the total resource amount of the node to be matched [S i ,T i ,C i ,G i ,B i ] is divided by the resource occupancy rates of multiple resource dimensions obtained after the matching node runs the target task, which can be expressed by the following expression:

[0158]

[0159] S1033: Divide the difference in resource occupancy rate of each resource dimension before and after the target task is run by the matching node by the resource occupancy rate of the corresponding resource dimension before the target task is run, to obtain resource change rates corresponding to the multiple resource dimensions before and after the target task is run by the matching node.

[0160] In this embodiment, the resource occupancy rate of each resource dimension after the to-be-matched node runs the target task obtained in S1032 is subtracted from the resource occupancy rate of each resource dimension before the to-be-matched node runs the target task obtained in S1031 according to the resource dimension, and the obtained difference can be expressed by the following expression:

[0161]

[0162] Divide the obtained difference by the resource occupancy rate of the corresponding resource dimension before running the target task, and obtain the resource change rate corresponding to multiple resource dimensions before and after the target task is run on the matching node, which can be expressed by the following expression:

[0163]

[0164] Simplifying the above expression, the resource change rates corresponding to multiple resource dimensions before and after the target task is run by the node to be matched can be expressed as follows:

[0165]

[0166] Among them, for example, Indicates the resource change rate of the disk storage capacity before and after the i-th matching node runs the target task; Indicates the resource change rate of the disk throughput resource dimension before and after the i-th matching node runs the target task.

[0167] Combination Figure 5 As shown, before executing S1031-S1033, you can refer to the previous embodiment and execute S101-S102 to determine the set of nodes to be matched corresponding to the target task from the cluster; and after executing S1031-S1033, you can refer to the previous embodiment and continue to execute S104-S106 to achieve the scheduling of the target task.

[0168] In this embodiment, the resource occupied amount and the total amount of resources in the resource information of the matching node are divided according to the resource dimension to obtain the resource occupation rates corresponding to the multiple resource dimensions before the matching node runs the target task; the resource occupied amount in the resource information of the matching node and the resource configuration value in the task resource configuration information of the target task are summed according to the resource dimension, and the sum is divided by the total amount of resources of the corresponding resource dimension in the resource information of the matching node to obtain the resource occupation rates corresponding to the multiple resource dimensions after the matching node runs the target task. Then, the difference between the resource occupation rates of each resource dimension after running the target task and before running the target task is divided by the resource occupation rate of the corresponding resource dimension before running the target task to obtain the resource change rates corresponding to the multiple resource dimensions of the matching node. Compared with the prior art, the method of the present application is calculated by the resource information of the matching node and the task resource configuration information of the target task to obtain the resource change rates corresponding to the multiple resource dimensions before and after the matching node runs the target task, which are used for the subsequent matching of the target task with the matching node, and the accuracy of the matching of the target task with the matching node is improved, thereby improving the utilization rate of node resources in the cluster.

[0169] Further, based on the above embodiment, a specific implementation of "determining a set of nodes to be matched corresponding to the target task from the cluster" in S102 includes:

[0170] Nodes whose node types match the target type are determined from the cluster, and the nodes whose node types match the target type are added to a to-be-matched node set corresponding to the target task.

[0171] In this embodiment, the node types in the cluster include but are not limited to one or more of computing node types, prediction node types, and analysis node types; taking the target type as computing node type as an example, the nodes corresponding to the computing node type are obtained from the cluster, and the obtained nodes are added to the set of nodes to be matched corresponding to the computing task.

[0172] In this embodiment, nodes whose node types match the target type are determined from the cluster, and the nodes whose node types match the target type are added to the set of nodes to be matched corresponding to the target task. Compared with the prior art, the method of the present application determines multiple nodes to be matched from the cluster whose node types match the target type, which can reduce the number of nodes to be processed in subsequent matching calculations, reduce the amount of calculations and improve the efficiency of target node determination, thereby improving the efficiency of target task scheduling.

[0173] Further, based on the above embodiment, the nodes to be matched are determined and obtained based on the nodes and corresponding node types obtained after pre-classification in the cluster. Therefore, a specific implementation method of the classification method of the nodes in the cluster is introduced in this embodiment, which may include:

[0174] According to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in multiple resource dimensions, resource occupancy indicator data of the target node to be classified is obtained.

[0175] In this embodiment, the resource occupancy index data includes multiple resource occupancy weights corresponding to multiple resource dimensions. For example, the multiple resource dimensions mentioned in step S101 may include but are not limited to one or more of memory, disk storage, disk throughput, CPU computing power, GPU computing power, and network bandwidth.

[0176] In this embodiment, the characteristics of the cluster in multiple resource dimensions include the total amount of resources of all nodes in the cluster in each resource dimension. For example, if the number of nodes in the cluster is n, and the multiple resource dimensions include disk storage S, disk throughput T, CPU computing power C, GPU computing power G, and network bandwidth B; then the total amount of resources of the cluster in multiple resource dimensions is C = [S c ,T c ,Cc ,G c ,B c ].

[0177] The total amount of resources of the cluster under the resource dimension of disk storage is expressed as Among them, S i It represents the resource usage of the disk storage capacity of the i-th node among the n nodes in the cluster. The total resource of the cluster under the resource dimension of disk throughput is expressed as Among them, T i It represents the resource usage of the disk throughput of the i-th node among the n nodes in the cluster. The total resources under other resource dimensions are expressed in the same way. The resource usage of the target node i to be classified in the cluster under multiple resource dimensions can be expressed as N i =[S i ,T i ,C i ,G i ,B i ].

[0178] According to the total amount of resources of the cluster in multiple resource dimensions C = [S c ,T c ,C c ,G c ,B c ], and the resource usage N of the target node i in the cluster in multiple resource dimensions i =[S i ,T i ,C i ,G i ,B i ], and obtain the resource occupancy index data of the target node to be classified through corresponding mathematical operations. Specifically, the resource occupancy index data can be constructed according to the ratio of the resource occupancy of the target node to be classified in the same resource dimension to the total amount of resources in the resource dimension in the cluster. Among them, the resource occupancy index data of the target node to be classified can be expressed as:

[0179]

[0180] According to the task running data of the target node to be classified for multiple task types, multiple groups of task type feature data of the target node to be classified are obtained.

[0181] Wherein, the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions;

[0182] In this embodiment, for example, tasks of multiple task types may include but are not limited to one or more of computing tasks, prediction tasks, and analysis tasks. Multiple sets of task running data corresponding to each of the multiple task types are obtained by running the target node to be classified, including resource change result values ​​corresponding to multiple resource dimensions when the task is running and the duration of running the target task until the running result of the target task is obtained.

[0183] The resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in each set of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and multiple task types.

[0184] For example, the resource occupancy weight corresponding to each resource dimension in the resource occupancy index data of the target node to be classified i is expressed as W i ; Task type characteristic data Cor of computing tasks i c The resource correlation weights Cor corresponding to each resource dimension in i c ; Task type feature data Cor of the prediction task i f The resource correlation weights Cor corresponding to each resource dimension in i f ; Task type feature data Cor i a The resource correlation weights Cor corresponding to each resource dimension in i a .

[0185] In this embodiment, for each group of task types, the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in the characteristic data of the task type are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types. The processing method can be to multiply the resource occupancy weight of the same resource dimension with the resource correlation weight coefficient, and accumulate the multiplication results of each resource dimension according to the same task type. Among them, the adaptability of the target node to be classified i and the multiple task types can be expressed as follows: Calculate the task adaptability Prediction task suitability F i f And analyze task suitability

[0186] According to the numerical values ​​of the adaptability of the target node to be classified and the various task types, the node category of the target node to be classified is configured.

[0187] In this embodiment, for example, according to the obtained adaptability of the target node to be classified and the corresponding adaptability of the multiple task types, F i f as well as The size of the cluster node is selected from multiple task types to be the most matched with the target node to be classified; and the task type is configured as the node category of the target node to be classified, thereby completing the classification of the cluster nodes.

[0188] In this embodiment, the resource occupancy index data of the target node to be classified is obtained according to the characteristics of the cluster and the target node to be classified in the cluster in multiple resource dimensions. In addition, according to the task running data of the target node to be classified for multiple task types, the task type characteristic data corresponding to the target node to be classified when executing different types of tasks is obtained. Then, the resource occupancy index data of the target node to be classified and each group of task type characteristic data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified to multiple task types; finally, according to the numerical value of the adaptability of the target node to be classified to multiple task types, the determined task type is configured as the node category of the target node to be classified, and the resources are allocated to the most suitable tasks for execution through the obtained matching degree, so as to make full use of the data sandbox cluster resources, configure the target node to be classified with a suitable node category, and thus improve the performance of the node running task.

[0189] See also Figure 6 , which is a schematic diagram of an embodiment of a cluster node classification method in a task scheduling method provided in an embodiment of the present application. Figure 6 A specific implementation method of a node classification method in a cluster is introduced, which may include:

[0190] S601: Calculate the ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension according to the total amount of resources of the cluster in multiple resource dimensions and the total amount of resources of the target node to be classified in the cluster in multiple resource dimensions.

[0191] Among them, the total amount of resources in multiple resource dimensions of the cluster is C = [S c ,T c ,C c ,G c ,B c ]; for example, the total amount of resources of the cluster under the resource dimension of disk storage is expressed as Among them, S i Indicates the resource usage of the disk storage capacity of the i-th node among the n nodes in the cluster. The total amount of resources of the target node to be classified in the cluster in multiple resource dimensions can be expressed as N i =[Si ,T i ,C i ,G i ,B i ].

[0192] S602: Normalize the ratios of the features of the obtained multiple resource dimensions to obtain the resource occupation weights of the target node to be classified corresponding to each resource dimension, and generate resource occupation index data of the target node to be classified.

[0193] The ratios of the features of the multiple resource dimensions obtained in S601 are normalized to convert the feature values ​​of all resource dimensions into the same numerical range, thereby eliminating the influence of the dimensions of different resource dimensions on the evaluation results, thereby generating resource occupancy index data of the target node to be classified:

[0194]

[0195] S603: For a target task type among the multiple task types, randomly generate initial task resource configuration information for the initial task according to the minimum resource values ​​respectively required for running the initial task in multiple resource dimensions.

[0196] Taking the target task type as the computing task type as an example, the initial task resource configuration information is randomly generated for the initial task

[0197] S604: Generate a task container for the initial task in the target node to be classified according to the initial task resource configuration information; and run the initial task through the task container.

[0198] On the premise that the initial configuration of the task resources has been completed, a task container is generated for the target task in the target node to be classified according to the initial task resource configuration information to run the task.

[0199] S605: Extracting, from the running data of the task container running the initial task, resource change result values ​​corresponding to the task container in multiple resource dimensions when the first running time and the running result of the initial task are obtained.

[0200] Taking the initial task as a computing task as an example, from the running data of the task container running the task, the duration from the task container starting to run the task j until the running result of the task is obtained is extracted as the first running time t j ; and extract the resource change result value Task′ corresponding to each of the task container in multiple resource dimensions j .

[0201] S606: Obtain characteristic change rates of the multiple resource dimensions according to the resource change result values ​​corresponding to the task container in the multiple resource dimensions and the resource configuration information of the multiple resource dimensions.

[0202] According to the resource change result value Task′ corresponding to multiple resource dimensions j , and initial task resource configuration information Task j , and calculate the characteristic change rate of each resource dimension separately, including ΔS j , ΔT j , ΔC j , ΔG j and ΔB j .

[0203] S607: Based on the resource change result values ​​corresponding to the multiple resource dimensions and the initial task resource configuration information, regenerate multiple groups of new task resource configuration information corresponding to the multiple resource dimensions one by one for the initial task.

[0204] Among them, multiple sets of new task resource configuration information include:

[0205]

[0206] In the above five sets of new task resource configuration information, the features with the ' symbol in the upper right corner indicate the resource change result value.

[0207] S608: Apply the multiple sets of new task resource configuration information to run the initial task respectively, and obtain multiple second running times corresponding to the multiple resource dimensions respectively.

[0208] According to the above five sets of new task resource configuration information, the initial tasks are run respectively and the corresponding second running times are obtained, which are: as well as

[0209] S609: Based on the first running time and the plurality of second running times, obtain running time change rates respectively corresponding to the plurality of resource dimensions.

[0210] Based on the first running time t j With multiple second running time as well as Get the runtime change rates corresponding to multiple resource dimensions as well as

[0211] S610: Calculate and process the characteristic change rates of multiple resource dimensions and the running time change rates corresponding to the multiple resource dimensions according to the same resource dimensions to obtain multiple correlation coefficients corresponding to the multiple resource dimensions one by one, and form a correlation matrix corresponding to the initial task.

[0212] The correlation array contains multiple correlation coefficients corresponding to multiple resource dimensions. For each resource dimension, taking the disk storage capacity as an example, the characteristic change rate of the resource dimension and the running time change rate are calculated using the following formula to obtain the correlation coefficient of the resource dimension:

[0213]

[0214] S611: Repeat S603-S610 to obtain multiple correlation arrays of the target task type; and average the correlation coefficients corresponding to the same resource dimension in the multiple correlation arrays to obtain a set of task type feature data corresponding to the target task type of the target node to be classified.

[0215] S603-S610 are repeatedly executed. For each task type, multiple tasks are executed and the corresponding relationship array corresponding to each task is calculated, and the average is calculated according to the same dimension to obtain a set of task type characteristic data of the task type.

[0216] S612: The resource occupancy weight corresponding to each resource dimension in the resource occupancy index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and multiple task types.

[0217] The resource occupancy index data corresponding to the target node to be classified and each task type feature data are represented by a vector to perform a dot product operation, thereby obtaining the degree of adaptation between the target node to be classified and the task type.

[0218] S613: Determine the task type with the highest value of adaptability to the target node to be classified among the multiple task types as the adaptation task type of the target node to be classified, so as to configure the node category of the target node to be classified as the target category.

[0219] The task type with the highest fitness value is selected from the fitness of the target node to be classified and multiple task types as the adaptation task type of the target node to be classified, and the node category configuration has been performed.

[0220] Furthermore, based on the above embodiments, in order to prevent data leakage and ensure the security and compliance of task operation, a multi-party collaborative container cluster based on blockchain is constructed for task operation.

[0221] The cluster includes nodes with multiple different identities in the blockchain network; the nodes with multiple different identities include at least: initiator nodes and participant nodes; the initiator nodes and the participant nodes reach a consensus on the same task.

[0222] In this embodiment, for example, the blockchain network contains multiple identities, including initiators, participants, auditors, and regulators. Among them, the initiator is the initiator of the data sandbox task, responsible for providing data and the task running environment; the participants are invited by the task initiator to jointly provide data and the running environment. The auditor is the initiator and the participant who provide the formula node, add the audit function to the processes such as character initiation, participation, resource allocation, and scheduling, and record it on the chain after reaching a consensus. The regulator is a data security management agency composed of a third party, which can authenticate the identities of the initiator and the participants and issue digital certificates, and is responsible for supervising the data flow to ensure that important data is not leaked. Figure 7 As shown, circles represent the nodes of the initiator and participants; the five-pointed star represents the consensus node of the initiator; the square represents the consensus node of the participants; and the triangle represents the consensus node of the regulator.

[0223] First, the initiator publishes the task and designates the participants to provide data and nodes. The auditor's consensus node verifies the initiator's identity certificate and initiates consensus on the task proposal; after the initiator, the participant and the auditor agree, the participant accepts the task application and opens his or her data permissions and node permissions. The initiator and the participants will form a container cluster with the nodes that have reached consensus for subsequent task operations. The resource nodes of the initiator and the participants can be added or reduced dynamically, and the auditor needs to review them.

[0224] In this embodiment, the cluster includes nodes with multiple different identities in the blockchain network; the multiple different identity nodes include at least: initiator nodes and participant nodes; the initiator nodes and the participant nodes reach a consensus on the same task. Compared with the prior art, the method of the present application builds a multi-party collaborative container cluster based on blockchain for task operation, which improves the efficiency and accuracy of task allocation and ensures the fairness and consistency of task execution.

[0225] See also Figure 8 , which is a schematic diagram of the structure of an embodiment of a task scheduling device provided in an embodiment of the present application. The device 70 includes: a resource configuration generation module 71, a first node determination module 72, a resource change calculation module 73, a node prediction module 74, a second node determination module 75 and a scheduling module 76.

[0226] Specifically, the resource configuration generation module 71 is used to generate task resource configuration information of the target task based on the similarity between the target task and the historical task, and the task resource configuration information of the historical task;

[0227] A first node determination module 72 is used to determine a set of to-be-matched nodes corresponding to a target task from a cluster; the set of to-be-matched nodes includes a plurality of to-be-matched nodes;

[0228] The resource change calculation module 73 is used to obtain the resource change rate of the to-be-matched node before and after the to-be-matched node runs the target task according to the resource information of the to-be-matched node and the task resource configuration information of the target task;

[0229] The node prediction module 74 is used to obtain the prediction score of the target task of the node to be matched according to the resource change rate before and after the node to be matched runs the target task, and the task type characteristic data of the node to be matched corresponding to the target type; the target type is the task type of the target task;

[0230] A second node determination module 75 is used to determine a target node from a plurality of nodes to be matched based on the predicted scores of the plurality of nodes to be matched respectively running the target task;

[0231] The scheduling module 76 is used to schedule the target task to the target node.

[0232] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0233] Further, based on the above embodiment, the resource configuration generation module 71 is specifically used to determine multiple pre-selected tasks from multiple historical tasks based on the similarity with the target task; the pre-selected tasks are historical tasks whose similarity with the target task is higher than the similarity threshold; based on at least one of the running time, resource occupancy and task resource configuration information of each of the multiple pre-selected tasks, a reference task is determined from the multiple pre-selected tasks; and the task resource configuration information of the target task is generated using the task resource configuration information of the reference task.

[0234] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0235] Further, based on the above-mentioned embodiment, the resource configuration generation module 71 is also specifically used to take the task with the shortest running time among multiple pre-selected tasks as a reference task; take the task with the least resource occupancy among multiple pre-selected tasks as a reference task; determine a group of task resource configuration information with the highest overlap rate based on the task resource configuration information of each of the multiple pre-selected tasks, and take any pre-selected task corresponding to the group of task resource configuration information with the highest overlap rate as a reference task.

[0236] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0237] Further, based on the above embodiments, see Fig. 9, which is a schematic diagram of the structure of another embodiment of a task scheduling device provided in an embodiment of the present application. The device 70 may also include: a similarity acquisition module 77.

[0238] Specifically, the similarity acquisition module 77 is used to obtain the task comparison reference feature set of the target task and the task comparison reference feature set of the historical tasks; wherein the task comparison reference feature set of the target task includes the data features of the target task, the task pre-operation features of the target task, and the features of multiple resource dimensions of the target task within the task pre-operation time period; the task comparison reference feature set of the historical tasks includes the data features of the historical tasks, the task operation features of the historical tasks, and the features of multiple resource dimensions of the historical tasks within the task operation time period; based on the differences in corresponding elements between the task comparison reference feature set of the target task and the task comparison reference feature set of the historical tasks, the similarity index value between the target task and the historical tasks is obtained; the similarity index value is negatively correlated with the similarity.

[0239] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0240] Further, based on the above embodiments, the node prediction module 74 is specifically used to calculate the resource load change rate of the node to be matched based on the resource occupancy index data of the node to be matched and the resource change rate before and after the node to be matched runs the target task; the resource occupancy index data is calculated based on the characteristics of the cluster in multiple resource dimensions and the characteristics of the node to be matched in the cluster in multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to multiple resource dimensions one by one; the task type feature data corresponding to the target type of the node to be matched is normalized to obtain the normalized task type feature data; based on the resource occupancy index data and the normalized task type feature data, the task type feature matching degree of the node to be matched corresponding to the target type is calculated; based on the resource load change rate and the task type feature matching degree, the predicted score of the node to be matched running the target task is calculated.

[0241] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0242] Further, based on the above embodiment, the second node determination module 75 is specifically used to determine the node with the highest prediction score for running the target task from the multiple nodes to be matched as the target node.

[0243] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0244] Further, on the basis of the above embodiment, the resource change calculation module 73 is specifically used to perform a division operation on the resource occupied amount and the total amount of resources in the resource information of the to-be-matched node according to the resource dimension, so as to obtain the resource occupation rates corresponding to the multiple resource dimensions before the to-be-matched node runs the target task; sum the resource occupied amount in the resource information of the to-be-matched node and the resource configuration value in the task resource configuration information of the target task according to the resource dimension, and divide the sum by the total amount of resources of the corresponding resource dimension in the resource information of the to-be-matched node, so as to obtain the resource occupation rates corresponding to the multiple resource dimensions after the to-be-matched node runs the target task; divide the difference between the resource occupation rates of each resource dimension after the to-be-matched node runs the target task and before the to-be-matched node runs the target task by the resource occupation rate of the corresponding resource dimension before the to-be-matched node runs the target task, so as to obtain the resource change rates corresponding to the multiple resource dimensions before and after the to-be-matched node runs the target task.

[0245] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0246] Further, based on the above embodiment, the first node determination module 72 is specifically used to determine nodes whose node types match the target type from the cluster, and add the nodes whose node types match the target type to the set of nodes to be matched corresponding to the target task.

[0247] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0248] Further, based on the above embodiments, see Fig.10 , which is a schematic diagram of the structure of another embodiment of a task scheduling device provided in an embodiment of the present application. The device 70 also includes: a node classification module 78.

[0249] Specifically, the node classification module 78 is used to obtain resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding one-to-one to the multiple resource dimensions; according to the task operation data of the target node to be classified for multiple task types, multiple groups of task type feature data of the target node to be classified are obtained; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions; the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in each group of task type feature data are calculated and processed according to the same resource dimension to obtain the adaptability of the target node to be classified to the multiple task types; according to the numerical value of the adaptability of the target node to be classified to the multiple task types, the node category of the target node to be classified is configured.

[0250] A task scheduling device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0251] The present application also provides a processor for executing the technical solution in any of the above method embodiments, and its implementation principle and technical effect are similar and will not be described in detail here.

[0252] An embodiment of the present application also provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the method described above. The implementation principles and technical effects are similar and will not be repeated here.

[0253] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0254] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A task scheduling method, characterized in that: include: Based on the similarity between the target task and the historical tasks, and the task resource configuration information of the historical tasks, generating the task resource configuration information of the target task; Determine a set of nodes to be matched corresponding to the target task from the cluster; the set of nodes to be matched includes multiple nodes to be matched; Obtaining a resource change rate of the node to be matched before and after running the target task according to the resource information of the node to be matched and the task resource configuration information of the target task; Obtaining a predicted score of the node to be matched running the target task according to the resource change rate before and after the node to be matched runs the target task, and the task type feature data of the node to be matched corresponding to the target type; the target type is the task type of the target task; Determine a target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched each running the target task; The target task is dispatched to the target node.

2. The method according to claim 1, characterized in that The generating of the task resource configuration information of the target task based on the similarity between the target task and the historical task and the task resource configuration information of the historical task includes: Based on the similarity with the target task, a plurality of pre-selected tasks are determined from the plurality of historical tasks; the pre-selected tasks are historical tasks whose similarity with the target task is higher than a similarity threshold; Determine a reference task from the plurality of pre-selected tasks based on at least one of the respective running time, resource occupancy, and task resource configuration information of the plurality of pre-selected tasks; The task resource configuration information of the target task is generated by using the task resource configuration information of the reference task.

3. The method according to claim 2, characterized in that Determining a reference task from the plurality of pre-selected tasks based on the respective running times of the plurality of pre-selected tasks comprises: Using a task with the shortest running time among the plurality of pre-selected tasks as a reference task; Determining a reference task from the plurality of pre-selected tasks based on the resource occupation of each of the plurality of pre-selected tasks includes: taking a task that occupies the least resources among the plurality of pre-selected tasks as a reference task; Determining a reference task from the plurality of pre-selected tasks based on the task resource configuration information of each of the plurality of pre-selected tasks includes: According to the task resource configuration information of each of the plurality of pre-selected tasks, a group of task resource configuration information with the highest overlap rate is determined, and any pre-selected task corresponding to the group of task resource configuration information with the highest overlap rate is used as a reference task.

4. The method according to any one of claims 1 to 3, characterized in that The similarity between the target task and the historical task is obtained as follows: Obtain a task comparison reference feature set of the target task and a task comparison reference feature set of the historical task; wherein the task comparison reference feature set of the target task includes data features of the target task, task pre-operation features of the target task, and features of multiple resource dimensions of the target task within a task pre-operation time period; the task comparison reference feature set of the historical task includes data features of the historical task, task operation features of the historical task, and features of multiple resource dimensions of the historical task within a task operation time period; Based on the differences in corresponding elements between the task comparison reference feature set of the target task and the task comparison reference feature set of the historical tasks, a similarity index value between the target task and the historical tasks is obtained; the similarity index value is negatively correlated with the similarity.

5. The method according to claim 1, characterized in that The task type characteristic data includes a plurality of resource correlation weights corresponding one-to-one to the plurality of resource dimensions; obtaining a predicted score of the to-be-matched node running the target task according to the resource change rate before and after the to-be-matched node runs the target task and the task type characteristic data corresponding to the target type of the to-be-matched node, including: The resource load change rate of the node to be matched is calculated according to the resource occupancy index data of the node to be matched and the resource change rate before and after the node to be matched runs the target task; the resource occupancy index data is calculated according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the node to be matched in the cluster in the multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to the multiple resource dimensions; Performing normalization processing on the task type feature data of the to-be-matched node corresponding to the target type to obtain normalized task type feature data; Calculate the task type feature matching degree of the to-be-matched node corresponding to the target type according to the resource occupancy index data and the normalized task type feature data; The predicted score of the to-be-matched node running the target task is calculated based on the resource load change rate and the task type feature matching degree.

6. The method according to claim 5, characterized in that The determining the target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched respectively running the target task includes: A node with the highest prediction score for running the target task is determined from the multiple nodes to be matched as the target node.

7. The method according to claim 1, characterized in that The resource information of the node to be matched includes the total amount of resources corresponding to the node to be matched in multiple resource dimensions, and the amount of resources occupied by the node to be matched in the multiple resource dimensions; the task resource configuration information of the target task includes the resource configuration values ​​of the multiple resource dimensions; The obtaining, according to the resource information of the node to be matched and the task resource configuration information of the target task, the resource change rate before and after the node to be matched runs the target task comprises: Performing a division operation on the occupied resource amount and the total resource amount in the resource information of the node to be matched according to the resource dimension, to obtain the resource occupation rates respectively corresponding to the multiple resource dimensions before the node to be matched runs the target task; According to the resource dimension, sum the resource occupied amount in the resource information of the node to be matched and the resource configuration value in the task resource configuration information of the target task, and divide the sum by the total amount of resources of the corresponding resource dimension in the resource information of the node to be matched, to obtain the resource occupation rates corresponding to the multiple resource dimensions after the node to be matched runs the target task; The difference in resource occupancy rate of each resource dimension after the node to be matched runs the target task and before running the target task is divided by the resource occupancy rate of the corresponding resource dimension before running the target task to obtain the resource change rates corresponding to the multiple resource dimensions before and after the node to be matched runs the target task.

8. The method according to claim 1, characterized in that: The step of determining a set of nodes to be matched corresponding to the target task from a cluster includes: A node whose node type matches the target type is determined from the cluster, and the node whose node type matches the target type is added to a to-be-matched node set corresponding to the target task.

9. The method according to claim 8, characterized in that The classification methods of the nodes in the cluster include: According to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions, obtaining resource occupation index data of the target node to be classified; the resource occupation index data includes multiple resource occupation weights corresponding to the multiple resource dimensions one by one; According to the task running data of the target node to be classified for multiple task types, multiple groups of task type feature data of the target node to be classified are obtained; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions; The resource occupation weight corresponding to each resource dimension in the resource occupation index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types; The node category of the target node to be classified is configured according to the numerical values ​​of the adaptability between the target node to be classified and the multiple task types.

10. The method according to claim 1, characterized in that The cluster includes nodes with multiple different identities in the blockchain network; the nodes with multiple different identities include at least: an initiator node and a participant node; the initiator node and the participant node reach a consensus on the same task.

11. A task scheduling device, characterized in that: include: A resource configuration generation module, used to generate task resource configuration information of the target task based on the similarity between the target task and the historical task, and the task resource configuration information of the historical task; A first node determination module is used to determine a set of nodes to be matched corresponding to the target task from a cluster; the set of nodes to be matched includes a plurality of nodes to be matched; A resource change calculation module, used to obtain the resource change rate of the to-be-matched node before and after running the target task according to the resource information of the to-be-matched node and the task resource configuration information of the target task; A node prediction module, used to obtain a prediction score of the node to be matched running the target task according to the resource change rate before and after the node to be matched runs the target task, and the task type feature data of the node to be matched corresponding to the target type; the target type is the task type of the target task; A second node determination module is used to determine a target node from the multiple nodes to be matched based on the predicted scores of the multiple nodes to be matched respectively running the target task; A scheduling module is used to schedule the target task to the target node.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

13. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 10 when running.

Images