Cluster node classification method and device, storage medium and processor

Abstract

The invention discloses a cluster node classification method and device, a storage medium and a processor. The method comprises the steps of obtaining resource occupation index data of a target to-be-classified node according to features of a cluster in multiple resource dimensions and features of the target to-be-classified node in the cluster in the multiple resource dimensions; obtaining multiple groups of task type feature data of the target to-be-classified node according to the task operation data of the target to-be-classified node for multiple task types; performing operation processing on a resource occupation weight corresponding to each resource dimension in the resource occupation index data and a resource correlation weight corresponding to each resource dimension in each group of task type feature data according to the same resource dimension to obtain an adaptation degree of the target to-be-classified node and multiple task types; according to the value of the adaptation degree of the target to-be-classified node and the various task types, the node type of the target to-be-classified node is configured, full utilization of data sandbox cluster resources is achieved, and then the performance of node operation tasks is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cluster node classification method, device, storage medium and processor. Background Art

[0002] Data Sandbox is a data management tool that provides an isolated environment for safely exploring and analyzing data. Data Sandboxes usually contain a copy of the data so that users can freely experiment without affecting the data in the production environment. A cluster is a group of servers connected by a fast network that work together to complete specific tasks. In a data sandbox environment, a cluster can provide higher processing power and storage capacity to support large-scale data analysis and processing. A cluster generally contains multiple nodes, each of which can be a separate computer or server. In a cluster, each node has its own processor, memory, and storage devices and can run tasks independently.

[0003] In a data sandbox cluster, nodes can be classified into different roles to meet different needs. For example, some nodes may be configured to specifically handle data loading and conversion tasks, while other nodes may be configured to run complex data analysis and machine learning algorithms. Currently, the nodes in the data sandbox cluster are classified, specifically by function. According to the function, the nodes can be divided into roles with different functions, such as data loading nodes, computing nodes, storage nodes, etc. Each type of node is responsible for processing specific tasks. For example, the data loading node is responsible for obtaining data from the data source, the computing node is responsible for performing data processing and analysis tasks, and the storage node is responsible for storing and managing data.

[0004] At present, the method of classifying cluster nodes by function mainly relies on the hardware capabilities of the nodes. For example, some nodes may have more powerful central processing units (CPUs) and graphics processing units (GPUs), which are suitable for executing computationally intensive tasks. When the tasks are running, the tasks with large computational requirements are placed on nodes with high hardware conditions for calculation. Some nodes may have larger storage space and are suitable as data storage nodes. However, in the actual operation of the data sandbox, multiple types of tasks are often performed, such as data analysis tasks, deep learning tasks, data prediction tasks, etc. The current node classification method only focuses on the hardware capabilities of the nodes, ignoring the differences in the performance of nodes running different types of tasks, resulting in the fact that the overall resources of the cluster are not fully utilized after the nodes are classified. How to make more full use of the data sandbox cluster resources and further improve the performance of node running tasks has become a technical problem that needs to be solved in this field. Summary of the invention

[0005] Based on the above problems, the present application provides a cluster node classification method, device, storage medium and processor, the purpose of which is to make more full use of cluster resources and improve the performance of node running tasks by classifying cluster nodes.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, the present application provides a cluster node classification method, comprising:

[0008] According to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions, obtaining resource occupation index data of the target node to be classified; the resource occupation index data includes multiple resource occupation weights corresponding to the multiple resource dimensions one by one;

[0009] According to the task running data of the target node to be classified for multiple task types, multiple groups of task type feature data of the target node to be classified are obtained; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions;

[0010] The resource occupation weight corresponding to each resource dimension in the resource occupation index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types;

[0011] The node category of the target node to be classified is configured according to the numerical values ​​of the adaptability between the target node to be classified and the multiple task types.

[0012] Optionally, in the above method, obtaining resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions includes:

[0013] Obtaining a ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension;

[0014] The ratios of the features of the multiple resource dimensions obtained are normalized to obtain multiple resource occupancy weights.

[0015] Optionally, the method as described above, for a target task type among the multiple task types, obtaining a set of task type feature data of the target node to be classified corresponding to the target task type according to the task running data of the target node to be classified for the target task type, comprises:

[0016] According to the running data of multiple tasks of the target task type respectively run by the target node to be classified, multiple correlation arrays are obtained; one correlation array corresponds to one task, and one correlation array includes multiple correlation coefficients corresponding to the multiple resource dimensions one by one, and the correlation coefficients represent the influence of the resource dimensions corresponding to the correlation coefficients on the running time when the target node to be classified runs the task;

[0017] The correlation coefficients corresponding to the same resource dimension in the plurality of correlation arrays are averaged to obtain a group of task type feature data corresponding to the target task type of the target node to be classified.

[0018] Optionally, the method as described above further comprises:

[0019] Randomly generate initial task resource configuration information for the target task according to the characteristics of the target node to be classified in the multiple resource dimensions and the minimum resource values ​​required for running the target task in the multiple resource dimensions; the initial task resource configuration information includes resource configuration values ​​of the multiple resource dimensions, and the resource configuration values ​​are greater than or equal to the minimum resource values ​​required for the corresponding resource dimensions; the target task is one of the multiple tasks;

[0020] Generating a task container for the target task in the target node to be classified according to the initial task resource configuration information;

[0021] The target task is run through the task container.

[0022] Optionally, the method as described above, obtaining a correlation array corresponding to the target task according to the running data of the target node to be classified running the target task, includes:

[0023] Extracting the first time information and the resource change result values ​​corresponding to the task container in the multiple resource dimensions when obtaining the running result of the target task from the running data of the target task run by the task container; the first time information is the duration from when the task container starts running the target task until the running result of the target task is obtained;

[0024] Obtaining characteristic change rates of the multiple resource dimensions according to the resource change result values ​​corresponding to the task container in the multiple resource dimensions and the resource configuration information of the multiple resource dimensions;

[0025] Based on the resource change result values ​​respectively corresponding to the multiple resource dimensions and the initial task resource configuration information, a plurality of groups of new task resource configuration information corresponding to the multiple resource dimensions are regenerated for the target task; in the new task resource configuration information, the resource configuration value of the corresponding resource dimension is the resource change result value corresponding to the corresponding resource dimension in the running data of the target task running by the task container; in the new task resource configuration information, the resource configuration value of the non-corresponding resource dimension is consistent with the resource configuration value of the same resource dimension in the initial task resource configuration information;

[0026] The target task is run by applying the multiple sets of new task resource configuration information respectively, and a plurality of second time information corresponding to the multiple resource dimensions are obtained; the second time information is the duration from the start of running the target task to obtaining the running result of the target task by applying a set of new task resource configuration information corresponding to the resource dimension;

[0027] Based on the first time information and the plurality of second time information, obtaining the runtime change rates respectively corresponding to the plurality of resource dimensions;

[0028] The characteristic change rates of the multiple resource dimensions and the running time change rates respectively corresponding to the multiple resource dimensions are processed according to the same resource dimensions to obtain multiple correlation coefficients corresponding to the multiple resource dimensions one by one, and form a correlation matrix corresponding to the target task.

[0029] Optionally, in the method described above, the resource occupancy index data is configured to be represented by a first vector, and the multiple groups of task type feature data are configured to be represented by multiple second vectors; in the first vector and the second vector, the same element position represents the same resource dimension;

[0030] The resource occupation weight corresponding to each resource dimension in the resource occupation index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types, including:

[0031] A dot product operation is performed on the first vector and each of the second vectors, and a vector dot product operation result obtained is used as the fitness of the target node to be classified and the task type corresponding to the second vector.

[0032] Optionally, in the above method, configuring the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types includes:

[0033] Determine the task type with the highest value of adaptability to the target node to be classified among the multiple task types as the adaptation task type of the target node to be classified;

[0034] The node category of the target node to be classified is configured as a target category; wherein the target category is a node category corresponding to the adaptation task type.

[0035] Optionally, in the method described above, before configuring the node category of the target node to be classified, the method further includes:

[0036] Obtaining the number of nodes to be allocated for each of the multiple node categories in the cluster, and obtaining the priorities of the multiple task types;

[0037] Based on the correspondence between node categories and task categories, the priority of the task type is used as the node allocation priority of the corresponding node category;

[0038] Sort the fitness of each node in the cluster with the same task type from large to small or from small to large to obtain a plurality of fitness sequences corresponding to the plurality of task types one by one;

[0039] The configuring the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types includes:

[0040] From the fitness sequence corresponding to the task type with the highest priority, determine the first L nodes with the highest fitness values, and configure the categories of the nodes corresponding to the first L fitness values ​​as the node categories corresponding to the task type with the highest priority; L is the number of nodes to be allocated in the node category corresponding to the task type with the highest priority;

[0041] From the fitness sequence corresponding to the task type with the second highest priority, after excluding the corresponding data of the classified nodes, determine the first M nodes with the highest fitness values, and configure the categories of the nodes corresponding to the first M fitness values ​​as the node categories corresponding to the task type with the second highest priority; M is the number of nodes to be allocated in the node category corresponding to the task type with the second highest priority;

[0042] The node categories of the remaining unclassified nodes in the cluster are configured as the node categories corresponding to the task types with the lowest priority.

[0043] In a second aspect, the present application provides a cluster node classification device, comprising:

[0044] A node resource occupancy index calculation module, used to obtain resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to the multiple resource dimensions one by one;

[0045] a task type feature calculation module, configured to obtain multiple groups of task type feature data of the target node to be classified according to the task running data of the target node to be classified for multiple task types; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions;

[0046] A fitness calculation module is used to calculate and process the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in each group of task type feature data according to the same resource dimension, so as to obtain the fitness of the target node to be classified and the multiple task types;

[0047] The node category configuration module is used to configure the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types.

[0048] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the cluster node classification method described in any of the above embodiments.

[0049] In a fourth aspect, the present application provides a processor, which is used to run a program, wherein the program executes the cluster node classification method described in any one of the above embodiments when running.

[0050] This application obtains resource occupancy index data of the target node to be classified according to the characteristics of the cluster and the target node to be classified in the cluster in multiple resource dimensions; wherein, the resource occupancy index data includes the resource occupancy weight of the target node to be classified compared with the cluster in different resource dimensions, and by quantifying the resource occupancy of the target node to be classified relative to the cluster as a whole, it helps to understand the allocation and use of resources more accurately. In addition, according to the task operation data of the target node to be classified for multiple task types, the corresponding task type feature data of the target node to be classified when executing different types of tasks is obtained; wherein, the task type feature data includes the resource correlation weight under each resource dimension, so as to obtain the operation impact degree of the target node to be classified under different resource dimensions when running different types of tasks. Then, by combining the resource occupancy index data of the target node to be classified with each set of task type feature data, and processing them according to the same resource dimension, the adaptability of the target node to be classified and multiple task types is obtained, and the matching degree between the current target node to be classified and each type of task is quantified, thereby mining the node type associated with the task type that the target node to be classified is adapted to; finally, according to the numerical value of the adaptability of the target node to be classified and the multiple task types, the node category of the target node to be classified is configured. In this solution, attention is paid to the differences in the running performance of nodes running different types of tasks, and thus the configuration of node categories is realized, making full use of the data sandbox cluster resources and improving the performance of node running tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A schematic diagram of an embodiment of a cluster node classification method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of another embodiment of a cluster node classification method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a flow chart of another embodiment of a cluster node classification method provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of another embodiment of a cluster node classification method provided in an embodiment of the present application;

[0056] Figure 5A schematic diagram of another embodiment of a cluster node classification method provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of an embodiment of a cluster node classification device provided by the present application;

[0058] Figure 7 A schematic structural diagram of another embodiment of a cluster node classification device provided in the present application. DETAILED DESCRIPTION

[0059] As described above, the current cluster node classification method mainly relies on the hardware capabilities of the nodes and is divided according to their functions. However, in the actual operation of the data sandbox, multiple types of tasks are often performed, such as data analysis tasks, deep learning tasks, data prediction tasks, etc. The current node classification method only focuses on the hardware capabilities of the nodes, ignoring the differences in the performance of nodes running different types of tasks, resulting in the overall cluster resources not being fully utilized after node classification.

[0060] After research, the inventor provides a cluster node classification method, device, storage medium and processor to solve the technical problem in the prior art that data sandbox cluster resources are not fully utilized, resulting in poor performance of node running tasks.

[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] See also Figure 1 , which is a schematic diagram of an embodiment of a cluster node classification method provided in an embodiment of the present application. The method includes:

[0063] S101: Obtain resource occupancy index data of a target node to be classified according to characteristics of a cluster in multiple resource dimensions and characteristics of a target node to be classified in the cluster in multiple resource dimensions.

[0064] The resource occupancy index data includes multiple resource occupancy weights corresponding to multiple resource dimensions. For example, the multiple resource dimensions mentioned in step S101 may include but are not limited to one or more of memory, disk storage, disk throughput, CPU computing power, GPU computing power, and network bandwidth.

[0065] In this embodiment, the characteristics of the cluster in multiple resource dimensions include the total amount of resources of all nodes in the cluster in each resource dimension. For example, if the number of nodes in the cluster is n, and the multiple resource dimensions include disk storage S, disk throughput T, CPU computing power C, GPU computing power G, and network bandwidth B; then the total amount of resources of the cluster in multiple resource dimensions is C = [S c ,T c ,C c ,G c ,B c ]. The total amount of resources of the cluster under the resource dimension of disk storage is expressed as Among them, S i It represents the resource usage of the disk storage capacity of the i-th node among the n nodes in the cluster. The total resource of the cluster under the resource dimension of disk throughput is expressed as Among them, T i It represents the resource usage of the disk throughput of the i-th node among the n nodes in the cluster. The total resources under other resource dimensions are expressed in the same way. The resource usage of the target node i to be classified in the cluster under multiple resource dimensions can be expressed as N i =[S i ,T i ,C i ,G i ,B i ]. According to the total amount of resources of the cluster in multiple resource dimensions C = [S c ,T c ,C c ,G c ,B c ], and the resource usage N of the target node i in the cluster in multiple resource dimensions i =[S i ,T i ,C i ,G i ,B i ], and obtain the resource occupancy index data of the target node to be classified through corresponding mathematical operations. Specifically, the resource occupancy index data can be constructed according to the ratio of the resource occupancy of the target node to be classified in the same resource dimension to the total amount of resources in the resource dimension in the cluster. Among them, the resource occupancy index data of the target node to be classified can be expressed as:

[0066]

[0067] S102: Acquire multiple groups of task type feature data of the target node to be classified according to the task running data of the target node to be classified for multiple task types.

[0068] Therein, multiple groups of task type feature data correspond one-to-one to multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to multiple resource dimensions.

[0069] In this embodiment, for example, tasks of multiple task types may include but are not limited to one or more of computing tasks, prediction tasks, and analysis tasks. Multiple sets of task running data corresponding to each of the multiple task types are obtained according to the target node to be classified, including resource change result values ​​corresponding to multiple resource dimensions when the task is running and the duration of running the target task until the running result of the target task is obtained.

[0070] Specifically, a set of task type feature data of the target node to be classified for the computing task type includes: multiple resource correlation weights corresponding to multiple resource dimensions. Assuming that the multiple resource dimensions include: disk storage capacity S, disk throughput T, CPU computing power C, GPU computing power G and network bandwidth B, a total of five resource dimensions, then a set of task type feature data of the target node to be classified i for the computing task type can be expressed as:

[0071]

[0072] Among them, Cor i c Indicates that the task type is the task type characteristic data corresponding to the computing task; It represents the resource relevance weight of the i-th node in the cluster for computing task type c under the resource dimension of disk storage capacity S; It represents the resource relevance weight of the i-th node in the cluster for computing task type c under the resource dimension of disk throughput T.

[0073] Similarly, the task type feature data corresponding to the prediction task for the target node i to be classified includes multiple resource correlation weights corresponding to multiple resource dimensions one by one, which can be expressed as Cor i f The target node i to be classified has a task type corresponding to the analysis task, and the task type feature data includes multiple resource correlation weights corresponding to multiple resource dimensions, which can be expressed as Cor i a .

[0074] S103: The resource occupancy weight corresponding to each resource dimension in the resource occupancy index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and multiple task types.

[0075] For example, the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data of the target node i to be classified are expressed as In each set of task type feature data, the resource correlation weight corresponding to each resource dimension is used to calculate the task type feature data Cor i c The resource relevance weights corresponding to each resource dimension in Task type feature data Cor i f The resource relevance weights corresponding to each resource dimension in And the task type characteristic data Cor i a The resource relevance weights corresponding to each resource dimension in

[0076] In this embodiment, for each group of task types, the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in the characteristic data of the task type are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types. The processing method can be to multiply the resource occupancy weights of the same resource dimension by the resource correlation weight coefficient, and accumulate the multiplication results of each resource dimension according to the same task type. Among them, the adaptability of the target node to be classified i and the multiple task types can be expressed as follows: Calculate the task adaptability Prediction task suitability F i f And analyze task suitability

[0077] S104: configuring the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types.

[0078] In this embodiment, for example, according to the obtained adaptability of the target node to be classified and the corresponding adaptability of the multiple task types, F i f as well as The size of the cluster node is selected from multiple task types to be the most matched with the target node to be classified; and the task type is configured as the node category of the target node to be classified, thereby completing the classification of the cluster nodes.

[0079] In an optional implementation, a method for selecting a task type that best matches the target node to be classified may be to compare the fitness of multiple task types and match the target node to be classified with the task type with the highest fitness. By comparing the fitness of different task types, it can be ensured that the target node to be classified is assigned to the task type that best suits its characteristics and needs, thereby reducing errors and uncertainties in the task execution process and improving the accuracy of task completion.

[0080] In another optional implementation, the node allocation priority of the node category corresponding to the task type is determined according to the preset number of tasks for each task type and the priority of the task type; a corresponding fitness sequence is generated for each task type; the fitness sequence is to sort the fitness of each node in the cluster with the same task type from large to small or from small to large; finally, according to the priority of the task type, the unclassified nodes that meet the preset number of tasks of the task type and have the highest fitness value are selected from the corresponding fitness sequence in turn, and the node category is configured for the selected nodes. In this implementation, high-priority task types usually have higher execution value or more urgent time requirements. Prioritizing the configuration of the highest fitness node for this task type can maximize the use of node resources and avoid wasting node resources on low-priority task types, thereby improving the efficiency of overall task execution.

[0081] It is understandable that when the resource configuration of the nodes in the cluster changes, or the number of nodes to be classified in the cluster changes (increases or decreases), the resource occupancy index data of the nodes will also change accordingly compared to before the above changes. Therefore, the originally configured node category may not be applicable after the change. For this reason, the technical solution of this application can be used to reclassify the cluster nodes.

[0082] In this embodiment, based on the characteristics of the cluster and the target node to be classified in the cluster in multiple resource dimensions, resource occupancy index data of the target node to be classified is obtained; wherein, the resource occupancy index data includes the resource occupancy weight of the target node to be classified compared with the cluster in different resource dimensions, and by quantifying the resource occupancy of the target node to be classified relative to the cluster as a whole, it helps to understand the allocation and usage of resources more accurately. Furthermore, according to the task running data of the target node to be classified for various task types, the task type characteristic data corresponding to the target node to be classified when executing different types of tasks is obtained; wherein the task type characteristic data includes the resource correlation weight under each resource dimension, so as to obtain the operation impact degree of the target node to be classified under different resource dimensions when running different types of tasks; then, by calculating and processing the resource occupancy index data of the target node to be classified and each group of task type characteristic data according to the same resource dimension, the adaptability of the target node to be classified to the various task types is obtained, and the matching degree between the current target node to be classified and each type of task is quantified, so as to mine the node type associated with the task type adapted by the target node to be classified; finally, according to the numerical value of the adaptability of the target node to be classified to the various task types, the node category of the target node to be classified is configured; according to the numerical value of the adaptability of the target node to be classified to the various task types, the determined task type is configured as the node category of the target node to be classified, and the resources are allocated to the task that is most suitable for its execution through the obtained matching degree. In this solution, attention is paid to the differences in the running performance of nodes running different types of tasks, and node category configuration is implemented accordingly, which fully utilizes the data sandbox cluster resources to configure suitable node categories for the target nodes to be classified, thereby improving the performance of node running tasks.

[0083] See also Figure 2 , which is a flow chart of another embodiment of a cluster node classification method provided by an embodiment of the present application. Based on the above embodiment, a specific implementation method of "obtaining resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions" in S101 includes:

[0084] S1011: Obtain the ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension.

[0085] In this embodiment, for example, according to the total amount of resources C of the cluster in multiple resource dimensions obtained = [S c ,T c ,C c ,G c ,B c ] and the resource usage N of the target node to be classified in the cluster in multiple resource dimensions i=[S i ,T i ,C i ,G i ,B i ], calculate the ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension, which can be expressed by the following expression:

[0086]

[0087] in, The disk storage ratio of the target node to be classified in the cluster; The disk throughput ratio of the target node to be classified in the cluster; The CPU computing power ratio of the target node to be classified in the cluster; The GPU computing power ratio of the target node to be classified in the cluster; It is the network bandwidth ratio of the target node to be classified in the cluster.

[0088] S1012: Normalize the ratios of the acquired features of the multiple resource dimensions to obtain multiple resource occupancy weights.

[0089] In this embodiment, the disk storage ratio, disk throughput ratio, CPU computing power ratio, GPU computing power ratio and network bandwidth ratio under the obtained multiple resource dimensions are normalized using the following formulas:

[0090]

[0091] In the above expression, k takes values ​​between 1 and 5, and k is an integer. When k takes different values, it represents different resource dimensions among the five resource dimensions. The value of x is the ratio calculated by S1011, and x is associated with the resource dimension represented by k. By normalizing the above expression, the resource occupancy weights corresponding to multiple resource dimensions of the target node to be classified in the cluster are obtained; among which, the disk storage occupancy weight is expressed as Disk throughput occupation weight CPU computing power usage weight GPU computing power usage weight and network bandwidth usage weight

[0092] In this embodiment, by obtaining the ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension; and normalizing the ratios of the characteristics of multiple resource dimensions obtained, multiple resource occupancy weights are obtained. Compared with the prior art, the method of the present application can quantify the resource occupancy of the target node relative to the entire cluster by calculating the ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension, which helps to understand the allocation and use of resources more accurately; and, by normalizing the ratios of the characteristics of multiple resource dimensions, the characteristic values ​​of all resource dimensions are converted to the same numerical range, thereby eliminating the influence of the dimension on the evaluation results, reflecting the degree of influence of different resource dimensions on the performance or status of the target node, and improving the accuracy of resource occupancy evaluation.

[0093] See also Figure 3 , which is a flow chart of another embodiment of a cluster node classification method provided by an embodiment of the present application. Based on the above embodiment, for a target task type among multiple task types, a specific implementation method of "acquiring a set of task type feature data corresponding to the target task type of the target node to be classified according to the task running data of the target node to be classified for the target task type" in S102 includes:

[0094] S1021: Acquire multiple related relationship arrays according to the running data of multiple tasks of the target task type respectively run by the target node to be classified.

[0095] Among them, a correlation array corresponds to a task run by the target node to be classified, and a correlation array includes multiple correlation coefficients corresponding to multiple resource dimensions one by one. The correlation coefficient represents the influence of the resource dimension corresponding to the correlation coefficient on the running time when the target node to be classified runs the task.

[0096] In this embodiment, taking the target task type as a computing task as an example, the running data corresponding to multiple computing tasks are respectively run according to the target node to be classified, and the corresponding relationship array corresponding to running each task is obtained. Among them, the running data corresponding to the computing task includes the resource change result values ​​corresponding to multiple resource dimensions before and after the execution of the computing task, and the total time required from the start of the computing task to the completion of the task and obtaining the running result.

[0097] For example, the correlation matrix can be used with Cor j Indicates, where j represents a task of a target task type running on the target node to be classified. Among them, a correlation array includes multiple correlation coefficients corresponding to multiple resource dimensions one by one, which can be expressed by the following expression:

[0098]

[0099] Among them, the correlation array Cor j The elements in represent the correlation coefficients corresponding to the resource dimension of disk storage, the correlation coefficient corresponding to the resource dimension of disk throughput, the correlation coefficient corresponding to the resource dimension of CPU computing power, the correlation coefficient corresponding to the resource dimension of GPU computing power, and the correlation coefficient corresponding to the resource dimension of network bandwidth in turn; the correlation coefficient corresponding to each resource dimension is used to represent the impact of the resource dimension on the running time when the target node to be classified runs the task. Specifically, the larger the correlation coefficient, the greater the impact on the running time; the smaller the correlation coefficient, the smaller the impact on the running time. In other words, the numerical value of the correlation coefficient is positively correlated with the degree of impact of the resource dimension on the running time.

[0100] S1022: Calculate the average of the correlation coefficients corresponding to the same resource dimension in multiple correlation arrays to obtain a set of task type feature data corresponding to the target task type of the target node to be classified.

[0101] In this embodiment, for example, if the target task type of the target node to be classified is a computing task, the task is m c ; then for this m c The correlation coefficients corresponding to the same resource dimension in the correlation array corresponding to each task are averaged using the following formula, that is, the average is calculated according to the same dimension:

[0102]

[0103] Among them, Cor k is the array of related information corresponding to the kth task whose target task type is computing task; specifically,

[0104] Cor i c The target node i to be classified corresponds to a set of task type feature data of the target task type being a computing task; specifically,

[0105] In this embodiment, according to the operation data of multiple tasks of the target task type respectively run by the target node to be classified, multiple related relationship arrays are obtained; the correlation coefficients corresponding to the same resource dimension in the multiple related relationship arrays are averaged to obtain a set of task type feature data corresponding to the target task type of the target node to be classified. Compared with the prior art, the method of the present application can avoid the influence of multiple random factors such as system load, memory allocation, etc. on a single task by running multiple tasks of the target task type and obtaining their operation data to obtain the average of the correlation coefficients in the same dimension, thereby effectively reducing the influence of these random errors on the final result, thereby improving the accuracy of classification; and, the degree of influence of different resource dimensions on the target task type can be deeply understood, and by understanding the performance characteristics of different nodes under different task types, more refined resource management can be achieved, thereby improving resource utilization.

[0106] Optionally, in the above method, a target task is used as an example to introduce a method for obtaining operation data. The target task is one of multiple tasks of the target task type. In order to obtain the operation data corresponding to the task, the method further includes:

[0107] According to the characteristics of the target node to be classified in multiple resource dimensions and the minimum resource values ​​required to run the target task in multiple resource dimensions, initial task resource configuration information is randomly generated for the target task.

[0108] The initial task resource configuration information includes resource configuration values ​​of multiple resource dimensions, and the resource configuration value is greater than or equal to the minimum resource value required by the corresponding resource dimension.

[0109] In this embodiment, for example, according to the characteristics of the target node to be classified in multiple resource dimensions and the minimum resource values ​​required to run the target task in multiple resource dimensions, the target task j in the task of the current target task type randomly generates the initial task resource configuration information Task j ; Among them, the initial task resource configuration information Task j The resource configuration values ​​of multiple resource dimensions are included in it, which can be expressed as

[0110] It is understandable that in order to avoid over-allocation or under-allocation of resources, the resource configuration value must be greater than or equal to the minimum resource value required by the corresponding resource dimension. That is, the initial task resource configuration should at least be able to meet the resource requirements for running the task.

[0111] On the premise that the initial configuration of the task resources has been completed, a task container is generated for the target task in the target node to be classified according to the initial task resource configuration information.

[0112] In actual applications, for each task to be run, the node will perform initial configuration of task resources for it.

[0113] In this embodiment, in the target node to be classified, a task container is generated for the target task according to the randomly generated initial task resource configuration information, so as to run the target task through the task container. It can be understood that in order to obtain the running data corresponding to multiple tasks of the target task type, it is necessary to run multiple tasks of the target task type. After a task container A is generated for the task in the target node to be classified according to the initial task resource configuration information of task A to run task A, task A is completed and the original task container A can be deleted. A new task container B is generated according to the initial task resource configuration information of task B to run task B. In addition, after task container A has completed running task A, task container A can also be reconfigured according to the initial task resource configuration information of target task B and updated to task container A′ to run task B.

[0114] In this embodiment, according to the characteristics of the target node to be classified in multiple resource dimensions and the minimum resource values ​​required to run the target task in multiple resource dimensions, the initial task resource configuration information is randomly generated for the target task, and a task container is generated for the target task in the target node to be classified according to the initial task resource configuration information; the target task is run through the task container. Compared with the prior art, the method of the present application randomly generates the initial task resource configuration information according to the minimum resource value, avoids over-allocation or under-allocation of resources, and ensures the effective use of resources; and generates a special task container for the target task to directly run the task in the container, which can reduce the preparation time and resource allocation time before the task starts, thereby improving the efficiency of node classification.

[0115] In order to more accurately analyze the impact of various resource dimensions on task running time when the target node to be classified runs different types of tasks, thereby improving the accuracy of node classification and the utilization of cluster resources, this solution proposes that for each task, it is run once with the initial task resource configuration, and then run again with the new task resource configuration. The main purpose is to obtain two key values: the characteristic change rate of the resource dimension, and the running time change rate of the resource dimension. Based on these two key values, the one-to-one correlation coefficients of each resource dimension are obtained to form a correlation matrix corresponding to the task. The following is an explanation of this process.

[0116] On the basis of the above embodiment, after a task container is generated according to the initial task resource configuration information to run a task of a target task type, for the running data of the task of the target task type, a specific implementation method of "obtaining a correlation array corresponding to the target task according to the running data of the target node to be classified" in S1021 includes:

[0117] The first time information and resource change result values ​​corresponding to the task container in multiple resource dimensions when the target task is run are extracted from the running data of the task container running the target task.

[0118] The first time information is the duration from when the task container starts running the target task until the running result of the target task is obtained.

[0119] In this embodiment, for example, from the running data of the target task running in the task container, the time from when the task container starts running the target task j to when the running result of the target task is obtained is extracted as the first time information t j ; and extract the resource change result values ​​corresponding to the task container in multiple resource dimensions when obtaining the running result of the target task j, and then calculate the resource change result values ​​corresponding to the task container in multiple resource dimensions. j The unified expression is:

[0120]

[0121] According to the resource change result values ​​corresponding to the task container in the multiple resource dimensions and the resource configuration information of the multiple resource dimensions, the characteristic change rates of the multiple resource dimensions are obtained.

[0122] In this embodiment, according to the resource change result values ​​Task′ corresponding to the task container in multiple resource dimensions, j , and initial task resource configuration information Task j , calculated separately in multiple resource dimensions to obtain the characteristic change rate of each resource dimension, including ΔS j , ΔT j , ΔC j , ΔG j and ΔB j .

[0123] Taking the resource dimension of disk storage as an example, the characteristic change rate of this resource dimension is calculated using the following formula. The characteristic change rates of other resource dimensions are similar to the following expressions:

[0124]

[0125] Based on the resource change result values ​​corresponding to the multiple resource dimensions and the initial task resource configuration information, multiple groups of new task resource configuration information corresponding to the multiple resource dimensions are regenerated for the target task.

[0126] Among them, in the new task resource configuration information, the resource configuration value of the corresponding resource dimension is the resource change result value corresponding to the corresponding resource dimension in the running data of the target task running in the task container; in the new task resource configuration information, the resource configuration value of the non-corresponding resource dimension is consistent with the resource configuration value of the same resource dimension in the initial task resource configuration information.

[0127] In this embodiment, based on Task′ j The resource change result values ​​corresponding to multiple resource dimensions in Task and the initial task resource configuration information j , regenerate multiple sets of new task resource configuration information corresponding to multiple resource dimensions for the target task. The generated new task resource configuration information includes:

[0128]

[0129] In the above five sets of new task resource configuration information, the feature with the ' symbol in the upper right corner indicates the resource change result value. Apply multiple sets of new task resource configuration information to run the target task respectively, and obtain multiple second time information corresponding to multiple resource dimensions.

[0130] The second time information is a set of new task resource configuration information corresponding to the resource dimension of the application, which is the duration from the start of running the target task to obtaining the running result of the target task.

[0131] In this embodiment, new task resource configuration information is applied respectively as well as Run the target task j, thereby obtaining the duration from the start of running the target task to obtaining the running result of the target task corresponding to multiple sets of new task resource configuration information as multiple second time information as well as

[0132] Based on the first time information and the plurality of second time information, runtime change rates respectively corresponding to the plurality of resource dimensions are obtained.

[0133] In this embodiment, based on the first time information t j and multiple second time information as well as Calculate the runtime change rate corresponding to multiple resource dimensions as well as For example, if the resource dimension is disk storage, the running time change rate of the current resource dimension is calculated using the following formula:

[0134]

[0135] The characteristic change rates of multiple resource dimensions and the running time change rates corresponding to the multiple resource dimensions are processed according to the same resource dimensions to obtain multiple correlation coefficients corresponding to the multiple resource dimensions one by one, forming a correlation matrix corresponding to the target task j.

[0136] In this embodiment, taking the calculation of the correlation coefficient corresponding to the disk storage capacity as the resource dimension as an example, the characteristic change rate ΔS corresponding to the disk storage capacity as the resource dimension is calculated. j The running time change rate corresponding to the resource dimension The following formula is used to perform calculations on the same resource dimension to obtain the correlation coefficient corresponding to the resource dimension of disk storage capacity:

[0137]

[0138] Similarly, the correlation coefficients corresponding to other resource dimensions can be obtained, thereby forming a correlation array corresponding to the target task j

[0139] It can be understood that in the process of obtaining multiple correlation coefficients corresponding one-to-one to multiple resource dimensions to form a correlation array corresponding to the target task, in addition to the method described in the above embodiment, new initial task resource configuration information can be repeatedly randomly generated for the target task, and the target task can be run using the new initial task resource configuration information to obtain multiple correlation coefficients corresponding to each resource dimension, and the average value of the multiple correlation coefficients obtained for each resource dimension can be calculated, and the average value can be used as the correlation coefficient corresponding to the resource dimension to form a correlation array corresponding to the target task.

[0140] In this embodiment, by extracting the first time information and the resource change result values ​​corresponding to the task container in multiple resource dimensions when obtaining the operation result of the target task from the operation data of the task container running the target task, and calculating the characteristic change rates of the multiple resource dimensions; based on the resource change result values ​​corresponding to the multiple resource dimensions and the initial task resource configuration information, regenerate multiple sets of new task resource configuration information corresponding to the multiple resource dimensions for the target task to run the target task, obtain multiple second time information corresponding to the multiple resource dimensions and calculate the running time change rates corresponding to the multiple resource dimensions; the characteristic change rates of the multiple resource dimensions and the running time change rates corresponding to the multiple resource dimensions are processed according to the same resource dimensions to form a relative relationship array corresponding to the target task. Compared with the prior art, the method of the present application quantifies the consumption of each resource dimension during the execution of the target task by extracting the resource change result values ​​of the task container in multiple resource dimensions; and, by calculating the characteristic change rates and running time change rates of multiple resource dimensions, it can reflect the changes in resource utilization efficiency and improve the accuracy of node classification.

[0141] See also Figure 4 , which is a flow chart of another embodiment of a cluster node classification method provided by an embodiment of the present application. Based on the above embodiment, the resource occupancy index data is configured to be represented by a first vector, and multiple groups of task type feature data are configured to be represented by multiple second vectors. Among them, in the first vector and the second vector, the same element position represents the same resource dimension.

[0142] At this time, a specific implementation method of "computing the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in each group of task type feature data according to the same resource dimension to obtain the adaptability of the target node to be classified and multiple task types" in step S103 includes:

[0143] S1031: performing a dot product operation on the first vector and each second vector, and obtaining a vector dot product operation result as a fitness degree between the target node to be classified and the task type corresponding to the second vector.

[0144] In this embodiment, for example, the first vector corresponding to the target node to be classified i is the resource occupancy index data corresponding to the current target node to be classified The second vector is multiple sets of task type feature data, including the task type feature data Cor of the computing task i c , feature data Cor of the prediction task i f And the task type characteristic data Cor ia Specifically, the second vectors corresponding to the multiple groups of task type feature data can be expressed as:

[0145]

[0146] A dot product operation is performed on the first vector corresponding to the target node to be classified and each second vector, and the vector dot product operation result obtained is used as the fitness of the target node to be classified and the task type corresponding to the second vector.

[0147] Specifically, the adaptability of the target node to be classified and various task types can be expressed as:

[0148]

[0149] F i f =Cor i f ·W i

[0150]

[0151] in, represents the adaptability between the target node i to be classified and the computing task; F i f Indicates the adaptability between the target node i to be classified and the computing task; Indicates the degree of adaptability between the target node i to be classified and the computing task.

[0152] In this embodiment, when the resource occupancy index data is configured to be represented by a first vector, and multiple groups of task type feature data are configured to be represented by multiple second vectors, a dot product operation is performed by the first vector and each second vector, and the vector dot product operation result is used as the adaptability of the target node to be classified to the task type corresponding to the second vector. Compared with the prior art, the method of the present application uses the first vector to represent the calculation of the resource occupancy index and multiple second vectors to represent multiple task type features, and calculates the dot product between the two vectors, which can quantify the matching degree between the current target node to be classified and each type of task, and allocate resources to the task that is most suitable for its execution through the obtained matching degree, thereby avoiding waste and unreasonable allocation of resources and improving resource utilization.

[0153] See also Figure 5 , which is a flow chart of another embodiment of a cluster node classification method provided by an embodiment of the present application. Based on the above embodiment, a specific implementation method of "configuring the node category of the target node to be classified according to the numerical value of the adaptability of the target node to be classified to multiple task types" in S104 includes:

[0154] S1041: Determine the task type with the highest value of adaptability to the target node to be classified among the multiple task types as the adaptation task type of the target node to be classified.

[0155] In this embodiment, for example, is the adaptability between the target node i to be classified and the computing task; F i f is the adaptability between the target node i to be classified and the computing task; is the adaptability between the target node i to be classified and the computing task. F i f as well as The task type corresponding to the highest value is compared with the size of the target node to be classified as the adaptation task type.

[0156] S1042: configuring the node category of the target node to be classified as the target category.

[0157] The target category is the node category corresponding to the adaptation task type.

[0158] In this embodiment, taking the adaptation task type as a computing task as an example, the node category of the target node to be classified is configured as a computing node; wherein the computing node is a node category corresponding to the computing task.

[0159] In this embodiment, the task type with the highest value of adaptability to the target node to be classified among multiple task types is determined as the adaptation task type of the target node to be classified; and the node category of the target node to be classified is configured as the target category. Compared with the prior art, the method of the present application improves the accuracy of classification by matching the target node to be classified with the task type with the highest adaptability, avoids waste and unreasonable use of resources, and enables more efficient use of cluster resources.

[0160] Further, different from the above embodiment in which the task type with the highest value of adaptability to the target node to be classified among the multiple task types is used as the adaptation task type of the target node to be classified, before executing S104, the following may also be included:

[0161] Get the number of nodes to be allocated for each of multiple node categories in the cluster, and get the priorities of multiple task types.

[0162] In this embodiment, for example, the number of nodes to be allocated for each of the multiple node categories in the cluster is respectively calculated as follows: c , prediction nodes m f and analysis nodes m a and the priority order of multiple task types from high to low is: computing task, prediction task, and analysis task.

[0163] Based on the correspondence between the node category and the task category, the priority of the task type is used as the node allocation priority of the corresponding node category.

[0164] In this embodiment, the task category corresponding to the computing node is a computing task, the task category corresponding to the prediction node is a prediction task, and the task category corresponding to the analysis node is an analysis task. Based on the correspondence between the node category and the task category, the priority of the task type is used as the node allocation priority of the corresponding node category, and the node allocation priority order can be determined from high to low as computing node, prediction node, and analysis node.

[0165] The fitness of each node in the cluster with the same task type is sorted from large to small or from small to large, and a plurality of fitness sequences corresponding to the plurality of task types are obtained.

[0166] In this embodiment, under each task type, all nodes in the cluster are sorted from large to small or from small to large according to the fitness, so as to obtain the fitness sequence corresponding to each task type. For example, if the number of all nodes in the cluster is n, for the task type of computing task, all nodes in the cluster are sorted from large to small according to the fitness, and the obtained fitness sequence can be expressed as Among them, F c The middle element is the value of the corresponding fitness of all nodes, and the fitness value decreases from left to right.

[0167] At this time, another specific implementation of "configuring the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified to multiple task types" in S104 includes:

[0168] From the fitness sequence corresponding to the task type with the highest priority, determine the first L with the highest fitness values, and configure the categories of the nodes corresponding to the first L fitness values ​​as the node categories corresponding to the task type with the highest priority.

[0169] Among them, L is the number of nodes to be allocated in the node category corresponding to the task type with the highest priority.

[0170] In this embodiment, for example, the task type with the highest priority is the computing task, and its corresponding fitness sequence is Among them, F c The elements in are sorted from large to small according to the fitness of all nodes. If the number of nodes to be assigned to the computing node is L, then the fitness sequence is F c , select L values ​​with the highest fitness from left to right; and configure the node categories of the nodes corresponding to the L values ​​with the highest fitness as computing nodes.

[0171] From the fitness sequence corresponding to the task type with the second highest priority, after excluding the corresponding data of the classified nodes, the top M nodes with the highest fitness values ​​are determined, and the categories of the nodes corresponding to the top M fitness values ​​are configured as the node categories corresponding to the task type with the second highest priority.

[0172] Wherein, M is the number of nodes to be allocated in the node category corresponding to the task type with the second highest priority.

[0173] In this embodiment, for example, the task type with the second highest priority is the prediction task, and its corresponding fitness sequence is Among them, F f The elements in are sorted from large to small according to the fitness of all nodes. After excluding the fitness data corresponding to the nodes that have been classified in the previous step, if the number of nodes to be allocated for the prediction node is M, then the fitness sequence is F. f , select M values ​​with the highest fitness from left to right; and configure the node categories of the nodes corresponding to the M values ​​with the highest fitness as prediction nodes.

[0174] Configure the node class of the remaining unclassified nodes in the cluster to the node class corresponding to the lowest priority task type.

[0175] In this embodiment, for example, the task type with the lowest priority is the analysis task; nodes configured as computing nodes and prediction nodes in the cluster are excluded, and the node categories of the remaining unclassified nodes are configured as analysis task nodes corresponding to the analysis task.

[0176] In another embodiment, for example, the obtained numbers of nodes to be allocated for each of the multiple node categories in the cluster are respectively calculated as follows: c , prediction nodes m f , analysis nodes m a and storage nodes m s and the priority order of multiple task types from high to low is: computing task, prediction task, analysis task and storage task. Based on the correspondence between node category and task category, the priority of task type is used as the node allocation priority of the corresponding node category. It can be determined that the node allocation priority order from high to low is computing node, prediction node, analysis node and storage node. Under each task type, all nodes in the cluster are sorted from large to small according to fitness, so as to obtain the fitness sequence corresponding to each task type.

[0177] At this time, after excluding the corresponding data of the classified nodes from the fitness sequence corresponding to the task type with the second highest priority, the top M nodes with the highest fitness values ​​are determined, and the categories of the nodes corresponding to the top M fitness values ​​are configured as the node categories corresponding to the task type with the second highest priority, including: from the fitness sequence corresponding to the task type with the second lowest priority, after excluding the corresponding data of the classified nodes, the top N nodes with the highest fitness values ​​are determined, and the categories of the nodes corresponding to the top N fitness values ​​are configured as the node categories corresponding to the task type with the second lowest priority.

[0178] Specifically, the task type with the second lowest priority is the analysis task, and its corresponding fitness sequence is Among them, F a The elements in are sorted from large to small according to the fitness of all nodes. After excluding the fitness data corresponding to the nodes that have been classified as computing nodes and prediction nodes, if the number of nodes to be assigned to the analysis node is N, then the fitness sequence is F. a , select N values ​​with the highest fitness from left to right; and configure the node categories of the nodes corresponding to the N values ​​with the highest fitness as analysis nodes.

[0179] Configure the node class of the remaining unclassified nodes in the cluster to the node class corresponding to the lowest priority task type.

[0180] In this embodiment, for example, the task type with the lowest priority is the storage task; nodes configured as computing nodes, prediction nodes, and analysis tasks in the cluster are excluded, and the node categories of the remaining unclassified nodes are configured as storage task nodes corresponding to the storage task.

[0181] In this embodiment, by obtaining the number of nodes to be allocated of each of the multiple node categories in the cluster, and obtaining the priorities of multiple task types; based on the correspondence between the node category and the task category, the priority of the task type is used as the node allocation priority of the corresponding node category; the fitness of each node in the cluster with the same task type is sorted from large to small or from small to large, and multiple fitness sequences corresponding to multiple task types are obtained; in order of priority, the unclassified node with the highest fitness is selected from the fitness sequence corresponding to the task type and configured as the node type corresponding to the task type. Compared with the prior art, the method of the present application can ensure that the node most suitable for executing a certain type of task is preferentially allocated to the task by obtaining the number of nodes to be allocated of different node categories in the cluster and performing node allocation in combination with the priority of the task type, thereby effectively improving the overall resource utilization efficiency and further improving the task execution efficiency.

[0182] See also Figure 6, which is a schematic diagram of the structure of an embodiment of a cluster node classification device provided by the present application. The device 60 includes: a node resource occupancy index calculation module 61, a task type feature calculation module 62, a fitness calculation module 63 and a node category configuration module 64.

[0183] The node resource occupancy index calculation module 61 is used to obtain the resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to the multiple resource dimensions.

[0184] The task type feature calculation module 62 is used to obtain multiple groups of task type feature data of the target node to be classified according to the task running data of the target node to be classified for multiple task types; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to multiple resource dimensions.

[0185] The adaptability calculation module 63 is used to calculate and process the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in each group of task type feature data according to the same resource dimension to obtain the adaptability of the target node to be classified and multiple task types.

[0186] The node category configuration module 64 is used to configure the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and multiple task types.

[0187] Further, based on the above embodiment, the node resource occupancy index calculation module 61 is specifically used to obtain the ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension; the ratios of the characteristics of multiple resource dimensions obtained are normalized to obtain multiple resource occupancy weights.

[0188] Further, on the basis of the above embodiments, for a target task type among multiple task types, a task type feature calculation module 62 is specifically used to obtain multiple correlation arrays according to the running data of multiple tasks of the target task type respectively running by the target node to be classified; one correlation array corresponds to one task, and one correlation array includes multiple correlation coefficients corresponding to multiple resource dimensions one by one, and the correlation coefficient represents the influence of the resource dimension corresponding to the correlation coefficient on the running time when the target node to be classified runs the task; the correlation coefficients corresponding to the same resource dimension in the multiple correlation arrays are averaged to obtain a set of task type feature data corresponding to the target task type of the target node to be classified.

[0189] Further, based on the above embodiment, the task type feature calculation module 62 is also specifically used to randomly generate initial task resource configuration information for the target task according to the characteristics of the target node to be classified in multiple resource dimensions and the minimum resource values ​​required to run the target task in multiple resource dimensions; the initial task resource configuration information includes resource configuration values ​​of multiple resource dimensions, and the resource configuration values ​​are greater than or equal to the minimum resource values ​​required for the corresponding resource dimensions; the target task is one of multiple tasks of the target task type; a task container is generated for the target task in the target node to be classified according to the initial task resource configuration information; and the target task is run through the task container.

[0190] Further, on the basis of the above embodiment, the task type feature calculation module 62, when obtaining the correlation array corresponding to the target task according to the running data of the target node to be classified running the target task, is specifically used to extract the first time information and the resource change result values ​​corresponding to the task container in multiple resource dimensions when obtaining the running result of the target task from the running data of the target task running in the task container; the first time information is the duration from the start of the running of the target task in the task container to the acquisition of the running result of the target task.

[0191] The task type feature calculation module 62 is specifically used to obtain the feature change rates of multiple resource dimensions based on the resource change result values ​​corresponding to the task container in multiple resource dimensions and the resource configuration information of the multiple resource dimensions; it is also used to regenerate multiple groups of new task resource configuration information corresponding to the multiple resource dimensions for the target task based on the resource change result values ​​corresponding to the multiple resource dimensions and the initial task resource configuration information; in the new task resource configuration information, the resource configuration value of the corresponding resource dimension is the resource change result value corresponding to the corresponding resource dimension in the running data of the task container running the target task; in the new task resource configuration information, the resource configuration value of the non-corresponding resource dimension is consistent with the resource configuration value of the same resource dimension in the initial task resource configuration information.

[0192] The task type feature calculation module 62 is also specifically used to apply multiple groups of new task resource configuration information to run the target task, and obtain multiple second time information corresponding to multiple resource dimensions respectively; the second time information is a group of new task resource configuration information for applying the corresponding resource dimensions, and the duration from the start of running the target task to obtaining the running result of the target task.

[0193] The task type feature calculation module 62 is further specifically configured to obtain the runtime change rates corresponding to the multiple resource dimensions based on the first time information and the multiple second time information.

[0194] The task type feature calculation module 62 is specifically used to process the feature change rates of multiple resource dimensions and the running time change rates corresponding to the multiple resource dimensions according to the same resource dimensions, obtain multiple correlation coefficients corresponding to the multiple resource dimensions one by one, and form a correlation array corresponding to the target task.

[0195] Further, based on the above embodiment, the resource occupancy index data is configured to be represented by a first vector, and multiple groups of task type feature data are configured to be represented by multiple second vectors; in the first vector and the second vector, the same element position represents the same resource dimension, and the fitness calculation module 63 is specifically used to perform a dot product operation on the first vector and each second vector, and the vector dot product operation result is used as the fitness of the target node to be classified and the task type corresponding to the second vector.

[0196] Further, based on the above embodiment, the node category configuration module 64 is specifically used to determine the task type with the highest value of adaptability with the target node to be classified among multiple task types as the adaptation task type of the target node to be classified; and configure the node category of the target node to be classified as the target category; wherein the target category is the node category corresponding to the adaptation task type.

[0197] See also Figure 7 , which is a schematic diagram of the structure of another embodiment of a cluster node classification device provided by the present application. Further, based on the above embodiment, the cluster node classification device 60 may also include a fitness sequence determination module 65;

[0198] Specifically, the fitness sequence determination module 65 is used to obtain the number of nodes to be allocated for each of the multiple node categories in the cluster, and to obtain the priorities of multiple task types; based on the correspondence between the node category and the task category, the priority of the task type is used as the node allocation priority of the corresponding node category; the fitness of each node in the cluster with the same task type is sorted from large to small or from small to large, and multiple fitness sequences corresponding to the multiple task types are obtained.

[0199] The node category configuration module 64 is specifically used to determine the first L nodes with the highest fitness values ​​from the fitness sequence corresponding to the task type with the highest priority, and configure the categories of the nodes corresponding to the first L fitness values ​​as the node categories corresponding to the task type with the highest priority; L is the number of nodes to be allocated in the node category corresponding to the task type with the highest priority; from the fitness sequence corresponding to the task type with the second highest priority, after excluding the corresponding data of the classified nodes, determine the first M nodes with the highest fitness values, and configure the categories of the nodes corresponding to the first M fitness values ​​as the node categories corresponding to the task type with the second highest priority; M is the number of nodes to be allocated in the node category corresponding to the task type with the second highest priority; and configure the node categories of the remaining unclassified nodes in the cluster as the node categories corresponding to the task type with the lowest priority.

[0200] An embodiment of the present invention provides a processor, which is used to run a program, wherein a cluster node classification method is executed when the program is running. The implementation principle and technical effect are similar and will not be described in detail here.

[0201] An embodiment of the present application also provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the method described above. The implementation principles and technical effects are similar and will not be repeated here.

[0202] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and equipment embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0203] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A cluster node classification method, characterized in that: include: According to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions, obtaining resource occupation index data of the target node to be classified; the resource occupation index data includes multiple resource occupation weights corresponding to the multiple resource dimensions one by one; According to the task running data of the target node to be classified for multiple task types, multiple groups of task type feature data of the target node to be classified are obtained; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions; The resource occupation weight corresponding to each resource dimension in the resource occupation index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types; The node category of the target node to be classified is configured according to the numerical values ​​of the adaptability between the target node to be classified and the multiple task types.

2. The method according to claim 1, characterized in that: The acquiring resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions includes: Obtaining a ratio of the characteristics of the target node to be classified and the cluster in the same resource dimension; The ratios of the features of the multiple resource dimensions obtained are normalized to obtain multiple resource occupancy weights.

3. The method according to claim 1, characterized in that: For a target task type among the multiple task types, according to the task running data of the target node to be classified facing the target task type, a set of task type feature data of the target node to be classified corresponding to the target task type is obtained, including: According to the running data of multiple tasks of the target task type respectively run by the target node to be classified, multiple correlation arrays are obtained; one correlation array corresponds to one task, and one correlation array includes multiple correlation coefficients corresponding to the multiple resource dimensions one by one, and the correlation coefficients represent the influence of the resource dimensions corresponding to the correlation coefficients on the running time when the target node to be classified runs the task; The correlation coefficients corresponding to the same resource dimension in the plurality of correlation arrays are averaged to obtain a group of task type feature data corresponding to the target task type of the target node to be classified.

4. The method according to claim 3, characterized in that The method further comprises: Randomly generate initial task resource configuration information for the target task according to the characteristics of the target node to be classified in the multiple resource dimensions and the minimum resource values ​​required for running the target task in the multiple resource dimensions; the initial task resource configuration information includes resource configuration values ​​of the multiple resource dimensions, and the resource configuration values ​​are greater than or equal to the minimum resource values ​​required for the corresponding resource dimensions; the target task is one of the multiple tasks; Generating a task container for the target task in the target node to be classified according to the initial task resource configuration information; The target task is run through the task container.

5. The method according to claim 4, characterized in that According to the running data of the target task being run by the target node to be classified, a related relationship array corresponding to the target task is obtained, including: Extracting the first time information and the resource change result values ​​corresponding to the task container in the multiple resource dimensions when obtaining the running result of the target task from the running data of the target task run by the task container; the first time information is the duration from when the task container starts running the target task until the running result of the target task is obtained; Obtaining characteristic change rates of the multiple resource dimensions according to the resource change result values ​​corresponding to the task container in the multiple resource dimensions and the resource configuration information of the multiple resource dimensions; Based on the resource change result values ​​respectively corresponding to the multiple resource dimensions and the initial task resource configuration information, a plurality of groups of new task resource configuration information corresponding to the multiple resource dimensions are regenerated for the target task; in the new task resource configuration information, the resource configuration value of the corresponding resource dimension is the resource change result value corresponding to the corresponding resource dimension in the running data of the target task running by the task container; in the new task resource configuration information, the resource configuration value of the non-corresponding resource dimension is consistent with the resource configuration value of the same resource dimension in the initial task resource configuration information; The target task is run by applying the multiple sets of new task resource configuration information respectively, and a plurality of second time information corresponding to the multiple resource dimensions are obtained; the second time information is the duration from the start of running the target task to obtaining the running result of the target task by applying a set of new task resource configuration information corresponding to the resource dimension; Based on the first time information and the plurality of second time information, obtaining the runtime change rates respectively corresponding to the plurality of resource dimensions; The characteristic change rates of the multiple resource dimensions and the running time change rates respectively corresponding to the multiple resource dimensions are processed according to the same resource dimensions to obtain multiple correlation coefficients corresponding to the multiple resource dimensions one by one, and form a correlation matrix corresponding to the target task.

6. The method according to claim 1, characterized in that The resource occupancy index data is configured to be represented by a first vector, and the multiple groups of task type feature data are configured to be represented by multiple second vectors; in the first vector and the second vector, the same element position represents the same resource dimension; The resource occupation weight corresponding to each resource dimension in the resource occupation index data and the resource correlation weight corresponding to each resource dimension in each group of task type feature data are processed according to the same resource dimension to obtain the adaptability of the target node to be classified and the multiple task types, including: A dot product operation is performed on the first vector and each of the second vectors, and a vector dot product operation result obtained is used as the fitness of the target node to be classified and the task type corresponding to the second vector.

7. The method according to claim 1, characterized in that The configuring the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types includes: Determine the task type with the highest value of adaptability to the target node to be classified among the multiple task types as the adaptation task type of the target node to be classified; The node category of the target node to be classified is configured as a target category; wherein the target category is a node category corresponding to the adaptation task type.

8. The method according to claim 1, characterized in that Before configuring the node category of the target node to be classified, the method further includes: Obtaining the number of nodes to be allocated for each of the multiple node categories in the cluster, and obtaining the priorities of the multiple task types; Based on the correspondence between node categories and task categories, the priority of the task type is used as the node allocation priority of the corresponding node category; Sort the fitness of each node in the cluster with the same task type from large to small or from small to large to obtain a plurality of fitness sequences corresponding to the plurality of task types one by one; The configuring the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types includes: From the fitness sequence corresponding to the task type with the highest priority, determine the first L nodes with the highest fitness values, and configure the categories of the nodes corresponding to the first L fitness values ​​as the node categories corresponding to the task type with the highest priority; L is the number of nodes to be allocated in the node category corresponding to the task type with the highest priority; From the fitness sequence corresponding to the task type with the second highest priority, after excluding the corresponding data of the classified nodes, determine the first M nodes with the highest fitness values, and configure the categories of the nodes corresponding to the first M fitness values ​​as the node categories corresponding to the task type with the second highest priority; M is the number of nodes to be allocated in the node category corresponding to the task type with the second highest priority; The node categories of the remaining unclassified nodes in the cluster are configured as the node categories corresponding to the task types with the lowest priority.

9. A cluster node classification device, characterized in that: include: A node resource occupancy index calculation module, used to obtain resource occupancy index data of the target node to be classified according to the characteristics of the cluster in multiple resource dimensions and the characteristics of the target node to be classified in the cluster in the multiple resource dimensions; the resource occupancy index data includes multiple resource occupancy weights corresponding to the multiple resource dimensions one by one; a task type feature calculation module, configured to obtain multiple groups of task type feature data of the target node to be classified according to the task running data of the target node to be classified for multiple task types; the multiple groups of task type feature data correspond one-to-one to the multiple task types, and each group of task type feature data includes multiple resource correlation weights corresponding one-to-one to the multiple resource dimensions; A fitness calculation module is used to calculate and process the resource occupancy weights corresponding to each resource dimension in the resource occupancy index data and the resource correlation weights corresponding to each resource dimension in each group of task type feature data according to the same resource dimension, so as to obtain the fitness of the target node to be classified and the multiple task types; The node category configuration module is used to configure the node category of the target node to be classified according to the numerical values ​​of the adaptability of the target node to be classified and the multiple task types.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

11. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 8 when running.

Images