Big data task distribution method oriented to resource cost balance

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a resource cost balance-oriented big data task distribution method, which comprises the following steps of: receiving a task request submitted by a user, and extracting a computing resource demand feature and a cost constraint condition in the task request; collecting dynamic resource indexes of each node in the distributed computing cluster in real time, and generating a resource availability portrait; based on the computing resource demand features and the resource availability portrait, constructing a multi-dimensional resource matching degree matrix; performing cost-benefit optimization analysis on the multi-dimensional resource matching degree matrix to generate a candidate node set meeting the cost constraint condition; executing a shunting decision in the candidate node set, and generating a task distribution path; distributing the task request to a target node according to the task distribution path; according to the invention, the task distribution efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and specifically relates to a big data task distribution method for resource cost balance. BACKGROUND

[0002] In the prior art, when processing big data task distribution, the dynamic resource changes of nodes and the task resource requirements are not effectively combined for accurate matching, resource adaptability is insufficient, efficient use of resources is difficult to achieve, and the consideration of resource cost lacks systematicness, which cannot take into account comprehensive cost factors such as energy consumption and hardware depreciation, so that the balance between resource use and cost control is difficult.

[0003] In the prior art, in the task distribution process, there is a lack of dynamic perception and shunting regulation mechanism for the real-time load state of nodes, which is prone to node load concentration, affecting the task processing efficiency, and the planning of the task distribution path does not fully consider the migration cost and other factors, and there is a lack of effective tracking means for the task distribution process, so that the overall distribution stability and reliability are poor.

[0004] Therefore, the present application provides a big data task distribution method for resource cost balance. SUMMARY

[0005] In order to make up for the deficiencies of the prior art and solve at least one technical problem proposed in the background art.

[0006] The technical scheme adopted by the present application to solve its technical problems is: the big data task distribution method for resource cost balance provided by the present application comprises S1, receiving a task request submitted by a user, extracting the computing resource requirement features and cost constraint conditions in the task request; S2, collecting dynamic resource indicators of each node in a distributed computing cluster in real time, and generating a resource availability portrait; S3, constructing a multi-dimensional resource matching degree matrix based on the computing resource requirement features and the resource availability portrait; S4, performing cost-benefit optimization analysis on the multi-dimensional resource matching degree matrix to generate a candidate node set that meets the cost constraint conditions; S5, performing shunting decision in the candidate node set to generate a task distribution path; S6, distributing the task request to a target node according to the task distribution path.

[0007] Preferably, the receiving of the task request submitted by the user and the extraction of the computing resource requirement features and the cost constraint conditions in the task request comprise: parsing the structured description field in the task request to generate an initial resource requirement set; vectorizing encoding the initial resource demand set to generate a computing resource demand feature; identifying a cost limit descriptor in the task request, and classifying the cost limit descriptor into a hard cost constraint and a flexible cost constraint according to a preset rule library to generate a constraint type identifier.

[0008] Preferably, the real-time collection of dynamic resource indicators of each node in the distributed computing cluster generates a resource availability portrait, including: periodically collecting an original resource indicator set through a monitoring agent deployed on the node; inputting the original resource indicator set into a sliding time window model to generate a time series regularized indicator; weighting and fusing the time series regularized indicator based on a resource weight configuration table to output the resource availability portrait.

[0009] Preferably, the weighting and fusing of the time series regularized indicator based on the resource weight configuration table to output the resource availability portrait includes: querying a preset weight strategy library according to a current cluster task type distribution feature to generate a dynamic weight vector matched with the task type; splitting the time series regularized indicator into an indicator component set according to CPU, memory, network, and storage dimensions; performing weighted summation operation on the indicator component set using the dynamic weight vector to output the resource availability portrait.

[0010] Preferably, the construction of a multi-dimensional resource matching degree matrix based on the computing resource demand feature and the resource availability portrait includes: parsing the computing resource demand feature into a four-dimensional vector; converting the resource availability portrait into a supply vector corresponding to the four-dimensional vector; determining an initial matching degree set according to the Euclidean distance between the four-dimensional vector and the supply vector; constructing a multi-dimensional resource matching degree matrix according to the initial matching degree set.

[0011] Preferably, the construction of a multi-dimensional resource matching degree matrix according to the initial matching degree set includes: normalizing the initial matching degree set; adjusting the normalized result according to a task priority coefficient to generate the multi-dimensional resource matching degree matrix.

[0012] Preferably, the cost-benefit optimization analysis of the multi-dimensional resource matching degree matrix includes: extracting an energy consumption cost indicator of a node unit computing resource from the multi-dimensional resource matching degree matrix; Superimpose the energy consumption cost index and the node hardware depreciation cost to generate a comprehensive cost coefficient; Set a cost threshold interval according to the cost constraint condition.

[0013] Preferably, the generating of the candidate node set meeting the cost constraint condition comprises: Establish a resource matching degree-cost balance function, wherein the calculation formula of the resource matching degree-cost balance function is as follows: ; In the formula, is the matching degree, is the matching degree weight, is the matching degree value of the multi-dimensional resource matching degree matrix, is the node hardware depreciation cost weight, is the node hardware depreciation cost, is the real-time energy consumption cost, is the real-time energy consumption cost weight; Filter the nodes with the matching degree located in the cost threshold interval to generate the candidate node set.

[0014] Preferably, the executing of the shunting decision in the candidate node set to generate a task distribution path comprises: Monitoring the real-time task queue depth of each node in the candidate node set; When it is detected that the real-time task queue depth exceeds an alarm threshold, activating a load shunt; Calculating a task migration cost prediction value through the load shunt; Constructing a shunting decision tree according to the task migration cost prediction value; Executing path optimization in the shunting decision tree, and selecting a path with the smallest migration cost increment as the task distribution path.

[0015] Preferably, the distributing of the task request to a target node according to the task distribution path comprises: Packaging the task request into a traceable data packet; Transmitting the traceable data packet to a target node through the task distribution path; Returning a distribution confirmation signal after receiving at the target node The present application has the following advantages: 1.The method for big data task distribution oriented to resource cost balance provided by the present application can accurately match tasks and node resources and improve the adaptability of resource utilization by collecting dynamic resource indicators of distributed computing cluster nodes in real time, generating resource availability portraits in combination with a sliding time window model and a dynamic weight vector, and constructing a multi-dimensional resource matching degree matrix based on computing resource demand characteristics and resource availability portraits.

[0016] 2.The method for big data task distribution oriented to resource cost balance provided by the present application can avoid node load concentration and improve task processing efficiency by making real-time task queue depth shunting decisions for candidate nodes, constructing a decision tree by calculating migration cost prediction values and selecting an optimal path. BRIEF DESCRIPTION OF DRAWINGS

[0017] The present application will be further described below in conjunction with the accompanying drawings.

[0018] Figure 1 The flowchart of the method for big data task distribution oriented to resource cost balance provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the present application will be further described below in conjunction with specific embodiments.

[0020] As shown in Figure 1 The method for big data task distribution oriented to resource cost balance provided by the present application embodiment includes S1, receiving a task request submitted by a user, extracting computing resource demand characteristics and cost constraint conditions in the task request; In the present application embodiment, the receiving of a task request submitted by a user and the extraction of computing resource demand characteristics and cost constraint conditions in the task request include: parsing a structured description field in the task request to generate an initial resource demand set; vectorizing and encoding the initial resource demand set to generate computing resource demand characteristics; identifying a cost limit descriptor in the task request, classifying the cost limit descriptor into a hard cost constraint and a flexible cost constraint according to a preset rule library, and generating a constraint type identifier.

[0021] Specifically, when parsing the structured description field in the task request, it is necessary to first determine the content contained in the structured description field. Such fields are usually resource-related information presented in a fixed format in the task request, such as resource type (such as CPU, memory, storage), resource quantity (such as 2 servers, 8 GB of memory), resource specification (such as CPU frequency 3.0 GHz, storage capacity 1 TB), etc.

[0022] Further, the specific content of each structured description field is extracted from the task request one by one, and all the extracted resource-related information is sorted according to the resource category to form an initial resource demand set containing resource type, quantity, specification, etc. The set needs to completely cover all the resource demands explicitly mentioned in the task request. When vectorizing the initial resource demand set, the encoding rule needs to be determined first, and each element in the initial resource demand set needs to be converted into the corresponding numerical value.

[0023] For example, "CPU" in the resource type corresponds to the value 1, "memory" corresponds to the value 2, and "storage" corresponds to the value 3; the resource quantity directly uses its original number; "3.0GHz" in the resource specification corresponds to the value 3.0, and "1TB" corresponds to the value 1024 (converted to GB).

[0024] Further, all the converted numerical values are arranged in a fixed order to form an ordered numerical vector, which is the computing resource demand feature. When identifying the cost limit descriptor in the task request, all cost-related descriptions need to be found in the task request. Such descriptors usually explicitly mention cost limits.

[0025] For example, "The total budget must not exceed 5000 yuan", "The cost is controlled between 3000-4000 yuan", "Try to control the cost around 2000 yuan", etc.

[0026] Further, referring to the definitions of hard cost constraints and flexible cost constraints in the preset rule library, hard cost constraints refer to cost limits that must be strictly followed and cannot be broken, such as descriptors containing "must not exceed" "must be lower than" etc. Flexible cost constraints refer to cost limits that can be adjusted within a certain range and are not strictly unbreakable, such as descriptors containing "try to control" "suggest around XX" etc. The identified cost limit descriptors are classified.

[0027] Further, an identifier is added to each type of cost limit descriptor, such as "hard cost limit content" for hard cost constraints and "flexible cost limit content" for flexible cost constraints. These identifiers together constitute the constraint type identifier.

[0028] S2, collect dynamic resource indicators of each node in the distributed computing cluster in real time, and generate a resource availability portrait; In the embodiment of the application, the real-time collection of dynamic resource indicators of each node in the distributed computing cluster and the generation of the resource availability portrait include: The monitoring agent deployed on the node periodically collects a set of original resource indicators; The set of original resource indicators is input into a sliding time window model to generate time-series regularized indicators; The time-series regularized indicators are weighted and fused based on a resource weight configuration table to output the resource availability portrait.

[0029] The weighted fusion of the time-series regularized indicators based on the resource weight configuration table and the output of the resource availability portrait include: According to the current cluster task type distribution characteristics, a preset weight strategy library is queried to generate a dynamic weight vector matched with the task type; The time-series regularized indicators are split into an indicator component set according to the CPU, memory, network, and storage dimensions; The dynamic weight vector is used to perform weighted summation operation on the indicator component set to output the resource availability portrait.

[0030] Specifically, the monitoring agent deployed on each node will carry out data collection work at a pre-set fixed time interval (such as every 5 minutes), and the collected content includes original resource indicators such as CPU usage, memory occupancy, network throughput, and storage read-write speed. The indicator data collected at different time points are summarized and integrated to form a set of original resource indicators.

[0031] Further, when the set of original resource indicators is input into the sliding time window model, the sliding time window model will take a fixed time length (such as 30 minutes) as a window, and will extract the indicator data in the corresponding time period from the set of original resource indicators in time sequence, and will arrange the data in time sequence to keep the data regular in the time dimension, and will generate time-series regularized indicators.

[0032] Further, when querying the preset weight strategy library according to the current cluster task type distribution characteristics, the current cluster task type distribution characteristics refer to the number proportion and running demand of various tasks (such as computing tasks, storage tasks, network tasks) in the cluster, and the preset weight strategy library stores weight rules corresponding to different task type distribution characteristics, for example, the CPU weight is higher in the cluster dominated by computing tasks, and the storage weight is higher in the cluster dominated by storage tasks, and by matching the current cluster task type distribution characteristics with the rules in the weight strategy library, the weight values corresponding to CPU, memory, network and storage are determined to form a dynamic weight vector.

[0033] Further, when the time sequence normalization index is split into an index component set according to the CPU, memory, network and storage dimensions, all CPU-related indexes (such as CPU usage) are extracted from the time sequence normalization index and classified as CPU components, all memory-related indexes (such as memory occupancy) are extracted and classified as memory components, and similarly, network-related indexes (such as network throughput) and storage-related indexes (such as storage read-write speed) are extracted to form an index component set including CPU components, memory components, network components and storage components. When performing weighted summation operation on the index component set by using the dynamic weight vector, the CPU components in the index component set are multiplied by the corresponding CPU weight in the dynamic weight vector, the memory components are multiplied by the corresponding memory weight, the network components are multiplied by the corresponding network weight, and the storage components are multiplied by the corresponding storage weight, and then the four product results are added together to obtain a value reflecting the overall availability of resources. The value and the corresponding weight of each component together constitute the resource availability image.

[0034] S3, constructing a multi-dimensional resource matching degree matrix based on the computing resource demand characteristics and the resource availability image; In the embodiment of the application, constructing a multi-dimensional resource matching degree matrix based on the computing resource demand characteristics and the resource availability image comprises: analyzing the computing resource demand characteristics into a four-dimensional vector; converting the resource availability image into a supply vector corresponding to the four-dimensional vector; determining an initial matching degree set according to the Euclidean distance between the four-dimensional vector and the supply vector; constructing a multi-dimensional resource matching degree matrix according to the initial matching degree set.

[0035] constructing a multi-dimensional resource matching degree matrix according to the initial matching degree set comprises: normalizing the initial matching degree set; adjusting the normalized result according to the task priority coefficient to generate the multi-dimensional resource matching degree matrix.

[0036] Specifically, when the computing resource requirement feature is parsed into a four-dimensional vector, the four dimensions correspond to CPU, memory, network, and storage. The CPU-related requirement value (e.g., a value corresponding to the required CPU processing capability) is extracted from the computing resource requirement feature as the first dimension value, the memory-related requirement value (e.g., a value corresponding to the required memory capacity) is extracted as the second dimension value, the network-related requirement value (e.g., a value corresponding to the required network bandwidth) is extracted as the third dimension value, and the storage-related requirement value (e.g., a value corresponding to the required storage capacity) is extracted as the fourth dimension value. The four values are arranged in the order of CPU, memory, network, and storage to form the four-dimensional vector. When the resource availability profile is converted into a supply vector corresponding to the four-dimensional vector, the four dimensions are also CPU, memory, network, and storage. The currently available CPU-related capability value (e.g., a value corresponding to the available CPU processing capability) is extracted from the resource availability profile as the first dimension value, the currently available memory-related capability value (e.g., a value corresponding to the available memory capacity) is extracted as the second dimension value, the currently available network-related capability value (e.g., a value corresponding to the available network bandwidth) is extracted as the third dimension value, and the currently available storage-related capability value (e.g., a value corresponding to the available storage capacity) is extracted as the fourth dimension value. The four values are arranged in the above order to generate the supply vector.

[0037] Further, when the initial matching degree set is determined according to the Euclidean distance between the four-dimensional vector and the supply vector, the value difference between the four-dimensional vector and the supply vector in each dimension is calculated first, i.e., the result of subtracting the corresponding dimension value. The smaller the difference, the closer the two vectors, the smaller the corresponding Euclidean distance, and the higher the matching degree. The matching degree values corresponding to each resource node are collected to form the initial matching degree set.

[0038] Further, when the initial matching degree set is normalized, the maximum matching degree value and the minimum matching degree value in the initial matching degree set are found first. Each matching degree value in the set is subtracted by the minimum matching degree value and then divided by the difference between the maximum matching degree value and the minimum matching degree value to obtain the normalized value. The normalized values range from 0 to 1. When the multi-dimensional resource matching degree matrix is generated by adjusting the normalized result according to the task priority coefficient, the task priority coefficient is pre-set and different tasks correspond to different coefficients (e.g., the coefficient of a high-priority task is 1.2 and the coefficient of an ordinary-priority task is 1.0). Each value in the normalized result is multiplied by the priority coefficient of the corresponding task to obtain the adjusted value. The adjusted values are arranged in the form of rows representing tasks and columns representing resource nodes to form the multi-dimensional resource matching degree matrix.

[0039] S4, performing cost-benefit optimization analysis on the multi-dimensional resource matching degree matrix to generate a candidate node set satisfying the cost constraint condition; In the embodiment of the present application, the cost-benefit optimization analysis on the multi-dimensional resource matching degree matrix comprises: extracting an energy consumption cost index of a node unit computing resource from the multi-dimensional resource matching degree matrix; superimposing the energy consumption cost index and a node hardware depreciation cost to generate a comprehensive cost coefficient; setting a cost threshold interval according to the cost constraint condition.

[0040] The generation of the candidate node set satisfying the cost constraint condition comprises: establishing a resource matching degree-cost balance function, wherein the calculation formula of the resource matching degree-cost balance function is as follows: ; In the formula, is a matching degree, is a matching degree weight, is a matching degree value of the multi-dimensional resource matching degree matrix, is a node hardware depreciation cost weight, is a node hardware depreciation cost, is a real-time energy consumption cost, is a real-time energy consumption cost weight; screening the nodes with the matching degree located in the cost threshold interval to generate the candidate node set.

[0041] Specifically, when extracting the energy consumption cost index of the node unit computing resource from the multi-dimensional resource matching degree matrix, the resource matching information and the corresponding energy consumption related data of each node recorded in the multi-dimensional resource matching degree matrix need to be determined first. For example, taking each unit of processing capacity (such as each GHz of computing capacity) as a unit, the energy consumption cost (such as the hourly electricity consumption) of each node under the unit computing resource is extracted from the matrix, and these cost data extracted are taken as the energy consumption cost index of the node unit computing resource. When the energy consumption cost index is superimposed with the node hardware depreciation cost, the node hardware depreciation cost refers to the cost of the hardware devices (such as servers, CPUs, memories, etc.) used by the node according to the usage time. The calculation method is to divide the initial purchase cost of the hardware device by the total expected usage time (such as a server with an initial cost of 10000 yuan and an expected usage of 10000 hours, the depreciation cost per hour is 1 yuan). The unit computing resource energy consumption cost index of each node is added to the unit computing resource hardware depreciation cost corresponding to the node (i.e. the value of the hardware depreciation cost after being allocated according to the unit computing resource), and the result is the comprehensive cost coefficient. When setting the cost threshold interval according to the cost constraint condition, the cost constraint condition includes the previously determined hard cost constraint and the elastic cost constraint. The highest cost limit in the hard cost constraint is taken as the upper limit of the threshold interval, and the lower limit of the reasonable cost range in the elastic cost constraint is taken as the lower limit of the threshold interval. If there is no explicit lower limit in the elastic cost constraint, 0 is taken as the lower limit, thereby determining the cost threshold interval.

[0042] Specifically, in the resource matching degree-cost balance function, is set in advance according to the importance of the task to the resource matching degree, and the setting basis is related to the task type. For example, a computing-intensive task will be assigned a higher value; from the multi-dimensional resource matching degree matrix, which is generated by normalizing the initial matching degree set and adjusting it in combination with the task priority coefficient, that is, the matching degree value of the corresponding node in the matrix; is set according to the importance of the hardware depreciation cost in the comprehensive cost. If the hardware device has a high value, a higher value will be set; is the node hardware depreciation cost, which is calculated by dividing the initial purchase cost of the hardware by the total expected usage time; is set according to the proportion of the energy consumption cost in the comprehensive cost. The of the node corresponding to the higher energy consumption will be appropriately improved; is the energy consumption cost index of the node unit computing resource, which is extracted from the multi-dimensional resource matching degree matrix. The greater the value, the better the node is in meeting the resource matching demand while controlling the cost, and the more suitable the node is as a candidate node for task deployment. When the value increases, the value will increase, which means that the node resource matching degree improves and the balance condition becomes better; when or increases, the value will decrease, which means that the node hardware depreciation cost or real-time energy consumption cost increases, and the balance condition becomes worse; and When the value increases, the influence on increases, or increases, the influence of the corresponding or on increases.

[0043] S5, performing a load distribution decision in the candidate node set to generate a task distribution path; In the embodiment of the application, the performing a load distribution decision in the candidate node set to generate a task distribution path comprises: monitoring real-time task queue depths of nodes in the candidate node set; activating a load distributor when it is detected that the real-time task queue depth exceeds an alarm threshold; calculating a task migration cost prediction value through the load distributor; constructing a load distribution decision tree according to the task migration cost prediction value; performing path optimization in the load distribution decision tree to select a path with the smallest migration cost increment as the task distribution path.

[0044] Specifically, when monitoring the real-time task queue depths of nodes in the candidate node set, the information of the task queues of the nodes is acquired in real time through a built-in task monitoring module of the nodes. The real-time task queue depth refers to the number of tasks currently waiting for processing of each node. The monitoring module records the number of tasks of each node once every fixed time (for example, 1 minute), and the recorded number is taken as the real-time task queue depth data. When it is detected that the real-time task queue depth exceeds an alarm threshold, the load distributor is activated. The alarm threshold is preset according to the processing capacity of the node. For example, if the node can process a maximum of 100 tasks per hour, 80 tasks are set as the alarm threshold. The monitoring module compares the real-time task queue depth with the alarm threshold. Once the real-time task queue depth of a node is greater than the alarm threshold, an activation signal is sent to the load distributor. After receiving the signal, the load distributor enters a working state.

[0045] Further, when calculating the task migration cost prediction value, the load distributor first determines the number of tasks that need to be migrated (i.e. the part of the real-time task queue depth exceeding the alarm threshold), and then collects the migration-related cost information, including the network transmission time of the tasks from the current node to other nodes, the preparation time of the target node to load the tasks after receiving the tasks, the cost of the network resources consumed during the migration process, etc. The estimated cost of migrating the excess tasks to other candidate nodes is calculated by comprehensively calculating these information, and the estimated cost is the task migration cost prediction value. When constructing the load distribution decision tree according to the task migration cost prediction value, the load distribution decision tree takes the overloaded node (the node whose real-time task queue depth exceeds the alarm threshold) as the root node, each branch under the root node corresponds to a candidate node that can receive migrated tasks, and each branch is marked with the task migration cost prediction value of migrating tasks to the corresponding candidate node. According to such a structure, all target nodes that can be migrated and the corresponding cost prediction values are arranged into a tree structure to form the load distribution decision tree.

[0046] Further, when performing path optimization in the load distribution decision tree, starting from the root node, the task migration cost prediction value on each branch is checked one by one, the prediction values of all branches are compared, and the branch with the smallest prediction value is selected. The path from the root node to the target candidate node corresponding to the branch is the path with the smallest migration cost increment, and this path is determined as the task distribution path.

[0047] S6, distributing the task request to the target node according to the task distribution path.

[0048] In the embodiments of the present application, the task request is distributed to the target node according to the task distribution path, including: packaging the task request into a traceable data packet; transmitting the traceable data packet to the target node through the task distribution path; returning a distribution confirmation signal after receiving the traceable data packet at the target node.

[0049] Specifically, when the task request is encapsulated as a traceable data packet, the task content, resource parameters, initiation time and initiation node identifier contained in the task request are collected first, and then the unique traceable identifier, encapsulation time and encapsulation node information are added, all information is integrated according to a preset format to form the traceable data packet. When the traceable data packet is transmitted to the target node through the task distribution path, according to the determined task distribution path, the traceable data packet is transmitted in sequence through the intermediate nodes from the sending node according to the path order, the intermediate node checks the data packet integrity after receiving, and continues to transmit after confirming that there is no error, until it reaches the target node. When the distribution confirmation signal is returned after the target node receives, the target node receives the traceable data packet and analyzes the content, checks the integrity and consistency, generates the distribution confirmation signal, the signal contains the target node identifier, the receiving time, the corresponding traceable identifier and the receiving state, and then is transmitted in reverse according to the original path, and finally returns to the task request initiation node.

[0050] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for distributing big data tasks in view of balancing resource cost, characterized in that: The method comprises the following steps: S1, receiving a task request submitted by a user, extracting a computing resource requirement feature and a cost constraint condition in the task request; S2, collecting dynamic resource indicators of each node in a distributed computing cluster in real time, and generating a resource availability portrait; S3, constructing a multi-dimensional resource matching degree matrix based on the computing resource requirement feature and the resource availability portrait; S4, performing cost-benefit optimization analysis on the multi-dimensional resource matching degree matrix, and generating a candidate node set satisfying the cost constraint condition; S5, performing shunt decision in the candidate node set, and generating a task distribution path; S6, distributing the task request to a target node according to the task distribution path.

2. The method of claim 1, wherein, The method comprises the following steps: parsing the structured description field in the task request to generate an initial resource requirement set; vectorizing the initial resource requirement set to generate a computing resource requirement feature; identifying the cost limit descriptor in the task request, and classifying the cost limit descriptor into hard cost constraint and elastic cost constraint according to a preset rule library to generate a constraint type identifier.

3. The method of claim 2, wherein, The method comprises the following steps: periodically collecting an original resource indicator set through a monitoring agent deployed on the node; inputting the original resource indicator set into a sliding time window model to generate a time series regularized indicator; weighting and fusing the time series regularized indicator based on a resource weight configuration table to output the resource availability portrait.

4. The method of claim 3, wherein, The method comprises the following steps: querying a preset weight strategy library according to the current cluster task type distribution feature to generate a dynamic weight vector matched with the task type; splitting the time series regularized indicator into an indicator component set according to CPU, memory, network and storage dimensions; performing weighted summation operation on the indicator component set by using the dynamic weight vector to output the resource availability portrait.

5. The method of claim 4, wherein, The method comprises the following steps: parsing the computing resource requirement feature into a four-dimensional vector; converting the resource availability portrait into a supply vector corresponding to the four-dimensional vector; determining an initial matching degree set according to the Euclidean distance between the four-dimensional vector and the supply vector; constructing a multi-dimensional resource matching degree matrix according to the initial matching degree set.

6. The method of claim 5, wherein, The method comprises the following steps: normalizing the initial matching degree set; adjusting the normalized result according to a task priority coefficient to generate the multi-dimensional resource matching degree matrix.

7. The method of claim 6, wherein, The method comprises the following steps: extracting an energy consumption cost indicator of a node unit computing resource from the multi-dimensional resource matching degree matrix; superimposing the energy consumption cost indicator and a node hardware depreciation cost to generate a comprehensive cost coefficient; setting a cost threshold interval according to the cost constraint condition.

8. The method of claim 7, wherein, The generating the candidate node set satisfying the cost constraint condition comprises: establishing a resource matching degree-cost balance function, wherein a calculation formula of the resource matching degree-cost balance function is as follows: ； wherein, is a match degree, is a match degree weight, is a match degree value of the multi-dimensional resource match degree matrix, is a node hardware depreciation cost weight, is a node hardware depreciation cost, is a real-time energy consumption cost, is a real-time energy consumption cost weight; screening the nodes with the matching degree located in the cost threshold interval to generate the candidate node set.

9. The method of claim 8, wherein, The executing the shunting decision in the candidate node set to generate the task distribution path comprises: monitoring real-time task queue depths of the nodes in the candidate node set; when detecting that the real-time task queue depth exceeds an alarm threshold, activating a load shunter; calculating a task migration cost prediction value through the load shunter; constructing a shunting decision tree according to the task migration cost prediction value; executing path optimization in the shunting decision tree, and selecting a path with the smallest migration cost increment as the task distribution path.

10. The method of claim 9, wherein, The distributing the task request to the target node according to the task distribution path comprises: encapsulating the task request into a traceable data packet; transmitting the traceable data packet to the target node through the task distribution path; returning a distribution confirmation signal after receiving the traceable data packet at the target node.

Images