Task allocation method

Abstract

The application discloses a task allocation method, and relates to the technical field of data processing. When responding to a task allocation request, target data and corresponding metadata are acquired, the resource consumption amount and the gain amount of computing power acceleration of the target data allocated to each node are respectively predicted based on the metadata through a preset algorithm, the matching values of the nodes are calculated through the consumption amount and the gain amount, the target data is allocated to the node with the highest matching value, and the target data is subjected to allocation processing, so that the task allocation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a task allocation method. BACKGROUND

[0002] With the explosive growth of big data processing, artificial intelligence training, cloud services and other businesses, distributed systems have become the core architecture supporting high concurrency and large-scale data operation. Through multi-node cooperation, resource pooling and task parallel processing are realized, and the system throughput and availability are significantly improved. Load balancing, as a key technology module of distributed systems, is responsible for reasonably distributing data to each node, and the core goal is to avoid single-point overload, optimize resource utilization, and ensure service response stability. In related technologies, represented by round robin and minimum connection number algorithms, the execution process relies on fixed allocation rules, and the node load state is judged by a preset threshold to achieve load balancing.

[0003] In the above process, it is difficult to cover the dynamic changes of load and the distributed environment of hardware heterogeneity, resulting in low task allocation efficiency. SUMMARY

[0004] The present application provides a task allocation method to at least solve the problem of low task allocation efficiency in related technologies.

[0005] The present application provides a task allocation method, comprising:

[0006] In response to the received task allocation request, the target data corresponding to the task allocation request and the metadata corresponding to the target data are determined;

[0007] According to the metadata, the overhead and the gain corresponding to each of the plurality of nodes are determined by a preset algorithm, the overhead is a predicted value of resource overhead required for allocating the target data to the node, and the gain is a predicted value of acceleration of computing power after allocating the target data to the node;

[0008] According to the plurality of overheads and the plurality of gains, the matching values corresponding to the plurality of nodes are determined, and the matching value is used to represent the matching degree of the target data allocated to the node;

[0009] The node with the highest matching value in the plurality of nodes is determined as the target node, and the target data is allocated and processed according to the target node.

[0010] The present application also provides a task allocation device, comprising:

[0011] The first determination module is configured to determine the target data corresponding to the task allocation request and the metadata corresponding to the target data in response to the received task allocation request;

[0012] The second determining module is configured to determine, according to the metadata, the overhead amount and the gain amount corresponding to each of the plurality of nodes by using a preset algorithm, wherein the overhead amount is a predicted value of resource overhead required for distributing the target data to the node, and the gain amount is a predicted value of computing power acceleration after the target data is distributed to the node.

[0013] The third determining module is configured to determine, according to the plurality of overhead amounts and the plurality of gain amounts, a matching value corresponding to each of the plurality of nodes, wherein the matching value is used to represent a matching degree of the target data distributed to the node.

[0014] The distributing module is configured to determine, as a target node, a node with the highest matching value among the plurality of nodes, and perform distribution processing on the target data according to the target node.

[0015] The application further provides an electronic device, which comprises a memory configured to store a computer program and a processor configured to execute the computer program to implement the steps of any of the above task distribution methods.

[0016] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above task distribution methods.

[0017] The application further provides a computer program product, which comprises a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above task distribution methods.

[0018] According to the application, when responding to a task distribution request, the target data and corresponding metadata are obtained, the resource overhead amount and the gain amount of computing power acceleration of distributing the target data to each node are predicted based on the metadata by using a preset algorithm, the matching value of each node is calculated based on the overhead amount and the gain amount, the target data is distributed to the node with the highest matching value, and the target data is processed. In this way, the resource overhead amount and the gain amount of computing power acceleration of processing data by each node are predicted, and the matching value is determined based on the two to select the optimal target node, which can accurately match the data processing demand and the node resource capability, reduces resource waste, optimizes resource utilization efficiency, improves the computing power acceleration effect, adapts to dynamic load changes, effectively improves the rationality and overall performance stability of task distribution, and improves the task distribution efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the application, the drawings required in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1A structural schematic diagram of a task allocation system provided by an embodiment of the present application is provided.

[0021] Figure 2 A flowchart of a task allocation method provided by an embodiment of the present application is provided.

[0022] Figure 3 A flowchart of another task allocation method provided by an embodiment of the present application is provided.

[0023] Figure 4 A flowchart of still another task allocation method provided by an embodiment of the present application is provided.

[0024] Figure 5 A structural schematic diagram of a task allocation device provided by an embodiment of the present application is provided.

[0025] Figure 6 A structural schematic diagram of an electronic device provided by the present application is provided. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0027] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0028] With the explosive growth of big data processing, artificial intelligence training, cloud services and other businesses, distributed systems have become the core architecture to support high concurrency and large-scale data operation. Through multi-node collaboration, resource pooling and task parallel processing are realized, and system throughput and availability are significantly improved. Load balancing, as a key technology module of distributed systems, is responsible for reasonably distributing data to each node, and the core goal is to avoid single-point overload, optimize resource utilization, and ensure service response stability. In related technologies, represented by polling and minimum connection number algorithm, the execution process relies on fixed allocation rules, and the node load state is judged by a preset threshold to achieve load balancing.

[0029] In the above process, it is difficult to cover the dynamic change of load, the distributed environment of heterogeneous hardware, resulting in low efficiency of task allocation.

[0030] To solve the above technical problems, the embodiments of the present application provide a task allocation method, when responding to a task allocation request, obtaining target data and corresponding metadata, predicting the resource consumption amount and the gain amount of computing power acceleration of allocating the target data to each node based on the metadata through a preset algorithm, calculating the matching value of each node through the consumption amount and the gain amount, allocating the target data to the node with the highest matching value, and performing allocation processing on the target data. In this way, the resource consumption amount and the gain amount of computing power acceleration of processing data by each node are predicted, and the matching value is determined based on the two to select the optimal target node, which can accurately match the data processing demand and the node resource capability, reduce resource waste, optimize resource utilization efficiency, improve the effect of computing power acceleration, adapt to dynamic load changes, effectively improve the rationality and overall performance stability of task allocation, and improve the efficiency of task allocation.

[0031] In order to make the person skilled in the art better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0032] Below, in combination with the specific application environment architecture or specific hardware architecture on which the execution of the task allocation method depends, the specific application environment architecture or specific hardware architecture is exemplified.

[0033] Figure 1 The structure diagram of a task allocation system provided by the embodiments of the present application is shown. Please refer to Figure 1 , Figure 1 The task allocation system can include a task receiving module, a metadata analysis module, a task allocation decision module, a decision auxiliary matrix module, a task allocation execution module, a computing node cluster, a node monitoring module, and a hardware event monitoring module.

[0034] The task receiving module is configured to receive task data to be allocated as an input source of the task allocation process.

[0035] The metadata analysis module is connected to the task receiving module and is configured to analyze the metadata corresponding to the task data and extract the attribute information of the task data.

[0036] The task allocation decision module is connected to the metadata analysis module, the decision auxiliary matrix module, the node monitoring module, and the hardware event monitoring module, receives the metadata output by the metadata analysis module, the real-time running state of the computing node cluster collected by the node monitoring module, and the hardware events captured by the hardware event monitoring module, and generates a task allocation strategy in combination with the multi-dimensional decision rules built in the decision auxiliary matrix module.

[0037] The decision assistance matrix module is connected with the task allocation decision module, and is configured to provide the basis for evaluating the matching degree of the task and the node for the task allocation decision module.

[0038] The task allocation execution module is connected with the task allocation decision module and the computing node cluster, and is configured to distribute the task to the target node in the computing node cluster according to the task allocation strategy.

[0039] The computing node cluster is composed of a plurality of computing nodes with data processing capability, and is configured to execute the distributed task.

[0040] The node monitoring module is connected with the computing node cluster and the task allocation decision module, and is configured to continuously monitor the resource usage, task execution and other state information of each node in the computing node cluster, and feed back the collected monitoring data to the task allocation decision module in real time.

[0041] The hardware event monitoring module is connected with the task allocation decision module, and is configured to monitor the events at the hardware level such as storage, network and power supply, and report the captured hardware event information to the task allocation decision module in real time.

[0042] Figure 2 A flowchart of a task allocation method provided by an embodiment of the application is shown. The execution subject of the embodiment of the application can be a processor. The processor can be realized by software, or realized by the combination of software and hardware. Please refer to Figure 2 The method comprises the following steps:

[0043] S201, in response to the received task allocation request, determining the target data corresponding to the task allocation request and the metadata corresponding to the target data.

[0044] The task allocation request can be a request for allocating a task to a suitable node for processing initiated by an application or a system.

[0045] The target data can refer to the data corresponding to the task allocation request that needs to be processed.

[0046] The metadata can be structured information for describing the properties and processing requirements of the target data.

[0047] The task allocation request can be received, analyzed and processed to determine the target data corresponding to the task allocation request, and the target data can be analyzed and processed to obtain the metadata corresponding to the target data.

[0048] S202, according to the metadata, determining the overhead and gain corresponding to each node by a preset algorithm.

[0049] The overhead is a predicted value of the resource overhead required for allocating the target data to the node.

[0050] The gain amount is a predicted value of the computing power acceleration after the target data is allocated to the node.

[0051] The preset algorithm can be a preset algorithm or a pre-trained prediction model.

[0052] Real-time state data of the plurality of nodes can be obtained by the node monitoring module, and the plurality of nodes respectively corresponding overhead amounts and gain amounts can be determined by the preset algorithm according to the metadata and the real-time state data.

[0053] The overhead prediction model and the gain prediction model can be obtained, the metadata and node information of the plurality of nodes are input into the overhead prediction model to obtain the plurality of overhead amounts, and the metadata and node information of the plurality of nodes are input into the gain prediction model to obtain the plurality of gain amounts.

[0054] It should be noted that the plurality of nodes respectively corresponding overhead amounts and gain amounts can be determined according to any feasible implementation manner, and the embodiments of the present application do not limit this.

[0055] S203, determining a plurality of nodes respectively corresponding matching values according to the plurality of overhead amounts and the plurality of gain amounts.

[0056] The matching value is used to represent the matching degree of the target data allocated to the node.

[0057] Optionally, for any one node, the gain amount of the node and the overhead amount of the node are determined, the ratio between the gain amount of the node and the overhead amount of the node is determined, and the plurality of nodes respectively corresponding ratios are determined as the plurality of nodes respectively corresponding matching values.

[0058] Optionally, for any one node, the gain amount of the node and the overhead amount of the node are determined, the difference between the gain amount of the node and the overhead amount of the node is determined, and the plurality of nodes respectively corresponding differences are determined as the plurality of nodes respectively corresponding matching values.

[0059] It should be noted that the plurality of nodes respectively corresponding matching values can be determined according to any feasible implementation manner, and the embodiments of the present application do not limit this.

[0060] S204, determining a node with the highest matching value in the plurality of nodes as a target node, and performing allocation processing on the target data according to the target node.

[0061] The target node can be a node with the highest comprehensive adaptability for processing the target data.

[0062] The allocation processing can be a process of distributing the target data from a source node to the target node.

[0063] The node with the highest matching value among the plurality of nodes can be determined as the target node, and real-time state checking is performed on the target node. After the checking passes, the target data is preprocessed according to the hardware characteristics of the target node, and the preprocessed target data is sent to the target node.

[0064] The task allocation method provided in this embodiment determines target data corresponding to a received task allocation request and metadata corresponding to the target data in response to the received task allocation request. According to the metadata, a plurality of nodes respectively correspond to an overhead amount and a gain amount determined by a preset algorithm. The overhead amount is a predicted value of resource overhead required for allocating the target data to the node, and the gain amount is a predicted value of acceleration of computing power after the target data is allocated to the node. According to the plurality of overhead amounts and the plurality of gain amounts, a plurality of nodes respectively correspond to a matching value, which is used to represent a matching degree of the target data allocated to the node. The node with the highest matching value among the plurality of nodes is determined as the target node, and the target data is allocated to the target node. In this way, the resource overhead amount and the acceleration gain amount of each node for processing data are predicted, and the matching value is determined based on the two to select the optimal target node, which can accurately match the data processing demand and the node resource capability, reduces resource waste and optimizes resource utilization efficiency, improves the acceleration effect of computing power, adapts to dynamic load changes, effectively improves the rationality of task allocation and overall performance stability, and improves the efficiency of task allocation.

[0065] Next, the task allocation method provided in this embodiment will be described in detail in combination with Figure 3 The specific process of determining the plurality of nodes respectively corresponding to the overhead amount and the gain amount according to the metadata by the preset algorithm will be explained.

[0066] Figure 3 Another flowchart of the task allocation method provided in this embodiment is shown in FIG. 6. Based on the above embodiment, the method can include the following steps. Figure 3

[0067] S301, real-time monitoring processing is performed on a cluster where a plurality of nodes are located to determine cluster monitoring data.

[0068] The real-time monitoring processing can be real-time monitoring of the state of the cluster and the state of the plurality of nodes.

[0069] The cluster monitoring data can be a structured data set reflecting the running state of the cluster and the nodes through real-time collection and processing.

[0070] The cluster where the plurality of nodes are located can be monitored in real time by a node monitoring module and a hardware event monitoring module to determine the cluster monitoring data.

[0071] S302, a plurality of current overhead weights and current gain weights are determined according to the cluster monitoring data. ​

[0072] The plurality of current overhead weights include a current storage weight, a current computation weight, and a current network weight.

[0073] The plurality of current overhead weights and the current gain weight can be dynamically adjusted.

[0074] For example, when the system finds that the current input / output bottleneck is serious, the value of the current storage weight can be increased, so that the proportion of the input / output corresponding overhead is increased, so that the scheduling will be more inclined to avoid operations with large input / output overhead.

[0075] The cluster monitoring data can include real-time statistical cluster state parameters, and can also include cluster state parameters driven by hardware events.

[0076] The cluster monitoring data can include processor usage, memory used capacity and usage, graphics processor load, network transceiver bandwidth, process survival state, hardware temperature, etc. of the plurality of nodes, and can also include cluster delay standard deviation, benchmark energy consumption, and task completion rate, etc.

[0077] The plurality of current overhead weights and the current gain weight can be dynamically determined according to the cluster state parameters in the cluster monitoring data and at least one hardware event.

[0078] Next, taking any one of the plurality of nodes as an example, the overhead corresponding to the node is determined according to the metadata and the plurality of current overhead weights, and the metadata includes the data amount.

[0079] S303, determining the storage overhead corresponding to the node according to the metadata and the current storage weight.

[0080] The storage overhead can be used to represent the overhead required for inputting / outputting target data from the storage device.

[0081] The storage duration can be determined according to the metadata and the node, and the storage overhead corresponding to the node is determined according to the storage duration and the current storage weight.

[0082] Optionally, the storage overhead corresponding to the node can be determined according to the metadata and the current storage weight by: determining the disk seek duration and the bandwidth corresponding to the node; determining the processing duration according to the data amount and the bandwidth; determining the sum of the processing duration and the disk seek duration as the storage duration; and determining the storage overhead corresponding to the node according to the storage duration and the current storage weight.

[0083] The disk seek duration can represent the duration required for positioning data to start reading, and the disk seek duration can be determined according to the hard disk type, for example, the disk seek duration of a mechanical hard disk is about 3-15 ms, and the disk seek duration of a solid state disk is about 0.1 ms.

[0084] The bandwidth can be a bandwidth of a storage device of the node.

[0085] Optionally, the storage overhead corresponding to the node can be determined according to the following formula:

[0086] T_io = a(t) x [t_seek + D_size / B_disk]

[0087] Wherein, T_io can represent the storage overhead, a(t) can represent the current storage weight, t_seek can represent a disk seek duration, D_size can represent a data amount, B_disk can represent a bandwidth, and D_size / B_disk can represent a duration required for reading target data.

[0088] S304, determining, according to the metadata and the current computing weight, a computing overhead corresponding to the node.

[0089] The computing overhead can be used to represent an overhead required for processing the target data on the node.

[0090] The duration and power required for the node to process the target data can be determined according to the metadata, and the computing overhead corresponding to the node can be determined according to the duration and power and the current computing weight.

[0091] Optionally, the computing overhead corresponding to the node can be determined according to the metadata and the current computing weight in the following manner: an energy consumption amount of a processor in the node is determined according to the metadata; and the computing overhead corresponding to the node is determined according to the energy consumption amount and the current computing weight.

[0092] Wherein, the energy consumption amount is an energy consumed by the processor for processing the target data.

[0093] Optionally, the computing overhead corresponding to the node can be determined according to the following formula:

[0094] E_comp = b(t) x [P_cpu x t_cpu + P_gpu x t_gpu]

[0095] Wherein, E_comp can represent the computing overhead, b(t) can represent the current computing weight, P_cpu can represent a power of a central processor, t_cpu can represent a computing duration of the central processor, P_gpu can represent a power of a graphics processor, and t_gpu can represent a computing duration of the graphics processor.

[0096] S305, determining, according to the metadata and the current network weight, a network overhead corresponding to the node.

[0097] The network overhead can be used to represent an overhead required for transmitting the target data to the node.

[0098] The transmission duration can be determined according to the metadata and the network link corresponding to the node, and the network cost corresponding to the node can be determined according to the transmission duration and the current network weight.

[0099] Optionally, the network cost corresponding to the node can be determined according to the metadata and the current network weight in the following manner: the network link bandwidth and the packet loss rate from the source node where the target data is located to the node are obtained; the transmission duration is determined according to the data size and the network link bandwidth; the packet loss penalty factor is determined according to the packet loss rate; and the network cost corresponding to the node is determined according to the packet loss penalty factor, the transmission duration and the current network weight.

[0100] When the packet loss rate is 0, the packet loss penalty factor can be 1, and when the packet loss rate is greater than 0, the packet loss penalty factor can be greater than 1.

[0101] Optionally, the network cost corresponding to the node can be determined by the following formula:

[0102] N_trans = γ(t) × [D_size / B_link × η_loss]

[0103] Wherein, N_trans can represent the network cost, γ(t) can represent the current network weight, D_size can represent the data size, B_link can represent the network link bandwidth, and η_loss can represent the packet loss penalty factor.

[0104] S306, determine the path hop count between the source node where the target data is located and the node.

[0105] The path hop count can be the number of network devices passed through from the source node to the target node.

[0106] The target link can be determined from the source node where the target data is located to the node, the number of network devices on the target link is determined, and the number is determined as the path hop count.

[0107] S307, determine the migration cost corresponding to the node according to the metadata and the path hop count.

[0108] The network link bandwidth and the network architecture type from the source node where the target data is located to the node can be obtained, the migration penalty coefficient is determined according to the network architecture type, and the migration cost corresponding to the node is determined according to the migration penalty coefficient, the data size, the network link bandwidth and the path hop count.

[0109] For example, the migration penalty coefficient can be determined by Table 1:

[0110] Table 1

[0111] Network architecture type Migration penalty coefficient In the same machine room 0.1 Cross availability zone 0.3 Cross region 0.5

[0112] For example, when detecting that there is a border gateway protocol route change in the source node to the node, the migration penalty coefficient is automatically increased by 0.1.

[0113] Optionally, the migration overhead corresponding to the node can be determined by the following formula:

[0114] K = λ × [(D_size / B_link) × (1 + 0.1×hop_count)]

[0115] Wherein, K can represent the migration overhead, λ can represent the migration penalty coefficient, D_size can represent the data size, B_link can represent the network link bandwidth, and hop_count can represent the path hop count.

[0116] S308, the sum of the storage overhead, the calculation overhead, the network overhead and the migration overhead is determined as the overhead corresponding to the node.

[0117] Optionally, the migration overhead corresponding to the node can be determined by the following formula:

[0118] C = T_io + E_comp + N_trans + K

[0119] Wherein, T_io can represent the storage overhead, E_comp can represent the calculation overhead, N_trans can represent the network overhead, and K can represent the migration overhead.

[0120] Next, taking any one of the plurality of nodes as an example, the gain corresponding to the node is determined according to the metadata and the current gain weight, wherein the metadata includes data type, data size, a plurality of fields and priority.

[0121] S309, according to the data type and the data size, the first coefficient is determined.

[0122] The first coefficient can avoid large size data from playing an absolute leading role in the calculation.

[0123] The base of the logarithmic function can be determined according to the data type, and the first coefficient can be determined according to the base, the logarithmic function and the data size.

[0124] For example, the base of the logarithmic function can be determined according to the data type by Table 2:

[0125] Table 2

[0126] Data type Base number Reason General data 10 Compatible with regular file size Scientific data e Match exponential growth characteristics Binary stream 2 Alignment bit calculation

[0127] Assuming that the data type is binary stream, the first coefficient can be log2(D_size), wherein D_size can represent the data size.

[0128] S310, determine the information entropy corresponding to the metadata according to the data type and the plurality of fields.

[0129] The information entropy can be used to represent the information richness or value density of the target data.

[0130] The field probability values corresponding to the plurality of fields can be calculated, the field coefficient can be determined according to the data type, and the information entropy corresponding to the metadata can be determined according to the field coefficient and the field probability values corresponding to the plurality of fields.

[0131] Different data types can correspond to different field coefficients.

[0132] Alternatively, the information entropy corresponding to the metadata can be determined according to the data type and the plurality of fields in the following manner: the field probability values corresponding to the plurality of fields are calculated; the field coefficients corresponding to the plurality of fields are determined according to the data type and the plurality of field probability values; for any one field, the field information entropy is determined according to the field probability value corresponding to the field and the corresponding field coefficient; and the information entropy corresponding to the metadata is determined by adding the field information entropies corresponding to the plurality of fields.

[0133] For example, it is assumed that the metadata includes field 1, field 2, and field 3. The occurrence frequencies of field 1, field 2, and field 3 are 1, 1, and 1 respectively, and the field probability values corresponding to field 1, field 2, and field 3 are determined to be 1 / 3, 1 / 3, and 1 / 3 respectively according to the occurrence frequencies of each field.

[0134] Alternatively, the field coefficient of a field can be determined in the following manner: the base of a logarithmic function is determined according to the data type, and the field coefficient is determined according to the base of the logarithmic function and the field probability value.

[0135] For example, the data type is a binary stream, the base of the logarithmic function is 2, and the field coefficient is log2(p), where p is the field probability value.

[0136] S311, determine the product of the first coefficient and the information entropy as the weighted information entropy.

[0137] The weighted information entropy can represent the weighted information richness or value density corresponding to the target data.

[0138] S312, determine the second coefficient according to the priority.

[0139] The priority can be determined according to the task allocation request.

[0140] The second coefficient can be determined according to the priority and the mapping relationship.

[0141] For example, the second coefficient corresponding to the emergency task is 1.5, the second coefficient corresponding to the general task is 1.2, and the second coefficient corresponding to the ordinary task is 1.0.

[0142] S313, determining the product of the weighted information entropy, the second coefficient and the current gain weight as the gain amount corresponding to the node.

[0143] The gain amount corresponding to the node can be determined by the following formula:

[0144] B = δ × (log(D_size) × Entropy(D_metadata)) × P_priority

[0145] Wherein, B can represent the gain amount, δ can represent the current gain weight, log(D_size) can represent the first coefficient, Entropy(D_metadata) can represent the information entropy, and P_priority can represent the second coefficient.

[0146] The implementation content of each step in the embodiments of the application can refer to the description of the corresponding steps or operations of the above method embodiments, and repeated content will not be described again.

[0147] The task allocation method provided in this embodiment determines the cluster monitoring data by performing real-time monitoring processing on the cluster where the plurality of nodes are located; determines a plurality of current overhead weights and a current gain weight according to the cluster monitoring data; determines a storage overhead amount corresponding to the node according to the metadata and the current storage weight; determines a computing overhead amount corresponding to the node according to the metadata and the current computing weight; determines a network overhead amount corresponding to the node according to the metadata and the current network weight; determines the path hop count between the source node where the target data is located and the node; determines a migration overhead amount corresponding to the node according to the metadata and the path hop count; determines the sum of the storage overhead amount, the computing overhead amount, the network overhead amount and the migration overhead amount as the overhead amount corresponding to the node; determines the first coefficient according to the data type and the data amount; determines the information entropy corresponding to the metadata according to the data type and the plurality of fields; determines the product of the first coefficient and the information entropy as the weighted information entropy; determines the second coefficient according to the priority; and determines the product of the weighted information entropy, the second coefficient and the current gain weight as the gain amount corresponding to the node. In this way, the dynamic weights are determined by real-time monitoring of the cluster, the storage, computing, network and migration overhead amounts of the node are accurately quantified in multiple dimensions, the weighted information entropy and the gain amount are calculated in combination with the data characteristics and the task priority, the dynamic adaptation and accurate evaluation of the overhead amount and the gain amount are realized, the matching accuracy of the target data and the node is improved, and the cluster resource utilization rate and the task processing efficiency are optimized.

[0148] Next, the application will be described in detail in combination with Figure 4The specific process of determining a plurality of current overhead weights and current gain weights according to the cluster monitoring data is explained, the plurality of current overhead weights including a current storage weight, a current computing weight and a current network weight.

[0149] Figure 4 A flowchart of another task allocation method provided by an embodiment of the present application is shown. Based on the above embodiment, the method can include Figure 4

[0150] S401, determining a cluster delay standard deviation, a benchmark energy consumption and a task completion rate according to cluster monitoring data.

[0151] The cluster delay standard deviation can be a quantitative value of the dispersion degree of all task processing delay data in the cluster.

[0152] The cluster delay standard deviation can be determined by the input / output delay data collected by the monitoring agent on each node in real time.

[0153] The benchmark energy consumption can be the energy consumption value of the cluster in a benchmark state of low load and no task execution.

[0154] The task completion rate can be the proportion of the number of successfully completed tasks to the total number of tasks within a certain time.

[0155] The cluster monitoring data can be preprocessed, and the cluster delay standard deviation, the benchmark energy consumption and the task completion rate can be calculated according to the preprocessed cluster monitoring data.

[0156] S402, determining a storage coefficient, a cluster average energy consumption and a network topology change number according to the cluster monitoring data.

[0157] The storage coefficient can be a dynamic coefficient triggered by a hardware event.

[0158] The cluster average energy consumption can be used to measure the energy consumption level of the cluster.

[0159] The number of network topology changes can be used to measure the stability of the network structure.

[0160] The cluster monitoring data can be parsed and processed, and the device information, energy consumption state data and topology change counter of a plurality of nodes obtained by a preset interface can be determined, a plurality of storage types can be determined according to the device information of the plurality of nodes, the storage coefficient can be calculated according to the plurality of storage types, the cluster average energy consumption can be calculated according to the energy consumption state data, and the number of network topology changes can be calculated according to the topology change counter.

[0161] ​Optionally, the storage coefficient, the cluster average energy consumption and the network topology change number are determined according to the cluster monitoring data: whether there is at least one new event is determined according to the cluster monitoring data; if yes, the storage coefficient, the cluster average energy consumption and the network topology change number are updated according to the at least one new event; if no, the storage coefficient, the cluster average energy consumption and the network topology change number are obtained from the storage space.

[0162] The new event can be a hardware event newly added in a preset time period.

[0163] Optionally, the storage coefficient, the cluster average energy consumption and the network topology change number can be updated according to the at least one new event in the following manner: if the at least one new event includes at least one storage event, the storage coefficient is updated according to the at least one storage event; if the at least one new event includes at least one energy consumption event, the cluster average energy consumption is updated according to the at least one energy consumption event; if the at least one new event includes at least one network event, the network topology change number is updated according to the at least one network event.

[0164] The storage coefficient can be updated according to the at least one storage event in the following manner: the storage medium type is determined according to the at least one storage event, and the storage coefficient is updated according to the storage medium type.

[0165] For example, when it is detected that the storage medium is changed from a solid state disk to a mechanical disk, the storage coefficient is increased by 0.15.

[0166] The cluster average energy consumption can be updated according to the at least one energy consumption event in the following manner: the cluster average energy consumption is recalculated according to the at least one energy consumption event.

[0167] The network topology change number can be updated according to the at least one network event in the following manner: the network topology change counter is updated according to the at least one network event, and the network topology change number is updated according to the updated network topology change counter.

[0168] S403, determining a current storage weight according to the cluster delay standard deviation and the storage coefficient.

[0169] The sum of the cluster delay standard deviation and the storage coefficient can be determined as the current storage weight.

[0170] Optionally, the current storage weight can be determined according to the cluster delay standard deviation and the storage coefficient in the following formula:

[0171] a(t) = 0.4 + 0.15 × σ_io(t) × e^(-0.1 × t_stable) + Δ_event

[0172] Wherein, a(t) can represent the current storage weight, σ_io(t) can represent the cluster delay standard deviation, the larger the cluster delay standard deviation, the greater the difference between the input and output performance of the nodes, the current storage weight should be improved to highlight the influence of the input and output performance difference on load balancing, Δ_event can represent the storage coefficient, e^(-0.1 × t_stable) can represent the exponential decay factor, t_stable can represent the length of time that the system is stably running, the more stable the system runs, the smaller the exponential decay factor.

[0173] S404, determining the current computing weight according to the benchmark energy consumption and the cluster average energy consumption.

[0174] The difference between the benchmark energy consumption and the cluster average energy consumption can be determined, and the current computing weight is determined according to the difference.

[0175] Optionally, the current computing weight can be determined according to the benchmark energy consumption and the cluster average energy consumption by the following formula:

[0176] β(t)=0.3 × (1+a × |E_avg - E_standard| / 100)

[0177] Wherein, β(t) can represent the current computing weight, a can represent the scaling factor, the scaling factor can be used to adjust the influence degree of deviation, E_avg can represent the cluster average energy consumption, and E_standard can represent the benchmark energy consumption.

[0178] S405, determining the current network weight according to the number of network topology changes.

[0179] Optionally, the current network weight can be determined according to the number of network topology changes and the first mapping relationship.

[0180] Optionally, the current network weight can be determined according to the number of network topology changes by the following formula:

[0181] γ(t)=0.3 × 1.2^topo_change_count

[0182] Wherein, γ(t) can represent the current network weight, and topo_change_count can represent the number of network topology changes.

[0183] S406, determining the current gain weight according to the task completion rate.

[0184] The current gain weight can be determined according to the task completion rate and the second mapping relationship.

[0185] The second mapping relationship can include a plurality of completion rate ranges, and the gain weight corresponding to the completion rate range.

[0186] The implementation of each step in the embodiments of the present application can refer to the description of the corresponding steps or operations of the above method embodiments, and repeated content will not be described again.

[0187] The task allocation method provided in this embodiment determines the cluster delay standard deviation, the benchmark energy consumption and the task completion rate according to the cluster monitoring data; determines the storage coefficient, the cluster average energy consumption and the number of network topology changes according to the cluster monitoring data; determines the current storage weight according to the cluster delay standard deviation and the storage coefficient; determines the current calculation weight according to the benchmark energy consumption and the cluster average energy consumption; determines the current network weight according to the number of network topology changes; and determines the current gain weight according to the task completion rate; wherein the plurality of current overhead weights include the current storage weight, the current calculation weight and the current network weight. In this way, the overhead weights and gain weights of storage, calculation and network are dynamically determined in multiple dimensions through cluster monitoring data, so that the calculation of overhead and gain is more in line with the real-time state of the cluster, thereby improving the matching accuracy of tasks and nodes and the utilization efficiency of cluster resources.

[0188] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better implementation.

[0189] Figure 5 A structural schematic diagram of a task allocation device provided in the embodiments of the present application is provided. Please refer to Figure 5 The task allocation device 500 includes a first determination module 501, a second determination module 502, a third determination module 503 and an allocation module 504.

[0190] The first determination module 501 is configured to determine target data corresponding to a received task allocation request and metadata corresponding to the target data in response to the received task allocation request.

[0191] The second determination module 502 is configured to determine, according to the metadata, a plurality of nodes respectively corresponding to an overhead and a gain through a preset algorithm, the overhead being a predicted value of resource overhead required for allocating the target data to the node, and the gain being a predicted value of computing power acceleration after the target data is allocated to the node.

[0192] The third determination module 503 is configured to determine, according to the plurality of overheads and the plurality of gains, a plurality of nodes respectively corresponding to a matching value, the matching value being used to represent a matching degree of the target data allocated to the node.

[0193] The allocation module 504 is configured to determine a node with the highest matching value in the plurality of nodes as a target node, and perform allocation processing on the target data according to the target node.

[0194] In a possible implementation, the second determining module 502 is specifically configured to:

[0195] perform real-time monitoring on a cluster where the plurality of nodes are located to determine cluster monitoring data;

[0196] determine a plurality of current overhead weights and current gain weights according to the cluster monitoring data;

[0197] determine a plurality of overhead amounts respectively corresponding to the plurality of nodes according to the metadata and the plurality of current overhead weights;

[0198] determine a plurality of gain amounts respectively corresponding to the plurality of nodes according to the metadata and the current gain weights.

[0199] In a possible implementation, the plurality of current overhead weights include a current storage weight, a current computing weight, and a current network weight, and for any one node, the second determining module 502 is specifically configured to:

[0200] determine a storage overhead amount corresponding to the node according to the metadata and the current storage weight;

[0201] determine a computing overhead amount corresponding to the node according to the metadata and the current computing weight;

[0202] determine a network overhead amount corresponding to the node according to the metadata and the current network weight;

[0203] determine a path hop count between a source node where the target data is located and the node;

[0204] determine a migration overhead amount corresponding to the node according to the metadata and the path hop count;

[0205] determine the overhead amount corresponding to the node as a sum of the storage overhead amount, the computing overhead amount, the network overhead amount, and the migration overhead amount.

[0206] In a possible implementation, the metadata includes a data amount, and the second determining module 502 is specifically configured to:

[0207] determine a disk seek time length and a bandwidth corresponding to the node;

[0208] determine a processing time length according to the data amount and the bandwidth;

[0209] determine the storage time length as a sum of the processing time length and the disk seek time length;

[0210] determine the storage overhead amount corresponding to the node according to the storage time length and the current storage weight.

[0211] In a possible implementation, the second determining module 502 is specifically configured to:

[0212] According to the metadata, a processor energy consumption amount of the node is determined, the processor energy consumption amount being energy consumed by the processor in processing the target data;

[0213] According to the processor energy consumption amount and a current computing weight, a computing overhead amount corresponding to the node is determined.

[0214] In a possible implementation, the metadata comprises a data amount, and the second determining module 502 is specifically configured to:

[0215] Obtain a network link bandwidth and a packet loss rate from a source node where the target data is located to the node;

[0216] According to the data amount and the network link bandwidth, a transmission time length is determined;

[0217] According to the packet loss rate, a packet loss penalty factor is determined;

[0218] According to the packet loss penalty factor, the transmission time length and a current network weight, a network overhead amount corresponding to the node is determined.

[0219] In a possible implementation, the metadata comprises a data type, a data amount, a plurality of fields and a priority, and for any one node, the second determining module 502 is specifically configured to:

[0220] According to the data type and the data amount, a first coefficient is determined;

[0221] According to the data type and the plurality of fields, an information entropy corresponding to the metadata is determined;

[0222] A product of the first coefficient and the information entropy is determined as a weighted information entropy;

[0223] According to the priority, a second coefficient is determined;

[0224] A product of the weighted information entropy, the second coefficient and a current gain weight is determined as a gain amount corresponding to the node.

[0225] In a possible implementation, the second determining module 502 is specifically configured to:

[0226] Calculate a field probability value corresponding to each field of the plurality of fields;

[0227] According to the data type and the plurality of field probability values, a field coefficient corresponding to each field of the plurality of fields is determined;

[0228] For any one field, according to a field probability value corresponding to the field and a field coefficient corresponding to the field, a field information entropy of the field is determined;

[0229] The field information entropies corresponding to the plurality of fields are added to determine an information entropy corresponding to the metadata.

[0230] In a possible implementation, the second determining module 502 is specifically configured to:

[0231] determine a cluster delay standard deviation, a benchmark energy consumption and a task completion rate according to the cluster monitoring data;

[0232] determine a storage coefficient, a cluster average energy consumption and a network topology change number according to the cluster monitoring data;

[0233] determine a current storage weight according to the cluster delay standard deviation and the storage coefficient;

[0234] determine a current computing weight according to the benchmark energy consumption and the cluster average energy consumption;

[0235] determine a current network weight according to the network topology change number;

[0236] determine a current gain weight according to the task completion rate;

[0237] wherein the plurality of current overhead weights comprise the current storage weight, the current computing weight and the current network weight.

[0238] In a possible implementation, the second determining module 502 is specifically configured to:

[0239] determine whether there is at least one new event according to the cluster monitoring data;

[0240] if yes, update the storage coefficient, the cluster average energy consumption and the network topology change number according to the at least one new event;

[0241] if no, obtain the storage coefficient, the cluster average energy consumption and the network topology change number in the storage space.

[0242] In a possible implementation, the updating of the storage coefficient, the cluster average energy consumption and the network topology change number according to the at least one new event comprises:

[0243] if the at least one new event comprises at least one storage event, update the storage coefficient according to the at least one storage event;

[0244] if the at least one new event comprises at least one energy consumption event, update the cluster average energy consumption according to the at least one energy consumption event;

[0245] if the at least one new event comprises at least one network event, update the network topology change number according to the at least one network event.

[0246] In a possible implementation, the third determining module 503 is specifically configured to:

[0247] determine a gain amount of the node and an overhead amount of the node for any one node;

[0248] determine a ratio between the gain amount of the node and the overhead amount of the node;

[0249] determine the ratio corresponding to each node as a matching value corresponding to each node.

[0250] The features of the embodiments of the task allocation device can be referred to the related descriptions of the embodiments of the task allocation method, which will not be repeated here.

[0251] Figure 6 The structural schematic diagram of the electronic device is provided in the present application. As shown in the figure, Figure 6 The electronic device 60 provided in the embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 60 further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected through a bus.

[0252] In the specific implementation process, the at least one processor 601 executes the computer execution instructions stored in the memory 602, so that the at least one processor 601 executes the task allocation method embodiments described above.

[0253] The specific implementation process of the processor 601 can be referred to the method embodiments described above, which has similar implementation principles and technical effects, and will not be repeated here.

[0254] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC) and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0255] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), for example, at least one disk memory.

[0256] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0257] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above task allocation method embodiments when executed.

[0258] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0259] Embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in any of the above task allocation method embodiments.

[0260] Embodiments of the present application also provide another computer program product, which includes a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above task allocation method embodiments.

[0261] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0262] The above describes in detail the task allocation method provided by the present application. The principles and implementation manners of the present application are described by using specific examples, and the above description of the examples is only used to help understand the method of the present application and the core idea thereof. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A task allocation method, characterized in that, include: In response to a received task allocation request, determine the target data corresponding to the task allocation request and the metadata corresponding to the target data; Based on the metadata, the opening volume and gain amount corresponding to multiple nodes are determined by a preset algorithm. The opening volume is the predicted value of the resource cost required to allocate the target data to the node, and the gain amount is the predicted value of the computing power acceleration after the target data is allocated to the node. Based on multiple opening volume and multiple gain volume, a matching value is determined for each of the multiple nodes. The matching value is used to indicate the degree of matching of the target data to the node. The node with the highest matching value among the plurality of nodes is determined as the target node, and the target data is allocated and processed according to the target node; Based on the metadata, the opening volume and gain of multiple nodes are determined by a preset algorithm, including: Real-time monitoring of the cluster containing the multiple nodes is performed to determine the cluster monitoring data. Based on the cluster monitoring data, determine multiple current overhead weights and current gain weights; Based on the metadata and the multiple current cost weights, determine the opening volume corresponding to each of the multiple nodes; Based on the metadata and the current gain weight, determine the gain amount corresponding to each of the multiple nodes; The metadata includes data type, data volume, multiple fields, and priority. For any given node, the gain amount corresponding to that node is determined based on the metadata and the current gain weight, including: The first coefficient is determined based on the data type and the data volume; Based on the data type and the multiple fields, determine the information entropy corresponding to the metadata; The product of the first coefficient and the information entropy is determined as the weighted information entropy; The second coefficient is determined based on the aforementioned priority. The product of the weighted information entropy, the second coefficient, and the current gain weight is determined as the gain amount corresponding to the node.

2. The method according to claim 1, characterized in that, The plurality of current overhead weights includes current storage weight, current computation weight, and current network weight. For any given node, the corresponding overhead volume is determined based on the metadata and the plurality of current overhead weights, including: Based on the metadata and the current storage weight, determine the storage capacity required for the node; Based on the metadata and the current computational weight, determine the computational cost corresponding to the node; Based on the metadata and the current network weight, determine the network access volume corresponding to the node; Determine the number of hops in the path from the source node containing the target data to the node; Based on the metadata and the path hop count, determine the migration cost corresponding to the node; The sum of the storage activation volume, the computing activation volume, the network activation volume, and the migration activation volume is determined as the activation volume corresponding to the node.

3. The method according to claim 2, characterized in that, The metadata includes data volume. Based on the metadata and the current storage weight, the storage capacity corresponding to the node is determined, including: Determine the disk seek time and bandwidth corresponding to the node; The processing time is determined based on the data volume and the bandwidth. The sum of the processing time and the disk seek time is determined as the storage time; Based on the storage duration and the current storage weight, determine the storage availability for the node.

4. The method according to claim 2, characterized in that, The metadata includes data volume. Based on the metadata and the current network weight, the network access volume corresponding to the node is determined, including: Obtain the network link bandwidth and packet loss rate from the source node where the target data is located to the node; The transmission duration is determined based on the data volume and the network link bandwidth. Determine the packet loss penalty factor based on the packet loss rate; The network access volume corresponding to the node is determined based on the packet loss penalty factor, the transmission duration, and the current network weight.

5. The method according to claim 4, characterized in that, Based on the data type and the multiple fields, the information entropy corresponding to the metadata is determined, including: Calculate the field probability values ​​corresponding to each of the multiple fields; Based on the data type and the probability values ​​of multiple fields, determine the field coefficients corresponding to each of the multiple fields; For any given field, determine the field information entropy based on the field probability value and the corresponding field coefficient. The information entropy of the data is determined by summing the field information entropy corresponding to each of the multiple fields.

6. The method according to any one of claims 1-4, characterized in that, Based on the cluster monitoring data, several current overhead weights and current gain weights are determined, including: Based on the cluster monitoring data, determine the cluster latency standard deviation, baseline energy consumption, and task completion rate; Based on the cluster monitoring data, determine the storage coefficient, average cluster energy consumption, and number of network topology changes; The current storage weight is determined based on the cluster latency standard deviation and the storage coefficient. The current calculation weight is determined based on the baseline energy consumption and the cluster average energy consumption; The current network weight is determined based on the number of network topology changes. The current gain weight is determined based on the task completion rate; The plurality of current overhead weights include the current storage weight, the current computation weight, and the current network weight.

7. The method according to claim 6, characterized in that, Based on the cluster monitoring data, determine the storage coefficient, average cluster energy consumption, and number of network topology changes, including: Based on the cluster monitoring data, determine whether there is at least one new event; If so, update the storage coefficient, average cluster energy consumption, and number of network topology changes based on the at least one new event; If not, obtain the storage coefficient, average cluster energy consumption, and number of network topology changes in the storage space.

8. The method according to claim 7, characterized in that, Based on the at least one new event, update the storage coefficient, average cluster energy consumption, and number of network topology changes, including: If the at least one newly added event includes at least one stored event, the storage coefficient is updated based on the at least one stored event; If the at least one new event includes at least one energy consumption event, update the cluster average energy consumption based on the at least one energy consumption event; If the at least one new event includes at least one network event, the number of network topology changes is updated based on the at least one network event.

Images