A task scheduling method for a big data processing platform

Abstract

The application discloses a task scheduling method under a big data processing platform. The method comprises the following steps: a resource manager updates a node state according to available resources and an energy consumption ratio of a node reported by a node manager; a scheduling manager performs secondary screening according to the node state, and adds and updates a node queue in descending order of a comprehensive energy consumption ratio of a primary screening node; when a task request is assigned to a container, a high energy consumption ratio node is preferentially selected, and the container is assigned in combination with resource reservation, label constraint and compatibility check. Small-scale test and performance analysis results on an open-source big data processing platform show that the method disclosed by the application can guarantee a minimum completion time, significantly improve the efficiency of task scheduling, and effectively reduce system energy consumption.

Description

Technical Field

[0001] This invention relates to the field of distributed computing resource scheduling technology, and more specifically to a task scheduling method under a big data processing platform. Background Technology

[0002] As society becomes increasingly information-driven, the efficient management and analysis of massive amounts of data has become a key challenge and urgent need across various industries. This need has driven the development of new technologies to address the complexity of big data processing. Currently, big data processing platforms have become the mainstream technical solution for big data analytics. However, with the continuous expansion of computing clusters, the energy consumption costs of big data processing platform applications continue to rise.

[0003] Existing mainstream big data processing platform task schedulers primarily allocate containers based on resource availability, which has two major drawbacks: first, they ignore energy efficiency differences and fail to consider the varying power consumption characteristics between nodes, leading to continuous operation of high-power nodes and energy waste; second, their strategies are static, and traditional scheduling rules cannot dynamically respond to real-time changes in node energy efficiency. How to reduce energy consumption and improve energy efficiency in big data processing platform data centers has become a critical issue urgently needing to be addressed in the field of distributed computing resource scheduling technology. Therefore, a task scheduling method for big data processing platforms is urgently needed—an intelligent scheduling method that integrates dynamic energy efficiency ratio assessment. Summary of the Invention

[0004] The purpose of this invention is to provide a task scheduling method under a big data processing platform, which can not only reduce the energy consumption of data center task execution, but also realize intelligent scheduling with dynamic energy consumption ratio evaluation.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a task scheduling method under a big data processing platform, comprising the following steps:

[0006] Step S1: The node manager periodically obtains the available resources and energy consumption ratio of the nodes, and reports them to the resource manager through the node manager's heartbeat signal. After receiving the obtained energy consumption ratio of the nodes, the resource manager updates the obtained energy consumption ratio of the nodes to obtain the updated energy consumption ratio of the nodes, and records the number of times the nodes report.

[0007] Step S2: The scheduling manager filters the energy consumption ratio of the nodes updated in step S1 to obtain the comprehensive energy consumption ratio of the secondary filtered nodes, adds the comprehensive energy consumption ratio of the secondary filtered nodes to the node queue in descending order, and updates the node queue.

[0008] Step S3: When the scheduler receives a new container request for a task, it selects a node from the node queue updated in step S2 and obtains the selected node;

[0009] Step S4: Perform a multi-dimensional compatibility check on the nodes selected in Step S3. The nodes that pass the multi-dimensional compatibility check will be the allocation containers for the new container requests of the tasks in Step S3.

[0010] Furthermore, in step S1, the node manager periodically obtains the available resources and energy consumption ratio of the nodes, and reports them to the resource manager via the node manager's heartbeat signal. After receiving the obtained energy consumption ratio of the nodes, the resource manager updates the obtained energy consumption ratio of the nodes to obtain the updated energy consumption ratio of the nodes, and records the number of times the nodes report; specifically:

[0011] Step S11: Node available resources refer to the node's idle hardware resources, including the number of available CPU cores, CPU utilization, available memory size, GPU utilization, disk load, and available network bandwidth. If the node has resource reservations, the node available resources are the node's available resources after deducting the resource reservations.

[0012] Step S12: Calculate the node's computing power based on its available resources, using the following formula:

[0013] ;

[0014] in, Indicates the computing power of a node. Indicates the number of available central processing unit cores. Indicates the utilization rate of the central processing unit. The value range is 0-1. Indicates the available memory size. Indicates graphics processor utilization. The value range is 0-1. Indicates disk load. The value range is 0-1. Indicates available network bandwidth. , , , , These represent the weight factors for the corresponding resources. The weighting factors represent the number of available CPU cores and CPU utilization. A weighting factor representing the available memory size. A weighting factor representing graphics processor utilization. Weighting factors representing disk load The weighting factor representing the available network bandwidth satisfies =1;

[0015] Step S13: Calculate the energy consumption ratio of the node, using the following formula:

[0016] ;

[0017] in, This represents the energy consumption ratio of a node. This represents the average power consumption of a node during the monitoring period. The monitoring period refers to the fixed time interval on which the node manager periodically collects, calculates, and reports the ratio of available resources to energy consumption of a node.

[0018] Furthermore, in step S2, the scheduling manager filters the energy consumption ratios of the nodes updated in step S1 to obtain the comprehensive energy consumption ratios of the secondary-level filtered nodes, adds the comprehensive energy consumption ratios of the secondary-level filtered nodes to the node queue in descending order, and updates the node queue; specifically:

[0019] Step S21, primary screening: exclude nodes whose energy consumption ratio is not included in the heartbeat signal of the node manager after a preset number of consecutive rejections, and obtain the primary screened nodes.

[0020] Step S22: Calculate the overall energy consumption ratio of the primary screening nodes, using the following formula;

[0021] ;

[0022] in, This represents the overall energy consumption ratio of the primary screening node. This represents the total number of reports submitted by the primary screening node, and its value typically ranges from [value range missing]. , Indicates the first primary screening node The energy consumption ratio reported next time. Indicates the first primary screening node The weighting factor of the energy consumption ratio reported this time satisfies and Number of reports The smaller the value, the better the primary filtering node. Weighting factor for the energy consumption ratio reported next time. The larger;

[0023] Step S23: Set an energy consumption ratio screening threshold, perform secondary screening on the primary screening nodes, and exclude nodes whose overall energy consumption ratio is less than the energy consumption ratio screening threshold to obtain the secondary screening nodes and their overall energy consumption ratios; the formula is as follows:

[0024] ;

[0025] in, This indicates the energy consumption ratio screening threshold. Represents the configurable coefficient, satisfying ;

[0026] ;

[0027] in, This represents the weighted average energy consumption ratio of the primary screening nodes. Indicates the total number of nodes. Indicates the primary filtering node. Indicates the primary filtering node The energy efficiency ratio;

[0028] ;

[0029] in, Indicates the primary filtering node Resource weighting factors, satisfying and , Indicates the primary filtering node The number of central processing unit cores, Indicates the primary filtering node CPU idle rate , Indicates the primary filtering node The normalized memory ratio, where the normalized memory ratio is the ratio of available memory to a memory baseline;

[0030] Step S24: Add the nodes selected in the secondary screening to the node queue in descending order of their overall energy consumption ratio, and update the node queue. When the nodes selected in the secondary screening have the same overall energy consumption ratio, sort them in descending order of their available resources.

[0031] Step S25: If no node is available in the secondary screening, wait for one heartbeat signal cycle and then return to step S21 to restart the primary screening. If no node is available after waiting for 2 to 3 heartbeat signal cycles, trigger the adaptive adjustment mechanism: return to step S23 and reduce the configurable coefficient. The value was then recalculated and the energy ratio screening threshold was recalculated. Then, proceed with the secondary screening.

[0032] Furthermore, in step S3, when the scheduler receives a new container request for a task, it selects a node from the node queue updated in step S2 to obtain the selected node; specifically:

[0033] Step S31: If the task's new container request specifies a resource reservation node, then the specified resource reservation node is directly set as the allocation container for the task's new container request. The specified resource reservation node is the selected node, but multi-dimensional compatibility checks of the selected node are not required.

[0034] Step S32: If the new container request for the task does not specify a resource reserved node, check if the new container request for the task has node label constraints. If there are node label constraints, check the node labels sequentially starting from the head of the updated node queue and select the first node that matches the node label constraint. If no node matching the node label constraint is found, select the head node of the updated node queue.

[0035] Furthermore, in step S4, a multi-dimensional compatibility check is performed on the nodes selected in step S3. The nodes that pass the multi-dimensional compatibility check become the allocation containers for the new container requests of the tasks in step S3; specifically:

[0036] Step S41: Check whether the available resources of the selected node meet the resource requirements of the new container of the task;

[0037] Step S42: According to the default locality policy of the big data processing platform, the locality policy is local > same rack > cross rack, check the data locality of the selected node;

[0038] Step S43: Check the hardware and software environment compatibility of the selected node;

[0039] Step S44: Check whether the security domain of the selected node meets the requirements;

[0040] Step S45: If the selected node passes the multi-dimensional compatibility check in steps S41 to S44, then the selected node will be the allocation container for the new container request of the task; otherwise, return to step S3 to reselect a node.

[0041] The beneficial effects of this invention are:

[0042] This invention periodically acquires the available resources and energy consumption ratio data of nodes to update the node status in the resource manager, and performs secondary filtering of nodes accordingly. Filtered nodes are added to or updated in the node queue. When a new container request is received, nodes with high energy consumption ratios are selected from the queue. After resource reservation, tag constraints and compatibility checks, containers are allocated, which significantly reduces data center energy consumption while ensuring job completion time. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0045] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any creative effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0047] This embodiment provides a task scheduling method under a big data processing platform (Hadoop platform), such as... Figure 1 As shown, it includes the following steps:

[0048] Step S1: The node manager periodically obtains the available resources and energy consumption ratio of the nodes, and reports them to the resource manager through the node manager's heartbeat signal. After receiving the obtained energy consumption ratio of the nodes, the resource manager updates the obtained energy consumption ratio of the nodes to obtain the updated energy consumption ratio of the nodes, and records the number of times the nodes report.

[0049] Step S2: The scheduling manager filters the energy consumption ratio of the nodes updated in step S1 to obtain the comprehensive energy consumption ratio of the secondary filtered nodes, adds the comprehensive energy consumption ratio of the secondary filtered nodes to the node queue in descending order, and updates the node queue.

[0050] Step S3: When the scheduler receives a new container request for a task, it selects a node from the node queue updated in step S2 and obtains the selected node;

[0051] Step S4: Perform a multi-dimensional compatibility check on the nodes selected in Step S3. The nodes that pass the multi-dimensional compatibility check will be the allocation containers for the new container requests of the tasks in Step S3.

[0052] Furthermore, in step S1, the node manager periodically obtains the available resources and energy consumption ratio of the nodes, and reports them to the resource manager via the node manager's heartbeat signal. After receiving the obtained energy consumption ratio of the nodes, the resource manager updates the obtained energy consumption ratio of the nodes to obtain the updated energy consumption ratio of the nodes, and records the number of times the nodes report; specifically:

[0053] Step S11: Node available resources refer to the node's idle hardware resources, including the number of available CPU cores, CPU utilization, available memory size, GPU utilization, disk load, and available network bandwidth. If the node has resource reservations, the node available resources are the node's available resources after deducting the resource reservations.

[0054] Step S12: Calculate the node's computing power based on its available resources, using the following formula:

[0055] ;

[0056] in, Indicates the computing power of a node. Indicates the number of available central processing unit cores. Indicates the utilization rate of the central processing unit. The value range is 0-1. Indicates the available memory size. Indicates graphics processor utilization. The value range is 0-1. Indicates disk load. The value range is 0-1. Indicates available network bandwidth. , , , , These represent the weight factors for the corresponding resources. The weighting factors represent the number of available CPU cores and CPU utilization. A weighting factor representing the available memory size. A weighting factor representing graphics processor utilization. Weighting factors representing disk load The weighting factor representing the available network bandwidth satisfies =1;

[0057] Step S13: Calculate the energy consumption ratio of the node, using the following formula:

[0058] ;

[0059] in, This represents the energy consumption ratio of a node. This represents the average power consumption of a node during the monitoring period. The monitoring period refers to the fixed time interval on which the node manager periodically collects, calculates, and reports the ratio of available resources to energy consumption of a node.

[0060] Furthermore, in step S2, the scheduling manager filters the energy consumption ratios of the nodes updated in step S1 to obtain the comprehensive energy consumption ratios of the secondary-level filtered nodes, adds the comprehensive energy consumption ratios of the secondary-level filtered nodes to the node queue in descending order, and updates the node queue; specifically:

[0061] Step S21, primary screening: exclude nodes whose energy consumption ratio is not included in the heartbeat signal of the node manager after a preset number of consecutive rejections, and obtain the primary screened nodes.

[0062] Step S22: Calculate the overall energy consumption ratio of the primary screening nodes, using the following formula;

[0063] ;

[0064] in, This represents the overall energy consumption ratio of the primary screening node. This represents the total number of reports submitted by the primary screening node, and its value typically ranges from [value range missing]. , Indicates the first primary screening node The energy consumption ratio reported next time. Indicates the first primary screening node The weighting factor of the energy consumption ratio reported this time satisfies and Number of reports The smaller the value, the better the primary filtering node. Weighting factor for the energy consumption ratio reported next time. The larger;

[0065] Step S23: Set an energy consumption ratio screening threshold, perform secondary screening on the primary screening nodes, and exclude nodes whose overall energy consumption ratio is less than the energy consumption ratio screening threshold to obtain the secondary screening nodes and their overall energy consumption ratios; the formula is as follows:

[0066] ;

[0067] in, This indicates the energy consumption ratio screening threshold. Represents the configurable coefficient, satisfying ;

[0068] ;

[0069] in, This represents the weighted average energy consumption ratio of the primary screening nodes. This indicates the total number of nodes in the initial screening. Indicates the primary filtering node. Indicates the primary filtering node The energy efficiency ratio;

[0070] ;

[0071] in, Indicates the primary filtering node Resource weighting factors, satisfying and , Indicates the primary filtering node The number of central processing unit cores, Indicates the primary filtering node CPU idle rate , Indicates the primary filtering node The normalized memory ratio, where the normalized memory ratio is the ratio of available memory to a memory baseline;

[0072] Step S24: Add the nodes selected in the secondary screening to the node queue in descending order of their overall energy consumption ratio, and update the node queue. When the nodes selected in the secondary screening have the same overall energy consumption ratio, sort them in descending order of their available resources.

[0073] Step S25: If no node is available in the secondary screening, wait for one heartbeat signal cycle and then return to step S21 to restart the primary screening. If no node is available after waiting for 2 to 3 heartbeat signal cycles, trigger the adaptive adjustment mechanism: return to step S23 and reduce the configurable coefficient. The value was then recalculated and the energy ratio screening threshold was recalculated. Then, proceed with the secondary screening.

[0074] Furthermore, in step S3, when the scheduler receives a new container request for a task, it selects a node from the node queue updated in step S2 to obtain the selected node; specifically:

[0075] Step S31: If the task's new container request specifies a resource reservation node, then the specified resource reservation node is directly set as the allocation container for the task's new container request. The specified resource reservation node is the selected node, but multi-dimensional compatibility checks of the selected node are not required.

[0076] Step S32: If the new container request for the task does not specify a resource reserved node, check if the new container request for the task has node label constraints. If there are node label constraints, check the node labels sequentially starting from the head of the updated node queue and select the first node that matches the node label constraint. If no node matching the node label constraint is found, select the head node of the updated node queue.

[0077] Furthermore, in step S4, a multi-dimensional compatibility check is performed on the nodes selected in step S3. The nodes that pass the multi-dimensional compatibility check become the allocation containers for the new container requests of the tasks in step S3; specifically:

[0078] Step S41: Check whether the available resources of the selected node meet the resource requirements of the new container of the task;

[0079] Step S42: According to the default locality policy of the big data processing platform, the locality policy is local > same rack > cross rack, check the data locality of the selected node;

[0080] Step S43: Check the hardware and software environment compatibility of the selected node;

[0081] Step S44: Check whether the security domain of the selected node meets the requirements;

[0082] Step S45: If the selected node passes the multi-dimensional compatibility check in steps S41 to S44, then the selected node will be the allocation container for the new container request of the task; otherwise, return to step S3 to reselect a node.

[0083] The following uses a data center of a big data processing platform as an example to illustrate the implementation process of the task scheduling method disclosed in this invention.

[0084] Step S1: First, the node manager periodically obtains the available resources of the nodes and the energy consumption ratio of the nodes, and reports them to the resource manager through the node manager's heartbeat signal.

[0085] In one embodiment, a network consisting of 10 nodes is divided into two security domains, namely the first security domain and the second security domain. The available resources of the nodes, the energy efficiency ratio of the nodes, the node tags, and the remarks are shown in Table 1. The "available resources of the node" refers to the currently idle hardware resources of the node, including the number of available central processing unit (CPU) cores. and CPU utilization Available memory size Graphics Processing Unit (GPU) Utilization Disk load and available network bandwidth If a node has reserved resources, the available resources must be reduced by the reserved portion.

[0086] Table 1. Information on available resources for nodes in a certain embodiment.

[0087]

[0088] Then, after the resource manager receives the energy consumption ratio of the acquired node, it updates the energy consumption ratio of the acquired node to obtain the updated energy consumption ratio of the node, and records the number of times the node reports.

[0089] In one embodiment, the resource manager maintains a node queue for each node. The queue length is equal to the number of times a node reports data. The node queue records the energy consumption ratio of each node for the most recent 5 times in ascending order, as shown in Table 2.

[0090] Table 2 Node queue of a certain embodiment

[0091]

[0092] Step S2: The scheduling manager filters the energy consumption ratios of the nodes updated in step S1 to obtain the comprehensive energy consumption ratios of the secondary-level filtered nodes. The comprehensive energy consumption ratios of the secondary-level filtered nodes are then added to the node queue in descending order, and the node queue is updated. The steps are as follows:

[0093] The first step is primary screening, which excludes nodes whose energy consumption ratio is not included in the heartbeat signal reported to the node manager for a preset number of consecutive times, thus obtaining the primary screened nodes.

[0094] The second step is to calculate the overall energy consumption ratio of the primary screening nodes. ;

[0095] In one embodiment, the primary screening node is... The weighting factor for the energy consumption ratio reported next time is: The overall energy consumption ratio of all primary screening nodes is shown in Table 3:

[0096] Table 3. Overall energy consumption ratio of the primary screening node in a certain embodiment

[0097]

[0098] The third step is to set the energy consumption ratio screening threshold. ;

[0099] In one embodiment, configurable coefficients The initial value is 0.8, and the resource weight factor of the primary screening node is calculated based on the proportion of available resources. ,in Indicates the primary filtering node Number of central processing unit cores Indicates the primary filtering node CPU idle rate , Indicates the primary filtering node The normalized memory ratio, which is the ratio of available memory to the baseline value of 32 gigabytes, was used to calculate the resource weight factor for each primary screening node. The calculation results were rounded to two decimal places, as shown in Table 4.

[0100] Table 4 Resource weight factors for primary screening nodes in a certain embodiment

[0101]

[0102] Weighted average energy consumption ratio of primary screening nodes Therefore, the energy consumption ratio screening threshold ;

[0103] The fourth step is secondary screening, which excludes nodes selected in the primary screening based on their overall energy consumption ratio. Less than the energy consumption ratio screening threshold The nodes further include:

[0104] If no nodes are available in the secondary screening, wait for one heartbeat signal cycle and then return to the first step of step S2 to restart the primary screening.

[0105] If no available node is found after waiting for 2 to 3 heartbeat signal cycles, the adaptive adjustment mechanism is triggered: return to step S2, step 3, and reduce the configurable coefficient. The value was then recalculated and the energy ratio screening threshold was recalculated. Then continue with step S2, step four: secondary screening;

[0106] In one embodiment, nodes with node number 4, node number 5, and node number 8 are excluded, leaving 7 nodes after secondary filtering;

[0107] The fifth step is to add the nodes selected in the secondary screening to the node queue in descending order of their overall energy consumption ratio, and update the node queue. When the overall energy consumption ratios of the nodes selected in the secondary screening are the same, they are sorted in descending order of their available resources. In one embodiment, the sorted node queue is shown in Table 5.

[0108] Table 5. Sorted Node Queue in a Certain Implementation Example

[0109]

[0110] After the node selection is complete, when the scheduler receives a new container request for a task, it selects a node from the node queue updated in step S2.

[0111] Step S3, in one embodiment, the new container request for the task is: {"Resource Requirements": {"Central Processing Unit (CPU)": 4 cores, "Memory": 8 gigabytes}, "Label Constraints": "Graphics Processing Unit (GPU)", "Resource Reservation": None}, and the processing steps are as follows:

[0112] The first step is to specify a resource reservation node for the new container request of the task. Then, the specified resource reservation node will be directly assigned to the allocation container of the new container request of the task. The specified resource reservation node is the selected node, but there is no need to perform multi-dimensional compatibility checks on the selected node.

[0113] In one embodiment, the task did not specify a resource reservation node;

[0114] The second step is to check if the new container request of the task has a node label constraint. If there is a node label constraint, check the node labels sequentially starting from the head of the updated node queue and select the first node that matches the node label constraint. If no node matching the node label constraint is found, select the head node of the updated node queue.

[0115] In one embodiment, the label constraint is "Graphics Processor (GPU)". The node queue is traversed and the node labels are checked. The labels of nodes 9 and 3 do not match the task label constraint, so they are skipped. The label of node 10 is "Graphics Processor (GPU)", and the selected node is obtained.

[0116] Third, if the above steps do not yield the selected node, the task enters a waiting state; in one embodiment, the selected node is node 10.

[0117] Step S4: Perform a multi-dimensional compatibility check on the selected nodes. The nodes that pass the multi-dimensional compatibility check will be assigned as the new container for the task. The processing steps are as follows:

[0118] The first step is to check whether the available resources of the selected node can meet the resource requirements of the new container for the task.

[0119] In one embodiment, the available resources for node 10 are: CPU: 8 cores (20%), memory: 16 gigabytes, which is greater than the container request;

[0120] The second step is to check the data locality of the selected node according to the default locality policy of the big data processing platform, which is local > same rack > cross rack.

[0121] The third step is to check the software and hardware environment compatibility of the selected node.

[0122] The fourth step is to check whether the security domain of the selected node meets the requirements.

[0123] Fifth step: If the selected node passes the multi-dimensional compatibility check from step one to step four of step S4, then the selected node will be the allocation container for the new container request of the task; otherwise, return to step S3 to reselect a node.

[0124] In one embodiment, after node 10 passes all the above checks, the scheduler allocates a container for the task on node 10.

[0125] Compared to related technologies, the big data processing platform task scheduling method provided by this invention improves overall energy efficiency by dynamically excluding low-energy-efficiency nodes and eliminating ineffective energy consumption; it optimizes resources by prioritizing scheduling to nodes with high energy efficiency ratios; while being compatible with traditional task scheduling mechanisms, it performs multi-dimensional condition checks before container allocation to ensure scheduling feasibility; in addition, to avoid scheduling deadlock, some embodiments support adaptive adjustment of the energy efficiency ratio screening threshold.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features is not contradictory, it should be considered to be within the scope of this specification.

[0127] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art will understand that adjustments to parameters such as the monitoring period, threshold coefficient α, and weighting factor, without departing from the core principles of this invention, all fall within the scope of protection of this patent. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1.A task scheduling method under a big data processing platform, characterized in that, The method comprises: Step S1: The node manager periodically acquires the available resources of the node and the energy consumption ratio of the node, and reports the same to the resource manager through the heartbeat signal of the node manager; after the resource manager receives the energy consumption ratio of the node, the energy consumption ratio of the node is updated to obtain the updated energy consumption ratio of the node, and the reporting times of the node are recorded; Step S2: The scheduling manager filters the updated energy consumption ratio of the node obtained in step S1 to obtain the comprehensive energy consumption ratio of the secondary filtering node, adds the comprehensive energy consumption ratio of the secondary filtering node to the node queue in descending order, and updates the node queue; specifically: Step S21, primary filtering, excluding nodes that do not contain the energy consumption ratio of the node in the heartbeat signal of the node manager for a continuous preset number of times, to obtain the primary filtering node; Step S22, calculating the comprehensive energy consumption ratio of the primary filtering node, the formula is as follows: ； wherein, represents the comprehensive energy consumption ratio of the primary screening node, represents the total number of reporting times of the primary screening node, and the value range is , represents the energy consumption ratio of the primary screening node in the th reporting, represents the energy consumption ratio of the primary screening node in the th reporting, and , the smaller the value of the reporting times , the greater the weight factor of the energy consumption ratio of the primary screening node in the th reporting . Step S23, setting an energy consumption ratio filtering threshold, performing secondary filtering on the primary filtering node, excluding nodes with a comprehensive energy consumption ratio of the primary filtering node less than the energy consumption ratio filtering threshold to obtain the secondary filtering node and the comprehensive energy consumption ratio of the secondary filtering node; the formula is as follows: ； wherein, represents an energy consumption ratio screening threshold value, represents a configurable coefficient, satisfying ; ； wherein, represents the weighted average energy consumption ratio of the primary screening nodes, represents the total number of nodes, represents the primary screening nodes, represents the primary screening nodes of the energy consumption ratio; ； wherein, represents a resource weight factor of the primary screening node , satisfying and , represents a number of central processor cores of the primary screening node , represents a central processor idle rate of the primary screening node , , represents a normalized memory ratio of the primary screening node , wherein the normalized memory ratio is a ratio of available memory to a memory reference value; Step S24, adding the comprehensive energy consumption ratio of the secondary filtering node to the node queue in descending order, and updating the node queue; when the comprehensive energy consumption ratio of the secondary filtering node is the same, the secondary filtering node is sorted in descending order according to the available resources; Step S25, if there is no available node in the secondary screening node, then wait for a heartbeat signal period, return to step S21 to start the primary screening, if there is still no available node after waiting for 2 to 3 heartbeat signal periods, trigger the adaptive adjustment mechanism: return to step S23, reduce the value of the configurable coefficient and recalculate the energy consumption ratio screening threshold , and then continue to perform secondary screening . Step S3: The scheduling manager selects a node from the node queue updated in step S2 when receiving a new container request of a task, to obtain the selected node; Step S4: Multi-dimensional compatibility check is performed on the node selected in step S3, and the node that passes the multi-dimensional compatibility check is the allocation container of the new container request of the task in step S3. 2.The task scheduling method under the big data processing platform of claim 1, wherein, In step S1, the node manager periodically acquires the available resources of the node and the energy consumption ratio of the node, and reports the same to the resource manager through the heartbeat signal of the node manager; after the resource manager receives the energy consumption ratio of the node, the energy consumption ratio of the node is updated to obtain the updated energy consumption ratio of the node, and the reporting times of the node are recorded; specifically: Step S11, the available resources of the node refer to the idle hardware resources of the node, including the available central processor core number, the central processor utilization rate, the available memory size, the graphics processor utilization rate, the disk load and the available network bandwidth; if the node has resource reservation, the available resources of the node are the available resources of the node after deducting the resource reservation; Step S12, calculating the computing power of the node based on the available resources of the node, the formula is as follows: ； wherein, represents the computing power of a node, represents the number of available central processor cores, represents the central processor utilization rate, the value range of which is 0-1, represents the available memory size, represents the graphics processor utilization rate, the value range of which is 0-1, represents the disk load, the value range of which is 0-1, represents the available network bandwidth, , , , , respectively represent the weight factor of the corresponding resource, represents the weight factor of the number of available central processor cores and the central processor utilization rate, represents the weight factor of the available memory size, represents the weight factor of the graphics processor utilization rate, represents the weight factor of the disk load, represents the weight factor of the available network bandwidth, and satisfies =1. Step S13, calculating the energy consumption ratio of the node, the formula is as follows: ； wherein, represents the energy consumption ratio of the node, represents the average power consumption of the node in a monitoring period, wherein the monitoring period refers to a fixed time interval in which the node manager periodically collects, calculates and reports the available resources of the node and the energy consumption ratio of the node. 3.The task scheduling method under the big data processing platform of claim 2, characterized in that, In step S3, the scheduling manager selects a node from the node queue updated in step S2 when receiving a new container request of a task, to obtain the selected node; specifically: Step S31: The new container request of the task specifies a resource reservation node, then the resource reservation node is directly taken as the allocation container of the new container request of the task, and the resource reservation node is the selected node but does not need to perform multi-dimensional compatibility check on the selected node. Step S32: The new container request of the task has no specified resource reservation node, checks whether the new container request of the task has node label constraint, if yes, sequentially checks node label from the head of the updated node queue, selects the first node matching the node label constraint, if not, selects the head node of the updated node queue. 4.The task scheduling method under the big data processing platform of claim 3, wherein, Step S4: Multi-dimensional compatibility check is performed on the node selected in step S3, and the node passing the multi-dimensional compatibility check is the allocated container of the new container request of the task in step S3; specifically: Step S41: Check whether the available resources of the selected node meet the resource amount of the new container request of the task; Step S42: According to the default locality strategy of the big data processing platform, the locality strategy is locality>same rack>cross rack, check the data locality of the selected node; Step S43: Check the software and hardware environment compatibility of the selected node; Step S44: Check whether the security domain of the selected node meets the requirements; Step S45: If the selected node passes the multi-dimensional compatibility check of steps S41 to S44, the selected node is the allocated container of the new container request of the task; Otherwise, return to step S3 to reselect a node.

Images