Capacity reduction method and device based on Yarn-Kubernetes, electronic equipment and storage medium

Abstract

The invention provides a capacity reduction method and device based on Yarn-Kubernetes, electronic equipment and a storage medium, and the method comprises the steps: obtaining preset index information of each management node from a Yarn cluster, including obtaining the resource utilization rate of the management node and the number of tasks operated in the management node, according to the method, the indexes of the management nodes are weighted and scored based on the weights set for different indexes in advance, and the target management nodes needing capacity reduction are screened based on the scoring result, so that the target management nodes are offline in the Yarn, the running containers corresponding to the target management nodes are deleted in the Kubernetes, and automatic capacity reduction of the Yarn-Kubernetes is realized. Moreover, weighted scoring is carried out based on each item of index information of the management node, so that the scoring result can reflect the resource demand condition of the management node more accurately, and the determination accuracy of the target management node needing capacity reduction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a capacity reduction method, device, electronic device and storage medium based on Yarn-Kubernetes. Background Art

[0002] YARN is a resource manager in the Hadoop ecosystem, mainly used for the allocation and management of resources in the cluster. YARN can automatically expand and shrink resources according to the needs of applications to provide better resource utilization. In YARN, resource shrinkage is usually automatic and dynamically adjusted according to the needs of applications and the load of the cluster.

[0003] Yarn-Kubernetes is a way to integrate Yarn with Kubernetes. It allows YARN applications to run on a Kubernetes cluster while leveraging Kubernetes' powerful container orchestration and resource management capabilities. Nodes on a Kubernetes cluster can run as Yarn's NodeManager (management node), and containers can host Yarn application tasks.

[0004] However, for Yarn-Kubernetes, Yarn's NodeManager is deployed on the Kubernetes cluster, and NodeManager is a resident process deployed in the Pod. This architecture cannot use YARN's native dynamic resource allocation mechanism to achieve Pod scaling after Kubernetes scales down after the NodeManager goes offline. In this way, idle resources cannot be scaled down when the cluster load is low. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a scaling-down method, device, electronic device and storage medium based on Yarn-Kubernetes to achieve automatic scaling-down of nodes in Yarn-Kubernetes.

[0006] According to one aspect of the present invention, a scaling-down method based on Yarn-Kubernetes is provided, which is applied to a Yarn-Kubernetes cluster. The method includes:

[0007] Query the preset indicator information of each management node in the Yarn cluster, the preset indicator information includes resource utilization and the number of tasks running in the management node;

[0008] Calculate the scaling-down score result of each management node based on the preset weight of each preset indicator information and each preset indicator information;

[0009] Determine the target management node that needs to be shrunk based on the shrunk scoring results of each management node according to the preset shrunk condition; the preset shrunk condition includes a shrunk scoring threshold and / or a shrunk quantity;

[0010] Each of the target management nodes is taken offline in the Yarn cluster, and the running containers where each of the target management nodes is located are deleted in the Kubernetes cluster.

[0011] In a possible embodiment, the preset weights are determined by the following steps:

[0012] Obtaining preset index records of each management node in the historical shrinking record, wherein the historical shrinking record includes the preset index records of each management node during the historical shrinking and the information of the management node actually shrunk;

[0013] Based on the initial weights set for each preset indicator, weighted scoring is performed on the preset indicators of each management node to obtain a weighted scoring result;

[0014] Determining the scaling-down nodes that need to be scaled down based on the preset scaling-down conditions and the weighted scoring results;

[0015] Determine the difference between the information of the shrunk node and the information of the actually shrunk management node to update the initial weight until the shrunk node is consistent with the actually shrunk management node;

[0016] The weights when determining that the shrunk node is consistent with the actually shrunk management node are preset weights.

[0017] In a possible embodiment, determining the target management node that needs to be shrunk based on the shrunk scoring result of each management node according to the preset shrunk condition includes:

[0018] Screening the management nodes whose scaling-down scores are lower than a preset score threshold, the target management nodes; and / or,

[0019] The management nodes are scored in descending order of the scaling-down scoring results, and a preset number of management nodes first in the scaling-down scoring result order are selected as target management nodes.

[0020] In a possible embodiment, the offline processing of each target management node in the Yarn cluster and the deleting of the running container where each target management node is located in the Kubernetes cluster include:

[0021] Mark the state of the target management node as retired in Yarn, and if there is a task running in the target management node, wait for the running task to complete before deleting the target management node;

[0022] In the case that the target management node has been deleted in the Yarn cluster, the Kubernetes interface is called according to the IP address of the target management node to delete the pod corresponding to the IP address.

[0023] According to another aspect of the present invention, a capacity reduction device based on Yarn-Kubernetes is provided, which is applied to a Yarn-Kubernetes cluster, and the device includes:

[0024] A query module is used to query the preset indicator information of each management node in the Yarn cluster, wherein the preset indicator information includes resource utilization and the number of tasks running in the management node;

[0025] A scoring module, used for calculating the scaling-down scoring result of each management node based on the preset weight of each preset indicator information and each preset indicator information;

[0026] A determination module, configured to determine a target management node that needs to be shrunk based on a shrinkage scoring result of each management node according to a preset shrinkage condition; the preset shrinkage condition includes a shrinkage scoring threshold and / or a shrinkage quantity;

[0027] The shrinking module is used to take each of the target management nodes offline in the Yarn cluster and delete the running containers where each of the target management nodes is located in the Kubernetes cluster.

[0028] In a possible embodiment, the preset weights are determined by the following steps:

[0029] Obtaining preset index records of each management node in the historical shrinking record, wherein the historical shrinking record includes the preset index records of each management node during the historical shrinking and the information of the management node actually shrunk;

[0030] Based on the initial weights set for each preset indicator, weighted scoring is performed on the preset indicators of each management node to obtain a weighted scoring result;

[0031] Determining the scaling-down nodes that need to be scaled down based on the preset scaling-down conditions and the weighted scoring results;

[0032] Determine the difference between the information of the shrunk node and the information of the actually shrunk management node to update the initial weight until the shrunk node is consistent with the actually shrunk management node;

[0033] The weights when determining that the shrunk node is consistent with the actually shrunk management node are preset weights.

[0034] In a possible embodiment, determining the target management node that needs to be shrunk based on the shrunk scoring result of each management node according to the preset shrunk condition includes:

[0035] Screening the management nodes whose scaling-down scores are lower than a preset score threshold, the target management nodes; and / or,

[0036] The management nodes are scored in descending order of the scaling-down scoring results, and a preset number of management nodes first in the scaling-down scoring result order are selected as target management nodes.

[0037] In a possible embodiment, the offline processing of each target management node in the Yarn cluster and the deleting of the running container where each target management node is located in the Kubernetes cluster include:

[0038] Mark the state of the target management node as retired in Yarn, and if there is a task running in the target management node, wait for the running task to complete before deleting the target management node;

[0039] In the case that the target management node has been deleted in the Yarn cluster, the Kubernetes interface is called according to the IP address of the target management node to delete the pod corresponding to the IP address.

[0040] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0041] Processor; and

[0042] Memory for storing programs,

[0043] The program includes instructions, which, when executed by the processor, cause the processor to execute any of the above-mentioned Yarn-Kubernetes-based scaling-down methods.

[0044] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned Yarn-Kubernetes-based scaling-down methods.

[0045] One or more technical solutions provided in the embodiments of the present invention obtain preset indicator information of each management node from the Yarn cluster, including obtaining the resource utilization of the management node and the number of tasks running in the management node, weighted scoring the indicators of each management node based on the weights pre-set for different indicators, and screening the target management node that needs to be scaled down based on the scoring results, thereby taking the target management node offline in Yarn and deleting the corresponding running container of the target management node in Kubernetes, thereby realizing automatic scaling down of Yarn-Kubernetes. Furthermore, by weighted scoring based on the various indicator information of the management node, the scoring results can more accurately reflect the resource demand of the management node, and improve the accuracy of determining the target management node that needs to be scaled down. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 A schematic diagram of a process of a capacity reduction method based on Yarn-Kubernetes provided in an embodiment of the present invention;

[0048] Figure 2 Another flowchart of a scaling-down method based on Yarn-Kubernetes provided in an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of an architecture for implementing a scaling-down method based on Yarn-Kubernetes provided in an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the structure of a capacity reduction device based on Yarn-Kubernetes provided in an embodiment of the present invention;

[0051] Figure 5 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0053] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0054] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0055] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0056] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0057] In order to realize automatic scaling down of Yarn-Kubernetes, the present invention provides a scaling down method, device, electronic device and storage medium based on Yarn-Kubernetes. The scaling down method based on Yarn-Kubernetes provided by the present invention can be applied to any electronic device with scaling down function, which can be a server, a computer or a mobile terminal, etc.

[0058] In a possible embodiment, the scaling-down method based on Yarn-Kubernetes provided by the present invention can be applied to Yarn-Kubernetes. Yarn-Kubernetes is a way to integrate YARN with Kubernetes, which allows YA RN applications to run on a Kubernetes cluster while taking advantage of Kubernetes' powerful container orchestration and resource management capabilities. Among them, the YARN cluster includes multiple resource managers ResourceManager, each of which can implement the allocation and management of resources in the cluster to support large-scale data processing jobs. Its main responsibilities are to coordinate jobs running on the cluster, allocate resources to applications and tasks, and monitor resource usage. Kubernetes is an open source container management platform for managing containerized applications on multiple hosts in a cloud platform, making the deployment of containerized applications simple and efficient.

[0059] In Yarn-Kubernetes, nodes in a Kubernetes cluster can run as Yarn's NodeManager. The YARN NodeManager is responsible for executing containers on cluster nodes, monitoring their resource usage, and coordinating resource allocation with the ResourceManager. The main responsibilities of the NodeManager are to maintain and manage resources on a single node and report information about resources on the node to the ResourceManager. In Yarn-Kubernetes, the Kubernetes process plays the role of the NodeManager in YARN, which is responsible for managing containers on Kubernetes nodes and communicating with the YARN ResourceManager to allocate and reclaim resources.

[0060] The scheme of the present invention is described below with reference to the accompanying drawings:

[0061] like Figure 1 As shown, Figure 1 A flowchart of a scaling-down method based on Yarn-Kubernetes provided in an embodiment of the present invention may include the following steps:

[0062] S101, querying preset indicator information of each management node in the Yarn cluster, wherein the preset indicator information includes resource utilization and the number of tasks running in the management node;

[0063] S102, calculating the scaling-down score result of each management node based on the preset weight of each preset indicator information and each preset indicator information;

[0064] S103, determining a target management node that needs to be shrunk based on the shrunk scoring results of each management node according to a preset shrunk condition; the preset shrunk condition includes a shrunk scoring threshold and / or a shrunk quantity;

[0065] S104: taking each of the target management nodes offline in the Yarn cluster, and deleting the running containers of each of the target management nodes in the Kubernetes cluster.

[0066] By applying the embodiment of the present invention, the preset indicator information of each management node is obtained from the Yarn cluster, including the resource utilization rate of the management node and the number of tasks running in the management node, and the indicators of each management node are weighted and scored based on the weights pre-set for different indicators, and the target management node that needs to be scaled down is screened based on the scoring results, so that the target management node is offline in Yarn, and the corresponding running container of the target management node is deleted in Kubernetes, thereby realizing the automatic scaling down of Yarn-Kubernetes. In addition, by weighted scoring based on the indicator information of the management node, the scoring result can more accurately reflect the resource demand of the management node, and improve the accuracy of determining the target management node that needs to be scaled down.

[0067] The above S101-S104 are exemplarily described below:

[0068] In a possible embodiment, in S101, the preset indicator information of each management node can be obtained through the resource manager in the Yarn cluster. As mentioned above, the management node will report its resource information to the resource manager, and the resource manager can store the information reported by the management node in the database of the Yarn cluster. Therefore, in this step, the indicator information of each management node can be obtained from the database.

[0069] The above-mentioned preset indicators can be set according to the actual application scenario, and specifically can be indicators related to the resources required for the operation of the management node. As a possible implementation method, the above-mentioned preset indicators may include resource utilization and business information running in the node, wherein the resource utilization may include CPU resource utilization, memory resource utilization, bandwidth resource utilization, IO resource utilization, and disk resource utilization, etc., and the above-mentioned business information may include the name of the task running in the management node, the number of tasks, and the user information to which the task belongs, etc. The above-mentioned tasks generally refer to big data processing tasks, such as data processing tasks sent by big data processing engines such as Spark and Hive. In a possible embodiment, the above-mentioned business information may also be the number of tasks of a specified task name or a specified user, etc.

[0070] After obtaining the preset indicator information of each management node, each indicator information can be weighted calculated based on the preset weights for different indicators. As a possible implementation, the product of each indicator and its corresponding weight can be added to obtain the shrinkage scoring result of the corresponding management node. For example, the shrinkage scoring result of the management node can be calculated by the formula of NodeManager score = weight 1*indicator 1+weight 2*indicator 2+weight 3*indicator 3+…+weight N*indicator N.

[0071] As another possible implementation, a basic score may be preset, and the sum of the products of each indicator and its corresponding weight may be multiplied by the basic score to obtain the scaling-down scoring result of the management node. For example, the scaling-down scoring result of the management node may be calculated by the formula of NodeManager score = (weight 1*indicator 1+weight 2*indicator 2+weight 3*indicator 3+…+weight N*indicator N)*basic score. The basic score is equivalent to normalizing the indicators and the products, so that the subsequent sorting is more referenceable.

[0072] The weight corresponding to the indicator can identify the percentage of the indicator in the overall indicator. The sum of the weights is usually 1. The above weights can be set according to the actual application scenario. In a possible embodiment, the weight of each indicator can also be set by the following steps:

[0073] S21. Obtaining a preset index record of each management node in a historical shrinking record, wherein the historical shrinking record includes a preset index record of each management node during historical shrinking and information of the management node actually shrunk;

[0074] S22, performing weighted scoring on the preset indicators of each management node based on the initial weights set for each preset indicator to obtain a weighted scoring result;

[0075] S23, determining the shrinking node that needs to be shrunk based on the preset shrinking condition and the weighted scoring results;

[0076] S24, determining the difference between the information of the shrunk node and the information of the actually shrunk management node, and updating the initial weight until the shrunk node is consistent with the actually shrunk management node;

[0077] S25. Determine that each weight when the shrunk node is consistent with the actually shrunk management node is a preset weight.

[0078] In the present invention, each time after the management node to be shrunk is determined, the shrinking situation can be stored in the historical shrinking record. As a possible implementation method, the final shrinking management node indicator information and identification information, the indicator information and identification information of each management node involved in the shrinking calculation, the shrinking time and the shrinking result, etc. can be stored. The identification information of the shrinking management node can be the name, IP or number of the management node that is actually shrunk, and the shrinking result can include success and failure.

[0079] In a possible embodiment, weights can be initialized for each preset indicator, and the initial weights can be set arbitrarily. Afterwards, for each shrinking situation record, weighted calculation can be performed according to the above weighted calculation formula using the initial weights and the indicator information of each management node in the shrinking, and the shrinking node can be selected based on the weighted calculation result. If the shrinking node is consistent with the actual shrinking node, it means that the weight setting is reasonable. If not, the initial weight can be updated.

[0080] As a possible implementation, one of the weights may be changed each time according to a preset fixed value and returned to S22 to recalculate the weighted scoring results of each management node. It is also possible to set a loss function and calculate the loss function value according to the difference between the indicator information of the shrunk node and the indicator information of the actual shrunk management node. The loss function may be a cross entropy loss, or the number of different indicator information, etc. The initial weight may be updated based on the difference. Specifically, the initial weight may be increased or decreased based on the cross entropy loss, or the weights of different indicators in the shrunk node and the actual shrunk management node may be changed. In other possible embodiments, the initial weight may also be updated by other parameter updating methods, which are not specifically limited in the present invention.

[0081] After scoring each management node, the management node list that needs to be scaled down can be screened based on the scoring results, that is, the management node list that needs to be deleted offline. In a possible embodiment, the management nodes that need to be scaled down can be determined by preset scaling conditions, which can include quantity conditions and score conditions.

[0082] Exemplarily, management nodes whose scaling-down scores are lower than a preset score threshold may be selected as the target management nodes; and / or,

[0083] The management nodes are scored in descending order of the scaling-down scoring results, and a preset number of management nodes first in the scaling-down scoring result order are selected as target management nodes.

[0084] The preset number of the above-mentioned scoring thresholds can be set according to the actual application scenario. For example, the preset number can be 5, 10, etc., that is, 5 or 10 management nodes need to be offline for each shrinking. Accordingly, after sorting the management nodes from low to high according to the shrinking score results, the top five or ten management nodes can be selected as the target management nodes, that is, the management nodes need to be offline.

[0085] The scoring threshold may be 50%, 30%, etc. of the basic score, that is, if the scoring result of the management node is lower than 50% or 30% of the basic score, it can be determined that the management node needs to be taken offline.

[0086] Taking the target management node offline means decommissioning the node from Yarn. In a possible embodiment, the target management node identifier may be added to an instruction so as to decommission the target management node from Yarn through a Yarn decommission instruction.

[0087] As a possible implementation method, the target management node identifier or IP contained in the target management node list can be added to a blacklist file, which is used to store node information that needs to be decommissioned. At the same time, the storage path of the blacklist file can also be configured in the ResourceManager file so that ResourceManager can also identify these nodes that need to be decommissioned.

[0088] You can then execute the refresh node command to refresh the node information in Yarn and ResourceManager based on the blacklist file to delete the management nodes that need to be decommissioned.

[0089] In a possible embodiment, there may be running tasks in the target management node. If the management node is directly decommissioned, it may cause abnormal operation of the tasks in the target management node. Therefore, the status of the target management node can be marked as retired in Yarn first, and if there are running tasks in the target management node, wait for the running tasks to complete and then delete the target management node.

[0090] As a possible implementation method, there is usually a status code during the running process of the task. If the status code indicates that the task in the target management node is running, you can wait for it to run. When the status code of the task indicates that the task has been completed, the target management node can be retired from Yarn, including deleting the target management node, releasing the resources occupied by the target management node, and deleting the node from the node list of Yarn.

[0091] In this way, by performing graceful decommissioning on the target management node, tasks such as Hive and Spark are not affected during the node offline period.

[0092] After the target management node is decommissioned from Yarn, the running container running the management node can be deleted from Kubernetes. Pod is the smallest management unit in Kubernetes, used to run actual tasks, and each pod has a unique IP address. The tasks of YARN applications are usually run as containers in Pod. Each Pod can contain one or more containers, each container represents a YARN task, and the containers in the Pod can share the same network and storage space.

[0093] Therefore, in a possible embodiment, when the target management node has been deleted in the Yarn cluster, the Kubernetes interface is called according to the IP address of the target management node to delete the pod corresponding to the IP address.

[0094] As a possible implementation method, after the target management node is offline in Yarn, the node list in Yarn and ResourceManager will be updated. Therefore, it is possible to determine whether the target management node has been successfully offline by querying the current node list. Specifically, if the target management node is found in the node list, it means that the target management node has not been successfully offline. If the target management node is not found in the node list, it means that the target management node has been successfully offline, and the corresponding pod can be deleted from Kubernetes.

[0095] As a possible implementation method, the IP address of the target management node can be obtained, and the IP address and deletion instructions can be sent to Kubernetes through the Kubernetes API interface. Kubernetes can delete the corresponding pod based on the IP address through the pod manager deployment, thereby completing the reduction of the target management node.

[0096] like Figure 2 As shown, Figure 2 A flowchart of a scaling-down method based on Yarn-Kubernetes provided in an embodiment of the present invention may include the following steps after starting the process:

[0097] S201. Query NodeManager information.

[0098] In this step, you can query the preset indicator information of each NodeManager through Yarn, including the resource utilization of the NodeManager and the number of tasks running.

[0099] S202: Score and rank each NodeManager.

[0100] In this step, various indicator information of the NodeManager may be weightedly calculated according to preset weights for different preset indicators to obtain scores of the NodeManager, and the NodeManagers may be sorted according to the scores.

[0101] S203, filter the reduction list.

[0102] In this step, each NodeManager can be screened according to the preset shrinking conditions to obtain the NodeManagers that need to be shrunk. Specifically, the preset number of NodeManagers with the lowest scores can be screened according to the above sorting to form a shrinking list, or the NodeManagers whose scores do not exceed the preset score threshold can be screened according to the above scoring to form a shrinking list. The shrinking list can include the preset indicator information of each NodeManager that needs to be shrunk and the NodeManager identifier, etc.

[0103] S204, Yarn takes NodeManager offline.

[0104] In this step, you can mark the NodeManager that needs to be scaled down with a retirement flag to indicate that the NodeManager needs to be offline, so as to avoid assigning new tasks to the NodeManager. If there are running tasks in the NodeManager that needs to be scaled down, you can wait for the task to complete before taking the NodeManager offline to achieve a graceful retirement of the NodeManager. If there are no running tasks in the NodeManager that needs to be scaled down, you can immediately take the NodeManager offline, specifically, you can delete the NodeManager.

[0105] S205. Reduce the capacity of the pod in k8s.

[0106] In this step, you can call the Kubernetes API interface to find the pod running the NodeManager based on the IP address of the offline NodeManager and delete the pod.

[0107] like Figure 3 As shown, Figure 3A schematic diagram of an architecture for implementing a Yarn-Kubernetes-based scaling-down method provided in an embodiment of the present invention. The architecture may include an automatic scaling-down service, a Yarn RM cluster, and a Kubernetes cluster. The Yarn RM cluster includes multiple resource managers, and the Kubernetes cluster includes multiple pod containers for running management node NodeManager.

[0108] The above-mentioned automatic scaling-down service may include an information collection module, a NodeManager scoring module, a NodeManager module and a Kubernetes common module, wherein the information collection module is used to collect preset indicator information of each resource manager in the Yarn RM cluster, and the preset indicator information may include indicator data such as CPU usage, memory usage, and the number of important tasks.

[0109] The NodeManager scoring module stores preset weights and corresponding scoring algorithms for each preset indicator information, and can calculate the scaling-down score of each NodeManager based on the weights and corresponding indicator information. The NodeManagers can be sorted based on the scaling-down score, and the NodeManager list that needs to be scaled down can be screened based on the preset quantity condition or score condition.

[0110] NodeManager module: It determines the node status of the relevant NodeManager nodes and calls YARN commands and interfaces to perform graceful retirement of the NodeManager. It does not affect the existing tasks that have been scheduled and can achieve lossless offline of the NodeManager.

[0111] The Kubernetes public module can provide pod query services. Specifically, it can query the pod information running the NodeManager in the Kubernetes cluster according to the IP address of each NodeManager that needs to be shrunk in the above NodeManager list, and delete these pods by calling the Kubernetes API interface.

[0112] The embodiment of the present invention is applied to deploy Kubernetes with Yarn, the resource manager of the Hadoop ecosystem, to develop an innovative scaling-down algorithm and system, which includes the definition of the scaling-down algorithm, supports the scoring and judgment of NodeManager with different indicators in multiple dimensions, reflects the scaling-down requirements in Yarn deployment of Kubernetes in a more standardized manner, and can scale down the Pod in k8s after the scoring is completed, thus solving the problem of automatic scaling-down after Yarn deployment of k8s. And by taking the node offline in an elegant retirement manner, the running stability of the tasks running in the node is guaranteed.

[0113] Based on the same inventive concept, according to another aspect of the present invention, a capacity reduction device based on Yarn-Kubernetes is provided, which is applied to a Yarn-Kubernetes cluster, such as Figure 4 As shown, the apparatus 400 may include:

[0114] The query module 401 is used to query the preset indicator information of each management node in the Yarn cluster, wherein the preset indicator information includes resource utilization and the number of tasks running in the management node;

[0115] A scoring module 402, configured to calculate a scoring result of scaling down of each management node based on a preset weight of each preset indicator information and each preset indicator information;

[0116] The determination module 403 is used to determine the target management node that needs to be shrunk based on the scaling-down scoring results of each management node according to a preset scaling-down condition; the preset scaling-down condition includes a scaling-down scoring threshold and / or a scaling-down quantity;

[0117] The shrinking module 404 is used to perform offline processing on each of the target management nodes in the Yarn cluster, and delete the running container where each of the target management nodes is located in the Kubernetes cluster.

[0118] In a possible embodiment, the preset weights are determined by the following steps:

[0119] Obtaining preset index records of each management node in the historical shrinking record, wherein the historical shrinking record includes the preset index records of each management node during the historical shrinking and the information of the management node actually shrunk;

[0120] Based on the initial weights set for each preset indicator, weighted scoring is performed on the preset indicators of each management node to obtain a weighted scoring result;

[0121] Determining the scaling-down nodes that need to be scaled down based on the preset scaling-down conditions and the weighted scoring results;

[0122] Determine the difference between the information of the shrunk node and the information of the actually shrunk management node to update the initial weight until the shrunk node is consistent with the actually shrunk management node;

[0123] The weights when determining that the shrunk node is consistent with the actually shrunk management node are preset weights.

[0124] In a possible embodiment, determining the target management node that needs to be shrunk based on the shrunk scoring result of each management node according to the preset shrunk condition includes:

[0125] Screening the management nodes whose scaling-down scores are lower than a preset score threshold, the target management nodes; and / or,

[0126] The management nodes are scored in descending order of the scaling-down scoring results, and a preset number of management nodes first in the scaling-down scoring result order are selected as target management nodes.

[0127] In a possible embodiment, the offline processing of each target management node in the Yarn cluster and the deleting of the running container where each target management node is located in the Kubernetes cluster include:

[0128] Mark the state of the target management node as retired in Yarn, and if there is a task running in the target management node, wait for the running task to complete before deleting the target management node;

[0129] In the case that the target management node has been deleted in the Yarn cluster, the Kubernetes interface is called according to the IP address of the target management node to delete the pod corresponding to the IP address.

[0130] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0131] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0132] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0133] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0134] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0135] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0136] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include, but is not limited to, a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0137] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, any of the above-described Yarn-Kubernetes-based scaling methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 may be configured to perform any of the above-described Yarn-Kubernetes-based scaling methods by any other appropriate means (e.g., by means of firmware).

[0138] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0142] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0143] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A scaling-down method based on Yarn-Kubernetes, characterized in that: Applied to a Yarn-Kubernetes cluster, the method includes: Query the preset indicator information of each management node in the Yarn cluster, the preset indicator information includes resource utilization and the number of tasks running in the management node; Calculate the scaling-down score result of each management node based on the preset weight of each preset indicator information and each preset indicator information; Determine the target management node that needs to be shrunk based on the shrunk scoring result of each management node according to the preset shrunk condition; the preset shrunk condition includes the shrunk scoring threshold and / or the shrunk quantity; Each of the target management nodes is taken offline in the Yarn cluster, and the running containers where each of the target management nodes is located are deleted in the Kubernetes cluster.

2. The method according to claim 1, characterized in that The preset weights are determined by the following steps: Obtaining preset index records of each management node in the historical shrinking record, wherein the historical shrinking record includes the preset index records of each management node during the historical shrinking and the information of the management node actually shrunk; Based on the initial weights set for each preset indicator, weighted scoring is performed on the preset indicators of each management node to obtain a weighted scoring result; Determining the scaling-down nodes that need to be scaled down based on the preset scaling-down conditions and the weighted scoring results; Determine the difference between the information of the shrunk node and the information of the actually shrunk management node to update the initial weight until the shrunk node is consistent with the actually shrunk management node; The weights when it is determined that the shrunk node is consistent with the actually shrunk management node are preset weights.

3. The method according to claim 1, characterized in that The step of determining the target management node that needs to be shrunk based on the shrunk scoring result of each management node according to the preset shrunk condition includes: Screening the management nodes whose scaling-down scores are lower than a preset score threshold, the target management nodes; and / or, The management nodes are scored in descending order of the scaling-down scoring results, and a preset number of management nodes first in the scaling-down scoring result order are selected as target management nodes.

4. The method according to claim 1, characterized in that The offline processing of each target management node in the Yarn cluster and deleting the running container where each target management node is located in the Kubernetes cluster includes: Mark the state of the target management node as retired in Yarn, and if there is a task running in the target management node, wait for the running task to complete before deleting the target management node; In the case that the target management node has been deleted in the Yarn cluster, the Kubernetes interface is called according to the IP address of the target management node to delete the pod corresponding to the IP address.

5. A capacity reduction device based on Yarn-Kubernetes, characterized in that: Applied to the Yarn-Kubernetes cluster, the device includes: A query module is used to query the preset indicator information of each management node in the Yarn cluster, wherein the preset indicator information includes resource utilization and the number of tasks running in the management node; A scoring module, used for calculating the scaling-down scoring result of each management node based on the preset weight of each preset indicator information and each preset indicator information; A determination module, configured to determine a target management node that needs to be shrunk based on a shrinkage scoring result of each management node according to a preset shrinkage condition; the preset shrinkage condition includes a shrinkage scoring threshold and / or a shrinkage quantity; The shrinking module is used to take each of the target management nodes offline in the Yarn cluster and delete the running containers where each of the target management nodes is located in the Kubernetes cluster.

6. The device according to claim 5, characterized in that The preset weights are determined by the following steps: Obtaining preset index records of each management node in the historical shrinking record, wherein the historical shrinking record includes the preset index records of each management node during the historical shrinking and the information of the management node actually shrunk; Based on the initial weights set for each preset indicator, weighted scoring is performed on the preset indicators of each management node to obtain a weighted scoring result; Determining the scaling-down nodes that need to be scaled down based on the preset scaling-down conditions and the weighted scoring results; Determine the difference between the information of the shrunk node and the information of the actually shrunk management node to update the initial weight until the shrunk node is consistent with the actually shrunk management node; The weights when it is determined that the shrunk node is consistent with the actually shrunk management node are preset weights.

7. The device according to claim 5, characterized in that The step of determining the target management node that needs to be shrunk based on the shrunk scoring result of each management node according to the preset shrunk condition includes: Screening the management nodes whose scaling-down scores are lower than a preset score threshold, the target management nodes; and / or, The management nodes are scored in descending order of the scaling-down scoring results, and a preset number of management nodes first in the scaling-down scoring result order are selected as target management nodes.

8. The device according to claim 5, characterized in that The offline processing of each target management node in the Yarn cluster and deleting the running container where each target management node is located in the Kubernetes cluster includes: Mark the state of the target management node as retired in Yarn, and if there is a task running in the target management node, wait for the running task to complete before deleting the target management node; In the case that the target management node has been deleted in the Yarn cluster, the Kubernetes interface is called according to the IP address of the target management node to delete the pod corresponding to the IP address.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-4.

Images