Resource allocation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a resource allocation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an operation index of each node in a target cluster; each node runs at least one component; analysis is carried out according to the operation indexes, current component distribution information and current resource allocation information of all the nodes are determined, and the current component distribution information is used for representing components operated in the nodes; according to the current component distribution information, adjusting a node where each component is located to obtain target component distribution information; and performing resource allocation on each component in combination with the target component distribution information and the current resource allocation information to obtain target resource allocation information. By adopting the method, the resource utilization rate can be improved.

Description

Technical Field

[0001] The present application relates to the field of distributed computing technology, and in particular to a resource allocation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In recent years, with the vigorous development of information technology and data science, the amount of data in various fields has shown explosive growth, which has promoted the widespread application of big data technology. Compared with traditional data processing, big data has become an important direction of current technological development due to its huge volume, diverse types, complex sources, and high processing timeliness requirements. In order to meet these challenges, a series of distributed computing technologies have emerged, including HDFS, MapReduce, Hive, HBase, and Spark. These technologies use distributed architecture to decompose tasks into smaller computing units and run them simultaneously on multiple nodes, achieving efficient data storage and processing, and providing users with rich functions and powerful performance support.

[0003] In a distributed computing environment, the proper allocation of resources is critical to system performance. The resources in a big data platform mainly include computing power, storage space, and network bandwidth. On these platforms, the operation of tasks is usually organized in the form of components, each of which is responsible for processing specific data or tasks. In order to improve parallelism and efficiency, components need to be distributed to various nodes in the cluster.

[0004] At present, the parameter configuration of distributed systems usually relies on manual experience. The system parameters are complex and influence each other. The manual tuning effect often cannot fully utilize the system resources. Summary of the invention

[0005] Based on this, it is necessary to provide a resource allocation method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve resource utilization in response to the above technical problems.

[0006] In a first aspect, the present application proposes a resource allocation method, the method comprising:

[0007] Collecting operating indicators of each node in the target cluster; each of the nodes runs at least one component;

[0008] Analyze according to the operation index to determine the current component distribution information and current resource allocation information of each node, wherein the current component distribution information is used to characterize the components running in the node;

[0009] According to the current component distribution information, adjusting the nodes where the components are located to obtain target component distribution information;

[0010] In combination with the target component distribution information and the current resource allocation information, resources are allocated to each of the components to obtain target resource allocation information.

[0011] In one embodiment, adjusting the nodes where each component is located according to the current component distribution information to obtain target node distribution information includes:

[0012] Get historical operating indicators;

[0013] Analyze the historical operation indicators to obtain component operation characteristics;

[0014] According to the component operation characteristics and the current component distribution information, the nodes where the components are located are adjusted to obtain the target component distribution information.

[0015] In one of the embodiments, the component operation characteristics include one or more of a component interaction mode, a component load characteristic, and a component behavior mode;

[0016] The component interaction mode is used to characterize whether there is data exchange between the components;

[0017] The component load characteristic is used to characterize the resource occupancy of the component during operation;

[0018] The component behavior pattern is used to characterize whether the component processes data-sensitive services.

[0019] In one embodiment, combining the target component distribution information and the current resource allocation information to allocate resources to each of the components to obtain the target resource allocation information includes:

[0020] Determining allocatable resources of each of the nodes according to the current resource allocation information;

[0021] Based on the preset step size, the target component distribution information and the allocatable resources of each of the nodes, resources are allocated to each of the components to obtain the target resource allocation information.

[0022] In one embodiment, allocating resources for each of the components based on the preset step size, the target component distribution information, and the allocatable resources of each of the nodes to obtain the target resource allocation information includes:

[0023] Based on the target component distribution information, determining the component allocated to each of the nodes;

[0024] According to the allocatable resources of each node and the preset step size, resources are allocated to each component allocated to each node in turn to obtain initial resource allocation information of each component;

[0025] When the initial resource allocation information of each component meets the allocation condition, the target resource allocation information is obtained.

[0026] In one embodiment, the method further comprises:

[0027] Verifying the target resource allocation information according to preset resource rationality verification conditions, and verifying the target component distribution information according to preset component rationality verification conditions;

[0028] Under the condition that the target resource allocation information and the target component distribution information are verified, the target resource allocation information and the target component distribution information are applied to the target cluster.

[0029] In a second aspect, the present application proposes a resource allocation method and apparatus, the apparatus comprising:

[0030] A collection module, used to collect operating indicators of each node in the target cluster; each of the nodes runs at least one component;

[0031] An analysis module, used to analyze according to the operation indicators to determine the current component distribution information and current resource allocation information of each node;

[0032] A component adjustment module, configured to adjust the distribution of each component in the target cluster according to the current component distribution information, to obtain target component distribution information;

[0033] The resource adjustment module is used to allocate resources to each of the components in combination with the target component distribution information and the current resource allocation information to obtain target resource allocation information.

[0034] In a third aspect, the present application proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0036] In a fifth aspect, the present application proposes a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0037] The above-mentioned resource allocation method, apparatus, computer equipment, computer-readable storage medium and computer program product collect and analyze the operating indicators of the target cluster, dynamically adjust the component distribution, and then allocate resources to each component by combining the target component distribution information and the current resource allocation information to obtain the target resource allocation information, which can effectively improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 An application environment diagram of a resource allocation method in an embodiment;

[0040] Figure 2 A schematic diagram of a flow chart of a resource allocation method in an embodiment;

[0041] Figure 3 A schematic diagram of target component distribution information in one embodiment;

[0042] Figure 4 A schematic diagram of resource allocation in another embodiment;

[0043] Figure 5 is a structural block diagram of a resource allocation device in an embodiment;

[0044] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The resource method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in the figure, the target cluster consists of multiple nodes, each of which has independent computing resources, such as CPU, memory, and storage, and is connected through a high-bandwidth network to achieve collaborative processing of tasks.

[0047] In the target cluster, each node runs multiple components, and each component performs a specific function, such as data storage, computing, or message processing. Components run on different nodes in a distributed manner, and their distribution and resource allocation directly affect the overall performance and stability of the cluster. Since the computing environment may change, for example, a component may migrate from node A to node B, the existing component distribution and resource allocation are difficult to adapt to the dynamic load requirements and resource utilization optimization requirements. Therefore, a resource allocation method is proposed in this application to optimize the distribution of components and the dynamic allocation of resources, achieve efficient utilization of cluster resources, improve overall operating performance, and ensure the stability and adaptability of the system.

[0048] In an exemplary embodiment, Figure 2 As shown, a resource allocation method is provided, including the following steps 202 to 208. Among them:

[0049] Step 202: collect the operating indicators of each node in the target cluster; each node runs at least one component.

[0050] Among them, the operating indicators are used to reflect the performance data of the current operating status of each node, including but not limited to CPU utilization, memory occupancy, network throughput and disk I / O; components refer to the basic functional units running in the target cluster, such as modules responsible for data storage, computing or message processing.

[0051] Optionally, deploy a metrics collection tool, such as Prometheus, in the target cluster to monitor the running status of each node in real time.

[0052] Optionally, the operation indicators of each node in the target cluster can be collected according to a preset time interval, where the preset time interval can be set according to the needs of the actual application scenario. For example, in a highly dynamic load scenario, the preset time interval can be set to a shorter second or minute level; while in a scenario with a relatively stable load change, it can be set to a longer hour level to balance the real-time monitoring and system overhead.

[0053] In this application, one or more components are run on each node in the target cluster, and different components perform specific functions, such as data storage, computing, or message processing. Each component runs on different nodes in a distributed manner, and its distribution and operation characteristics directly affect the resource utilization efficiency and overall performance of the target cluster. Therefore, by collecting operation indicators in real time or periodically, data support can be provided for subsequent resource optimization and component scheduling.

[0054] Step 204 , analyzing according to the operation index, determining the current component distribution information and current resource allocation information of each node, where the current component distribution information is used to characterize the components running in the node.

[0055] The component distribution information is used to indicate the distribution of each component, for example, component A is distributed on node A, and component B is distributed on node B.

[0056] Optionally, the amount of resources actually occupied by each component can be calculated by analyzing the operating indicators on the node, for example, the CPU utilization corresponds to the number of CPU cores actually occupied by the component.

[0057] Optionally, in a distributed environment, the operating indicators are usually accompanied by component identification information, such as container name, task ID, or service ID. By parsing the identification information in the operating indicators, a mapping relationship between the component and the corresponding node can be directly established. For example, if a certain operating indicator record shows that the CPU utilization is 80%, and it is identified that the indicator comes from task Task_A and the task runs on node Node_1, it can be inferred that component Task_A is currently running on node Node_1. Similarly, combined with other indicators (such as memory usage, I / O operations), the resource usage of Task_A can be further deduced, thereby obtaining the current resource allocation information.

[0058] Alternatively, in some distributed environments, a single running indicator may not be able to directly provide complete component distribution information, which can be achieved by analyzing multiple related indicators. For example, by combining the node's network traffic indicator and the task's input / output volume information, it can be determined that tasks with high-frequency data transmission are likely to run on the current node.

[0059] Optionally, in many distributed systems, component startup is recorded in logs or metadata storage, and operating indicators are only a supplement to these historical records. By combining historical startup logs (such as the last startup time and startup node of a component) with real-time operating indicators, the distribution information of the current component can be cross-verified. For example, if a log record shows that Component_X was recently started on node Node_2, and the current CPU usage of Node_2 matches the resource characteristics of Component_X, the distribution location of Component_X can be confirmed.

[0060] Optionally, a method based on running indicator pattern matching can also be used. For some distributed systems, the resource usage patterns of components have a high regularity, such as a specific computing component that regularly consumes peak CPU or a specific storage component that has a stable I / O pattern. By analyzing the time series characteristics of the running indicators, the actual indicators on the nodes can be matched with the predefined component patterns to determine which components run on which nodes.

[0061] Step 206: According to the current component distribution information, adjust the nodes where each component is located to obtain the target component distribution information.

[0062] The target component distribution information is used to characterize the distribution of components on each node after adjustment, including the specific components running on each node and their quantity distribution.

[0063] Optionally, the component interaction mode, load characteristics, and node resource conditions are determined from the current component distribution information, and the components on each node are migrated to obtain the target component distribution information.

[0064] Optionally, based on the interaction patterns between components, define affinity (e.g., component A and component B must run on the same node) or anti-affinity rules (e.g., component C and component D must run on different nodes) to reduce the communication overhead between high-frequency interacting components and avoid sharing nodes between components with high resource competition. For example, if the data interaction between two components is extremely high, they can be scheduled to the same node first to reduce network latency; while for components with high memory usage, they need to be distributed to different nodes to prevent resource contention.

[0065] Optionally, by analyzing the resource utilization of each node, non-critical components on high-load nodes can be migrated to low-load nodes to achieve balanced utilization of cluster resources. This approach usually uses load balancing algorithms, such as polling, least connections, or weighted distribution, to redistribute the load to each node. Especially in scenarios where cluster resources change dynamically, load balancing can respond quickly by monitoring the real-time resource utilization of nodes, such as transferring tasks on nodes with high CPU utilization to idle nodes.

[0066] Optionally, by globally analyzing the current state of component distribution and resource allocation, a new component distribution scheme is generated using an optimization algorithm (such as a genetic algorithm, simulated annealing algorithm, or integer linear programming). This approach usually aims to reduce the overall resource usage cost, reduce cross-node communication overhead, or improve resource utilization. For example, an objective function is defined with minimizing the communication delay between nodes as the optimization goal, and a new component distribution scheme is obtained through iterative optimization calculation, and the nodes where the components are located are adjusted accordingly.

[0067] Step 208: allocate resources to each component based on the target component distribution information and the current resource allocation information to obtain target resource allocation information.

[0068] The target resource allocation information is used to describe the resource reallocation of each component, including resource information such as the allocated CPU, memory, storage, and network bandwidth.

[0069] Optionally, resources can be allocated in a fixed ratio according to the priority or historical operation characteristics of the components according to the preset resource allocation rules. For example, for high-priority components, a higher resource ratio can be allocated to ensure their operating performance requirements; while low-priority components are allocated the remaining resources.

[0070] Optionally, you can analyze the historical operation data of a component to predict its future resource requirements and adjust resource allocation accordingly. For example, if the historical operation indicators of a component show that it usually requires an 8-core CPU and 16GB of memory under high load, you can combine the target component distribution information to prioritize node resources that meet this requirement to the component.

[0071] Optionally, reasonable resource allocation is achieved through optimization model construction and solution. The resource requirements of components and the resource capacity of nodes are abstracted into mathematical models. The optimization goals are to maximize the resource utilization of nodes and improve system performance. By building resource constraints, the optimal resource allocation plan is solved.

[0072] In the above resource allocation method, by collecting and analyzing the operating indicators of the target cluster, dynamically adjusting the component distribution, and then combining the target component distribution information and the current resource allocation information, resources are allocated to each component to obtain the target resource allocation information, which can effectively improve resource utilization.

[0073] In one embodiment, the above-mentioned adjusting the nodes where each component is located according to the current component distribution information to obtain the target node distribution information includes: obtaining historical operation indicators; parsing the historical operation indicators to obtain component operation characteristics; adjusting the nodes where each component is located according to the component operation characteristics and the current component distribution information to obtain the target component distribution information.

[0074] In this embodiment, the acquired historical operation indicators are parsed to obtain the operation characteristics of the component, and the component operation characteristics include one or more of component interaction mode, component load characteristics and component behavior mode.

[0075] Among them, the component interaction mode is used to characterize whether there is data exchange between components; for example, if the data transmission flow between component A and component B is high, it can be inferred that there is frequent data interaction between them, and they need to be deployed on the same node as much as possible during distribution to reduce communication overhead.

[0076] Among them, the component load characteristics are used to characterize the resource usage of the component during operation; for example, if the historical indicators of component C show that its CPU utilization has remained above 90% for a long time, it can be judged as a highly computing-intensive component and needs to be allocated to nodes with high computing power during distribution.

[0077] Among them, the component behavior mode is used to characterize whether the component processes data-sensitive services. For example, if a component frequently processes sensitive data during operation, it needs to be isolated and deployed to a specific security node to improve data security.

[0078] Optionally, based on the component operation characteristics obtained through analysis and combined with the current component distribution information, the distribution position of each component is adjusted to generate target distribution information.

[0079] Exemplarily, for components with high data interaction, such as component A and component B, they are preferentially adjusted to the same node to reduce cross-node communication overhead.

[0080] Exemplarily, it is distributed to nodes with sufficient resources to avoid resource contention.

[0081] Exemplarily, for components that process sensitive data, they are adjusted to isolated nodes to ensure data security.

[0082] Finally, based on the adjusted component distribution plan, the target component distribution information is generated to clarify the specific components and their quantity distribution on each node, providing a basis for subsequent resource allocation and operation optimization.

[0083] Exemplary, combined Figure 3 As shown, Figure 3 is a schematic diagram of component distribution information in an embodiment, Figure 3 The first line in the table is the components in the distributed system and their specific submodules or services, including HDFS (such as nn, zkfc, jn, dn), Yarn (such as rm, nm), Trino (such as tc, tw), HBase (such as hm, rs), etc. These represent the services or roles of each functional component in the system. The first column lists the nodes or roles in the cluster, such as master1, master2, and master3 are master nodes, core-n represents core nodes, and task-1 to task-n represent task nodes. Different components run on each node. The number 1 in the table indicates that the corresponding component runs on the node. For example: HDFS's nn, zkfc, jn, and HBase's hm are running on the master1 node.

[0084] In the above embodiments, by combining component operation characteristics and current distribution information, resource utilization efficiency can be significantly improved.

[0085] In one embodiment, the above-mentioned combination of target component distribution information and current resource allocation information performs resource allocation on each component to obtain target resource allocation information, including: determining the allocatable resources of each node according to the current resource allocation information; performing resource allocation on each component based on a preset step size, target component distribution information and allocatable resources of each node to obtain target resource allocation information.

[0086] First, determine the allocatable resources of each node based on the current resource allocation information. The allocatable resources of a node refer to the remaining portion after deducting the system reserved resources from the total resources of the current node. For example, if the total resources of a node are 16-core CPU and 64GB memory, and 12-core CPU and 48GB memory are currently allocated, then the allocatable resources of the node are 4-core CPU and 16GB memory. Through this process, accurate resource capacity information can be provided for subsequent resource allocation.

[0087] In this embodiment, the resource allocation problem is constructed as a multi-dimensional optimization problem, in which components are regarded as optimization dimensions, and the resource capacity of each node is modeled as a constraint. Specifically, the resource requirements of each component, such as CPU, memory, and network bandwidth, are quantified as a multi-dimensional vector of the component in the resource allocation model. These vectors define the demand intensity and priority of the components when allocating resources. The goal is to determine the resource allocation ratio of each component on each node through an optimization algorithm, which not only meets the resource requirements of the components, but also maximizes the resource utilization efficiency.

[0088] Optionally, in one embodiment, the above-mentioned resource allocation is performed for each component based on a preset step size, target component distribution information and allocatable resources of each node to obtain target resource allocation information, including: determining the component to be allocated to each node based on the target component distribution information; allocating resources to each component allocated to each node in turn according to the allocatable resources of each node and the preset step size to obtain initial resource allocation information for each component; and obtaining the target resource allocation information when the initial resource allocation information of each component meets the allocation conditions.

[0089] First, according to the target component distribution information, clarify which components are distributed on each node, for example, there are components 1, 2, and 3 on node A, and component 4 on node B. Then, according to the allocatable resources of the node, resources are gradually allocated to each component by preset step size. For example, the preset step size is 2-core CPU and 4GB memory, and resources are allocated to each component allocated to each node in turn to obtain resource allocation information for each component. Optionally, the components allocated to each node can be prioritized, and resources can be allocated to the components allocated to each node in turn according to the priority.

[0090] For example, the allocatable resources of node A are 6-core CPU and 12 GB memory, and the requirements of components 1, 2, and 3 are as follows:

[0091] Component 1: Requires a 3-core CPU and 4GB of memory;

[0092] Component 2: requires a 2-core CPU and 6GB of memory;

[0093] Component 3: Requires a 2-core CPU and 5GB of memory.

[0094] The resource allocation problem can be modeled as a constrained optimization problem. The goal here is to allocate the component requirements to nodes A so that the resource utilization is maximized within the total resource constraints while satisfying as many component requirements as possible.

[0095] In the allocation process, components are regarded as items, their resource requirements are regarded as the volume of items, and the allocatable resources of node A are regarded as the "capacity of the backpack". The allocation rule is to select combinations that meet the node resource constraints from the component set, giving priority to meeting the complete requirements of more components.

[0096] First round of allocation:

[0097] Component 1: First allocate 2 CPU cores and 4 GB memory (to meet some requirements). The remaining resources of node A are updated to 4 CPU cores and 8 GB memory.

[0098] Component 2: Allocate 2 CPU cores and 4 GB memory (partially meet the demand). The remaining resources of Node A are updated to 2 CPU cores and 4 GB memory.

[0099] Component 3: 2 CPU cores and 4 GB memory are allocated (partially meeting the requirements). The remaining resources of Node A are updated to 0 CPU cores and 0 GB memory.

[0100] In the second round of allocation, it is found that the CPU and memory of node A have been used up, but the requirements of some components are not fully met. Therefore, the allocated resources are gradually recycled to new components according to the demand balance.

[0101] Optionally, in the recycling strategy, check the matching degree between the resources allocated to each component in the first round and its demand. Give priority to recycling the resources of low-priority components or over-allocated resources, and release resources for components with higher demand or higher priority. For example:

[0102] Component 1 requires a 3-core CPU and 4 GB of memory, which are fully met. Resources will not be recycled for the time being.

[0103] Component 2 requires a 2-core CPU and 6 GB of memory, but currently only a 2-core CPU and 4 GB of memory are allocated. The requirements are not fully met and it is a high-demand component.

[0104] Component 3 requires a 2-core CPU and 5 GB of memory, but currently only 2-core CPU and 4 GB of memory are allocated. The requirements are not fully met, but the priority is low.

[0105] Some resources can be reclaimed from low-priority components. For example, 0.5 CPU cores and 1 GB of memory can be reclaimed from component 3 to prioritize the needs of component 2.

[0106] The reclaimed resources are reallocated to components with unmet needs based on priority.

[0107] Optionally, if resources are limited, you can reduce the preset length, for example, reduce the step size of 2-core CPU and 4GB memory to 1-core CPU and 2GB memory, and meet some of the needs of more components through more refined allocation.

[0108] The resource recovery and reallocation cycle is performed multiple times to gradually optimize the resource allocation plan until the needs of all components are met as much as possible, or the node resources are completely exhausted.

[0109] Optionally, weights can be assigned to each component according to priority, and component resources on each node can be allocated according to priority and weight. For example, a higher priority component is given a greater weight so that it can obtain more resources in resource allocation. For example, the weight of component 1 is 0.5, and the weights of components 2 and 3 are 0.25. When allocating resources, the allocation order and resource amount are adjusted according to the weight ratio. The allocation of resources to each component according to priority can refer to the description process in the above embodiment, which will not be repeated here.

[0110] Exemplary, combined Figure 4 As shown, Figure 4 This is a schematic diagram of resource allocation in an embodiment. First, the components on each node are determined, and then resources are reserved for the operating system first, and then allocated according to the minimum requirements of the service. If resources are insufficient, the distribution strategy will be readjusted to ensure that the service can meet basic requirements. Based on the initial allocation, the remaining resources are allocated to each component by gradually increasing resources. For example, CPU and memory are increased for key components first, and the allocation is rounded to optimize resource utilization.

[0111] In one embodiment, the above method also includes: verifying the target resource allocation information according to preset resource rationality verification conditions, and verifying the target component distribution information according to preset component rationality verification conditions; under the condition that the target resource allocation information and the target component distribution information are verified, applying the target resource allocation information and the target component distribution information to the target cluster.

[0112] In this embodiment, after generating the target resource allocation information and the target component distribution information, rationality verification is performed to ensure the feasibility and effectiveness of this information before it is applied to the target cluster. Among them, the preset resource rationality verification conditions refer to the standards used to check whether the target resource allocation information meets the system operation requirements, including but not limited to the following: whether the resource allocation of the node exceeds its actual available capacity, such as whether the allocation of CPU, memory or network bandwidth exceeds the remaining resources of the current node; whether the resource requirements of the component are met, especially whether the resource allocation of the key component meets the minimum requirements for operation; whether the overall resource allocation complies with the load balancing principle to avoid extreme situations where the resource utilization between nodes is too high or too low. Through these conditions, it can be ensured that the resource allocation plan can both meet the needs of the components and effectively utilize system resources.

[0113] Among them, the preset component rationality verification conditions refer to the standards used to verify whether the target component distribution information meets the component operation characteristics and business requirements, including the following: whether high-interaction components are allocated to nodes that are physically close or the same to reduce network communication overhead; whether high-load components are allocated to nodes with sufficient resources to avoid performance bottlenecks caused by insufficient resources; whether sensitive data processing components are isolated to specific security nodes to ensure data security. In addition, it is also necessary to check whether the component distribution complies with affinity or anti-affinity rules. For example, some specific components need to be distributed to the same node to achieve collaborative operation, or need to be distributed to different nodes to avoid resource competition.

[0114] Through the above verification conditions, the target resource allocation information and target component distribution information are checked separately to ensure that the generated solution meets the system requirements in terms of resource usage, component distribution, performance and security. After verification, this information can be officially applied to the target cluster. Specifically, according to the target resource allocation information, reallocate resources such as CPU, memory, storage and network bandwidth to the components; according to the target component distribution information, migrate and deploy the components from the original node to the new target node location. Ultimately, these verified resource allocation and component distribution solutions can significantly improve the overall performance and operational stability of the target cluster while meeting the specific needs of different business scenarios.

[0115] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0116] Based on the same inventive concept, the embodiment of the present application also provides a resource allocation device for implementing the resource allocation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more resource allocation device embodiments provided below can refer to the limitations on the resource allocation method above, and will not be repeated here.

[0117] In an exemplary embodiment, Figure 5 As shown, a resource device is provided, including: a collection module, an analysis module, a component adjustment module and a resource adjustment module, wherein:

[0118] The collection module is used to collect the operating indicators of each node in the target cluster; each of the nodes runs at least one component.

[0119] The analysis module is used to analyze according to the operation index to determine the current component distribution information and current resource information of each node.

[0120] The component adjustment module is used to adjust the distribution of each component in the target cluster according to the current component distribution information to obtain target component distribution information.

[0121] The resource adjustment module is used to combine the target component distribution information and the current resource information to adjust the resources of each component to obtain the target resource information.

[0122] In one embodiment, the component adjustment module includes:

[0123] The indicator acquisition unit is used to obtain historical operation indicators.

[0124] The indicator parsing unit is used to parse the historical operation indicators to obtain component operation characteristics.

[0125] The adjustment unit is used to adjust the nodes where each component is located according to the component operation characteristics and the current component distribution information to obtain the target component distribution information.

[0126] In one embodiment, the component characteristics in the above-mentioned indicator analysis unit include one or more of component interaction mode, component load characteristics and component behavior mode; wherein the component interaction mode is used to characterize whether there is data exchange between the components; wherein the component load characteristics are used to characterize the resource occupancy of the component during operation; wherein the component behavior mode is used to characterize whether the component processes data-sensitive services.

[0127] In one embodiment, the resource adjustment module includes:

[0128] The available resource determining unit is used to determine the available resources of each of the nodes according to the current resource information.

[0129] The resource unit is used to allocate resources for each of the components based on a preset step size, the target component distribution information and the available resources of each of the nodes to obtain the target resource information.

[0130] In one embodiment, the resource unit further includes:

[0131] The node component determination subunit is used to determine the component to each of the nodes based on the target component distribution information.

[0132] The component resource subunit is used to perform resource allocation for each component of each node in turn according to the available resources of each node and the preset step size, so as to obtain initial resource information of each component.

[0133] The resource judgment subunit is used to obtain the target resource information when the initial resource information of each component meets the conditions.

[0134] In one embodiment, the above device further includes a verification module, and the verification module further includes:

[0135] The rationality verification unit is used to verify the target resource information according to the preset resource rationality verification conditions, and to verify the target component distribution information according to the preset component rationality verification conditions.

[0136] An application unit is used to apply the target resource information and the target component distribution information to the target cluster under the condition that the target resource information and the target component distribution information are verified.

[0137] Each module in the resource allocation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0138] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store operating indicators. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a resource allocation method is implemented.

[0139] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0142] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A resource allocation method, characterized in that: The method comprises: Collecting operating indicators of each node in the target cluster; each of the nodes runs at least one component; Analyze according to the operation index to determine the current component distribution information and current resource allocation information of each node, wherein the current component distribution information is used to characterize the components running in the node; According to the current component distribution information, adjusting the nodes where the components are located to obtain target component distribution information; In combination with the target component distribution information and the current resource allocation information, resources are allocated to each of the components to obtain target resource allocation information.

2. The method according to claim 1, characterized in that The step of adjusting the nodes where the components are located according to the current component distribution information to obtain target node distribution information includes: Get historical operating indicators; Analyze the historical operation indicators to obtain component operation characteristics; According to the component operation characteristics and the current component distribution information, the nodes where the components are located are adjusted to obtain the target component distribution information.

3. The method according to claim 2, characterized in that The component operation characteristics include one or more of component interaction mode, component load characteristics and component behavior mode; The component interaction mode is used to characterize whether there is data exchange between the components; The component load characteristic is used to characterize the resource occupancy of the component during operation; The component behavior pattern is used to characterize whether the component processes data-sensitive services.

4. The method according to claim 1, characterized in that The combining the target component distribution information and the current resource allocation information to allocate resources to each of the components to obtain the target resource allocation information includes: Determining allocatable resources of each of the nodes according to the current resource allocation information; Based on the preset step size, the target component distribution information and the allocatable resources of each of the nodes, resources are allocated to each of the components to obtain the target resource allocation information.

5. The method according to claim 4, characterized in that The allocating resources for each of the components based on the preset step size, the target component distribution information, and the allocatable resources of each of the nodes to obtain the target resource allocation information includes: Based on the target component distribution information, determining the component allocated to each of the nodes; According to the allocatable resources of each node and the preset step size, resources are allocated to each component allocated to each node in turn to obtain initial resource allocation information of each component; When the initial resource allocation information of each component meets the allocation condition, the target resource allocation information is obtained.

6. The method according to claim 1, characterized in that The method further comprises: Verifying the target resource allocation information according to preset resource rationality verification conditions, and verifying the target component distribution information according to preset component rationality verification conditions; Under the condition that the target resource allocation information and the target component distribution information are verified, the target resource allocation information and the target component distribution information are applied to the target cluster.

7. A resource allocation method and device, characterized in that: The device comprises: A collection module, used to collect operating indicators of each node in the target cluster; each of the nodes runs at least one component; An analysis module, used to analyze according to the operation indicators to determine the current component distribution information and current resource allocation information of each node; A component adjustment module, configured to adjust the distribution of each component in the target cluster according to the current component distribution information, to obtain target component distribution information; The resource adjustment module is used to allocate resources to each of the components in combination with the target component distribution information and the current resource allocation information to obtain target resource allocation information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images