A method for dynamic resource management, an electronic device, and a computer-readable storage medium

Abstract

The present application discloses a method for dynamic resource management, an electronic device, and a computer-readable storage medium. The present application obtains a target resource operation data set for each service, where the target resource operation data set includes target resource operation data arranged according to the acquisition time sequence, and the target resource operation data represents the target resource usage amount when the big data cluster runs the service within a preset time period; determines a target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set, the target service pair includes a first service and a second service, and the overall change trend of the resource operation data during the running of the first service by the big data cluster is opposite to the overall change trend of the resource operation data during the running of the second service by the big data cluster; performs resource sharing processing on the initial configured resource data of the first service and the second service. Thereby, the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource management, and in particular, to a method for dynamically managing resources, an electronic device, and a computer-readable storage medium. Background Art

[0002] A big data cluster is a distributed computing architecture composed of multiple servers. It runs services by providing high storage capacity and powerful computing capabilities, such as online transaction processing, data mining, machine learning, etc. Currently, it is widely used in many fields such as finance, healthcare, and e-commerce. There are usually multiple services running in a big data cluster, and insufficient resource allocation for services will lead to a decrease in the service processing efficiency of the big data cluster.

[0003] Currently, when the business resources in a big data cluster are insufficiently allocated, virtual machines are usually created to expand the big data cluster, and then the reserved resources are added to the virtual machines. This method requires reserving resources for dynamic expansion, and the resources reserved before expansion are in an idle state, resulting in resource waste and low resource utilization rate. Summary of the Invention

[0004] The main technical problem to be solved by this application is to provide a method for dynamically managing resources, an electronic device, and a computer-readable storage medium, which can improve resource utilization rate.

[0005] To solve the above technical problem, this application provides a method for dynamically managing resources.

[0006] In one embodiment, a method for dynamically managing resources is applied to a big data cluster, and the method includes:

[0007] Obtain the target resource operation data sets of each service, where the target resource operation data in the target resource operation data sets includes target resource usage amounts when the big data cluster runs the service within a preset time period, arranged according to the acquisition time sequence;

[0008] Determine target service pairs from the services according to the data attributes of the target resource operation data in each target resource operation data set. The target service pairs include a first service and a second service, and the total change trend of the resource operation data during the big data cluster running the first service is opposite to the total change trend of the resource operation data during the big data cluster running the second service;

[0009] Perform resource sharing processing on the initial configured resource data of the first service and the second service.

[0010] In one embodiment, the data attribute includes the data change trend of the target resource operation data in the target resource operation data set, and the step of determining the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set includes:

[0011] Traverse each service, perform matching processing on the data change trend corresponding to the current service and the data change trends corresponding to other services to obtain a matching result; in response to the matching result indicating that the data change trends between the current service and the other services match, use the current service and the other services as alternative service pairs to obtain multiple alternative service pairs;

[0012] Determine the target service pair from the multiple alternative service pairs.

[0013] In one embodiment, the data change trend corresponding to the current service includes the data change sub-trends of the target resource operation data, and the step of performing matching processing on the data change trend corresponding to the current service and the data change trends corresponding to other services to obtain a matching result includes:

[0014] Determine the target value corresponding to each target resource operation data from a preset change trend mapping table according to the data change sub-trends of the target resource operation data in the current service, and each target resource operation data corresponds to a time series;

[0015] Calculate the numerical sum between the target values corresponding to the time series in the current service and the target values corresponding to the time series in the other services;

[0016] In response to the numerical sum being equal to a preset value, determine that the data change trends between the current service and the other services match.

[0017] In one embodiment, the step of determining the target service pair from the multiple alternative service pairs includes:

[0018] Traverse each alternative service pair, and determine whether one alternative service in the current alternative service pair has the same service type as one alternative service in the other alternative service pairs;

[0019] If so, perform duplicate removal processing on the current alternative service pair and the other alternative service pairs to obtain the target service pair.

[0020] In one embodiment, the step of performing duplicate removal processing on the current alternative service pair and the other alternative service pairs to obtain the target service pair includes:

[0021] Obtain the total amount of the first resource used when the big data cluster runs the current alternative service pair and the total amount of the second resource used when the big data cluster runs the other alternative service pairs;

[0022] Determine the maximum total resource usage from the total amount of the first resource usage and the total amount of the second resource usage;

[0023] Use the alternative service pair corresponding to the maximum total resource usage as the target service pair.

[0024] In one embodiment, the step of obtaining the target resource operation dataset of each service includes:

[0025] Obtain the initial resource operation data of the big data cluster when running each service;

[0026] Preprocess the initial resource operation data to obtain the preprocessed initial resource operation data, where the initial resource operation data includes sub-resource operation data of multiple resource types;

[0027] Perform weighted processing on the sub-resource operation data of each resource type in the initial resource operation data to obtain the target resource operation dataset of the corresponding service.

[0028] In one embodiment, the step of preprocessing the initial resource operation data to obtain the preprocessed initial resource operation data includes:

[0029] Clean the initial resource operation data to obtain the cleaned initial resource operation data;

[0030] Perform segmentation processing on the cleaned initial resource operation data according to the acquisition time sequence corresponding to each of them to obtain the initial resource operation dataset corresponding to each preset time period;

[0031] Use the largest initial resource operation data in each initial resource operation dataset as the preprocessed initial resource operation data.

[0032] In one embodiment, the step of performing resource sharing processing on the initial configured resource data of the first service and the second service includes:

[0033] In response to the total resource operation data required for the big data cluster to run the first service being less than the initial configured resource data of the first service, and the total resource operation data required for the big data cluster to run the second service being greater than the initial configured resource data, then call the initial configured resource data of the first service to run the second service.

[0034] To solve the above technical problems, the present application provides an electronic device, including a memory and a processor. The memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above-mentioned resource dynamic management method.

[0035] To solve the above technical problems, the present application provides a computer-readable storage medium, including: storing program data, which is used to implement the above-mentioned resource dynamic management method when executed by a processor.

[0036] The present application obtains a target resource operation data set for each service. The target resource operation data set includes target resource operation data arranged according to the collection time sequence, and the target resource operation data represents the target resource usage amount when the big data cluster runs services within a preset time period. According to the data attributes of the target resource operation data in each target resource operation data set, target service pairs are determined from the services. The target service pairs include a first service and a second service, and the overall change trend of the resource operation data during the running of the first service by the big data cluster is opposite to the overall change trend of the resource operation data during the running of the second service by the big data cluster. The initial configured resource data of the first service and the second service are subjected to resource sharing processing. Thus, target service pairs with complementary resources are determined through the data attributes of the target resource operation data in each target resource operation data set, and the first service and the second service in the target service pairs are subjected to resource sharing, so as to ensure that idle resources can be obtained from idle services to run services during the peak period of service operation, and the idle resources can be given to the services during the peak period of operation during the low period of service operation to run services, realizing resource sharing between services and improving resource utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings, where:

[0038] Figure 1 is a schematic flowchart of an exemplary embodiment of a resource dynamic management method shown in the present application;

[0039] Figure 2 is a schematic diagram of a resource curve corresponding to a service shown in the present application;

[0040] Figure 3 is a schematic diagram of resource curve LA and resource curve LC shown in the present application;

[0041] Figure 4It is a schematic diagram of the resource curve LA and the resource curve LD shown in this application;

[0042] Figure 5 It is Figure 1 a schematic flowchart of an exemplary embodiment of step S110 in the resource dynamic management method shown;

[0043] Figure 6 It is Figure 1 a schematic flowchart of an exemplary embodiment of step S120 in the resource dynamic management method shown;

[0044] Figure 7 It is a schematic diagram of the initial configured resource data of services in the big data cluster shown in this application;

[0045] Figure 8 It is a schematic diagram of the target resource operation data when the big data cluster runs services shown in this application;

[0046] Figure 9 It is a schematic diagram of performing resource sharing processing on the initial configured resource data of services shown in this application;

[0047] Figure 10 It is a block diagram of a resource dynamic management device shown in an exemplary embodiment of this application;

[0048] Figure 11 It is a schematic structural diagram of an embodiment of an electronic device provided by this application;

[0049] Figure 12 It is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by this application. Detailed Embodiments

[0050] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. It can be understood that the specific embodiments described herein are only used to explain this application, rather than limiting this application. Additionally, it should be noted that for the sake of description, only parts related to this application rather than all structures are shown in the accompanying drawings. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0051] First of all, it should be noted that a big data cluster is a distributed computing architecture composed of multiple servers. It runs services by providing high storage capacity and powerful computing capabilities, such as online transaction processing, data mining, machine learning, etc., and is currently widely used in multiple fields such as finance, healthcare, and e-commerce. Usually, multiple services run in a big data cluster, and insufficient resource allocation for services will lead to a reduction in the service processing efficiency of the big data cluster.

[0052] Currently, when the business resource allocation of a big data cluster is insufficient, virtual machines are usually created to expand the big data cluster, and then the reserved resources are added to the virtual machines. This method requires reserving the resources needed for dynamic expansion, and the resources reserved before expansion are in an idle state, resulting in resource waste and low resource utilization.

[0053] Based on this, the present application provides a method for dynamic resource management, an electronic device, and a computer-readable storage medium. For details, please refer to Figure 1 , Figure 1 is a schematic flowchart of an exemplary embodiment of a method for dynamic resource management shown in the present application.

[0054] The execution subject of a method for dynamic resource management can be a terminal device, a server, or other processing devices. Among them, the terminal device can be a computer, a mobile device, a terminal, a computing device, a vehicle-mounted device, etc. The execution subject of the method for dynamic resource management can also be a dynamic resource management device. In some possible implementation manners, the method for dynamic resource management can be implemented by a processor calling computer-readable instructions stored in a memory. The execution subject of the method for dynamic resource management can also be a big data cluster, which is a computer system architecture formed by connecting multiple computers through a network. The big data cluster can be deployed on a private cloud built by K8S (Kubernetes, a container orchestration engine).

[0055] Specifically, a method for dynamic resource management in this embodiment includes the following steps:

[0056] Step S110: Obtain the target resource operation data sets of each service. The target resource operation data sets include target resource operation data arranged in the acquisition time sequence, and the target resource operation data represents the target resource usage when the big data cluster runs services within a preset time period.

[0057] A service refers to using the computing and storage capabilities of a big data cluster to process and analyze large-scale data. For example, a service can be storing images, data query, etc. A service can include multiple sub-services, and the sub-services can be zookeeper (an open-source distributed coordination service), hdfs (Hadoop Distributed File System), yarn (Yet Another Resource Negotiator), spark (an open-source distributed coordination service), hive (a data warehouse tool based on Hadoop), an open-source distributed stream processing platform Kafka (Apache Kafka), etc.

[0058] The target resource operation dataset is the target resource operation data during the operation of the big data cluster arranged according to the acquisition time sequence. For example, the target resource operation dataset includes the target resource operation data collected at 1:01, the target resource operation data collected at 1:01:15, the target resource operation data collected at 1:01:30, and so on.

[0059] The target resource operation data is the target resource usage amount during the operation of the big data cluster within a preset time period. The target resource usage amount is the resource used by the big data cluster during operation. The target resource operation data can be CPU (Central Processing Unit) data, storage data, network IO (Input / Output) data, disk IO data, etc.

[0060] The resource dynamic management device obtains the target resource operation datasets of each service. Specifically, the resource dynamic management device collects the target resource operation data during the operation of the big data cluster at preset time intervals; arranges the collected target resource operation data according to the acquisition time sequence to form the target resource operation dataset. For example, the preset time interval can be 15 seconds.

[0061] Step S120: Determine the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation dataset. The target service pair includes a first service and a second service. The total change trend of the resource operation data during the operation of the first service by the big data cluster is opposite to the total change trend of the resource operation data during the operation of the second service by the big data cluster.

[0062] The data attribute is a data change trend characterizing the resource operation data. The data attribute can be the size change of the target resource operation data compared to the previous target resource operation data except for the first target resource operation data, such as an increase, decrease, or no change in the data. The data attribute can also be the change trend of the resource curve drawn based on the target resource operation data, such as no change, rising, or falling of the resource curve. As an example, the resource dynamic management device marks points in a preset coordinate system with each target resource operation data as the ordinate and the acquisition time sequence corresponding to each target resource operation data as the abscissa, connects the points to form a resource curve, and takes the change trend of the resource curve as the data attribute of the target resource operation data.

[0063] The overall change trend of resource operation data refers to the overall data change trend of the target resource operation data. The overall change trend of resource operation data can be that the resource operation data generally shows an increasing trend. For example, the overall change trend of resource operation data can be gradually increasing, or remaining unchanged first, then gradually increasing, and then remaining unchanged. The overall change trend of resource operation data can also be that the resource operation data generally shows a decreasing trend. For example, the overall change trend of resource operation data can be gradually decreasing, or remaining unchanged first, then gradually decreasing, and then remaining unchanged. The overall change trend of resource operation data can also be increasing first and then decreasing, or decreasing first and then increasing.

[0064] The overall change trends of the resource operation data of two services being opposite can be that the target resource operation data of one service shows an increasing trend while the target resource operation data of the other service shows a decreasing trend. The overall change trends of the resource operation data of two services being opposite can also be that the target resource operation data of one service increases first and then decreases, while the target resource operation data of the other service decreases first and then increases.

[0065] A target service pair refers to at least two services selected from the services running in the big data cluster. A target service pair can include a first service and a second service. A target service pair can also include a first service, a second service, and a third service.

[0066] The resource dynamic management device determines the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set. Specifically, the data attribute can be the change trend of the resource curve drawn based on the target resource operation data. The resource dynamic management device determines the corresponding array from the preset resource curve change trend mapping table according to the change trend of the resource curve; compares the sum of any two arrays with a preset value to obtain a comparison result; if the comparison result indicates that the sum of the two arrays is equal to the preset value, the two services corresponding to the target resource operation data of the two arrays are determined as the target service pair. In this way, the data change trends of the services in the target service pair are matched in each time period, so that resource complementarity can be achieved during resource sharing, thus enabling full utilization of resources and saving resources.

[0067] In one embodiment, the resource curve drawn based on each target resource operation data is as Figure 2 shown, in Figure 2Among them, the resource curves plotted according to the target resource operation data of Service A, Service B, Service C, and Service D in the target resource operation data set of the operation data are the resource curve LA, the resource curve LB, the resource curve LC, and the resource curve LD respectively. In the preset resource curve change trend mapping table, the corresponding value for unchanged is 0, the corresponding value for rising is 1, and the corresponding value for falling is -1. The change trends of each time period of the resource curve LA are: unchanged, falling, and unchanged in sequence, so the corresponding array is [0, -1, 0]; the change trends of each time period of the resource curve LB are: unchanged, unchanged, and unchanged, so the corresponding array is [0, 0, 0]. The change trends of each time period of the resource curve LC are: unchanged, rising, and unchanged, so the corresponding array is [0, 1, 0]. The change trends of each time period of the resource curve LD are: unchanged, rising, and unchanged, so the corresponding array is [0, 1, 0]. Comparing the sum of any two arrays with the value 0, the arrays with a sum of 0 are the [0, -1, 0] corresponding to the resource curve LA and the [0, 1, 0] corresponding to the resource curve LC, and the [0, -1, 0] corresponding to the resource curve LA and the [0, 1, 0] corresponding to the resource curve LD; the service A corresponding to the resource curve LA and the service C corresponding to the resource curve LC are determined as alternative service pairs, as shown in Figure 3 shown. The service A corresponding to the resource curve LA and the service D corresponding to the resource curve LD are determined as alternative service pairs, as shown in Figure 4 shown; select the target service pair from the alternative service pairs.

[0068] Step S130: Perform resource sharing processing on the initial configured resource data of the first service and the second service.

[0069] The initial configured resource data refers to the starting resource data configured for the service in the big data cluster.

[0070] Resource sharing processing means sharing the initial configured resource data of the first service with the second service, or sharing the initial configured resource data of the second service with the first service.

[0071] The resource dynamic management device performs resource sharing processing on the initial configured resource data of the first service and the second service. As an example, the big data cluster includes multiple physical nodes, and the service schedules the initial configured resource data through the physical nodes to realize service operation. The resource dynamic management device adjusts the first service and the second service to the same physical node to realize resource sharing between the first service and the second service.

[0072] It can be seen that by obtaining the target resource operation data sets of each service, the target resource operation data in the target resource operation data sets includes the target resource operation data arranged according to the acquisition time sequence, and the target resource operation data represents the target resource usage amount when the big data cluster runs services within a preset time period; determining the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set, the target service pair includes a first service and a second service, and the total change trend of the resource operation data during the operation of the first service by the big data cluster is opposite to the total change trend of the resource operation data during the operation of the second service by the big data cluster; performing resource sharing processing on the initial configured resource data of the first service and the second service. Thus, on the one hand, by determining the target service pair with complementary resources according to the data attributes of the target resource operation data in each target resource operation data set, and sharing the resources of the first service and the second service in the target service pair, it is ensured that idle resources can be obtained from the service with idle resources for service operation during the peak period of service operation, and redundant resources can be obtained from the service with sufficient resources for service operation during the low period of service operation, realizing resource sharing between services, improving resource utilization rate, and at the same time avoiding the risk of data loss caused by adding or reducing external resources. On the other hand, by sharing the initial configured resource data between the services of the big data cluster, dynamic resource management of the big data cluster is realized.

[0073] Based on the above embodiments, please continue to refer to Figure 5 , Figure 5 which Figure 1 is a schematic flowchart of an exemplary embodiment of step S110 in the resource dynamic management method shown. Specifically, the process of step S110 for obtaining the target resource operation data sets of each service includes the following steps:

[0074] Step S510, obtaining the initial resource operation data of the big data cluster when running each service.

[0075] The initial resource operation data refers to the resource data when the big data cluster runs each service.

[0076] The resource dynamic management device obtains the initial resource operation data of the big data cluster when running each service. As an example, the resource dynamic management device retrieves the initial resource operation data of the big data cluster when running each service from a preset resource data repository. As another example, the resource dynamic management device collects the resource usage amount of the big data cluster when running each service, and takes the resource usage amount of each service as the initial resource operation data of the big data cluster when running each service.

[0077] Step S520, preprocessing the initial resource operation data to obtain the preprocessed initial resource operation data, and the initial resource operation data includes sub-resource operation data of multiple resource types.

[0078] Preprocessing refers to cleaning or deduplicating data.

[0079] Resource type refers to the type to which the resource operation data belongs. The resource type can be CPU type, storage type, network IO type, and disk IO type.

[0080] The resource dynamic management device preprocesses the initial resource operation data to obtain the preprocessed initial resource operation data. As an example, the resource dynamic management device cleans the initial resource operation data to obtain the preprocessed initial resource operation data. By cleaning, the resource dynamic management device removes jitter data and invalid data, which is beneficial to improving the quality of the target resource operation data set. As another example, the resource dynamic management device cleans the initial resource operation data to obtain the cleaned initial resource operation data; segments the data according to the acquisition time sequence corresponding to each cleaned initial resource operation data to obtain the initial resource operation data sets corresponding to each preset time period; and uses the largest initial resource operation data in each initial resource operation data set as the preprocessed initial resource operation data.

[0081] In one embodiment, the initial resource operation data includes initial sub-resource data of multiple resource types, as follows:

[0082]

[0083] Where M represents the matrix of the initial resource operation data, represents the first initial sub-resource data of the first resource type, represents the th initial sub-resource data of the first resource type, represents the th first initial sub-resource data of the th resource type, represents the th initial sub-resource data of the

[0084] In one embodiment, the resource dynamic management device segments the time sequence at preset intervals to obtain multiple time periods. For example, the preset interval can be 6 hours. Classify each cleaned initial resource operation data according to the time period to which the acquisition time sequence belongs to obtain the initial resource operation data sets corresponding to each preset time period, and use the largest initial resource operation data in each initial resource operation data set as the preprocessed initial resource operation data to achieve the merging of the initial resource operation data in the initial resource operation data set. The preprocessed initial resource operation data includes sub-resource operation data of multiple resource types, as follows:

[0085]

[0086] Among them, represents the matrix of the initial resource operation data after preprocessing, represents the sub-resource operation data of the first resource type in the time period, represents the sub-resource operation data of the first resource type in the time period, represents the sub-resource operation data of the k-th resource type in the time period, represents the sub-resource operation data of the k-th resource type in the time period.

[0087] Furthermore, the resource dynamic management device classifies the initial resource operation data after cleaning according to a preset quantity, and obtains a plurality of initial resource operation data sets, and one initial resource operation data set corresponds to one time period.

[0088] Step S530: Perform weighted processing on the sub-resource operation data of each resource type in the initial resource operation data to obtain a target resource operation data set corresponding to the service.

[0089] The resource dynamic management device performs weighted processing on the sub-resource operation data of each resource type in the initial resource operation data to obtain a target resource operation data set corresponding to the service. Specifically, the resource dynamic management device determines the corresponding weight value according to the resource type in the preset weight mapping table, and the preset weight mapping table stores the one-to-one correspondence between the preset resource type and the preset weight value; calculate the total resource data of the sub-resource operation data, the weight value corresponding to the sub-resource operation data, and the resource type of the sub-resource operation data in each time period to obtain the target resource operation data in each time period, and aggregate the target resource operation data in each time period to form a target resource operation data set for the service.

[0090] In one embodiment, the sub-resource operation data, the weight value corresponding to the sub-resource operation data, the total resource data of the resource type of the sub-resource operation data, and the target resource operation data satisfy the following formula:

[0091]

[0092] Among them, represents the target resource operation data in the time period, represents the weight value corresponding to the k-th resource type, represents the sub-resource operation data of the k-th resource type in the time period, represents the total number of resource types, Represents the total resource data of the k-th resource type in the big data cluster.

[0093] In one embodiment, the target resource operation data of each time period and the target resource operation data set of the service satisfy the following formula:

[0094]

[0095] Wherein, Represents the target resource operation data set, Represents The target resource operation data of the time period, Represents The target resource operation data of the time period, Represents The target resource operation data of the time period, Represents The target resource operation data of the time period.

[0096] It can be seen that the initial resource operation data of each service running in the big data cluster is obtained, the initial resource operation data is preprocessed to obtain the preprocessed initial resource operation data, and the sub-resource operation data of each resource type in the initial resource operation data is weighted to obtain the target resource operation data set corresponding to the service. The data quality of the target resource operation data set is improved through preprocessing, and the quality of the target resource operation data set is further improved through weighting.

[0097] Based on the above embodiment, please continue to refer to Figure 6 , Figure 6 For Figure 1 A schematic flowchart of an exemplary embodiment of step S120 in the resource dynamic management method shown. Specifically, the data attribute includes the data change trend of the target resource operation data in the target resource operation data set, and the process of step S120 determining the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set includes the following steps:

[0098] Step S610, traverse each service, match the data change trend corresponding to the current service with the data change trends corresponding to other services, and obtain a matching result.

[0099] The data change trend includes the data change sub-trend of the target resource operation data.

[0100] The matching result includes the data change trend matching between the current service and other services, and the data change trend mismatch between the current service and other services.

[0101] The resource dynamic management device performs matching processing on the data change trend corresponding to the current service and the data change trend corresponding to other services to obtain a matching result. Specifically, the resource dynamic management device determines the target value corresponding to each target resource operation data from a preset change trend mapping table according to the data change sub-trend of each target resource operation data in the current service, and each target resource operation data corresponds to a time series; the preset change trend mapping table stores the one-to-one correspondence between the preset data change sub-trend and the preset target value; calculate the sum of the target values corresponding to each time series in the current service and the target values corresponding to the time series in other services; in response to the sum of the values being equal to the preset value, it is determined that the data change trends between the current service and other services match. The preset value can be 0. For example, if the sum of the values is equal to 0, it is determined that the data change trends between the current service and other services match, and if the sum of the values is greater than 0 or less than 0, it is determined that the data change trends between the current service and other services do not match.

[0102] Step S620, in response to the matching result indicating that the data change trends between the current service and other services match, the current service and other services are used as alternative service pairs to obtain multiple alternative service pairs.

[0103] The resource dynamic management device, in response to the matching result indicating that the data change trends between the current service and other services match, uses the current service and other services as alternative service pairs to obtain multiple alternative service pairs. For example, if the matching result indicates that the data change trends between service E and service F match, then service E and service F are used as an alternative service pair.

[0104] Step S630, determine the target service pair from multiple alternative service pairs.

[0105] The resource dynamic management device determines the target service pair from multiple alternative service pairs. Specifically, traverse each alternative service pair, and determine whether the service type of an alternative service in the current alternative service pair is the same as that of an alternative service in other alternative service pairs; if so, perform duplicate removal processing on the current alternative service pair and other alternative service pairs to obtain the target service pair, and if not, use the current alternative service pair as the target service pair. For example, the current alternative service pair includes service A and service C, and other alternative service pairs include service A and service D. It is determined that the service A in the current alternative service pair is of the same service type as the service A in other alternative service pairs, then duplicate removal processing is performed on the current alternative service pair and other alternative service pairs, and the target service pair is selected from the current alternative service pair and other alternative service pairs.

[0106] The resource dynamic management device performs deduplication processing on the current alternative service pair and other alternative service pairs to obtain the target service pair. Specifically, the resource dynamic management device obtains the first total resource usage amount when the big data cluster runs the current alternative service pair and the second total resource usage amount when the big data cluster runs other alternative service pairs; determines the maximum total resource usage amount from the first total resource usage amount and the second total resource usage amount; and takes the alternative service pair corresponding to the maximum total resource usage amount as the target service pair.

[0107] The first total resource usage amount refers to the sum of the target resource operation data of all services in the current alternative service pair. The first total resource usage amount can be the sum of the target resource operation data of all services in the current alternative service pair over all time periods, or it can be the maximum value of the sum of the target resource operation data of all services in the current alternative service pair over a certain time period.

[0108] The second total resource usage amount refers to the sum of the target resource operation data of all services in other alternative service pairs. The second total resource usage amount can be the sum of the target resource operation data of all services in other alternative service pairs over all time periods, or it can be the maximum value of the sum of the target resource operation data of all services in other alternative service pairs over a certain time period.

[0109] The resource dynamic management device determines the maximum total resource usage amount from the first total resource usage amount and the second total resource usage amount. Specifically, the resource dynamic management device compares the first total resource usage amount and the second total resource usage amount. If the first total resource usage amount is greater than or equal to the second total resource usage amount, the first total resource usage amount is determined as the maximum total resource usage amount; otherwise, the second total resource usage amount is determined as the maximum total resource usage amount.

[0110] It can be seen that by traversing each service, matching the data change trend corresponding to the current service with the data change trend corresponding to other services to obtain a matching result; in response to the matching result indicating that the data change trends between the current service and other services match, taking the current service and other services as alternative service pairs to obtain multiple alternative service pairs; and determining the target service pair from the multiple alternative service pairs. Thus, on the one hand, the target service pair is selected from the services, and on the other hand, through the matching process, the current service and other services with matching data change trends are obtained, and based on this, the current service and other services with complementary resources are obtained, so as to realize the resource sharing of the initial configuration resource data of subsequent services.

[0111] Specifically, Figure 1In the resource dynamic management method shown above, the process of performing resource sharing processing on the initial configured resource data of the first service and the second service in step S130 includes: in response to the total resource operation data required for the big data cluster to run the first service being less than the initial configured resource data of the first service, and the total resource operation data required for the big data cluster to run the second service being greater than the initial configured resource data, the initial configured resource data of the first service is called to run the second service.

[0112] The total resource operation data required to run the first service may be the target resource operation data when all sub-services in the first service run simultaneously. Exemplarily, the resource dynamic management device respectively collects the target resource operation data when the big data cluster runs each sub-service in the first service, and takes the sum of the target resource operation data of each sub-service in the first service as the total resource operation data required to run the first service. The total resource operation data required to run the first service may also be the sum of the target resource operation data in the target resource operation data set.

[0113] The total resource operation data required to run the second service may be the target resource operation data when all sub-services in the second service run simultaneously. Exemplarily, the resource dynamic management device respectively collects the target resource operation data when the big data cluster runs each sub-service in the second service, and takes the sum of the target resource operation data of each sub-service in the second service as the total resource operation data required to run the second service. The total resource operation data required to run the second service may also be the sum of the target resource operation data in the target resource operation data set.

[0114] The resource dynamic management device calls the initial configured resource data of the first service to run the second service. Exemplarily, the resource dynamic management device generates an instruction to call the initial configured resource data of the first service to run the second service, and sends the call instruction to the K8S platform deployed by the resource dynamic management device. After receiving the call instruction, K8S searches for the corresponding initial configured resource data in the preset resource database according to the first service, and configures the found initial configured resource data for the second service.

[0115] The resource dynamic management device performs resource sharing processing on the initial configured resource data of the first service and the second service. Specifically, in response to the total resource operation data required for the big data cluster to run the first service being greater than the initial configured resource data of the first service, and the total resource operation data required for the big data cluster to run the second service being less than the initial configured resource data, the initial configured resource data of the second service is called to run the first service.

[0116] Further, after the resource dynamic management device performs resource sharing processing on the initial configured resource data of the first service and the second service, the first service and the second service are on the same physical node. The resource dynamic management device determines the sum of the initial configured resource data of the first service and the second service as the total configured resource data available at the physical node; determines the resource usage permission of the first service in the total configured resource data according to the target resource operation data of the first service; and determines the resource usage permission of the second service in the total configured resource data according to the target resource operation data of the second service.

[0117] The resource dynamic management device determines the resource usage permission of the first service in the total configured resource data according to the target resource operation data of the first service. Specifically, the resource usage permission includes a resource usage upper limit and a resource usage lower limit. The resource dynamic management device determines the sum of the maximum target resource operation data in the target resource operation data of the first service and the preset redundancy value as the resource usage upper limit of the first service in the total configured resource data; and determines the minimum target resource operation data in the target resource operation data of the first service as the resource usage lower limit of the first service in the total configured resource data. For example, the resource usage upper limit of the resource dynamic management device can be , represents the maximum target resource operation data of the first service, represents the preset redundancy value; the resource usage lower limit of the resource dynamic management device can be , represents the minimum target resource operation data of the first service.

[0118] The resource dynamic management device determines the resource usage permission of the second service in the total configured resource data according to the target resource operation data of the second service. Specifically, the resource usage permission includes a resource usage upper limit and a resource usage lower limit. The resource dynamic management device determines the sum of the maximum target resource operation data in the target resource operation data of the second service and the preset redundancy value as the resource usage upper limit of the second service in the total configured resource data; and determines the minimum target resource operation data in the target resource operation data of the second service as the resource usage lower limit of the second service in the total configured resource data.

[0119] It can be seen that by invoking the initial configured resource data of the first service to run the second service, resource sharing of the initial configured resource data of the first service and the second service is realized, so that when the initial configured resource data of the second service is insufficient, the redundant initial configured resource data of the first service can be shared, improving resource utilization.

[0120] In one embodiment, a POD (container group, the smallest deployment unit) in the Kubernetes container orchestration platform can run services in a big data cluster, such as Figure 7As shown, Service A, Service B, Service C, and Service D in the big data cluster are respectively configured with initial configuration resource data in the POD. After the big data cluster runs Service A, Service B, Service C, and Service D for a preset period of time, the load conditions of the big data cluster running each service are different, and the usage conditions of the initial configuration resource data are also different in different time periods. The resource dynamic management device obtains the target resource operation data in the target resource operation data set of each service, and the target resource operation data represents the target resource usage amount when the big data cluster runs the service within the preset period of time. As Figure 8 shown, in the big data cluster, the target resource operation data of Service A in the first time period is A1, the target resource operation data of Service A in the second time period is A2, and the target resource operation data of Service B at all times is B1; the target resource operation data of Service C in the first time period is C1, and the target resource operation data of Service C in the second time period is C2; the target resource operation data of Service D in the first time period is D1, and the target resource operation data of Service D in the second time period is D2. The resource dynamic management device determines the target service pair from Service A, Service B, Service C, and D according to the data attributes of the target resource operation data in the target resource operation data set of Service A, B, C, and D. The determined target service pair includes Service A and Service C; the initial configuration resources of Service A and Service C are processed for resource sharing. Figure 9 is the resource allocation situation of Service A, B, C, and D. As Figure 9 shown, in the big data cluster, the initial configuration resource data of Service B and Service D remains unchanged. In the first time period, Service A calls the redundant initial configuration resource data of Service C in the business low peak period during the business peak period, improving the operation efficiency of Service A during the business peak period. In the second time period, Service A shares the redundant initial configuration resource data with Service C during the business low peak period, improving the operation efficiency of Service C during the business peak period. Thus, all resources in the big data cluster can be fully utilized at all times, and it is ensured that there are sufficient resources for the service to use during the business peak period, improving the operation efficiency of the big data cluster.

[0121] Figure 10 is the block diagram of the resource dynamic management device shown in an exemplary embodiment of the present application. As Figure 10 shown, the exemplary resource dynamic management device 1000 includes: a target resource operation data set acquisition module 1010, a target service pair determination module 1020, and a resource sharing module 1030. Specifically:

[0122] The target resource operation data set acquisition module 1010 is used to acquire the target resource operation data set of each service. The target resource operation data set includes the target resource operation data arranged according to the acquisition time sequence, and the target resource operation data represents the target resource usage amount when the big data cluster runs the service within the preset period of time.

[0123] A target service determination module 1020 is configured to determine a target service pair from services according to data attributes of target resource operation data in each target resource operation data set. The target service pair includes a first service and a second service, and the overall change trend of the resource operation data during the operation of the first service by the big data cluster is opposite to the overall change trend of the resource operation data during the operation of the second service by the big data cluster.

[0124] A resource sharing module 1030 is configured to perform resource sharing processing on the initial configured resource data of the first service and the second service.

[0125] In this exemplary resource dynamic management device, by obtaining the target resource operation data sets of each service, the target resource operation data set includes target resource operation data arranged according to the acquisition time sequence, and the target resource operation data represents the target resource usage amount when the big data cluster runs the service within a preset time period; determining a target service pair from services according to the data attributes of the target resource operation data in each target resource operation data set, the target service pair includes a first service and a second service; performing resource sharing processing on the initial configured resource data of the first service and the second service. Thus, a target service pair with complementary resources is determined through the data attributes of the target resource operation data in each target resource operation data set, and the first service and the second service in the target service pair are resource-shared, so as to ensure that idle resources can be obtained from the service with idle resources for service operation during the peak period of service operation, and redundant resources can be obtained from the service with sufficient resources for service operation during the low period of service operation, realizing resource sharing between services and improving resource utilization rate.

[0126] Among them, the functions of each module can be referred to the embodiments of the resource dynamic management method, which will not be elaborated here.

[0127] To implement the resource dynamic management method of the above embodiments, the present application proposes another electronic device. For details, please refer to Figure 11 , Figure 11 which is a schematic structural diagram of an embodiment of the electronic device provided by the present application.

[0128] The electronic device 1100 includes a memory 1101 and a processor 1102. Among them, the memory 1101 and the processor 1102 are coupled.

[0129] The memory 1101 is used to store program data, and the processor 1102 is used to execute the program data to implement the resource dynamic management method of the above embodiments.

[0130] In this embodiment, the processor 1102 can also be referred to as a CPU (Central Processing Unit). The processor 1102 may be an integrated circuit chip with signal processing capabilities. The processor 1102 can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 1102 can also be any conventional processor, etc.

[0131] This application also provides a computer-readable storage medium, such as Figure 12 As shown, the computer-readable storage medium 1200 is used to store program data 1201. When the program data 1201 is executed by the processor, it is used to implement the resource dynamic management method in the method embodiment of this application.

[0132] Based on the method involved in the method embodiment of resource dynamic management, when this application is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

[0133] The above description is only the implementation mode of this application, and does not limit the patent scope of this application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be included in the patent protection scope of this application by the same token.

Claims

1. A resource dynamic management method, characterized in that: The resource dynamic management method is applied to a big data cluster, and the method comprises: Acquire a target resource operation data set for each business, wherein the target resource operation data set includes target resource operation data arranged in a collection time sequence, and the target resource operation data represents a target resource usage when the big data cluster runs the business within a preset time period; Determine a target business pair from the business according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair includes a first business and a second business, and the total change trend of the resource operation data in the process of the big data cluster running the first business is opposite to the total change trend of the resource operation data in the process of the big data cluster running the second business; Perform resource sharing processing on the initial configuration resource data of the first service and the second service; After the step of performing resource sharing processing on the initial configuration resource data of the first service and the second service, the method further includes: the first service and the second service are located in the same physical node, and the sum of the initial configuration resource data of the first service and the second service is determined as the total configuration resource data available at the physical node; the resource use rights of the first service in the total configuration resource data are determined according to the target resource operation data of the first service; and the resource use rights of the second service in the total configuration resource data are determined according to the target resource operation data of the second service. The data attributes include data change trends of the target resource operation data in the target resource operation data set, and the step of determining the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set includes: Traverse each business, match the data change trend corresponding to the current business with the data change trend corresponding to other businesses, and obtain a matching result; In response to the matching result indicating that the data change trends between the current service and the other services match, the current service and the other services are used as candidate service pairs to obtain a plurality of candidate service pairs; Determine the target service pair from the multiple candidate service pairs; The data change trend corresponding to the current business includes the data change sub-trend of each target resource operation data, and the step of matching the data change trend corresponding to the current business with the data change trends corresponding to other businesses to obtain a matching result includes: Determine the target value corresponding to each target resource operation data from a preset change trend mapping table according to the data change sub-trend of each target resource operation data in the current business, each target resource operation data corresponds to a time series; Calculate the sum of the target values ​​corresponding to each time sequence in the current service and the target values ​​corresponding to the time sequences in other services; In response to the sum of the numerical values ​​being equal to a preset numerical value, it is determined that the data change trends between the current business and the other business match.

2. The resource dynamic management method according to claim 1, characterized in that: The step of determining the target service pair from the multiple candidate service pairs comprises: Traversing each candidate service pair, determining whether a candidate service in the current candidate service pair has the same service type as a candidate service in another candidate service pair; If so, deduplication processing is performed on the current candidate service pair and the other candidate service pairs to obtain the target service pair.

3. The resource dynamic management method according to claim 2, characterized in that: The step of performing deduplication processing on the current candidate service pair and the other candidate service pairs to obtain the target service pair includes: Acquire a first total resource usage when the big data cluster runs the current candidate service pair and a second total resource usage when the big data cluster runs the other candidate service pairs; Determine a maximum total resource usage from the first total resource usage and the second total resource usage; The candidate service pair corresponding to the maximum total resource usage is used as the target service pair.

4. The resource dynamic management method according to claim 1, characterized in that: The step of obtaining the target resource operation data set of each business includes: Obtaining initial resource operation data of the big data cluster running each business; Preprocessing the initial resource operation data to obtain preprocessed initial resource operation data, wherein the initial resource operation data includes sub-resource operation data of multiple resource types; The sub-resource operation data of each resource type in the initial resource operation data are weighted to obtain a target resource operation data set of the corresponding business.

5. The resource dynamic management method according to claim 4, characterized in that: The step of preprocessing the initial resource operation data to obtain the preprocessed initial resource operation data includes: Cleaning the initial resource operation data to obtain cleaned initial resource operation data; Perform segment processing according to the collection time sequence corresponding to each cleaned initial resource operation data to obtain the initial resource operation data set corresponding to each preset time period; The largest initial resource operation data set among the initial resource operation data sets is used as the preprocessed initial resource operation data set.

6. The resource dynamic management method according to claim 1, characterized in that: The step of performing resource sharing processing on the initial configuration resource data of the first service and the second service comprises: In response to the fact that the total resource operation data required by the big data cluster to run the first business is less than the initial configuration resource data of the first business, and the total resource operation data required by the big data cluster to run the second business is greater than the initial configuration resource data, the initial configuration resource data of the first business is called to run the second business.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the resource dynamic management method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the resource dynamic management method according to any one of claims 1 to 6.

Images