Resource dynamic management method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a resource dynamic management method, electronic equipment and a computer readable storage medium. A target resource operation data set of each service is obtained, the target resource operation data set comprises target resource operation data arranged according to a collection time sequence, and the target resource operation data represents a target resource usage amount when a big data cluster operates the service within a preset time period; determining a target service pair from services according to the data attribute of the target resource operation data in each target resource operation data set, the target service pair comprising a first service and a second service, the total change trend of the resource operation data in the process that the big data cluster operates the first business is opposite to the total change trend of the resource operation data in the process that the big data cluster operates the second business; and performing resource sharing processing on the initial configuration resource data of the first service and the second service. Therefore, the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource management, and in particular to a resource dynamic management method, electronic equipment and computer-readable storage medium. Background Art

[0002] A big data cluster is a distributed computing architecture consisting of multiple servers. It provides high storage capacity and powerful computing power to run services, such as online transaction processing, data mining, and machine learning. It is currently widely used in finance, medical care, e-commerce, and other fields. A big data cluster usually runs multiple services, and insufficient resource allocation for services will reduce the service processing efficiency of the big data cluster.

[0003] Currently, when business resources are insufficiently allocated to a big data cluster, a method is usually adopted to expand the capacity of the big data cluster by creating a virtual machine, and then adding the reserved resources to the virtual machine. This method requires reserving the resources required for dynamic expansion, and the reserved resources are idle before the expansion, resulting in resource waste and low resource utilization. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a resource dynamic management method, an electronic device and a computer-readable storage medium, which can improve resource utilization.

[0005] In order to solve the above technical problems, the present application provides a resource dynamic management method.

[0006] In one embodiment, a resource dynamic management method is applied to a big data cluster, the method comprising: Acquire a target resource operation data set for each business, wherein the target resource operation data set includes target resource operation data arranged in a collection time sequence, and the target resource operation data represents a target resource usage when the big data cluster runs the business within a preset time period; Determine a target business pair from the business according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair includes a first business and a second business, and the total change trend of the resource operation data in the process of the big data cluster running the first business is opposite to the total change trend of the resource operation data in the process of the big data cluster running the second business; The initial configuration resource data of the first service and the second service are processed for resource sharing.

[0007] In one embodiment, the data attribute includes a data change trend of the target resource operation data in the target resource operation data set, and the step of determining the target business pair from the business according to the data attribute of the target resource operation data in each target resource operation data set includes: Traversing each business, matching the data change trend corresponding to the current business with the data change trend corresponding to other businesses to obtain a matching result; in response to the matching result indicating that the data change trends between the current business and the other business match, taking the current business and the other business as candidate business pairs to obtain multiple candidate business pairs; The target service pair is determined from the multiple candidate service pairs.

[0008] In one embodiment, the data change trend corresponding to the current business includes data change sub-trends of the operation data of each target resource, and the step of matching the data change trend corresponding to the current business with the data change trends corresponding to other businesses to obtain a matching result includes: Determine the target value corresponding to each target resource operation data from a preset change trend mapping table according to the data change sub-trend of each target resource operation data in the current business, each target resource operation data corresponds to a time series; Calculate the sum of the target values ​​corresponding to each time sequence in the current service and the target values ​​corresponding to the time sequences in other services; In response to the sum of the numerical values ​​being equal to a preset numerical value, it is determined that the data change trends between the current business and the other business match.

[0009] In one embodiment, the step of determining the target service pair from the multiple candidate service pairs includes: Traversing each candidate service pair, determining whether a candidate service in the current candidate service pair has the same service type as a candidate service in another candidate service pair; If so, deduplication processing is performed on the current candidate service pair and the other candidate service pairs to obtain the target service pair.

[0010] In one embodiment, the step of performing deduplication processing on the current candidate service pair and the other candidate service pairs to obtain the target service pair includes: Acquire a first total resource usage when the big data cluster runs the current candidate service pair and a second total resource usage when the big data cluster runs the other candidate service pairs; Determine a maximum total resource usage from the first total resource usage and the second total resource usage; The candidate service pair corresponding to the maximum total resource usage is used as the target service pair.

[0011] In one embodiment, the step of obtaining the target resource operation data set of each service includes: Obtaining initial resource operation data of the big data cluster running each business; Preprocessing the initial resource operation data to obtain preprocessed initial resource operation data, wherein the initial resource operation data includes sub-resource operation data of multiple resource types; The sub-resource operation data of each resource type in the initial resource operation data are weighted to obtain a target resource operation data set of the corresponding business.

[0012] In one embodiment, the step of preprocessing the initial resource operation data to obtain the preprocessed initial resource operation data includes: Cleaning the initial resource operation data to obtain cleaned initial resource operation data; Perform segment processing according to the collection time sequence corresponding to each cleaned initial resource operation data to obtain the initial resource operation data set corresponding to each preset time period; The largest initial resource operation data set among the initial resource operation data sets is used as the preprocessed initial resource operation data set.

[0013] In one embodiment, the step of performing resource sharing processing on the initial configuration resource data of the first service and the second service includes: In response to the fact that the total resource operation data required by the big data cluster to run the first business is less than the initial configuration resource data of the first business, and the total resource operation data required by the big data cluster to run the second business is greater than the initial configuration resource data, the initial configuration resource data of the first business is called to run the second business.

[0014] In order to solve the above technical problems, the present application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above resource dynamic management method.

[0015] In order to solve the above technical problems, the present application provides a computer-readable storage medium, including: storing program data, and the program data is used to implement the above resource dynamic management method when executed by a processor.

[0016] The present application obtains the target resource operation data set of each business, the target resource operation data set includes the target resource operation data arranged in the acquisition time sequence, and the target resource operation data represents the target resource usage when the big data cluster runs the business within a preset time period; according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair is determined from the business, the target business pair includes the first business and the second business, and the total change trend of the resource operation data in the process of the big data cluster running the first business is opposite to the total change trend of the resource operation data in the process of the big data cluster running the second business; the initial configuration resource data of the first business and the second business are processed for resource sharing. Thus, the target business pair with complementary resources is determined by the data attributes of the target resource operation data in each target resource operation data set, and the first business and the second business in the target business pair are shared with resources, so as to ensure that idle resources can be obtained from the idle business for business operation during the peak period of business operation, and the idle resources can be given to the business during the peak period of business operation during the trough period of business operation, so as to realize resource sharing between businesses and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flowchart of an exemplary embodiment of a resource dynamic management method shown in the present application; Figure 2 is a schematic diagram of a resource curve corresponding to a service shown in this application; Figure 3 is a schematic diagram of a resource curve LA and a resource curve LC shown in the present application; Figure 4 is a schematic diagram of a resource curve LA and a resource curve LD shown in the present application; Figure 5 for Figure 1 A flow chart of an exemplary embodiment of step S110 in the resource dynamic management method is shown; Figure 6 for Figure 1 A flow chart of an exemplary embodiment of step S120 in the resource dynamic management method is shown; Figure 7 It is a schematic diagram of the initial configuration resource data of the business in the big data cluster shown in this application; Figure 8It is a schematic diagram of target resource operation data when a big data cluster runs a business as shown in this application; Fig. 9 It is a schematic diagram of resource sharing processing of initial configuration resource data of the service shown in this application; Fig.10 is a block diagram of a resource dynamic management device shown in an exemplary embodiment of the present application; Fig.11 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Fig.12 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0019] First of all, it should be noted that a big data cluster is a distributed computing architecture consisting of multiple servers. It runs services such as online transaction processing, data mining, and machine learning by providing high storage capacity and powerful computing power. It is currently widely used in finance, medical care, e-commerce, and other fields. There are usually multiple services running in a big data cluster, and insufficient resource allocation for services will lead to reduced service processing efficiency of the big data cluster.

[0020] Currently, when business resources are insufficiently allocated to a big data cluster, a method is usually adopted to expand the capacity of the big data cluster by creating a virtual machine, and then adding the reserved resources to the virtual machine. This method requires reserving the resources required for dynamic expansion, and the reserved resources are idle before the expansion, resulting in resource waste and low resource utilization.

[0021] Based on this, the present application provides a resource dynamic management method, an electronic device and a computer-readable storage medium. Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of a resource dynamic management method shown in the present application.

[0022] The execution subject of a resource dynamic management method may be a terminal device or a server or other processing device, wherein the terminal device may be a computer, a mobile device, a terminal, a computing device, a vehicle-mounted device, etc. The execution subject of the resource dynamic management method may also be a resource dynamic management device. In some possible implementations, the resource dynamic management method may be implemented by a processor calling a computer-readable instruction stored in a memory. The execution subject of the resource dynamic management method may also be a big data cluster, which is a computer system architecture formed by multiple computers connected through a network. The big data cluster may be deployed on a private cloud built by K8S (Kubernetes, a container orchestration engine).

[0023] Specifically, a resource dynamic management method of this embodiment includes the following steps: Step S110: Obtain a target resource operation data set for each business, wherein the target resource operation data set includes target resource operation data arranged in a collection time sequence, and the target resource operation data represents target resource usage when the big data cluster runs the business within a preset time period.

[0024] Business refers to the use of the computing and storage capabilities of big data clusters to process and analyze large-scale data. For example, business can be image storage, data query, etc. Business can include multiple sub-services, which can be zookeeper (open source distributed coordination service), hdfs (Hadoop Distributed File System), yarn (Yet Another Resource Negotiator), spark (open source distributed coordination service), hive (data warehouse tool built on Hadoop), open source distributed stream processing platform Kafka (Apache Kafka), etc.

[0025] The target resource operation data set is the target resource operation data when the big data cluster runs the business, which is arranged in the collection time sequence. For example, the target resource operation data set includes the target resource operation data collected at 1:01, the target resource operation data collected at 1:01:15, the target resource operation data collected at 1:01:30, and so on.

[0026] The target resource operation data is the target resource usage when the big data cluster runs the business within the preset time period. The target resource usage is the resources used by the big data cluster to run the business. The target resource operation data can be CPU (Central Processing Unit) data, storage data, network IO (Input / Output) data, disk IO data, etc.

[0027] The resource dynamic management device obtains the target resource operation data set of each business. Specifically, the resource dynamic management device collects the target resource operation data when the big data cluster runs the business at a preset time interval; arranges the collected target resource operation data according to the collection time sequence to form a target resource operation data set. For example, the preset time interval can be 15 seconds.

[0028] Step S120: determine a target business pair from the business according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair including a first business and a second business, and the total change trend of the resource operation data in the process of the big data cluster running the first business is opposite to the total change trend of the resource operation data in the process of the big data cluster running the second business.

[0029] Data attributes are data change trends that characterize resource operation data. Data attributes can be changes in the size of target resource operation data other than the first target resource operation data compared to the previous target resource operation data, such as an increase, decrease, or no change in the data. Data attributes can also be change trends in resource curves drawn based on target resource operation data, such as no change, increase, or decrease in the resource curve. As an example, the resource dynamic management device uses each target resource operation data as the vertical coordinate and the collection time sequence corresponding to each target resource operation data as the horizontal coordinate to mark points in a preset coordinate system, connects the points to form a resource curve, and uses the change trend of the resource curve as the data attribute of the target resource operation data.

[0030] The overall change trend of resource operation data refers to the overall change trend of target resource operation data. The overall change trend of resource operation data may be that the resource operation data is generally increasing, for example, the overall change trend of resource operation data may be gradually increasing, or first unchanged, then gradually increasing, and then unchanged. The overall change trend of resource operation data may also be that the resource operation data is generally decreasing, for example, the overall change trend of resource operation data may be gradually decreasing, or first unchanged, then gradually decreasing, and then unchanged. The overall change trend of resource operation data may also be first increasing and then decreasing, or first decreasing and then increasing.

[0031] The overall change trends of the resource operation data of two businesses may be opposite, that is, the target resource operation data of one business shows an increasing trend, while the target resource operation data of the other business shows a decreasing trend. The overall change trends of the resource operation data of two businesses may also be opposite, that is, the target resource operation data of one business first increases and then decreases, while the target resource operation data of the other business first decreases and then increases.

[0032] The target business pair refers to at least two businesses selected from the businesses running in the big data cluster. The target business pair may include a first business and a second business. The target business pair may also include a first business, a second business, and a third business.

[0033] The resource dynamic management device determines the target business pair from the business according to the data attributes of the target resource operation data in each target resource operation data set. Specifically, the data attribute can be a change trend of a resource curve drawn based on the target resource operation data. The resource dynamic management device determines the corresponding array from a preset resource curve change trend mapping table according to the change trend of the resource curve; compares the sum of any two arrays with a preset value to obtain a comparison result; if the comparison result indicates that the sum of the two arrays is equal to the preset value, the two businesses corresponding to the target resource operation data of the two arrays are determined as the target business pair. In this way, the data change trends of the businesses in the target business pair are matched in each time period, so that resources can be complementary when resources are shared, thereby making full use of resources and saving resources.

[0034] In one embodiment, a resource curve is drawn based on the operation data of each target resource. Figure 2 As shown, in Figure 2 In the example, the resource curves drawn according to the target resource operation data in the target resource operation data set of business A, business B, business C and business D are resource curves LA, resource curve LB, resource curve LC and resource curve LD respectively. In the preset resource curve change trend mapping table, the value corresponding to unchanged is 0, the value corresponding to rising is 1, and the value corresponding to falling is -1. The change trends of resource curve LA in each time period are: unchanged, falling and unchanged, and the corresponding array is [0, -1, 0]; the change trends of resource curve LB in each time period are: unchanged, unchanged and unchanged, and the corresponding array is [0, 0, 0]. The change trends of resource curve LC in each time period are: unchanged, rising and unchanged, and the corresponding array is [0, 1, 0]. The change trends of resource curve LD in each time period are: unchanged, rising and unchanged, and the corresponding array is [0, 1, 0]. Compare the sum of any two arrays with the value 0, and the arrays with the sum of 0 are [0, -1, 0] corresponding to resource curve LA and [0, 1, 0] corresponding to resource curve LC, and [0, -1, 0] corresponding to resource curve LA and [0, 1, 0] corresponding to resource curve LD; determine business A corresponding to resource curve LA and business C corresponding to resource curve LC as candidate business pairs, and resource curve LA and resource curve LC are as follows: Figure 3 As shown, the service A corresponding to the resource curve LA and the service D corresponding to the resource curve LD are determined as a candidate service pair. The resource curve LA and the resource curve LD are as shown in FIG. Figure 4 As shown; select a target business pair from the candidate business pairs.

[0035] Step S130: Perform resource sharing processing on the initial configuration resource data of the first service and the second service.

[0036] Initial configuration resource data refers to the starting resource data configured for the business in the big data cluster.

[0037] Resource sharing processing refers to sharing the initial configuration resource data of the first service with the second service, or sharing the initial configuration resource data of the second service with the first service.

[0038] The resource dynamic management device performs resource sharing processing on the initial configuration resource data of the first service and the second service. As an example, the big data cluster includes multiple physical nodes, and the service implements service operation by scheduling the initial configuration resource data through the physical nodes. The resource dynamic management device adjusts the first service and the second service to the same physical node to implement resource sharing of the first service and the second service.

[0039] It can be seen that by obtaining the target resource operation data set of each business, the target resource operation data set includes the target resource operation data arranged in the acquisition time sequence, and the target resource operation data represents the target resource usage when the big data cluster runs the business within the preset time period; according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair is determined from the business, the target business pair includes the first business and the second business, and the total change trend of the resource operation data in the process of the big data cluster running the first business is opposite to the total change trend of the resource operation data in the process of the big data cluster running the second business; the initial configuration resource data of the first business and the second business are processed for resource sharing. Therefore, on the one hand, the target business pair with complementary resources is determined by the data attributes of the target resource operation data in each target resource operation data set, and the first business and the second business in the target business pair are shared. Resources, so as to ensure that idle resources can be obtained from idle resources for business operation during the peak period of business operation, and redundant resources can be obtained from sufficient resources for business operation during the trough period of business operation, so as to realize resource sharing between businesses, improve resource utilization, and also avoid the risk of data loss caused by increasing or decreasing external resources. On the other hand, by sharing the initial configuration resource data between the businesses of the big data cluster, dynamic resource management of the big data cluster is realized.

[0040] Based on the above examples, please continue to refer to Figure 5 , Figure 5 for Figure 1 The flowchart of an exemplary embodiment of step S110 in the resource dynamic management method is shown. Specifically, the process of obtaining the target resource operation data set of each business in step S110 includes the following steps: Step S510: Obtain initial resource operation data of each business running in the big data cluster.

[0041] Initial resource operation data refers to the resource data when the big data cluster runs various businesses.

[0042] The resource dynamic management device obtains the initial resource operation data of each service run by the big data cluster. As an example, the resource dynamic management device retrieves the initial resource operation data of each service run by the big data cluster from a preset resource data storage library. As another example, the resource dynamic management device collects the resource usage when the big data cluster runs each service, and uses the resource usage of each service as the initial resource operation data of each service run by the big data cluster.

[0043] Step S520: pre-process the initial resource operation data to obtain pre-processed initial resource operation data, where the initial resource operation data includes sub-resource operation data of multiple resource types.

[0044] Preprocessing refers to cleaning or deduplication of data.

[0045] Resource type refers to the type of resource operation data. Resource type can be CPU type, storage type, network IO type, and disk IO type.

[0046] The resource dynamic management device preprocesses the initial resource operation data to obtain the preprocessed initial resource operation data. As an example, the resource dynamic management device cleans the initial resource operation data to obtain the preprocessed initial resource operation data. The resource dynamic management device removes jitter data and invalid data through cleaning, which is beneficial to improving the quality of the target resource operation data set. As another example, the resource dynamic management device cleans the initial resource operation data to obtain the cleaned initial resource operation data; performs segmentation processing according to the collection timing corresponding to each cleaned initial resource operation data set to obtain the initial resource operation data set corresponding to each preset time period; and uses the largest initial resource operation data in each initial resource operation data set as the preprocessed initial resource operation data set.

[0047] In one embodiment, the initial resource operation data includes initial sub-resource data of multiple resource types, as shown below:

[0048] Where M represents the matrix of initial resource operation data, Represents the first initial sub-resource data of the first resource type, Indicates the first resource type Initial sub-resource data, Indicates The first initial sub-resource data of the resource type, Indicates The resource type Initial sub-resource data.

[0049] In one embodiment, the resource dynamic management device segments the time series according to a preset interval time to obtain multiple time periods. For example, the preset interval time may be 6 hours. The initial resource operation data after each cleaning is classified according to the time period to which the acquisition time series belongs, and the initial resource operation data set corresponding to each preset time period is obtained. The largest initial resource operation data in each initial resource operation data set is used as the preprocessed initial resource operation data to realize the merging of the initial resource operation data in the initial resource operation data set. The preprocessed initial resource operation data includes sub-resource operation data of multiple resource types, as shown below:

[0050] in, The matrix representing the initial resource operation data after preprocessing, express The sub-resource operation data of the first resource type in the time period, express The sub-resource operation data of the first resource type in the time period, express Time period Sub-resource operation data of resource types, express Time period Sub-resource operation data of a resource type.

[0051] Furthermore, the resource dynamic management device classifies each cleaned initial resource operation data according to a preset number to obtain a plurality of initial resource operation data sets, and one initial resource operation data set corresponds to one time period.

[0052] Step S530: weighting the sub-resource operation data of each resource type in the initial resource operation data to obtain a target resource operation data set of the corresponding business.

[0053] The resource dynamic management device performs weighted processing on the sub-resource operation data of each resource type in the initial resource operation data to obtain the target resource operation data set of the corresponding business. Specifically, the resource dynamic management device determines the corresponding weight value in the preset weight mapping table according to the resource type, and the preset weight mapping table stores a one-to-one correspondence between the preset resource type and the preset weight value; calculates the sub-resource operation data of each time period, the weight value corresponding to the sub-resource operation data, and the total resource data of the resource type of the sub-resource operation data to obtain the target resource operation data of each time period, and aggregates the target resource operation data of each time period to form the target resource operation data set of the business.

[0054] In one embodiment, the sub-resource operation data, the weight value corresponding to the sub-resource operation data, the total resource data of the resource type of the sub-resource operation data, and the target resource operation data satisfy the following formula:

[0055] in, express Target resource operation data for a time period, Indicates the weight value corresponding to the kth resource type, express Sub-resource operation data of the kth resource type in the time period, Indicates the total number of resource types, Represents the total resource data of the kth resource type in the big data cluster.

[0056] In one embodiment, the target resource operation data of each time period and the target resource operation data set of the business satisfy the following formula:

[0057] in, Indicates the target resource running data set. express Target resource operation data for a time period, express Target resource operation data for a time period, express Target resource operation data for a time period, express Target resource operation data for a time period.

[0058] It can be seen that the initial resource operation data of each business running in the big data cluster is obtained, the initial resource operation data is preprocessed to obtain the preprocessed initial resource operation data, and the sub-resource operation data of each resource type in the initial resource operation data is weighted to obtain the target resource operation data set of the corresponding business. The data quality of the target resource operation data set is improved through preprocessing, and the quality of the target resource operation data set is further improved through weighted processing.

[0059] Based on the above examples, please continue to refer to Figure 6 , Figure 6 for Figure 1 The flowchart of an exemplary embodiment of step S120 in the resource dynamic management method is shown. Specifically, the data attribute includes the data change trend of the target resource operation data in the target resource operation data set, and the process of step S120 determining the target business pair from the business according to the data attribute of the target resource operation data in each target resource operation data set includes the following steps: Step S610: traverse each business, match the data change trend corresponding to the current business with the data change trend corresponding to other businesses, and obtain a matching result.

[0060] The data change trend includes the data change sub-trend of the target resource operation data.

[0061] The matching results include a match in the data change trend between the current business and other businesses, and a mismatch in the data change trend between the current business and other businesses.

[0062] The resource dynamic management device matches the data change trend corresponding to the current business with the data change trend corresponding to other businesses to obtain a matching result. Specifically, the resource dynamic management device determines the target value corresponding to each target resource operation data from the preset change trend mapping table according to the data change sub-trend of each target resource operation data in the current business, and each target resource operation data corresponds to a time series; the preset change trend mapping table stores a one-to-one correspondence between the preset data change sub-trend and the preset target value; calculates the numerical sum between the target value corresponding to each time series in the current business and the target value corresponding to the time series in other businesses; in response to the numerical sum being equal to the preset numerical value, it is determined that the data change trend between the current business and other businesses matches. The preset numerical value can be 0. For example, if the numerical sum is equal to 0, it is determined that the data change trend between the current business and other businesses matches. If the numerical sum is greater than 0 or less than 0, it is determined that the data change trend between the current business and other businesses does not match.

[0063] Step S620: In response to the matching result indicating that the data change trends between the current business and other businesses match, the current business and other businesses are used as candidate business pairs to obtain a plurality of candidate business pairs.

[0064] In response to the matching result indicating that the data change trend between the current service and other services matches, the resource dynamic management device uses the current service and other services as candidate service pairs to obtain multiple candidate service pairs. For example, if the matching result indicates that the data change trend between service E and service F matches, service E and service F are used as candidate service pairs.

[0065] Step S630: Determine a target service pair from a plurality of candidate service pairs.

[0066] The resource dynamic management device determines a target business pair from a plurality of alternative business pairs. Specifically, it traverses each alternative business pair to determine whether an alternative business in the current alternative business pair has the same business type as an alternative business in other alternative business pairs; if so, it deduplicates the current alternative business pair and other alternative business pairs to obtain a target business pair; if not, it uses the current alternative business pair as the target business pair. For example, the current alternative business pair includes business A and business C, and other alternative business pairs include business A and business D. It is determined that the business type of business A in the current alternative business pair is the same as that of business A in other alternative business pairs, then it deduplicates the current alternative business pair and other alternative business pairs, and selects the target business pair from the current alternative business pair and other alternative business pairs.

[0067] The resource dynamic management device performs deduplication processing on the current candidate service pair and other candidate service pairs to obtain the target service pair. Specifically, the resource dynamic management device obtains the first resource usage total amount when the big data cluster runs the current candidate service pair and the second resource usage total amount when the big data cluster runs other candidate service pairs; determines the maximum resource usage total amount from the first resource usage total amount and the second resource usage total amount; and uses the candidate service pair corresponding to the maximum resource usage total amount as the target service pair.

[0068] The first resource usage total refers to the sum of the target resource operation data of all services in the current candidate service pair. The first resource usage total can be the sum of the target resource operation data of all services in the current candidate service pair in all time periods, or the maximum value of the sum of the target resource operation data of all services in the current candidate service pair in a time period.

[0069] The second resource usage total refers to the sum of target resource operation data of all services in other candidate service pairs. The second resource usage total can be the sum of target resource operation data of all services in other candidate service pairs in all time periods, or the maximum value of the sum of target resource operation data of all services in other candidate service pairs in a time period.

[0070] The resource dynamic management device determines the maximum resource usage total amount from the first resource usage total amount and the second resource usage total amount. Specifically, the resource dynamic management device compares the first resource usage total amount and the second resource usage total amount. If the first resource usage total amount is greater than or equal to the second resource usage total amount, the first resource usage total amount is determined as the maximum resource usage total amount. Otherwise, the second resource usage total amount is determined as the maximum resource usage total amount.

[0071] It can be seen that by traversing each business, the data change trend corresponding to the current business is matched with the data change trend corresponding to other businesses to obtain a matching result; in response to the matching result indicating that the data change trend between the current business and other businesses matches, the current business and other businesses are used as candidate business pairs to obtain multiple candidate business pairs; the target business pair is determined from the multiple candidate business pairs. Thus, on the one hand, the target business pair is selected from the business, and on the other hand, the current business and other businesses with matching data change trends are obtained through matching processing, and based on this, the current business and other businesses with complementary resources are obtained, so that the resource sharing of the initial configuration resource data of subsequent businesses can be realized.

[0072] Specifically, Figure 1 The process of processing the initial configuration resource data of the first business and the second business for resource sharing in step S130 of the resource dynamic management method shown includes: in response to the total resource operation data required by the big data cluster to run the first business being less than the initial configuration resource data of the first business, and the total resource operation data required by the big data cluster to run the second business being greater than the initial configuration resource data, the initial configuration resource data of the first business is called to run the second business.

[0073] The total resource operation data required to run the first business may be the target resource operation data when all sub-services in the first business are running at the same time. Exemplarily, the resource dynamic management device collects the target resource operation data when the big data cluster runs each sub-service in the first business, and takes the sum of the target resource operation data of each sub-service in the first business as the total resource operation data required to run the first business. The total resource operation data required to run the first business may also be the sum of the target resource operation data in the target resource operation data set.

[0074] The total resource operation data required to run the second business may be the target resource operation data when all sub-services in the second business are running at the same time. Exemplarily, the resource dynamic management device collects the target resource operation data when the big data cluster runs each sub-service in the second business, and takes the sum of the target resource operation data of each sub-service in the second business as the total resource operation data required to run the second business. The total resource operation data required to run the second business may also be the sum of the target resource operation data in the target resource operation data set.

[0075] The resource dynamic management device calls the initial configuration resource data of the first service to run the second service. Exemplarily, the resource dynamic management device generates an instruction to call the initial configuration resource data of the first service to run the second service, and sends the call instruction to the K8S platform where the resource dynamic management device is deployed. After receiving the call instruction, K8S searches for the corresponding initial configuration resource data in the preset resource database according to the first service, and configures the found initial configuration resource data to the second service.

[0076] The resource dynamic management device performs resource sharing processing on the initial configuration resource data of the first service and the second service. Specifically, in response to the total resource operation data required by the big data cluster to run the first service being greater than the initial configuration resource data of the first service, and the total resource operation data required by the big data cluster to run the second service being less than the initial configuration resource data, the resource dynamic management device calls the initial configuration resource data of the second service to run the first service.

[0077] Furthermore, after the resource dynamic management device performs resource sharing processing on the initial configuration resource data of the first service and the second service, the first service and the second service are located in the same physical node. The resource dynamic management device determines the sum of the initial configuration resource data of the first service and the second service as the total configuration resource data available at the physical node; determines the resource usage rights of the first service in the total configuration resource data according to the target resource operation data of the first service; and determines the resource usage rights of the second service in the total configuration resource data according to the target resource operation data of the second service.

[0078] The resource dynamic management device determines the resource usage rights of the first service in the total resource configuration data according to the target resource operation data of the first service. Specifically, the resource usage rights include an upper limit and a lower limit of resource usage. The resource dynamic management device determines the sum of the maximum target resource operation data and the preset redundancy value in the target resource operation data of the first service as the upper limit of resource usage of the first service in the total resource configuration data; and determines the minimum target resource operation data in the target resource operation data of the first service as the lower limit of resource usage of the first service in the total resource configuration data. For example, the resource usage upper limit of the resource dynamic management device can be , Indicates the maximum target resource operation data of the first business, represents the preset redundancy value; the lower limit of resource usage of the resource dynamic management device can be , Indicates the minimum target resource operation data of the first service.

[0079] The resource dynamic management device determines the resource usage rights of the second service in the total configuration resource data according to the target resource operation data of the second service. Specifically, the resource usage rights include the upper limit of resource usage and the lower limit of resource usage. The resource dynamic management device determines the sum of the maximum target resource operation data and the preset redundancy value in the target resource operation data of the second service as the upper limit of resource usage of the second service in the total configuration resource data; and determines the minimum target resource operation data in the target resource operation data of the second service as the lower limit of resource usage of the second service in the total configuration resource data.

[0080] It can be seen that by calling the initial configuration resource data of the first business to run the second business, the initial configuration resource data of the first business and the second business can be shared. Therefore, when the initial configuration resource data of the second business is insufficient, the excess initial configuration resource data of the first business can be shared, thereby improving resource utilization.

[0081] In one embodiment, a POD (container group, the smallest deployment unit) in the Kubernetes container orchestration platform can run services in a big data cluster, such as Figure 7 As shown, services A, B, C and D in the big data cluster are respectively configured with initial configuration resource data in POD. After the big data cluster runs services A, B, C and D for a preset period of time, the load of each service in the big data cluster is different, and the usage of the initial configuration resource data is also different in different time periods. The resource dynamic management device obtains the target resource operation data in the target resource operation data set of each service. The target resource operation data represents the target resource usage when the big data cluster runs the service within the preset time period. Figure 8 As shown, in the big data cluster, the target resource operation data of business A in the first time period is A1, the target resource operation data of business A in the second time period is A2, and the target resource operation data of business B at all times is B1; the target resource operation data of business C in the first time period is C1, and the target resource operation data of business C in the second time period is C2; the target resource operation data of business D in the first time period is D1, and the target resource operation data of business D in the second time period is D2. The resource dynamic management device determines the target business pair from business A, B, C and D according to the data attributes of the target resource operation data in the target resource operation data set of business A, B, C and D, and the determined target business pair includes business A and business C; and performs resource sharing processing on the initial configuration resources of business A and business C. Fig. 9 The resource allocation for business A, B, C and D. Fig. 9 As shown in the figure, in the big data cluster, the initial configuration resource data of business B and business D remain unchanged. In the first time period, business A calls the redundant initial configuration resource data of business C during the business trough period during the business peak period, which improves the operation efficiency of business A during the business peak period. In the second time period, business A shares the redundant initial configuration resource data with business C during the business trough period, which improves the operation efficiency of business C during the business peak period. As a result, all resources in the big data cluster can be fully utilized at all times, and it is ensured that the service has sufficient resources to use during the business peak period, which improves the operation efficiency of the big data cluster.

[0082] Fig.10 FIG. 1 is a block diagram of a resource dynamic management device shown in an exemplary embodiment of the present application. Fig.10As shown, the exemplary resource dynamic management device 1000 includes: a target resource operation data set acquisition module 1010, a target business pair determination module 1020 and a resource sharing module 1030. Specifically: The target resource operation data set acquisition module 1010 is used to obtain the target resource operation data set of each business. The target resource operation data set includes target resource operation data arranged in a collection time sequence. The target resource operation data represents the target resource usage when the big data cluster runs the business within a preset time period.

[0083] The target business pair determination module 1020 is used to determine the target business pair from the business according to the data attributes of the target resource operation data in each target resource operation data set. The target business pair includes a first business and a second business. The total change trend of the resource operation data during the big data cluster running the first business is opposite to the total change trend of the resource operation data during the big data cluster running the second business.

[0084] The resource sharing module 1030 is used to perform resource sharing processing on the initial configuration resource data of the first service and the second service.

[0085] In this exemplary resource dynamic management device, by acquiring the target resource operation data set of each business, the target resource operation data set includes the target resource operation data arranged in the acquisition time sequence, and the target resource operation data represents the target resource usage when the big data cluster runs the business within a preset time period; according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair is determined from the business, and the target business pair includes the first business and the second business; the initial configuration resource data of the first business and the second business are processed for resource sharing. Thus, the target business pair with complementary resources is determined by the data attributes of the target resource operation data in each target resource operation data set, and the first business and the second business in the target business pair are processed for resource sharing, so as to ensure that idle resources can be obtained from idle resources for business operation during the peak period of business operation, and redundant resources can be obtained from the business with sufficient resources for business operation during the trough period of business operation, so as to realize resource sharing between businesses and improve resource utilization.

[0086] The functions of each module can be found in the implementation example of the dynamic resource management method, which will not be described in detail here.

[0087] In order to implement the resource dynamic management method of the above embodiment, the present application proposes another electronic device, for details, please refer to Fig.11 , Fig.11 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0088] The electronic device 1100 includes a memory 1101 and a processor 1102 , wherein the memory 1101 and the processor 1102 are coupled.

[0089] The memory 1101 is used to store program data, and the processor 1102 is used to execute the program data to implement the resource dynamic management method of the above embodiment.

[0090] In this embodiment, the processor 1102 may also be referred to as a CPU (Central Processing Unit). The processor 1102 may be an integrated circuit chip having signal processing capabilities. The processor 1102 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor 1102 may also be any conventional processor, etc.

[0091] The present application also provides a computer-readable storage medium, such as Fig.12 As shown, the computer-readable storage medium 1200 is used to store program data 1201. When the program data 1201 is executed by the processor, it is used to implement the resource dynamic management method in the method embodiment of the present application.

[0092] The method involved in the embodiment of the resource dynamic management method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0093] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A resource dynamic management method, characterized in that: The resource dynamic management method is applied to a big data cluster, and the method comprises: Acquire a target resource operation data set for each business, wherein the target resource operation data set includes target resource operation data arranged in a collection time sequence, and the target resource operation data represents a target resource usage when the big data cluster runs the business within a preset time period; Determine a target business pair from the business according to the data attributes of the target resource operation data in each target resource operation data set, the target business pair includes a first business and a second business, and the total change trend of the resource operation data in the process of the big data cluster running the first business is opposite to the total change trend of the resource operation data in the process of the big data cluster running the second business; The initial configuration resource data of the first service and the second service are processed for resource sharing.

2. The resource dynamic management method according to claim 1, characterized in that: The data attributes include data change trends of the target resource operation data in the target resource operation data set, and the step of determining the target service pair from the services according to the data attributes of the target resource operation data in each target resource operation data set includes: Traverse each business, match the data change trend corresponding to the current business with the data change trend corresponding to other businesses, and obtain a matching result; In response to the matching result indicating that the data change trends between the current service and the other services match, the current service and the other services are used as candidate service pairs to obtain a plurality of candidate service pairs; The target service pair is determined from the multiple candidate service pairs.

3. The resource dynamic management method according to claim 2, characterized in that: The data change trend corresponding to the current business includes data change sub-trends of the operation data of each target resource, and the step of matching the data change trend corresponding to the current business with the data change trends corresponding to other businesses to obtain a matching result includes: Determine the target value corresponding to each target resource operation data from a preset change trend mapping table according to the data change sub-trend of each target resource operation data in the current business, each target resource operation data corresponds to a time series; Calculate the sum of the target values ​​corresponding to each time sequence in the current service and the target values ​​corresponding to the time sequences in other services; In response to the sum of the numerical values ​​being equal to a preset numerical value, it is determined that the data change trends between the current business and the other business match.

4. The resource dynamic management method according to claim 2, characterized in that: The step of determining the target service pair from the multiple candidate service pairs comprises: Traversing each candidate service pair, determining whether a candidate service in the current candidate service pair has the same service type as a candidate service in another candidate service pair; If so, deduplication processing is performed on the current candidate service pair and the other candidate service pairs to obtain the target service pair.

5. The resource dynamic management method according to claim 4, characterized in that: The step of performing deduplication processing on the current candidate service pair and the other candidate service pairs to obtain the target service pair includes: Acquire a first total resource usage when the big data cluster runs the current candidate service pair and a second total resource usage when the big data cluster runs the other candidate service pairs; Determine a maximum total resource usage from the first total resource usage and the second total resource usage; The candidate service pair corresponding to the maximum total resource usage is used as the target service pair.

6. The resource dynamic management method according to claim 1, characterized in that: The step of obtaining the target resource operation data set of each business includes: Obtaining initial resource operation data of the big data cluster running each business; Preprocessing the initial resource operation data to obtain preprocessed initial resource operation data, wherein the initial resource operation data includes sub-resource operation data of multiple resource types; The sub-resource operation data of each resource type in the initial resource operation data are weighted to obtain a target resource operation data set of the corresponding business.

7. The resource dynamic management method according to claim 6, characterized in that: The step of preprocessing the initial resource operation data to obtain the preprocessed initial resource operation data includes: Cleaning the initial resource operation data to obtain cleaned initial resource operation data; Perform segment processing according to the collection time sequence corresponding to each cleaned initial resource operation data to obtain the initial resource operation data set corresponding to each preset time period; The largest initial resource operation data set among the initial resource operation data sets is used as the preprocessed initial resource operation data set.

8. The resource dynamic management method according to claim 1, characterized in that: The step of performing resource sharing processing on the initial configuration resource data of the first service and the second service comprises: In response to the fact that the total resource operation data required by the big data cluster to run the first business is less than the initial configuration resource data of the first business, and the total resource operation data required by the big data cluster to run the second business is greater than the initial configuration resource data, the initial configuration resource data of the first business is called to run the second business.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the resource dynamic management method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the resource dynamic management method according to any one of claims 1 to 8.

Images