Computer system resource dynamic allocation adjustment method based on big data analysis

Abstract

This invention relates to the field of computer system resource management technology, and in particular to a method for dynamic allocation and adjustment of computer system resources based on big data analysis. This method involves collecting and standardizing hardware resource status data, business load characteristic data, and historical allocation data. A two-dimensional evaluation model is established to determine resource allocation priorities. An optimal resource allocation scheme is obtained by constructing an objective function that maximizes resource utilization and minimizes business performance deviations. This reduces resource allocation problems caused by human error. Furthermore, a dynamic adjustment mechanism prioritizes high-urgency business needs in mild conflict scenarios. The allocation effect is monitored in real time through a three-dimensional scoring system of resource utilization, business performance, and system stability, with feedback processing of abnormal indicators to obtain a list of abnormal indicators. This effectively avoids the risk of the optimal allocation scheme failing, ensuring that the implemented scheme matches theoretical expectations and preventing resource waste.

Description

Technical Field

[0001] This invention relates to the field of computer system resource management technology, and in particular to a method for dynamic allocation and adjustment of computer system resources based on big data analysis. Background Technology

[0002] With the deepening of digital transformation, computer systems have become the core carriers of enterprise business operations, data processing and service delivery. From cloud computing platforms and large data centers to edge computing nodes, the types of business carried by the systems are becoming increasingly diversified (such as real-time transactions, offline data analysis, high-frequency communication interaction, and massive data storage). Moreover, the business load exhibits significant characteristics of "dynamic, sudden and heterogeneous". Against this background, traditional computer system resource allocation methods have gradually exposed their limitations and are unable to meet the dual requirements of modern systems for "efficient resource utilization" and "business performance assurance".

[0003] However, the current lack of quantitative analysis of the relationship between business load characteristics and resource status makes it difficult to accurately identify resource demand gaps and business performance deviations. Allocation schemes are prone to problems of "over-allocation" or "under-allocation," and it is difficult to reasonably make dynamic allocation adjustments and evaluate the effects after allocation to ensure the maximization of resource utilization and the stability of business performance. Therefore, a solution is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a method for dynamic allocation and adjustment of computer system resources based on big data analysis. This method involves collecting multi-dimensional data and performing standardized processing to construct the correlation between business and resources and establish a two-dimensional evaluation model to determine resource allocation priorities. The optimal allocation scheme is solved using a multi-objective optimization function, and dynamic adjustment and effect evaluation are achieved. Ultimately, the dual objectives of "maximizing resource utilization" and "minimizing business performance deviation" are achieved, thereby addressing the aforementioned technical deficiencies.

[0005] The objective of this invention can be achieved through the following technical solution: a method for dynamically allocating and adjusting computer system resources based on big data analysis, comprising the following steps:

[0006] Step 1: Collect hardware resource status data, business load characteristic data, and historical allocation data of the computer system, and preprocess them to obtain a standardized dataset;

[0007] Step 2: Construct a business process-resource status association table and multi-dimensional data samples based on in-depth processing of standardized datasets;

[0008] Step 3: Based on the resource allocation demand assessment of multi-dimensional data samples, construct a two-dimensional assessment model of resource demand gap value and performance deviation rate, calculate the demand priority score of each business process, and determine the resource allocation priority order table;

[0009] Step 4: Construct an objective function that maximizes resource utilization and minimizes business performance deviation, set resource allocation constraints, and solve for the optimal allocation scheme;

[0010] Step 5: The dynamic adjustment process of resource allocation within the optimal allocation scheme and the evaluation process of allocation effect within the preset evaluation period after resource adjustment.

[0011] Preferably, the construction process of the business process-resource status association table and the multi-dimensional data sample is as follows:

[0012] S1: Assign globally unique identifiers to business processes and hardware resources based on standardized datasets;

[0013] S2: Based on the allocation of a globally unique identifier, the collected service load characteristic data and hardware resource status data are bound in real time to form associated record data;

[0014] S3: Preprocess the obtained associated record data to obtain the business process-resource status association table;

[0015] S4: Based on the constructed business process-resource status association table, a multi-dimensional data sample is formed with business processes as the unit.

[0016] Preferably, the process for obtaining the resource allocation priority list is as follows:

[0017] Based on multi-dimensional data samples, the resource demand gap value of each business process is obtained. Resource demand gap value = current resource consumption of the business / historical average resource consumption of the business - current resource allocation ratio of the business / total available resource ratio.

[0018] Set the offline computing task completion rate or the real-time transaction response latency as key performance indicators for the business.

[0019] The performance deviation rate of each business process is calculated based on (business performance key indicators - business performance key indicator thresholds) / business performance key indicator thresholds.

[0020] The weighting coefficient 'a' of the resource demand gap value of each business process and the weighting coefficient 'b' of the key business performance indicators are obtained, and both 'a' and 'b' are greater than zero.

[0021] The demand priority score is calculated by multiplying the resource demand gap value by a weighting coefficient a and the key business performance indicators by a weighting coefficient b. The business processes are then sorted from high to low based on the demand priority scores to obtain a resource allocation priority order table.

[0022] Preferably, the process for obtaining the optimal allocation scheme is as follows:

[0023] Pre-construct objective functions to maximize resource utilization (Umax) and minimize business performance deviation (Dmin), where business processes are labeled as i, i = 1, 2, 3, ..., m, and m is a natural number greater than zero;

[0024] Simultaneously set constraints to maximize resource utilization (Umax) and minimize business performance deviation (Dmin);

[0025] The optimal resource allocation scheme is obtained by solving based on the pre-set solution algorithm and constraints.

[0026] Preferably, the dynamic adjustment process is as follows:

[0027] T1: Based on the optimal resource allocation scheme, the resource demand saturation during the resource allocation period is obtained. Resource demand saturation = (∑ target resource demand of business processes) / current available resource capacity of the system × 100%;

[0028] T2: Retrieve the preset maximum resource demand saturation value BHmax and the preset minimum resource demand saturation value BHmin;

[0029] T3: The resource demand saturation level is judged to determine whether there is no conflict, a slight conflict, or a severe conflict.

[0030] Preferably, when a minor conflict is obtained, the business priority data corresponding to the business process is obtained, the business priority data is set to o, o = 1, 2, 3, and the priority weight vo of each business priority data is obtained, that is, the priority weight v1 of high business priority data, the priority weight v2 of medium business priority data and the priority weight v3 of low business priority data, v1 > v2 > v3 > 0.

[0031] Calculate the urgency of each business process. Urgency = priority weight (vo) of each business priority data × resource demand gap value. Sort the processes by urgency from high to low to obtain a dynamic priority allocation list.

[0032] Preferably, the evaluation process for the allocation effect within the preset evaluation period after resource adjustment is as follows:

[0033] Based on the optimal resource allocation scheme, the allocation effect data within the preset evaluation period after resource adjustment is obtained. The allocation effect data includes resource utilization rate indicators, business performance indicators, and system stability indicators.

[0034] Obtain the allocation ratio scores B1, B2, and B3 for resource utilization, business performance, and system stability indicators, where B1 > B2 > B3 > 0, and B1 + B2 + B3 = 100.

[0035] The multidimensional allocation effect score is calculated based on the resource utilization rate index × B1 + business performance index × B2 + system stability index × B3. The multidimensional allocation effect score is then processed to obtain a list of abnormal indicators or qualified signals.

[0036] Preferably, the resource utilization rate index = min(actual utilization rate / target utilization rate, 1); the business performance index is calculated based on ∑(1 - |key business performance index - key business performance index threshold| / key business performance index threshold) × priority weight vo; the system stability index = 1 - abnormal event occurrence rate.

[0037] The beneficial effects of this invention are as follows:

[0038] (1) This invention collects hardware resource status data, business load characteristic data and historical allocation data and performs standardized processing, constructs the correlation between business and resources and establishes a two-dimensional evaluation model to determine the priority of resource allocation, and obtains the optimal resource allocation scheme by constructing an objective function that maximizes resource utilization and minimizes business performance deviation, thereby reducing resource allocation problems caused by human judgment errors.

[0039] (2) This invention combines a dynamic adjustment mechanism to prioritize high-urgency business needs in mild conflict scenarios, avoid resource idleness and waste, and monitors the allocation effect in real time through a three-dimensional scoring of resource utilization rate, business performance and system stability, and obtains an abnormal indicator list by feedback processing of abnormal indicators, thereby effectively avoiding the risk of failure of the optimal allocation scheme, making the scheme implementation effect conform to theoretical expectations, and avoiding resource waste. Attached Figure Description

[0040] The invention will now be further described with reference to the accompanying drawings;

[0041] Figure 1 This is a reference diagram of the method of the present invention;

[0042] Figure 2 This is a partial reference diagram of Embodiment 1 of the present invention;

[0043] Figure 3 This is a partial reference diagram of Embodiment 3 of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments;

[0046] Example 1: Please refer to Figures 1 to 3 As shown, this invention is a method for dynamically allocating and adjusting computer system resources based on big data analysis, comprising the following steps:

[0047] Step 1: Collect hardware resource status data, business load characteristic data, and historical allocation data of the computer system, and preprocess them to obtain a standardized dataset;

[0048] Step 2: Construct a business process-resource status association table and multi-dimensional data samples based on in-depth processing of standardized datasets;

[0049] Step 3: Based on the resource allocation demand assessment of multi-dimensional data samples, construct a two-dimensional assessment model of resource demand gap value and performance deviation rate, calculate the demand priority score of each business process, and determine the resource allocation priority order table;

[0050] Step 4: Construct an objective function that maximizes resource utilization and minimizes business performance deviation, set resource allocation constraints, and solve for the optimal allocation scheme;

[0051] Step 5: The dynamic adjustment process of resource allocation within the optimal allocation scheme and the evaluation process of allocation effect within the preset evaluation period after resource adjustment;

[0052] The first step, obtaining the standardized dataset, is as follows:

[0053] The system collects hardware resource status data, business load characteristic data, and historical resource allocation data of the computer system in real time. It then performs preprocessing such as cleaning and standardization on the collected hardware resource status data, business load characteristic data, and historical resource allocation data to obtain a standardized dataset, which is then output and stored.

[0054] This means obtaining comprehensive and accurate resource data from computer systems to provide data support for subsequent resource demand forecasting and allocation decisions;

[0055] Hardware resource status data includes CPU utilization, memory usage, and other data.

[0056] Business load characteristic data includes the request concurrency, data processing volume, response latency requirements, and business priority data (high / medium / low) of each business process.

[0057] Historical resource allocation data includes the allocation ratio and allocation duration of each hardware resource across different business processes within a historical time period;

[0058] The process of constructing the business process-resource status association table and multi-dimensional data samples is as follows:

[0059] S1: Assign globally unique identifiers to business processes and hardware resources based on standardized datasets;

[0060] S2: Based on the allocation of a globally unique identifier, the collected service load characteristic data and hardware resource status data are bound in real time to form associated record data;

[0061] The associated record data includes unique identifiers for business processes, timestamps, business load characteristic data (such as request volume and response latency), and hardware resource status data (such as CPU utilization and memory usage).

[0062] For example, real-time status data of a business process's current resources (such as CPU utilization and memory usage) and load characteristic data of the process (such as request volume and response latency) are aligned with timestamps and bound to the same business process identifier;

[0063] S3: Preprocess the obtained associated record data (such as cleaning) to obtain the business process-resource status association table;

[0064] S4: Based on the constructed business process-resource status association table, a multi-dimensional data sample is formed with business processes as the unit. Then, the multi-dimensional data sample is output and stored.

[0065] This involves associating hardware resource data, load characteristic data, and historical allocation data corresponding to the same business process to form a multi-dimensional data sample based on the business process, providing a structured data foundation for subsequent evaluation and algorithm input;

[0066] The "Business Process-Resource Status" association table is the core data carrier for achieving accurate matching between resources and business. Its construction requires "unique identifier binding, multi-dimensional data association, and structured storage".

[0067] Example 2:

[0068] The process of obtaining the resource allocation priority list is as follows:

[0069] Based on multi-dimensional data samples, the resource demand gap value of each business process is obtained. Resource demand gap value = current resource consumption of the business / historical average resource consumption of the business - current resource allocation ratio of the business / total available resource ratio.

[0070] Set the offline computing task completion rate or the real-time transaction response latency as key performance indicators for the business.

[0071] The performance deviation rate of each business process is calculated based on (business performance key indicators - business performance key indicator thresholds) / business performance key indicator thresholds.

[0072] The weighting coefficient 'a' of the resource demand gap value of each business process and the weighting coefficient 'b' of the key business performance indicators are obtained, and both 'a' and 'b' are greater than zero.

[0073] The priority score for demand is calculated based on the resource demand gap value × weighting coefficient a + key business performance indicators × weighting coefficient b.

[0074] The business processes are sorted from high to low based on the priority scores of the requirements to obtain a resource allocation priority order table. The resource allocation priority order table is then output and displayed immediately to provide support for subsequent resource allocation.

[0075] The process of obtaining the optimal allocation scheme is as follows:

[0076] Pre-construct objective functions to maximize resource utilization Umax (representing (∑ actual resource consumption of each business process) / total available resources of the system) and minimize business performance deviation Dmin (representing ∑ (performance deviation rate of business process i × weight of business process i)). Here, business processes are labeled as i, i = 1, 2, 3, ..., m, where m is a natural number greater than zero.

[0077] Simultaneously set constraints to maximize resource utilization (Umax) and minimize business performance deviation (Dmin);

[0078] The constraints include that the CPU allocation percentage of a single business process is less than or equal to the preset CPU allocation percentage, the memory allocation percentage of a single business process is less than or equal to the preset memory allocation percentage, the resource allocation of high-priority businesses shall not be less than the preset E% of their historical average allocation, and E>80. The network bandwidth allocation must meet the minimum bandwidth requirements of all businesses (e.g., the minimum bandwidth of real-time transaction business is ≥100Mbps).

[0079] If the optimal resource allocation scheme is obtained by solving based on the pre-set solution algorithm (such as the non-dominated sorting genetic algorithm: NSGA-II) and constraints, the optimal resource allocation scheme will be output and displayed immediately.

[0080] Example 3:

[0081] Dynamic adjustment of execution is a crucial step in transforming the resource allocation scheme generated by multi-objective optimization algorithms into executable system instructions and achieving coordinated adaptation of hardware resources, business processes, and the system. It requires smooth switching and precise allocation of resources while ensuring business continuity. The dynamic adjustment process is as follows:

[0082] T1: Based on the optimal resource allocation scheme, the resource demand saturation during the resource allocation period is obtained. Resource demand saturation = (∑ target resource demand of business processes) / current available resource capacity of the system × 100%;

[0083] T2: Retrieve the preset maximum resource demand saturation value BHmax and the preset minimum resource demand saturation value BHmin;

[0084] T3: Determine the resource demand saturation level. If the resource demand saturation level is less than BHmin, it is determined that there is no conflict and the allocation continues.

[0085] If BHmin≤ResourceDemandSaturation≤BHmax, it is determined to be a minor conflict. Then, the business priority data corresponding to the business process is obtained, and the business priority data is set to o, o=1,2,3. The priority weight vo of each business priority data is obtained, that is, the priority weight v1 of high business priority data, the priority weight v2 of medium business priority data, and the priority weight v3 of low business priority data, v1>v2>v3>0;

[0086] Calculate the urgency of each business process. Urgency = priority weight (vo) of each business priority data × resource demand gap value. Sort the processes by urgency from high to low to obtain a dynamic priority allocation list. Output the dynamic priority allocation list and then make dynamic allocation adjustments according to the dynamic priority allocation list.

[0087] If the resource demand saturation is greater than BHmax, it is judged as a severe conflict and an emergency alarm command is triggered. The emergency alarm command will be sent to the operation and maintenance personnel to remind them to adjust or reacquire the current optimal resource allocation plan.

[0088] The process for evaluating the allocation effectiveness is as follows:

[0089] Based on the optimal resource allocation scheme, the allocation effect data within the preset evaluation period after resource adjustment is obtained. The allocation effect data includes resource utilization rate indicators, business performance indicators, and system stability indicators.

[0090] Wherein, the resource utilization rate index = min(actual utilization rate / target utilization rate, 1);

[0091] The business performance index is calculated based on ∑(1-|Business performance key index-Business performance key index threshold| / Business performance key index threshold)×priority weight vo;

[0092] System stability index = 1 - abnormal event occurrence rate;

[0093] Obtain the allocation ratio scores B1, B2, and B3 for resource utilization, business performance, and system stability indicators, where B1 > B2 > B3 > 0, and B1 + B2 + B3 = 100; for example, (resource utilization: B1 = 40, business performance: B2 = 40, and system stability: B3 = 20).

[0094] The multidimensional allocation effect score is calculated based on the resource utilization index × B1 + business performance index × B2 + system stability index × B3. The multidimensional allocation effect score is then judged. If the multidimensional allocation effect score is less than the preset multidimensional allocation effect score threshold, an unqualified signal is generated. When an unqualified signal is generated, abnormal indicators that exceed the preset threshold (such as offline computing business task completion time exceeding the corresponding preset threshold, storage read / write latency exceeding the corresponding preset threshold) are filtered out to form an abnormal indicator list.

[0095] If the multidimensional allocation effect score is greater than or equal to the preset multidimensional allocation effect score threshold, a qualified signal is generated, and an abnormal indicator list or a qualified signal is output. The abnormal indicator list or the preset warning text corresponding to the qualified signal is immediately displayed to the operation and maintenance personnel so as to intuitively understand the allocation effect.

[0096] In summary, the process involves collecting and standardizing hardware resource status data, business load characteristic data, and historical allocation data. This establishes a correlation between business needs and resources, and builds a two-dimensional evaluation model to determine resource allocation priorities. Furthermore, by constructing an objective function that maximizes resource utilization and minimizes business performance deviations, the optimal resource allocation scheme is obtained. This reduces resource allocation problems caused by human error. Simultaneously, a dynamic adjustment mechanism prioritizes high-urgency business needs in mild conflict scenarios, avoiding resource idleness and waste. Additionally, a three-dimensional scoring system—resource utilization, business performance, and system stability—monitors the allocation effect in real time, along with feedback processing of abnormal indicators to obtain a list of abnormal indicators. This effectively avoids the risk of the optimal allocation scheme failing during implementation, ensuring that the scheme's implementation effect matches theoretical expectations and preventing resource waste.

[0097] The threshold is set for comparative analysis of results to determine whether they are good or bad. The value of the threshold is determined by a combination of large-scale model analysis of sample data and human experience. It can also be adjusted appropriately based on seasonal or common-sense influencing factors.

[0098] The size of the coefficient is a specific value obtained by quantifying each parameter to facilitate subsequent comparison. The size of the coefficient depends on the amount of sample data and the corresponding operating coefficient initially set by those skilled in the art for each set of sample data; as long as it does not affect the proportional relationship between the parameter and the quantified value.

[0099] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for dynamically adjusting computer system resource allocation based on big data analysis, characterized in that, Includes the following steps: Step 1: Collect hardware resource status data, business load characteristic data, and historical allocation data of the computer system, and preprocess them to obtain a standardized dataset; Step 2: Construct a business process-resource status association table and multi-dimensional data samples based on in-depth processing of standardized datasets; Step 3: Based on the resource allocation demand assessment of multi-dimensional data samples, construct a two-dimensional assessment model of resource demand gap value and performance deviation rate, calculate the demand priority score of each business process, and determine the resource allocation priority order table; Step 4: Construct an objective function that maximizes resource utilization and minimizes business performance deviation, set resource allocation constraints, and solve for the optimal allocation scheme; Step 5: The dynamic adjustment process of resource allocation within the optimal allocation scheme and the evaluation process of allocation effect within the preset evaluation period after resource adjustment.

2. The method of claim 1, wherein, The process of constructing the business process-resource status association table and the multi-dimensional data samples is as follows: S1: Assign globally unique identifiers to business processes and hardware resources based on standardized datasets; S2: Based on the allocation of a globally unique identifier, the collected service load characteristic data and hardware resource status data are bound in real time to form associated record data; S3: Preprocess the obtained associated record data to obtain the business process-resource status association table; S4: Based on the constructed business process-resource status association table, a multi-dimensional data sample is formed with business processes as the unit.

3. The method of claim 1, wherein the method further comprises: The process for obtaining the resource allocation priority list is as follows: Based on multi-dimensional data samples, the resource demand gap value of each business process is obtained. Resource demand gap value = current resource consumption of the business / historical average resource consumption of the business - current resource allocation ratio of the business / total available resource ratio. Set the offline computing task completion rate or the real-time transaction response latency as key performance indicators for the business. The performance deviation rate of each business process is calculated based on (business performance key indicators - business performance key indicator thresholds) / business performance key indicator thresholds. The weighting coefficient 'a' of the resource demand gap value of each business process and the weighting coefficient 'b' of the key business performance indicators are obtained, and both 'a' and 'b' are greater than zero. The demand priority score is calculated by multiplying the resource demand gap value by a weighting coefficient a and the key business performance indicators by a weighting coefficient b. The business processes are then sorted from high to low based on the demand priority scores to obtain a resource allocation priority order table.

4. The method of claim 1, wherein the method further comprises: The process for obtaining the optimal allocation scheme is as follows: Pre-construct objective functions to maximize resource utilization (Umax) and minimize business performance deviation (Dmin), where business processes are labeled as i, i = 1, 2, 3, ..., m, and m is a natural number greater than zero; Simultaneously set constraints to maximize resource utilization (Umax) and minimize business performance deviation (Dmin); The optimal resource allocation scheme is obtained by solving based on the pre-set solution algorithm and constraints.

5. The method of claim 1, wherein, The dynamic adjustment process is as follows: T1: Based on the optimal resource allocation scheme, the resource demand saturation during the resource allocation period is obtained. Resource demand saturation = (∑ target resource demand of business processes) / current available resource capacity of the system × 100%; T2: Retrieve the preset maximum resource demand saturation value BHmax and the preset minimum resource demand saturation value BHmin; T3: The resource demand saturation level is judged to determine whether there is no conflict, a slight conflict, or a severe conflict.

6. The method of claim 5, wherein the method further comprises: When a minor conflict is detected, the business priority data corresponding to the business process is obtained, and the business priority data is set to o, o = 1, 2, 3. The priority weight vo of each business priority data is obtained, that is, the priority weight v1 of high business priority data, the priority weight v2 of medium business priority data, and the priority weight v3 of low business priority data, v1 > v2 > v3 > 0. Calculate the urgency of each business process. Urgency = priority weight (vo) of each business priority data × resource demand gap value. Sort the processes by urgency from high to low to obtain a dynamic priority allocation list.

7. The method of claim 1, wherein the method further comprises: The evaluation process for the allocation effect within the preset evaluation period after resource adjustment is as follows: Based on the optimal resource allocation scheme, the allocation effect data within the preset evaluation period after resource adjustment is obtained. The allocation effect data includes resource utilization rate indicators, business performance indicators, and system stability indicators. Obtain the allocation ratio scores B1, B2, and B3 for resource utilization, business performance, and system stability indicators, where B1 > B2 > B3 > 0, and B1 + B2 + B3 = 100. The multidimensional allocation effect score is calculated based on the resource utilization rate index × B1 + business performance index × B2 + system stability index × B3. The multidimensional allocation effect score is then processed to obtain a list of abnormal indicators or qualified signals.

8. The method of claim 7, wherein the method further comprises: The resource utilization rate index = min(actual utilization rate / target utilization rate, 1); the business performance index is calculated based on ∑(1-|business performance key index - business performance key index threshold| / business performance key index threshold) × priority weight vo; the system stability index = 1-abnormal event occurrence rate.

Images