A method for dynamically allocating data processing resources for data middle platform

Abstract

The present invention discloses a method for dynamically allocating data processing resources for a data middle station, and relates to the technical field of data processing in power generation enterprises. The present invention processes the real-time data of the data middle station through a Kafka system and generates real-time data processing results; processes the offline data of the data middle station through a Hadoop system and generates offline data processing results; predicts the data volume of the real-time data processing results and the offline data processing results in a future unit time, and outputs a real-time data volume prediction value and an offline data volume prediction value; adopts a dynamic weight resource allocation algorithm to calculate the allocation of resource volume for the real-time data volume prediction value and the offline data volume prediction value; this method can more accurately match the resource demand in the future unit time, avoid over-allocation or under-allocation of resources, and greatly improve resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for dynamically allocating data processing resources for a data center. Background Art

[0002] The data middle platform consolidates the business and data of existing and newly built information systems, serving as an intermediate and supportive platform for data-enabled new businesses and applications. With the advancement of smart power plant development, thermal power generation companies are relying on the data middle platform to integrate multi-source, heterogeneous data (including real-time and historical streaming DCS data, equipment inventory maintenance and inspection data, equipment vibration monitoring, and environmental protection CEMS data). This platform supports the real-time computation of key algorithm modules such as combustion optimization, equipment failure prediction, and economic analysis. These modules require frequent access to massive, multi-source, heterogeneous datasets, with high real-time and bursty performance. For example, resource competition between different real-time intelligent prediction services, as well as resource allocation conflicts between existing and new business model training, pose significant challenges to the overall allocation of system resources. Especially with the continuous expansion of business scale and increasing model complexity, the need for efficient and rational allocation of system resources has become increasingly urgent, significantly increasing the pressure of resource competition.

[0003] Publication No. CN113836235B discloses a data processing method based on a data middle platform and related equipment. It processes real-time data through the Kafka system to obtain real-time data calculation results, and processes offline data through the Hadoop system. The Kafka system and the Hadoop system are integrated, while giving full play to their respective advantages and compensating for their own shortcomings through other systems.

[0004] However, the above application still has the following problems: real-time data processing and offline data processing are two core business scenarios. When performing real-time data processing and offline data processing, the above application lacks allocation of computing resources. Traditional data processing systems usually adopt static resource allocation strategies to pre-allocate a fixed proportion of computing resources for real-time and offline modules, such as the queue quota of Hadoop YARN in the Hadoop system, or the resource request limit of Kubernetes. The pre-allocation of fixed proportions of computing resources for real-time and offline modules has the problem of rigid resource allocation and inability to perform forward-looking resource scheduling based on future data volume predictions. Summary of the Invention

[0005] In order to solve the technical problems existing in the background technology, the present invention proposes a method for dynamically allocating data processing resources for a data center.

[0006] The present invention proposes a method for dynamically allocating data processing resources for a data center, which is characterized by comprising:

[0007] Step S1: Process the real-time data of the data center through the Kafka system and generate real-time data processing results;

[0008] Step S2: Process the offline data of the data center through the Hadoop system and generate offline data processing results;

[0009] Step S3: predicting the data volume of the real-time data processing results and the offline data processing results within a future unit time based on the time series analysis model, and outputting the real-time data volume prediction value and the offline data volume prediction value;

[0010] Step S4: using a dynamic weight resource allocation algorithm to allocate computing resources to the real-time data volume prediction value and the offline data volume prediction value.

[0011] Furthermore, the process of outputting the real-time data volume prediction value and the offline data volume prediction value includes:

[0012] The stability of the real-time data processing results and the offline data processing results is judged by time series diagrams, autocorrelation diagrams and statistical test methods;

[0013] If it is stable, the real-time data processing result or the offline data processing result adopts the ARIMA model to generate the corresponding real-time data volume prediction value in the future unit time and the offline data volume prediction value in the offline data processing module;

[0014] If it is non-stationary, the real-time data processing results or offline data processing results are differentially processed, and then fitted using the ARIMA model. The ARIMA model is used to generate the corresponding real-time data volume prediction value in the future unit time and the offline data volume prediction value in the offline data processing module.

[0015] Furthermore, the process of the dynamic weighted resource allocation algorithm includes:

[0016] The data complexity, current system load, and response speed requirements corresponding to the real-time data volume prediction value or the offline data volume prediction value are set as the influencing factors of resource allocation, and the resource weights are obtained through calculation; the resource volume allocation is calculated based on the resource weights.

[0017] Furthermore, the process of obtaining resource weights includes:

[0018] Let f1 be the real-time data volume prediction value or the offline data volume prediction value; obtain the data type of f1, marked as {image; video; text; value; BP neural network; LSTM neural network; MSET structure; office file; other file}; and obtain the value corresponding to the data type;

[0019] Let f2 be the data complexity, which is used to record the data call and resource usage at each moment;

[0020] Let f3 be the current load of the system;

[0021] Let f4 be the response speed requirement, and f4 be the preset response time threshold;

[0022] Normalize f1, f2, f3 and f4 to get 、 、 and ;

[0023] Among them, the normalization process of f1 includes:

[0024] Set the weight corresponding to the data type of f1, marked as data type weight, and weight and obtain the value of the data type corresponding to f1 according to the data type weight ;

[0025] Calculate resource weight RW:

[0026] The RW calculation formula is: ;

[0027] Among them, α, β, γ and δ are 、 、 and The corresponding weight.

[0028] Furthermore, the process of data complexity f2 includes:

[0029] The value range of f2 is [0,1], and the calculation formula of f2 is: ;

[0030] Among them, C ty is the data type diversity index, , dt is the total number of data types in f1, tdt is the preset maximum total number of data types;

[0031] C st is the data structure complexity index, , nl is the total number of nested layers of the data structure of each data type in f1, fc is the total number of data fields of each data type in f1, mnl is the preset maximum number of nested layers, and mfc is the preset maximum number of fields;

[0032] C re is the autocorrelation degree of f1, and its value range is [0,1];

[0033] C vo is the data volatility index, , is the standard deviation of the values ​​in f1, is the average value of the values ​​in f1;

[0034] 、 、 and C ty 、C st 、C re and C vo The weight of .

[0035] Furthermore, the resource allocation process includes:

[0036] Substitute the real-time data volume prediction value into the RW calculation formula to calculate the resource weight of the real-time data volume prediction value ;

[0037] Substitute the offline data volume prediction value obtained in the data prediction module into the resource weight calculation unit to calculate the resource weight of the offline data volume prediction value ;

[0038] Calculate the resource weight of the real-time data processing module , ;

[0039] Calculate the resource weight of the offline data processing module , ;

[0040] The total design computing resources are R, and the computing resources of the real-time data processing module are ;

[0041] The computing resources of the offline data processing module are .

[0042] Furthermore, the security enhancement module is used to encrypt and store the real-time data processing results and offline data processing results in the unified data storage module, and distinguish the processing permissions of internal and external data requests through access control policies.

[0043] Furthermore, it is characterized by comprising the following modules:

[0044] Real-time data processing module: processes the real-time data of the data center through the Kafka system and generates real-time data processing results;

[0045] Offline data processing module: processes offline data in the data center through the Hadoop system and generates offline data processing results;

[0046] Unified data storage module: used to store the real-time data processing results in the real-time data processing module and the offline data processing results in the offline data processing module;

[0047] Data prediction module: obtains the real-time data processing results and offline data processing results in the unified data storage module, predicts the data volume of the real-time data processing module and offline data processing module in the future unit time based on the time series analysis model, and outputs the real-time data volume prediction value and the offline data volume prediction value;

[0048] Intelligent scheduling module: obtains the real-time data volume prediction value and offline data volume prediction value of the data prediction module, and uses the dynamic weight resource allocation algorithm to allocate computing resources to the real-time data processing module and offline data processing module in the future unit time.

[0049] The present invention proposes a method for dynamically allocating data processing resources for a data center, which has the following beneficial technical effects:

[0050] The real-time data volume prediction value or offline data volume prediction value, data complexity, current system load and response speed requirements are introduced as influencing factors for resource allocation, and the resource weight can be dynamically adjusted according to the real-time data volume prediction value or offline data volume prediction value. This method can more accurately match the resource requirements of the real-time data processing module and the offline data processing module in the future unit time, realize dynamic differentiated resource allocation for the real-time data processing module and the offline data processing module, avoid over-allocation or under-allocation of resources, solve the problem of rigid resource allocation in traditional static resource allocation strategies, and greatly improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0052] During the implementation and deployment of a smart power plant system at a thermal power generation enterprise, actual operation monitoring revealed the following technical problems: with the continuous increase in the number of algorithm modules such as equipment safety warning and economic analysis, the overall system load increased significantly, and the response delay of key business data processing increased significantly.

[0053] After in-depth analysis and stress testing, we determined the root cause of the problem is as follows:

[0054] (1) Early warning modules generally use technologies such as neural networks and multivariate state estimation. Their reasoning process requires high-frequency access and integration of multi-source heterogeneous real-time data from different systems. The linear growth in the number of modules leads to an exponential increase in the concurrent use of computing resources (CPU), memory, and I / O bandwidth, becoming the primary factor in increasing system load.

[0055] (2) When deploying a new business module, it is necessary to call on massive amounts of historical offline data to train and optimize model parameters. This training task is computationally intensive and competes fiercely with existing online warning services for cluster resources (especially GPU / CPU computing power and storage I / O), further exacerbating real-time business response delays and even triggering task timeout alarms.

[0056] These issues severely restrict the scalability and reliability of smart power plant systems. To address the core conflict between real-time service latency and offline training resource conflicts, a dynamic allocation method for data processing resources must be implemented on the data center platform to optimize resource scheduling efficiency, ensure the real-time performance of key services, and improve overall system availability.

[0057] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention, and are not to be construed as limiting the present invention.

[0058] Hadoop YARN is the core component of the Hadoop system in existing technologies. It is a distributed resource management and scheduling framework that is mainly used to efficiently allocate and manage computing resources in a cluster and supports multiple distributed computing frameworks running on the same cluster.

[0059] Kubernetes is an open source container orchestration platform in existing technologies, used to automate the deployment, expansion, and management of containerized applications.

[0060] like Figure 1 A method for dynamically allocating data processing resources for a data center is shown, comprising:

[0061] Step S1: Process the real-time data of the data center through the Kafka system and generate real-time data processing results;

[0062] Step S2: Process the offline data of the data center through the Hadoop system and generate offline data processing results;

[0063] Step S3: predicting the data volume of the real-time data processing results and the offline data processing results within a future unit time based on the time series analysis model, and outputting the real-time data volume prediction value and the offline data volume prediction value;

[0064] Step S4: using a dynamic weight resource allocation algorithm to allocate computing resources to the real-time data volume prediction value and the offline data volume prediction value.

[0065] In an optional embodiment, the process of outputting the real-time data volume prediction value and the offline data volume prediction value includes:

[0066] The stability of the real-time data processing results and the offline data processing results is judged by time series diagrams, autocorrelation diagrams and statistical test methods;

[0067] If it is stable, the real-time data processing result or the offline data processing result adopts the ARIMA model to generate the corresponding real-time data volume prediction value in the future unit time and the offline data volume prediction value in the offline data processing module;

[0068] If it is non-stationary, the real-time data processing results or offline data processing results are differentially processed, and then fitted using the ARIMA model. The ARIMA model is used to generate the corresponding real-time data volume prediction value in the future unit time and the offline data volume prediction value in the offline data processing module.

[0069] The ARIMA model parameters are generated through residual analysis.

[0070] The ARIMA model is an autoregressive integrated moving average model, which is a statistical model used in the prior art to analyze and predict time series data;

[0071] In an optional embodiment, the dynamic weight resource allocation algorithm includes the following steps:

[0072] Obtain the real-time data volume prediction value and offline data volume prediction value of the data prediction module, set the real-time data volume prediction value or offline data volume prediction value, data complexity, current system load and response speed requirements as the influencing factors of resource allocation, and obtain resource weights through calculation;

[0073] The real-time data volume prediction value and offline data volume prediction value obtained in the data prediction module are substituted into the resource weight calculation step, the resource weight of the real-time data processing module and the resource weight of the offline data processing module are calculated, and the calculation resource allocation is performed for the real-time data processing module and the offline data processing module.

[0074] The intelligent scheduling module uses a dynamic weight resource allocation algorithm to allocate computing resources to the real-time data processing module and the offline data processing module in the future unit time. Unlike the traditional static resource allocation method, this algorithm introduces the real-time data volume prediction value or the offline data volume prediction value, data complexity, the current system load and response speed requirements as influencing factors for resource allocation, and can dynamically adjust the resource weight according to the real-time data volume prediction value or the offline data volume prediction value. This method can more accurately match the system's resource requirements in the future unit time, avoid over-allocation or under-allocation of resources, solve the problem of rigid resource allocation in traditional static resource allocation strategies, and greatly improve resource utilization.

[0075] In an optional embodiment, in the resource weight calculation step:

[0076] Let f1 be the real-time data volume prediction value or the offline data volume prediction value;

[0077] Let f2 be the data complexity, which is used to record the data call and resource usage at each moment; get the data type of f1, marked as {image; video; text; value; BP neural network; LSTM neural network; MSET structure; office-type file; other file}; and get the value corresponding to the data type;

[0078] Let f3 be the current load of the system;

[0079] Let f4 be the response speed requirement, and f4 be the preset response time threshold;

[0080] Normalize f1, f2, f3 and f4 to get 、 、 and ;

[0081] Among them, the normalization process of f1 includes:

[0082] Set the weight corresponding to the data type of f1, marked as data type weight, and weight and obtain the value of the data type corresponding to f1 according to the data type weight ;

[0083] It should be further explained that f1 means [total size of image files in Mb; total size of video files in Mb; total size of text files in Mb; number of numerical values; total number of neurons in BP neural network; total number of neurons in LSTM; total dimension of MSET structure; total size of OFFICE; total size of other files].

[0084] The normalized data of the above F1 real-time and F1 offline data sets are calculated as follows: the weighted sum of each item in F1, with weights of [b1; b2; b3; ...; bi], that is, [0.5; 2; 1; ...; bi],

[0085] Then f1=∑ai*bi;

[0086] Where i represents the number of the data type corresponding to f1, and its value is a positive integer; ai represents the value of the data type corresponding to i; bi represents the weight of the data type corresponding to i;

[0087] Calculate resource weight RW:

[0088] The RW calculation formula is: ;

[0089] Among them, α, β, γ and δ are 、 、 and The corresponding weight.

[0090] In an optional embodiment, in the resource allocation step, the data complexity f2 is calculated as follows:

[0091] Preferably, in the resource allocation step, the data complexity f2 is calculated as follows:

[0092] The value range of f2 is [0,1], and the calculation formula of f2 is: ;

[0093] Among them, C ty is the data type diversity index, , dt is the total number of data types in f1, tdt is the preset maximum total number of data types;

[0094] C st is the data structure complexity index, , nl is the total number of nested layers of the data structure of each data type in f1, fc is the total number of data fields of each data type in f1, mnl is the preset maximum number of nested layers, and mfc is the preset maximum number of fields;

[0095] C re is the autocorrelation degree of f1, and its value range is [0,1];

[0096] C vo is the data volatility index, , is the standard deviation of the values ​​in f1, is the average value of the values ​​in f1;

[0097] 、 、 and C ty 、C st 、C re and C vo The weight of .

[0098] In an optional embodiment, 、 、 、 The value of the normalized data C ty 、C st 、C re and C vo , and then calculated by entropy weight method 、 、 and The numerical value of .

[0099] In an optional embodiment, in the resource allocation step:

[0100] Substitute the real-time data volume prediction value obtained in the data prediction module into the resource weight calculation unit to calculate the resource weight of the real-time data volume prediction value ;

[0101] Substitute the offline data volume prediction value obtained in the data prediction module into the resource weight calculation unit to calculate the resource weight of the offline data volume prediction value ;

[0102] Calculate the resource weight of the real-time data processing module , ;

[0103] Calculate the resource weight of the offline data processing module , ;

[0104] The total design computing resources are R, and the computing resources of the real-time data processing module are ;

[0105] The computing resources of the offline data processing module are .

[0106] The present invention also provides a system for a method for dynamically allocating data processing resources for a data center, comprising the following modules:

[0107] Real-time data processing module: processes the real-time data of the data center through the Kafka system and generates real-time data processing results;

[0108] Offline data processing module: processes offline data in the data center through the Hadoop system and generates offline data processing results;

[0109] Unified data storage module: used to store the real-time data processing results in the real-time data processing module and the offline data processing results in the offline data processing module;

[0110] Data prediction module: obtains the real-time data processing results and offline data processing results in the unified data storage module, predicts the data volume of the real-time data processing module and offline data processing module in the future unit time based on the time series analysis model, and outputs the real-time data volume prediction value and the offline data volume prediction value;

[0111] Intelligent scheduling module: obtains the real-time data volume prediction value and offline data volume prediction value of the data prediction module, and uses the dynamic weight resource allocation algorithm to allocate computing resources to the real-time data processing module and offline data processing module in the future unit time.

[0112] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0113] In the embodiments provided by the present invention, it should be understood that the disclosed systems or methods can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative. For example, the division of modules is only a logical function division, and other division methods may be used in actual implementation.

[0114] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, and may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the objectives of this embodiment based on actual needs.

[0115] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing module, each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or hardware plus software functional modules.

[0116] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0117] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for dynamically allocating data processing resources for a data center, characterized in that: include: Step S1: Process the real-time data of the data center through the Kafka system and generate real-time data processing results; Step S2: Process the offline data of the data center through the Hadoop system and generate offline data processing results; Step S3: predicting the data volume of the real-time data processing results and the offline data processing results within a future unit time, and outputting the real-time data volume prediction value and the offline data volume prediction value; Step S4: using a dynamic weight resource allocation algorithm to allocate computing resources to the real-time data volume prediction value and the offline data volume prediction value; The stability of the real-time data processing results and the offline data processing results is judged by time series diagrams, autocorrelation diagrams and statistical test methods; If it is stable, the real-time data processing result or the offline data processing result adopts the ARIMA model to generate the corresponding real-time data volume prediction value in the future unit time and the offline data volume prediction value in the offline data processing module; If it is not stationary, the real-time data processing results or offline data processing results are differentially processed, and then fitted using the ARIMA model. The ARIMA model is used to generate the corresponding real-time data volume prediction value in the future unit time and the offline data volume prediction value in the offline data processing module; Set the data complexity, current system load, and response speed requirements corresponding to the real-time data volume prediction value or offline data volume prediction value as the influencing factors of resource allocation, and obtain the resource weight through calculation; Allocate computing resources based on resource weights; Let f1 be the real-time data volume prediction value or the offline data volume prediction value; obtain the data type of f1, marked as {image; video; text; value; BP neural network; LSTM neural network; MSET structure; office file; other file}; and obtain the value corresponding to the data type; Let f2 be the data complexity, which is used to record the data call and resource usage at each moment; Let f3 be the current load of the system; Let f4 be the response speed requirement, and f4 be the preset response time threshold; Normalize f1, f2, f3 and f4 to get and ; Among them, the normalization process of f1 includes: Set the weight corresponding to the data type of f1, marked as data type weight, and weight and obtain the value of the data type corresponding to f1 according to the data type weight ; Calculate resource weight RW: The RW calculation formula is: ; Among them, α, β, γ and δ are and The corresponding weight; The value range of f2 is [0,1], and the calculation formula of f2 is: ; Among them, C ty is the data type diversity index, , dt is the total number of data types in f1, tdt is the preset maximum total number of data types; C st is the data structure complexity index, , nl is the total number of nested layers of the data structure of each data type in f1, fc is the total number of data fields of each data type in f1, mnl is the preset maximum number of nested layers, and mfc is the preset maximum number of fields; C re is the autocorrelation degree of f1, and its value range is [0,1]; C vo is the data volatility index, , is the standard deviation of the values ​​in f1, is the average value of the values ​​in f1; and C ty 、C st 、C re and C vo The weight of .

2. A method for dynamically allocating data processing resources for a data center according to claim 1, characterized in that: The resource allocation process includes: Substitute the real-time data volume prediction value into the RW calculation formula to calculate the resource weight of the real-time data volume prediction value ; Substitute the offline data volume prediction value obtained in the data prediction module into the resource weight calculation unit to calculate the resource weight of the offline data volume prediction value ; Calculate the resource weight of the real-time data processing module ; Calculate the resource weight of the offline data processing module ; The total design computing resources are R, and the computing resources of the real-time data processing module are ; The computing resources of the offline data processing module are .

3. A method for dynamically allocating data processing resources for a data center according to claim 2, characterized in that: Security enhancement module: used to encrypt and store the real-time data processing results and offline data processing results in the unified data storage module, and distinguish the processing permissions of internal and external data requests through access control policies.

4. A method for dynamically allocating data processing resources for a data middle station according to any one of claims 1 to 3, characterized in that: Includes the following modules: Real-time data processing module: processes the real-time data of the data center through the Kafka system and generates real-time data processing results; Offline data processing module: processes offline data in the data center through the Hadoop system and generates offline data processing results; Unified data storage module: used to store the real-time data processing results in the real-time data processing module and the offline data processing results in the offline data processing module; Data prediction module: obtains the real-time data processing results and offline data processing results in the unified data storage module, predicts the data volume of the real-time data processing module and the offline data processing module in the future unit time, and outputs the real-time data volume prediction value and the offline data volume prediction value; Intelligent scheduling module: obtains the real-time data volume prediction value and offline data volume prediction value of the data prediction module, and uses the dynamic weight resource allocation algorithm to allocate computing resources to the real-time data processing module and offline data processing module in the future unit time.

Images