A method, system, equipment, and medium for dynamic resource optimization of ETL pipelines

Abstract

This application relates to the field of ETL resource optimization technology, specifically to a method, system, equipment, and medium for dynamic resource optimization of ETL pipelines. The method includes: collecting time-series data of the ETL pipeline operating environment, dividing it into time windows, and then standardizing it; generating a linear prediction sequence of system load indicators using a linear load prediction model; generating a coupled prediction sequence of system load indicators using a pre-trained multi-dimensional time-series load prediction model; integrating the two prediction sequences and then performing de-standardization to obtain a comprehensive prediction sequence of the original system load indicators; identifying peak periods, off-peak periods, and periods of abnormal resource scarcity based on the comprehensive prediction sequence of the original system load indicators; and dynamically adjusting and executing the scheduling strategy of ETL tasks based on the identification results. This application achieves automated optimization of ETL pipeline resources and improved system efficiency by collecting multi-source time-series data, combining two models to generate a comprehensive prediction sequence, and dynamically adjusting the scheduling strategy.

Description

Technical Field

[0001] This application relates to the field of ETL resource optimization technology, specifically to an ETL pipeline dynamic resource optimization method, system, equipment, and medium. Background Technology

[0002] In the current big data processing technology field, the ETL (Extract, Transform, Load) process, as a core component of data integration, is widely used in manufacturing, finance, and other scenarios. It is responsible for efficiently integrating data from heterogeneous data sources and loading it into the target system. The performance of the ETL pipeline directly affects the timeliness and business continuity of enterprise data processing. Especially against the backdrop of surging data volumes and increasing real-time requirements, resource optimization of the ETL pipeline has become a focus of industry attention.

[0003] In existing technologies, some ETL tools optimize resource utilization through predefined resource configurations and static scheduling rules. For example, they employ fixed parallelism and memory allocation strategies, combined with task priority sorting, to alleviate resource conflicts. Some methods also introduce simple threshold alarm mechanisms to trigger manual intervention when the system load exceeds a preset value, in order to maintain processing efficiency. These methods reduce the risk of resource idleness or overload to some extent and use historical data analysis to assist decision-making.

[0004] However, existing technologies have significant shortcomings: fixed resource configurations cannot adapt to dynamic fluctuations in load, resulting in resource shortages during peak hours and resource waste during off-peak hours; static scheduling strategies lack the ability to respond to real-time load changes and cannot predict and avoid potential bottlenecks; at the same time, data collection is limited to a single dimension (such as monitoring only CPU or memory), ignoring the coupling effect of industrial equipment indicators, ETL task scheduling indicators and system load indicators, resulting in insufficient prediction accuracy and difficulty in achieving intelligent adjustment. Summary of the Invention

[0005] To address the technical problems of existing ETL tools relying on fixed resource configurations and static scheduling strategies for resource optimization, which cannot adapt to load fluctuations in real time, leading to resource waste and performance bottlenecks, and lacking multi-dimensional data coupling and prediction capabilities, this application provides a dynamic resource optimization method, system, device, and medium for ETL pipelines. By collecting multi-source time-series data, combining ARIMA and LSTM models to generate a comprehensive prediction sequence, and dynamically adjusting the ETL scheduling strategy, it achieves automated optimization of ETL pipeline resources and improves system efficiency.

[0006] Firstly, this application provides a method for dynamic resource optimization of an ETL pipeline, comprising the following steps:

[0007] S1. Collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment;

[0008] S2. Divide the ETL pipeline operating environment time series data into time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window;

[0009] S3. Input the standardized observations of the time series data of the cluster system load index into the linear load prediction model to generate linear prediction values ​​of the system load index with a specified continuous prediction step size, thus forming a linear prediction sequence of the system load index. The linear load prediction model is constructed based on the ARIMA algorithm.

[0010] S4. Input the standardized feature vector of the continuous time window into the pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, thus forming a coupled prediction sequence of system load indicators. The multi-dimensional time-series load prediction model is constructed based on the LSTM algorithm.

[0011] S5. Integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, and perform destandardization on the comprehensive prediction sequence of the system load index to obtain the original comprehensive prediction sequence of the system load index.

[0012] S6. Based on the comprehensive prediction sequence of the original system load indicators, identify the peak periods, low periods and periods of abnormal resource shortage in the system load within the ETL pipeline operating environment, and dynamically adjust the scheduling strategy of ETL tasks according to the identification results;

[0013] S7. Execute the adjusted scheduling strategy.

[0014] It should be further noted that, in step S1, the industrial equipment indicators include:

[0015] Number of gateways operating normally;

[0016] Number of production equipment operating normally;

[0017] Number of quality inspection equipment in operation;

[0018] Number of other operating devices;

[0019] Number of alarms;

[0020] Quantity of products produced;

[0021] Quantity of products tested;

[0022] Number of faulty devices;

[0023] Number of items that failed equipment inspection.

[0024] It should be further noted that the ETL task scheduling metrics in step S1 include:

[0025] Number of offline tasks;

[0026] Real-time task count;

[0027] Number of task execution failures;

[0028] Number of tasks that failed due to external reasons;

[0029] Number of tasks failed due to resource constraints;

[0030] Number of tasks that failed due to internal reasons;

[0031] Number of parallel tasks;

[0032] Duration of non-ETL task execution.

[0033] It should be further noted that in step S1, the calculation of cluster system load metrics includes:

[0034] CPU utilization;

[0035] Network traffic;

[0036] Memory usage.

[0037] It should be further noted that the formula for standardization in step S2 is:

[0038]

[0039] In the formula, The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Observed values;

[0040] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The sample mean within the corresponding time window;

[0041] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The standard deviation of the samples within the corresponding time window;

[0042] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Standardized observations.

[0043] It should be further noted that step S3, which involves generating a linear prediction sequence of system load indicators with a specified prediction step size using a linear load prediction model, includes:

[0044] S301. Input the standardized observations of the time-series data of the cluster system load index into the linear load prediction model, and automatically select the optimal parameter combination of the ARIMA algorithm formula in the linear load prediction model using the AIC criterion. The ARIMA algorithm formula is as follows:

[0045]

[0046] in, For the first i The load metrics of the computing cluster system at a given time point Standardized observations;

[0047] For the first i The load metrics of the computing cluster system at a given time point Standardized observations;

[0048] for Order difference operator;

[0049] For constant terms;

[0050] These are the autoregressive coefficients;

[0051] The moving average coefficient;

[0052] This is the white noise error term;

[0053] These are the core parameters, where:

[0054] The number of autoregressive terms represents the number of lagged observations used in the linear load forecasting model;

[0055] The difference order represents the number of differences required to make the time series stationary;

[0056] The number of moving average terms represents the amount of lag error used in the linear load forecasting model;

[0057] The optimal parameter combination automatically selected using the AIC criterion is expressed as follows: ;

[0058] S302. Using the optimal parameter combination as the core parameter of the ARIMA algorithm recursive formula, the ARIMA algorithm recursive formula is used to generate linear prediction values ​​of the system load index with a specified continuous prediction step size. The ARIMA algorithm recursive formula is as follows:

[0059]

[0060] in, The time point of the last known observation;

[0061] To predict the step size;

[0062] For the first i The load metrics of the computing cluster system at a given time point The linear predicted value;

[0063] The load metrics of the computing cluster system at a given time point Standardized observations, when hour, = .

[0064] It should be further noted that in step S2, the length of the time window is 1 hour;

[0065] In step S302, the prediction step size is determined. =1-24, with a unit prediction step size of 1 hour.

[0066] It should be further noted that step S4, which involves using a pre-trained multi-dimensional time-series load prediction model to generate coupled predicted values ​​of system load metrics with a specified continuous prediction step size, includes:

[0067] S401. Each standardized feature vector is split into an input feature vector and a corresponding multidimensional label. The standardized feature vector consists of standardized observations of industrial equipment indicators and ETL task scheduling indicators, and the multidimensional label consists of standardized observations of computing cluster system load indicators.

[0068] S402. Use principal component analysis to reduce the dimensionality of the standardized eigenvectors to obtain the dimensionality-reduced eigenvectors. , For the first A feature after dimensionality reduction, The dimension after dimensionality reduction;

[0069] S403. Dimensionally reduce the eigenvectors of continuous time windows. Input a pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size.

[0070] It should be further noted that the training steps for the multidimensional time-series load forecasting model include:

[0071] S411. Collect historical data of ETL pipeline operation environment time series data to construct a sample set. 20% of the recent continuous historical data in the sample set is retained as the validation set, and the remaining 80% of the samples are used as the training set.

[0072] S412. Perform the same processing as steps S2, S401, and S402 on the validation set and training set to obtain the dimensionality-reduced feature vector of the training set, the multidimensional label of the training set, the dimensionality-reduced feature vector of the validation set, and the multidimensional label of the validation set.

[0073] S413. Input the dimensionality-reduced feature vector of the training set into the multidimensional time-series load prediction model to generate preliminary system load index coupled prediction values ​​with a specified continuous prediction step size. Each preliminary system load index coupled prediction value corresponds to a prediction time point.

[0074] S414. Use the multidimensional labels of the training set corresponding to each prediction time point as the true labels of that prediction time point, use the mean squared error as the loss function to calculate the loss and backpropagate the error to update the function.

[0075] Repeat steps S413-S414 to train the model. During training, use the validation set to monitor the multidimensional time-series load prediction model and retain the optimal model as the pre-trained multidimensional time-series load prediction model.

[0076] It should be further explained that in step S5, the weighted average method is used to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index. The formula is as follows:

[0077]

[0078] In the formula, For the first i The load metrics of the computing cluster system at a given time point The overall forecast value;

[0079] For the first i The load metrics of the computing cluster system at a given time point The linear predicted value;

[0080] For the first i The load metrics of the computing cluster system at a given time point The coupled predicted value;

[0081] For linear load prediction weighting coefficients;

[0082] Weighting coefficients for multidimensional time-series load prediction.

[0083] It should be further explained that obtaining and The steps are as follows:

[0084] S501. Calculate the mean squared error of the linear load prediction model on the validation set. Mean squared error of pre-trained multi-dimensional time-series load prediction model on validation set ;

[0085] S502. Calculate the weighting coefficients:

[0086]

[0087] .

[0088] It should be further noted that the formula used for destandardization is:

[0089]

[0090] For the first i The load metrics of the computing cluster system at a given time point The original composite forecast value.

[0091] It should be further explained that, in step S6, the method for identifying peak periods, off-peak periods, and periods of abnormal resource strain within the ETL pipeline operating environment is as follows:

[0092] The period during which the average load index of any cluster system is greater than or equal to 80% within a specified time window is defined as a peak period.

[0093] The period in which the average load index of all cluster systems is below 50% within a specified time window is defined as a low period.

[0094] The period during which the highest value of any cluster system load indicator is greater than or equal to 90% within a specified time window is defined as the peak period.

[0095] It should be further noted that, in step S6, the method for dynamically adjusting the scheduling strategy of the ETL task based on the recognition results includes:

[0096] The following measures will be implemented during peak hours:

[0097] Disable retrying for failed ETL scheduling and reschedule the recalculation to off-peak hours;

[0098] Non-critical ETL tasks are postponed to off-peak hours;

[0099] The total concurrent users during peak hours are limited to 80% of the original concurrent users, while the total concurrent users during off-peak hours are increased to 120% of the original concurrent users.

[0100] Provide users with suggestions for configuring new ETL scheduling times. If the user sets the scheduling time during peak hours, an alarm will pop up and indicate the off-peak hours.

[0101] During off-peak hours, tasks can increase memory usage and intra-task concurrency to speed up execution.

[0102] Send alarm notifications to relevant maintenance personnel;

[0103] During periods of extreme resource scarcity, the following measures shall be implemented:

[0104] Immediately cease non-emergency dispatching;

[0105] Send an emergency alert notification to the operations and maintenance team;

[0106] Record abnormal events in a log for later analysis.

[0107] Analysis of abnormally high resource scheduling.

[0108] Secondly, this application provides an ETL pipeline dynamic resource optimization system for implementing the above-mentioned ETL pipeline dynamic resource optimization method, including:

[0109] The time-series data acquisition module is used to collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment.

[0110] The data processing module is used to divide the ETL pipeline operating environment time series data into time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window.

[0111] The linear load forecasting module is used to input the standardized observations of the time series data of the system load index of the computing cluster into the linear load forecasting model, generate linear forecasts of the system load index with a specified continuous forecasting step size, and form a linear forecast sequence of the system load index.

[0112] The coupled load prediction module is used to input the standardized feature vector of the continuous time window into the pre-trained multi-dimensional time series load prediction model, generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, and form a coupled prediction sequence of system load indicators.

[0113] The prediction sequence integration and de-standardization module is used to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, and to de-standardize the comprehensive prediction sequence of the system load index to obtain the original comprehensive prediction sequence of the system load index.

[0114] The load period identification and scheduling strategy adjustment module is used to identify the peak periods, low periods and abnormally tight resource periods of the system load in the ETL pipeline operating environment based on the comprehensive prediction sequence of the original system load index, and dynamically adjust the scheduling strategy of ETL tasks according to the identification results.

[0115] The scheduling policy execution module is used to execute the adjusted scheduling policy.

[0116] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described ETL pipeline dynamic resource optimization method.

[0117] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described ETL pipeline dynamic resource optimization method.

[0118] As can be seen from the above technical solutions, this application has the following advantages:

[0119] 1. This application solves the problem of single data collection dimension and inability to fully reflect the system status in the prior art by collecting time-series data of the ETL pipeline operating environment (including industrial equipment indicators, ETL task scheduling indicators and computing cluster system load indicators), realizes the foundation for integrated analysis of multi-source data, and provides support for accurate prediction.

[0120] 2. This application uses the ARIMA algorithm to construct a linear load forecasting model to generate a linear forecasting sequence of system load indicators, and combines the LSTM algorithm to construct a multidimensional time-series load forecasting model to generate a coupled forecasting sequence of system load indicators. This solves the problem that the existing technology has a single forecasting model and cannot capture linear and nonlinear load trends, and realizes a comprehensive prediction of future resource demand.

[0121] 3. This application solves the problem of rigid scheduling strategies and inability to dynamically adapt to load changes in the prior art by integrating the linear prediction sequence and coupled prediction sequence of system load indicators into a comprehensive prediction sequence, and based on this, identifying peak periods, off-peak periods and periods of abnormal resource shortage, thereby realizing real-time optimization and adjustment of ETL task scheduling strategies.

[0122] 4. By implementing the adjusted scheduling strategy (such as limiting the number of concurrent users during peak hours and postponing non-critical tasks), the problem of the lack of automated execution mechanism in the existing technology is solved, the automated optimization of resource allocation is realized, and the operating efficiency and stability of the ETL pipeline are improved. Attached Figure Description

[0123] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0124] Figure 1 This is a flowchart of an ETL pipeline dynamic resource optimization method in one embodiment of this application.

[0125] Figure 2 This is a schematic block diagram of an ETL pipeline dynamic resource optimization system in one embodiment of this application.

[0126] Figure 3 This is a schematic diagram of the hardware structure of an electronic device in one embodiment of this application. Detailed Implementation

[0127] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this patent, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this patent.

[0128] The following will describe in detail the dynamic resource optimization method for ETL pipelines involved in this application. Specific details such as particular system structures and technologies are presented for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0129] In the ETL pipeline dynamic resource optimization method involved in this application, the term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or sets thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0130] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0131] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0132] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0133] The ETL pipeline dynamic resource optimization method provided in this application embodiment is executed by a computer device, and correspondingly, the ETL pipeline dynamic resource optimization system runs in the computer device.

[0134] Figure 1 This is a flowchart of an embodiment of the ETL pipeline dynamic resource optimization method of this application. Wherein, Figure 1 The executing entity can be an ETL pipeline dynamic resource optimization system. Depending on different requirements, the order of steps in this flowchart can be changed, and some can be omitted.

[0135] like Figure 1 As shown, the dynamic resource optimization method for the ETL pipeline includes:

[0136] Step S1: Collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment.

[0137] By limiting the collection of three types of time-series data—industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators—comprehensive and multi-dimensional data collection of the ETL pipeline operating environment status (underlying support, task requirements, and system resources) was achieved, providing the necessary data foundation for subsequent analysis and prediction.

[0138] In some specific embodiments, industrial equipment specifications include:

[0139] Number of gateways operating normally;

[0140] Number of production equipment operating normally;

[0141] Number of quality inspection equipment in operation;

[0142] Number of other operating devices;

[0143] Number of alarms;

[0144] Quantity of products produced;

[0145] Quantity of products tested;

[0146] Number of faulty devices;

[0147] Number of items that failed equipment inspection.

[0148] By limiting the selection range of industrial equipment indicators to cover key aspects such as equipment operating status, production efficiency, alarms, and quality, a comprehensive quantitative perception of the industrial environment operating status supporting ETL pipeline operation is achieved, providing key input data reflecting the intensity of underlying industrial activities for subsequent load prediction models.

[0149] In some specific embodiments, ETL task scheduling metrics include:

[0150] Number of offline tasks;

[0151] Real-time task count;

[0152] Number of task execution failures;

[0153] Number of tasks that failed due to external reasons;

[0154] Number of tasks failed due to resource constraints;

[0155] Number of tasks that failed due to internal reasons;

[0156] Number of parallel tasks;

[0157] Duration of non-ETL task execution.

[0158] By limiting the selection range of ETL task scheduling metrics to cover core dimensions such as task type, execution status (especially the breakdown of failure reasons), parallelism, and idle status, fine-grained monitoring of ETL task execution behavior characteristics is achieved, providing key input data for subsequent load prediction models that reflect the needs and status changes of ETL tasks themselves.

[0159] In some specific embodiments, the calculation of cluster system load metrics includes:

[0160] CPU utilization;

[0161] Network traffic;

[0162] Memory usage.

[0163] By limiting the selection range of computing cluster system load indicators to focus on three core performance dimensions—CPU utilization, network traffic, and memory usage—standardized measurements of key aspects of resource consumption in the computing environment where the ETL pipeline resides are achieved, providing a clear target for load prediction models and a benchmark for system status assessment.

[0164] Step S2: Divide the ETL pipeline operating environment time series data into time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window.

[0165] By dividing the collected time-series data into time windows and performing standardization processing, and organizing them into standardized feature vectors for each time window, the original heterogeneous data was standardized and structured, eliminating dimensional differences and preparing it for input into subsequent prediction models.

[0166] In some specific embodiments, the formula for standardization is:

[0167]

[0168] In the formula, The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Observed values;

[0169] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The sample mean within the corresponding time window;

[0170] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The standard deviation of the samples within the corresponding time window;

[0171] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Standardized observations.

[0172] By using the Z-score formula based on the sample mean and sample standard deviation within a time window for standardized processing, the normalization of raw time series data of different dimensions and magnitudes was achieved, eliminating scale differences between indicators and providing a data foundation for subsequent feature vector construction and model training and prediction.

[0173] Step S3: Input the standardized observations of the time series data of the cluster system load index into the linear load prediction model to generate linear prediction values ​​of the system load index with a specified continuous prediction step size, thus forming a linear prediction sequence of the system load index. The linear load prediction model is constructed based on the ARIMA algorithm.

[0174] By inputting standardized system load index data into a linear load forecasting model based on ARIMA to generate a short-term linear forecast sequence, efficient and statistically supported short-term trend extrapolation forecasting of system load is achieved, relying solely on historical load data.

[0175] In some specific embodiments, the step of generating a linear prediction sequence of system load metrics with a specified prediction step size using a linear load prediction model includes:

[0176] S301. Input the standardized observations of the time-series data of the cluster system load index into the linear load prediction model, and automatically select the optimal parameter combination of the ARIMA algorithm formula in the linear load prediction model using the AIC criterion. The ARIMA algorithm formula is as follows:

[0177]

[0178] in, For the first i The load metrics of the computing cluster system at a given time point Standardized observations;

[0179] For the first i The load metrics of the computing cluster system at a given time point Standardized observations;

[0180] for Order difference operator;

[0181] For constant terms;

[0182] These are the autoregressive coefficients;

[0183] The moving average coefficient;

[0184] This is the white noise error term;

[0185] These are the core parameters, where:

[0186] The number of autoregressive terms represents the number of lagged observations used in the linear load forecasting model;

[0187] The difference order represents the number of differences required to make the time series stationary;

[0188] The number of moving average terms represents the amount of lag error used in the linear load forecasting model;

[0189] The optimal parameter combination automatically selected using the AIC criterion is expressed as follows: ;

[0190] S302. Using the optimal parameter combination as the core parameter of the ARIMA algorithm recursive formula, the ARIMA algorithm recursive formula is used to generate linear prediction values ​​of the system load index with a specified continuous prediction step size. The ARIMA algorithm recursive formula is as follows:

[0191]

[0192] in, The time point of the last known observation;

[0193] To predict the step size;

[0194] For the first i The load metrics of the computing cluster system at a given time point The linear predicted value;

[0195] The load metrics of the computing cluster system at a given time point Standardized observations, when hour, = .

[0196] By defining the specific application steps of the linear load forecasting model, including automatically selecting the optimal parameter combination using the AIC criterion and performing multi-step forecasting using recursive formulas, efficient and interpretable short-term linear extrapolation forecasting of system load based on the temporal dependencies of historical load data is achieved.

[0197] In some specific embodiments, in step S2, the length of the time window is 1 hour;

[0198] In step S302, the prediction step size is determined. =1-24, with a unit prediction step size of 1 hour.

[0199] By limiting the time window length to 1 hour and the prediction step size to 1-24 hours, consistency between the input data and the prediction output of the prediction model in terms of time granularity is achieved, ensuring the feasibility and practicality of short-term prediction of system load in hourly units.

[0200] Step S4: Input the standardized feature vector of the continuous time window into the pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, thus forming a coupled prediction sequence of system load indicators. The multi-dimensional time-series load prediction model is constructed based on the LSTM algorithm.

[0201] By inputting the standardized feature vector of a continuous time window into a multi-dimensional time-series load prediction model based on LSTM to generate a coupled prediction sequence, nonlinear system load prediction considering the complex interactions of multiple environmental factors is realized.

[0202] In some specific embodiments, the step of generating coupled predicted values ​​of system load metrics with a specified continuous prediction step size using a pre-trained multi-dimensional time-series load prediction model includes:

[0203] S401. Each standardized feature vector is split into an input feature vector and a corresponding multidimensional label. The standardized feature vector consists of standardized observations of industrial equipment indicators and ETL task scheduling indicators, and the multidimensional label consists of standardized observations of computing cluster system load indicators.

[0204] S402. Use principal component analysis to reduce the dimensionality of the standardized eigenvectors to obtain the dimensionality-reduced eigenvectors. , For the first A feature after dimensionality reduction, The dimension after dimensionality reduction;

[0205] S403. Dimensionally reduce the eigenvectors of continuous time windows. Input a pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size.

[0206] By limiting the input of the multidimensional time-series load prediction model to a continuous feature vector that has undergone dimensionality reduction and integrates industrial equipment indicators and ETL task scheduling indicators, nonlinear system load prediction that considers the complex coupling relationship of multi-source heterogeneous factors in the ETL pipeline operating environment is realized.

[0207] In some specific embodiments, the training steps of the multidimensional time-series load prediction model include:

[0208] S411. Collect historical data of ETL pipeline operation environment time series data to construct a sample set. 20% of the recent continuous historical data in the sample set is retained as the validation set, and the remaining 80% of the samples are used as the training set.

[0209] S412. Perform the same processing as steps S2, S401, and S402 on the validation set and training set to obtain the dimensionality-reduced feature vector of the training set, the multidimensional label of the training set, the dimensionality-reduced feature vector of the validation set, and the multidimensional label of the validation set.

[0210] S413. Input the dimensionality-reduced feature vector of the training set into the multidimensional time-series load prediction model to generate preliminary system load index coupled prediction values ​​with a specified continuous prediction step size. Each preliminary system load index coupled prediction value corresponds to a prediction time point.

[0211] S414. Use the multidimensional labels of the training set corresponding to each prediction time point as the true labels of that prediction time point, use the mean squared error as the loss function to calculate the loss and backpropagate the error to update the function.

[0212] Repeat steps S413-S414 to train the model. During training, use the validation set to monitor the multidimensional time-series load prediction model and retain the optimal model as the pre-trained multidimensional time-series load prediction model.

[0213] By clarifying the training process of the multidimensional time-series load prediction model (LSTM), including dataset partitioning, standardization and dimensionality reduction, model training and validation set monitoring, a prediction model with stronger generalization ability is constructed based on learning the complex nonlinear mapping relationship between multi-source features of the environment and system load from historical data.

[0214] Step S5: Integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, and perform inverse standardization on the comprehensive prediction sequence of the system load index to obtain the original comprehensive prediction sequence of the system load index.

[0215] By integrating linear and coupled prediction sequences into a comprehensive prediction sequence and performing denormalization, a more robust prediction result that combines the advantages of different prediction models is achieved, and the predicted values ​​are restored to their original dimensions for practical application.

[0216] In some specific embodiments, a weighted average method is used to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, as shown in the formula:

[0217]

[0218] In the formula, For the first i The load metrics of the computing cluster system at a given time point The overall forecast value;

[0219] For the first i The load metrics of the computing cluster system at a given time point The linear predicted value;

[0220] For the first i The load metrics of the computing cluster system at a given time point The coupled predicted value;

[0221] For linear load prediction weighting coefficients;

[0222] Weighting coefficients for multidimensional time-series load prediction.

[0223] By limiting the use of the weighted average method to integrate linear and coupled prediction sequences, and calculating dynamic weight coefficients based on the prediction errors of each model on the validation set, a more robust and accurate comprehensive prediction of system load is achieved, which combines the advantages of linear extrapolation and nonlinear coupling.

[0224] In some specific embodiments, obtaining and The steps are as follows:

[0225] S501. Calculate the mean squared error of the linear load prediction model on the validation set. Mean squared error of pre-trained multi-dimensional time-series load prediction model on validation set ;

[0226] S502. Calculate the weighting coefficients:

[0227]

[0228] .

[0229] By defining a specific formula for calculating the weight coefficients (based on the reciprocal of the mean squared error loss of each model on the validation set), the contribution weights of each model in the overall prediction can be objectively and quantitatively allocated according to its historical prediction performance, thereby optimizing the accuracy of the prediction results.

[0230] In some specific embodiments, the formula used for denormalization is:

[0231]

[0232] For the first i The load metrics of the computing cluster system at a given time point The original composite forecast value.

[0233] By explicitly using the sample mean and standard deviation of the original time window for de-standardization to restore the comprehensive forecast value, the normalized forecast value is converted back to the original dimensional and practically meaningful system load index value, providing directly understandable forecast data for subsequent resource period identification.

[0234] Step S6: Based on the comprehensive prediction sequence of the original system load indicators, identify the peak periods, low periods, and periods of abnormal resource shortage in the system load within the ETL pipeline operating environment, and dynamically adjust the scheduling strategy of the ETL task according to the identification results.

[0235] By identifying peak periods, off-peak periods, and periods of abnormal resource shortage based on the original comprehensive prediction sequence, and dynamically adjusting the ETL task scheduling strategy according to the identification results, a forward-looking and differentiated optimization configuration of task scheduling based on the predicted system load status is achieved.

[0236] In some specific embodiments, the method for identifying peak periods, off-peak periods, and periods of abnormal resource strain in the system load within the ETL pipeline operating environment is as follows:

[0237] The period during which the average load index of any cluster system is greater than or equal to 80% within a specified time window is defined as a peak period.

[0238] The period in which the average load index of all cluster systems is below 50% within a specified time window is defined as a low period.

[0239] The period during which the highest value of any cluster system load indicator is greater than or equal to 90% within a specified time window is defined as the peak period.

[0240] By defining the threshold rules for determining periods of abnormal resource scarcity during peak and off-peak hours, the system achieves automated and standardized identification of periods with different future load risk levels based on comprehensive prediction sequences.

[0241] In some specific embodiments, the method for dynamically adjusting the scheduling strategy of ETL tasks based on the identification results includes:

[0242] The following measures will be implemented during peak hours:

[0243] Disable retrying for failed ETL scheduling and reschedule the recalculation to off-peak hours;

[0244] Non-critical ETL tasks are postponed to off-peak hours;

[0245] The total concurrent users during peak hours are limited to 80% of the original concurrent users, while the total concurrent users during off-peak hours are increased to 120% of the original concurrent users.

[0246] Provide users with suggestions for configuring new ETL scheduling times. If the user sets the scheduling time during peak hours, an alarm will pop up and indicate the off-peak hours.

[0247] During off-peak hours, tasks can increase memory usage and intra-task concurrency to speed up execution.

[0248] Send alarm notifications to relevant maintenance personnel;

[0249] During periods of extreme resource scarcity, the following measures shall be implemented:

[0250] Immediately cease non-emergency dispatching;

[0251] Send an emergency alert notification to the operations and maintenance team;

[0252] Record abnormal events in a log for later analysis.

[0253] Analysis of abnormally high resource scheduling.

[0254] By implementing specific scheduling strategy adjustments (such as task retry control, task priority adjustment, concurrency limiting, and alarm notification) during identified peak periods and periods of abnormal resource stress, proactive and refined intervention in ETL task scheduling based on predicted resource stress is achieved. This avoids resource bottlenecks, optimizes task execution efficiency, and ensures system stability.

[0255] Step S7: Execute the adjusted scheduling strategy.

[0256] By implementing the adjusted scheduling strategy, prediction-based optimization decisions were put into practice, ultimately achieving the goals of dynamically adjusting resource allocation, optimizing task execution, and ensuring stable system operation.

[0257] In one specific embodiment, the steps of the ETL pipeline dynamic resource optimization method include:

[0258] Step S1: Collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment, wherein:

[0259] Industrial equipment specifications include:

[0260] Number of gateways operating normally;

[0261] Number of production equipment operating normally;

[0262] Number of quality inspection equipment in operation;

[0263] Number of other operating devices;

[0264] Number of alarms;

[0265] Quantity of products produced;

[0266] Quantity of products tested;

[0267] Number of faulty devices;

[0268] Number of items failing equipment inspection;

[0269] ETL task scheduling metrics include:

[0270] Number of offline tasks;

[0271] Real-time task count;

[0272] Number of task execution failures;

[0273] Number of tasks that failed due to external reasons;

[0274] Number of tasks failed due to resource constraints;

[0275] Number of tasks that failed due to internal reasons;

[0276] Number of parallel tasks;

[0277] Duration of non-ETL task execution;

[0278] The load metrics of a computing cluster system include:

[0279] CPU utilization;

[0280] Network traffic;

[0281] Memory usage.

[0282] Step S2: Divide the ETL pipeline operating environment time series data into 1-hour time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window. The standardization formula is as follows:

[0283]

[0284] In the formula, The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Observed values;

[0285] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The sample mean within the corresponding time window;

[0286] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The standard deviation of the samples within the corresponding time window;

[0287] The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Standardized observations.

[0288] Step S3 involves inputting the standardized observations of the time-series data of the cluster system load index into the linear load prediction model to generate linear prediction values ​​of the system load index with a specified continuous prediction step size, thus forming a linear prediction sequence of the system load index. The linear load prediction model is constructed based on the ARIMA algorithm, and the steps include:

[0289] S301. Input the standardized observations of the time-series data of the cluster system load index into the linear load prediction model, and automatically select the optimal parameter combination of the ARIMA algorithm formula in the linear load prediction model using the AIC criterion. The ARIMA algorithm formula is as follows:

[0290]

[0291] in, For the first i The load metrics of the computing cluster system at a given time point Standardized observations;

[0292] For the first i The load metrics of the computing cluster system at a given time point Standardized observations;

[0293] for Order difference operator;

[0294] For constant terms;

[0295] These are the autoregressive coefficients;

[0296] The moving average coefficient;

[0297] This is the white noise error term;

[0298] These are the core parameters, where:

[0299] The number of autoregressive terms represents the number of lagged observations used in the linear load forecasting model;

[0300] The difference order represents the number of differences required to make the time series stationary;

[0301] The number of moving average terms represents the amount of lag error used in the linear load forecasting model;

[0302] The optimal parameter combination automatically selected using the AIC criterion is expressed as follows: ;

[0303] S302. Using the optimal parameter combination as the core parameter of the ARIMA algorithm recursive formula, the ARIMA algorithm recursive formula is used to generate linear prediction values ​​of the system load index with a specified continuous prediction step size. The ARIMA algorithm recursive formula is as follows:

[0304]

[0305] in, The time point of the last known observation;

[0306] To predict the step size, =1-24, with a prediction step size of 1 hour;

[0307] For the first i The load metrics of the computing cluster system at a given time point The linear predicted value;

[0308] The load metrics of the computing cluster system at a given time point Standardized observations, when hour, = .

[0309] Step S4: Input the standardized feature vectors of continuous time windows into the pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, forming a coupled prediction sequence of system load indicators. The multi-dimensional time-series load prediction model is constructed based on the LSTM algorithm, and the steps include:

[0310] S401. Each standardized feature vector is split into an input feature vector and a corresponding multidimensional label. The standardized feature vector consists of standardized observations of industrial equipment indicators and ETL task scheduling indicators, and the multidimensional label consists of standardized observations of computing cluster system load indicators.

[0311] S402. Use principal component analysis to reduce the dimensionality of the standardized eigenvectors to obtain the dimensionality-reduced eigenvectors. , For the first A feature after dimensionality reduction, The dimension after dimensionality reduction;

[0312] S403. Dimensionally reduce the feature vectors of the past 168 consecutive time windows. Input a pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, where each time window is 1 hour.

[0313] The training steps for a multidimensional time-series load forecasting model include:

[0314] S411. Collect historical data of ETL pipeline operation environment time series data to construct a sample set. 20% of the recent continuous historical data in the sample set is retained as the validation set, and the remaining 80% of the samples are used as the training set.

[0315] S412. Perform the same processing as steps S2, S401, and S402 on the validation set and training set to obtain the dimensionality-reduced feature vector of the training set, the multidimensional label of the training set, the dimensionality-reduced feature vector of the validation set, and the multidimensional label of the validation set.

[0316] S413. Input the dimensionality-reduced feature vector of the training set into the multidimensional time-series load prediction model to generate preliminary system load index coupled prediction values ​​with a specified continuous prediction step size. Each preliminary system load index coupled prediction value corresponds to a prediction time point.

[0317] S414. Use the multidimensional labels of the training set corresponding to each prediction time point as the true labels of that prediction time point, use the mean squared error as the loss function to calculate the loss and backpropagate the error to update the function.

[0318] Repeat steps S413-S414 to train the model. During training, use the validation set to monitor the multidimensional time-series load prediction model and retain the optimal model as the pre-trained multidimensional time-series load prediction model.

[0319] Step S5: Use the weighted average method to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index. The formula is:

[0320]

[0321] In the formula, For the first i The load metrics of the computing cluster system at a given time point The overall forecast value;

[0322] For the first i The load metrics of the computing cluster system at a given time point The linear predicted value;

[0323] For the first i The load metrics of the computing cluster system at a given time point The coupled predicted value;

[0324] For linear load prediction weighting coefficients;

[0325] Weighting coefficients for multi-dimensional time-series load prediction;

[0326] Get and The steps are as follows:

[0327] S501. Calculate the mean squared error of the linear load prediction model on the validation set. Mean squared error of pre-trained multi-dimensional time-series load prediction model on validation set ;

[0328] S502. Calculate the weighting coefficients:

[0329]

[0330] ;

[0331] The original system load index comprehensive prediction sequence is obtained by inverse standardization of the comprehensive prediction sequence, and the formula is:

[0332]

[0333] For the first i The load metrics of the computing cluster system at a given time point The original composite forecast value.

[0334] Step S6: Based on the comprehensive prediction sequence of the original system load indicators, identify the peak periods, off-peak periods, and periods of abnormal resource shortage in the system load within the ETL pipeline operating environment, and dynamically adjust the scheduling strategy of the ETL task according to the identification results.

[0335] The method for identifying peak, off-peak, and abnormally resource-constrained periods of system load within the ETL pipeline operating environment is as follows:

[0336] The period during which the average load index of any cluster system is greater than or equal to 80% within a specified time window is defined as a peak period.

[0337] The period in which the average load index of all cluster systems is below 50% within a specified time window is defined as a low period.

[0338] The period during which the highest value of any cluster system load indicator is greater than or equal to 90% within a specified time window is defined as the peak period.

[0339] Methods for dynamically adjusting the scheduling strategy of ETL tasks based on the identification results include:

[0340] The following measures will be implemented during peak hours:

[0341] Disable retrying for failed ETL scheduling and reschedule the recalculation to off-peak hours;

[0342] Non-critical ETL tasks are postponed to off-peak hours;

[0343] The total concurrent users during peak hours are limited to 80% of the original concurrent users, while the total concurrent users during off-peak hours are increased to 120% of the original concurrent users.

[0344] Provide users with suggestions for configuring new ETL scheduling times. If the user sets the scheduling time during peak hours, an alarm will pop up and indicate the off-peak hours.

[0345] During off-peak hours, tasks can increase memory usage and intra-task concurrency to speed up execution.

[0346] Send alarm notifications to relevant maintenance personnel;

[0347] During periods of extreme resource scarcity, the following measures shall be implemented:

[0348] Immediately cease non-emergency dispatching;

[0349] Send an emergency alert notification to the operations and maintenance team;

[0350] Record abnormal events in a log for later analysis.

[0351] Analysis of abnormally high resource scheduling includes:

[0352] For peak periods, especially periods of severe resource shortage, high resource consumption scheduling screening is carried out. The resource consumption of each ETL job is combined with the predicted value for the next 24 hours to assess the potential impact of each job on system resources in a specific time period.

[0353] ETL jobs for each study period The average CPU utilization rate is specified as follows: Average memory usage is Average network traffic is The predicted resource utilization rate for the next hour t is... , , Define the impact factor To measure homework At the point of timet Impact on system resources:

[0354]

[0355] Weighting coefficients are set based on the importance of the three predicted values: , , ;

[0356] Based on the calculated impact factors This allows us to identify which ETL jobs have a significant impact on system resources within a specific time period, and define ETL jobs that have a significant impact on system resources as high-impact-factor tasks.

[0357] Step S7: Execute the adjusted scheduling strategy. Based on the APIs of DolphinScheduler and Seatunnel, dynamically modify the ETL job configuration parameters to improve the system's automation and response speed. DolphinScheduler, as a workflow scheduling system, is responsible for scheduling and managing ETL tasks. Seatunnel, as a lightweight data synchronization tool, is responsible for data extraction, transformation, and loading (ETL) operations. By calling the Seatunnel API through DolphinScheduler, the dynamic configuration and execution of ETL tasks are achieved.

[0358] The specific steps include:

[0359] The scheduling policy output in step S6 is converted into an interface. The DolphinScheduler calls the Seatunnel's modification interface to dynamically adjust the configuration parameters of the ETL job. The adjustments are made according to the specific decision API, including parallelism, memory allocation, etc.

[0360] Based on the adjusted ETL job configuration, the execution time and order of ETL tasks during peak hours are rearranged through the task scheduling interface of DolphinScheduler. Based on load prediction, suggestions and alarms are provided for the execution time of newly added ETL schedules.

[0361] For the high-impact factor tasks identified and analyzed in step S6, an alert will be pushed out and information such as suggested execution time will be provided.

[0362] Trigger the execution of ETL tasks and monitor their running status through DolphinScheduler;

[0363] During the execution of ETL tasks, the running status and resource usage of the tasks can be obtained through the real-time monitoring interfaces provided by DolphinScheduler and Seatunnel.

[0364] If any abnormal situation or resource shortage is detected, an alarm should be triggered in a timely manner and corresponding adjustment measures should be taken.

[0365] The following are embodiments of the ETL pipeline dynamic resource optimization system provided in this application. This ETL pipeline dynamic resource optimization system and the ETL pipeline dynamic resource optimization methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the ETL pipeline dynamic resource optimization system, please refer to the embodiments of the above ETL pipeline dynamic resource optimization methods.

[0366] like Figure 2 As shown, the ETL pipeline dynamic resource optimization system includes:

[0367] The time-series data acquisition module is used to collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment.

[0368] The data processing module is used to divide the ETL pipeline operating environment time series data into time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window.

[0369] The linear load forecasting module is used to input the standardized observations of the time series data of the system load index of the computing cluster into the linear load forecasting model, generate linear forecasts of the system load index with a specified continuous forecasting step size, and form a linear forecast sequence of the system load index.

[0370] The coupled load prediction module is used to input the standardized feature vector of the continuous time window into the pre-trained multi-dimensional time series load prediction model, generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, and form a coupled prediction sequence of system load indicators.

[0371] The prediction sequence integration and de-standardization module is used to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, and to de-standardize the comprehensive prediction sequence of the system load index to obtain the original comprehensive prediction sequence of the system load index.

[0372] The load period identification and scheduling strategy adjustment module is used to identify the peak periods, low periods and abnormally tight resource periods of the system load in the ETL pipeline operating environment based on the comprehensive prediction sequence of the original system load index, and dynamically adjust the scheduling strategy of ETL tasks according to the identification results.

[0373] The scheduling policy execution module is used to execute the adjusted scheduling policy.

[0374] The ETL pipeline dynamic resource optimization system in this embodiment is used to implement the ETL pipeline dynamic resource optimization method.

[0375] This application also provides an electronic device for implementing the various embodiments of this application. Figure 3 To illustrate the hardware structure of an electronic device according to various embodiments of this application, as shown in the following diagram... Figure 3 As shown, the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor.

[0376] Those skilled in the art will understand that the electronic device structure involved in the embodiments of this application does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0377] In embodiments of this application, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0378] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0379] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0380] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0381] This application also provides a storage medium storing a program product capable of implementing the ETL pipeline dynamic resource optimization method. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the foregoing "Exemplary Methods" section of this specification according to the various exemplary embodiments of this application.

[0382] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0383] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dynamic resource optimization method for ETL pipelines, characterized in that, include: S1. Collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment; S2. Divide the ETL pipeline operating environment time series data into time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window; S3. Input the standardized observations of the time series data of the cluster system load index into the linear load prediction model to generate linear prediction values ​​of the system load index with a specified continuous prediction step size, thus forming a linear prediction sequence of the system load index. The linear load prediction model is constructed based on the ARIMA algorithm. S4. Input the standardized feature vector of the continuous time window into the pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, thus forming a coupled prediction sequence of system load indicators. The multi-dimensional time-series load prediction model is constructed based on the LSTM algorithm. The steps for generating coupled predicted values ​​of system load metrics with a specified continuous prediction step size using a pre-trained multidimensional time-series load prediction model include: S401. Each standardized feature vector is split into an input feature vector and a corresponding multidimensional label. The standardized feature vector consists of standardized observations of industrial equipment indicators and ETL task scheduling indicators, and the multidimensional label consists of standardized observations of computing cluster system load indicators. S402. Use principal component analysis to reduce the dimensionality of the standardized eigenvectors to obtain the dimensionality-reduced eigenvectors. , For the first A feature after dimensionality reduction, The dimension after dimensionality reduction; S403. Dimensionally reduce the eigenvectors of continuous time windows. Input a pre-trained multi-dimensional time-series load prediction model to generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size; S5. Integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, and perform destandardization on the comprehensive prediction sequence of the system load index to obtain the original comprehensive prediction sequence of the system load index. S6. Based on the comprehensive prediction sequence of the original system load indicators, identify the peak periods, low periods and periods of abnormal resource shortage in the system load within the ETL pipeline operating environment, and dynamically adjust the scheduling strategy of ETL tasks according to the identification results; S7. Execute the adjusted scheduling strategy.

2. The ETL pipeline dynamic resource optimization method as described in claim 1, characterized in that, In step S1, the calculation of cluster system load metrics includes: CPU utilization; Network traffic; Memory usage.

3. The ETL pipeline dynamic resource optimization method as described in claim 1, characterized in that, In step S2, the formula for standardization is: In the formula, The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Observed values; The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The sample mean within the corresponding time window; The first in the ETL pipeline operating environment timing data i Each indicator at a given time point The standard deviation of the samples within the corresponding time window; The first in the ETL pipeline operating environment timing data i Each indicator at a given time point Standardized observations.

4. The ETL pipeline dynamic resource optimization method as described in claim 1, characterized in that, Step S3, which involves generating a linear prediction sequence of system load indices with a specified prediction step size using a linear load prediction model, includes: S301. Input the standardized observations of the time-series data of the cluster system load index into the linear load prediction model, and automatically select the optimal parameter combination of the ARIMA algorithm formula in the linear load prediction model using the AIC criterion. The ARIMA algorithm formula is as follows: in, For the first i The load metrics of the computing cluster system at a given time point Standardized observations; For the first i The load metrics of the computing cluster system at a given time point Standardized observations; for Order difference operator; For constant terms; These are the autoregressive coefficients; The moving average coefficient; This is the white noise error term; These are the core parameters, where: The number of autoregressive terms represents the number of lagged observations used in the linear load forecasting model; The difference order represents the number of differences required to make the time series stationary; The number of moving average terms represents the amount of lag error used in the linear load forecasting model; The optimal parameter combination automatically selected using the AIC criterion is expressed as follows: ; S302. Using the optimal parameter combination as the core parameter of the ARIMA algorithm recursive formula, the ARIMA algorithm recursive formula is used to generate linear prediction values ​​of the system load index with a specified continuous prediction step size. The ARIMA algorithm recursive formula is as follows: in, The time point of the last known observation; To predict the step size; For the first i The load metrics of the computing cluster system at a given time point The linear predicted value; The load metrics of the computing cluster system at a given time point Standardized observations, when hour, = .

5. The ETL pipeline dynamic resource optimization method as described in claim 1, characterized in that, The training steps for a multidimensional time-series load forecasting model include: S411. Collect historical data of ETL pipeline operation environment time series data to construct a sample set. 20% of the recent continuous historical data in the sample set is retained as the validation set, and the remaining 80% of the samples are used as the training set. S412. Perform the same processing as steps S2, S401, and S402 on the validation set and training set to obtain the dimensionality-reduced feature vector of the training set, the multidimensional label of the training set, the dimensionality-reduced feature vector of the validation set, and the multidimensional label of the validation set. S413. Input the dimensionality-reduced feature vector of the training set into the multidimensional time-series load prediction model to generate preliminary system load index coupled prediction values ​​with a specified continuous prediction step size. Each preliminary system load index coupled prediction value corresponds to a prediction time point. S414. Use the multidimensional labels of the training set corresponding to each prediction time point as the true labels of that prediction time point, use the mean squared error as the loss function to calculate the loss and backpropagate the error to update the function. Repeat steps S413-S414 to train the model. During training, use the validation set to monitor the multidimensional time-series load prediction model and retain the optimal model as the pre-trained multidimensional time-series load prediction model.

6. The ETL pipeline dynamic resource optimization method as described in claim 1, characterized in that, In step S5, the weighted average method is used to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index. The formula is as follows: In the formula, For the first i The load metrics of the computing cluster system at a given time point The overall forecast value; For the first i The load metrics of the computing cluster system at a given time point The linear predicted value; For the first i The load metrics of the computing cluster system at a given time point The coupled predicted value; For linear load prediction weighting coefficients; Weighting coefficients for multidimensional time-series load prediction.

7. An ETL pipeline dynamic resource optimization system, characterized in that, To implement the ETL pipeline dynamic resource optimization method as described in any one of claims 1-6, the method includes: The time-series data acquisition module is used to collect time-series data of the ETL pipeline operating environment, including time-series data of industrial equipment indicators, ETL task scheduling indicators, and computing cluster system load indicators within the ETL pipeline operating environment. The data processing module is used to divide the ETL pipeline operating environment time series data into time windows, then perform standardization processing to obtain standardized observations of the ETL pipeline operating environment time series data, and organize them into standardized feature vectors for each time window. The linear load forecasting module is used to input the standardized observations of the time series data of the system load index of the computing cluster into the linear load forecasting model, generate linear forecasts of the system load index with a specified continuous forecasting step size, and form a linear forecast sequence of the system load index. The coupled load prediction module is used to input the standardized feature vector of the continuous time window into the pre-trained multi-dimensional time series load prediction model, generate coupled prediction values ​​of system load indicators with a specified continuous prediction step size, and form a coupled prediction sequence of system load indicators. The prediction sequence integration and de-standardization module is used to integrate the linear prediction sequence and the coupled prediction sequence of the system load index into a comprehensive prediction sequence of the system load index, and to de-standardize the comprehensive prediction sequence of the system load index to obtain the original comprehensive prediction sequence of the system load index. The load period identification and scheduling strategy adjustment module is used to identify the peak periods, low periods and abnormally tight resource periods of the system load in the ETL pipeline operating environment based on the comprehensive prediction sequence of the original system load index, and dynamically adjust the scheduling strategy of ETL tasks according to the identification results. The scheduling policy execution module is used to execute the adjusted scheduling policy.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When a processor executes a computer program, it implements the steps of the ETL pipeline dynamic resource optimization method as described in any one of claims 1-6.

9. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the ETL pipeline dynamic resource optimization method as described in any one of claims 1-6.

Images