Big data cluster deployment method and device and storage medium
Through interactive introspective differential learning algorithm and collaborative trust voting algorithm, the problem of low node selection efficiency in large-scale Hadoop cluster deployment is solved, and efficient and stable component deployment and resource utilization are achieved.
Patent Information
- Application Number
- CN202410996822.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-26
AI Technical Summary
In large-scale Hadoop cluster deployment, the number of nodes is huge and the monitoring indicators is large. Administrators need to view resource utilization one by one, which is inefficient. The Ambari system cannot automatically recommend nodes with suitable resource utilization, resulting in poor deployment stability and prediction accuracy.
An interactive introspective differential learning algorithm is used to train timing prediction large models and small models, and the prediction results of multiple models are fused through the collaborative trust voting algorithm to determine the optimal node for component deployment.
It improves the stability and training efficiency of the model, reduces manpower investment, improves deployment efficiency and resource utilization, and recommends the accuracy of deployment nodes.
Smart Images

Figure CN119025263A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to a big data cluster deployment method, device and storage medium. Background Art
[0002] In the related art, before deploying components in a large-scale Hadoop cluster, due to the large number of nodes and the large number of monitoring indicators of various component types, the administrator needs to check the resource utilization of each node and different components one by one, which requires a lot of manpower and is inefficient. When using Ambari to deploy components in a big data cluster, the system cannot automatically recommend nodes with suitable resource utilization. In addition to considering the current resource utilization of each node, the administrator also needs rich experience to predict the resource usage after the component is deployed, which is difficult and the stability is difficult to guarantee. In addition, the use of historical data for simple model training has poor accuracy of the prediction results and poor stability in large-scale deployment. Summary of the invention
[0003] A first aspect embodiment of the present disclosure proposes a big data cluster deployment method, which includes: reading the timing monitoring data of the components on each node of the big data cluster, inputting the timing monitoring data into a timing prediction large model and one or more small models with different architectures, using an interactive introspection difference learning algorithm to perform model training, and obtaining trained large models and small models; inputting the timing monitoring data of the components to be deployed into the trained large model and small model respectively for processing to obtain corresponding prediction results, and fusing the prediction results corresponding to the trained large model and small model through a collaborative trust voting algorithm to obtain a fused prediction result; determining the optimal node from each node of the data cluster based on the fused prediction result, and deploying the components to be deployed on the optimal node.
[0004] In some embodiments of the present disclosure, the timing monitoring data of the components on each node of the big data cluster is read, the timing monitoring data is input into a timing prediction large model and one or more small models of different architectures, and an interactive introspective difference learning algorithm is used to perform model training, and the trained large model and small model are obtained, including: calling the Ambari indicator system interface to read the timing monitoring data of each component, the timing monitoring data including at least one of CPU utilization, memory utilization, disk usage, disk input / output, and network speed bandwidth; preprocessing and feature engineering the timing monitoring data to obtain valid features in the timing monitoring data; inputting the valid features into a timing prediction large model and one or more small models of different architectures, and using an interactive introspective difference learning algorithm to perform model training with the goal of minimizing the empirical accuracy of the timing prediction large model and the small model and minimizing the prediction error of the small model, and obtaining the trained large model and small model.
[0005] In some embodiments of the present disclosure, effective features are input into a large time series prediction model and one or more small models with different architectures, and an interactive self-reflection difference learning algorithm is used to perform model training with the goal of minimizing the empirical accuracy of the large time series prediction model and the small model and minimizing the prediction error of the small model to obtain the trained large model and small model, including: inputting effective features into the iTransformer large model and the LightGBM model, the Prophet model, and the long short-term memory network LSTM model for processing to obtain the outputs of the iTransformer large model and the LightGBM model, the Prophet model, and the LSTM model; using the outputs of the iTransformer large model as soft labels for the LightGBM model, the Prophet model, and the LSTM model respectively; based on the soft labels, the LightGBM model, the Prophet model, and the LSTM model are trained through a self-reflection prompt learning algorithm to obtain the trained iTransformer large model, the LightGBM model, the Prophet model, and the LSTM model.
[0006] In some embodiments of the present disclosure, the time series monitoring data of the components to be deployed are respectively input into the trained large model and the small model for processing to obtain corresponding prediction results, and the prediction results corresponding to the trained large model and the small model are fused through a collaborative trust voting algorithm to obtain the fused prediction results, including: reading the components to be deployed from the list of components to be deployed, and judging whether the components to be deployed are new components; if the components to be deployed are not new components, performing data preprocessing and feature engineering on the time series monitoring data of the components to be deployed to obtain effective features of the components to be deployed; inputting the effective features of the components to be deployed into the trained iTransformer large model, LightGBM model, Prophet model and LSTM model for processing to obtain the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model; adopting a collaborative trust voting algorithm to fuse the prediction results corresponding to the trained iTransformer large model, Light GBM model, Prophet model and LSTM model to obtain the fused prediction results.
[0007] In some embodiments of the present disclosure, a collaborative trust voting algorithm is used to fuse the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, and the fused prediction results include: for each of the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, trust assessment scores are performed on models other than the model based on preset metrics to obtain trust assessment scoring data; the weight of each model is determined according to the trust assessment scoring data, and the prediction results corresponding to all models are fused based on the weights to obtain the fused prediction results.
[0008] In some embodiments of the present disclosure, the method further includes: performing weight fitting on the trust assessment scoring data and the soft label to obtain the weight fitting result of the soft label; and using the weight fitting result to guide the model training of the iTransformer large model.
[0009] The second aspect of the present disclosure proposes a big data cluster deployment device, which includes: a training module, which is used to read the timing monitoring data of the components on each node of the big data cluster, input the timing monitoring data into a timing prediction large model and one or more small models with different architectures, and use an interactive introspection difference learning algorithm to perform model training to obtain the trained large model and small model; a prediction module, which is used to input the timing monitoring data of the components to be deployed into the trained large model and small model respectively for processing to obtain corresponding prediction results, and fuse the prediction results corresponding to the trained large model and small model through a collaborative trust voting algorithm to obtain a fused prediction result; a deployment module, which is used to determine the optimal node from each node of the data cluster based on the fused prediction result, and deploy the components to be deployed on the optimal node.
[0010] The third aspect embodiment of the present disclosure proposes a communication device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.
[0011] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0012] The fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect embodiment of the present disclosure when executed by a processor.
[0013] In summary, the big data cluster deployment method proposed in the present disclosure includes reading the time series monitoring data of the components on each node of the big data cluster, inputting the time series monitoring data into the time series prediction large model and one or more small models with different architectures, using the interactive introspection difference learning algorithm to perform model training, and obtaining the trained large model and small model; inputting the time series monitoring data of the components to be deployed into the trained large model and small model respectively to obtain the corresponding prediction results, and fusing the prediction results corresponding to the trained large model and small model through the collaborative trust voting algorithm to obtain the fused prediction results; determining the optimal node from each node of the data cluster based on the fused prediction results, and deploying the components to be deployed on the optimal node. This solution generates an efficient and stable training model by collecting the monitored time series data and adopting the advanced training method of the interactive introspection difference learning algorithm, which saves the training cost and improves the stability and training efficiency of the model; when deploying components, after fusing the prediction results of multiple training models, the optimal deployment node is recommended for deployment, without the need to manually read the monitoring data for judgment, thereby improving the deployment efficiency and resource utilization.
[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0016] Figure 1 An example diagram of an Ambari system framework in a related technology provided in an embodiment of the present disclosure;
[0017] Figure 2 A flowchart of a big data cluster deployment method provided in an embodiment of the present disclosure;
[0018] Figure 3 A flowchart of a big data cluster deployment method provided in an embodiment of the present disclosure;
[0019] Figure 4 An example diagram of an Ambari system framework provided in an embodiment of the present disclosure;
[0020] Figure 5 An example diagram of a process for model training on monitoring data of each node component provided in an embodiment of the present disclosure;
[0021] Figure 6 An example diagram of a use scenario of a big data cluster deployment method provided in an embodiment of the present disclosure;
[0022] Figure 7 An example diagram of a process for recommending optimal node deployment for a component to be deployed provided by an embodiment of the present disclosure;
[0023] Figure 8 An example diagram of a use scenario of a big data cluster deployment method provided in an embodiment of the present disclosure;
[0024] Fig. 9 A schematic diagram of the structure of a big data cluster deployment device provided in an embodiment of the present disclosure;
[0025] Fig.10 A schematic diagram of the hardware structure of a communication device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0026] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0027] First, the technical problems to be solved by the technical solution of the present disclosure and related technologies are described in detail.
[0028] Ambari is an open source, web-based management platform for managing and monitoring big data clusters, especially Apache Hadoop clusters. It provides cluster-level management and monitoring functions, including deployment, configuration, start, stop, monitoring, alarm, upgrade, and expansion. The core of Ambari is Ambari Server, which is a web application that provides an easy-to-use user interface for accessing and managing clusters through a browser. Ambari deploys Hadoop software and can automatically handle dependencies between services and components; manages the start, stop, and restart of Hadoop service components, and multi-version management of configuration files; Ambari Metrics performance indicators monitor the current status of Hadoop services, such as CPU utilization, memory utilization, disk usage, disk IO, network speed and bandwidth, etc., and can customize alarm events.
[0029] The overall architecture of Ambari in related technologies is as follows Figure 1As shown in the figure, Ambari Server is responsible for the management and coordination of the entire cluster, and supports administrators to manage the cluster through UI or API; Coordinator can also be called API handler, which is used to receive operation requests from the web side; Ambari Metrics System (AMS) is used to collect, store and query cluster performance indicator data; Ambari Agent is installed on each cluster node, responsible for collecting node status information and executing commands issued by Ambari Server; Ambari Web is a web-based user interface provided by Ambari.
[0030] At present, the process of deploying components in Ambari generally includes the following steps: select the service to be deployed in the user interface of Ambari Server, such as HDFS, Hbase, Kafka, Hive, etc.; the administrator manually selects to deploy each component to a specific node, and Ambari will deploy and configure the component on the specified node in the cluster; after the components are deployed and configured, the service can be started. Ambari will automatically start all necessary components and provide a visual progress bar to show the deployment progress.
[0031] The relevant technology has the following disadvantages: before deploying components in a large-scale Hadoop cluster, due to the large number of nodes and the large number of monitoring indicators of various component types, the administrator needs to check the resource utilization of each node and different components one by one, which requires a lot of manpower and is inefficient; when using Ambari to deploy components in a big data cluster, the system cannot automatically recommend nodes with suitable resource utilization. In addition to considering the current resource utilization of each node, the administrator also needs rich experience to predict the resource usage after the component is deployed, which is difficult and the stability is difficult to guarantee.
[0032] In order to solve the above problems, the present invention proposes a new big data cluster deployment solution, which generates efficient and stable training models by collecting monitored time series data and adopting advanced interactive introspection difference learning algorithm training, thus saving training costs and improving model stability and training efficiency. For components to be deployed, the prediction results of multiple models are integrated through a collaborative trust voting algorithm, which improves the prediction accuracy and saves manpower. It can be applied to the Ambari system to realize efficient deployment of large-scale Hadoop clusters.
[0033] The present disclosure is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] Figure 2 A flowchart of a big data cluster deployment method provided in an embodiment of the present disclosure.
[0035] like Figure 2 As shown, the big data cluster deployment method provided by the embodiment of the present disclosure includes the following steps 101-103:
[0036] Step 101, read the time series monitoring data of the components on each node of the big data cluster, input the time series monitoring data into the time series prediction large model and one or more small models with different architectures, use the interactive introspection difference learning algorithm to perform model training, and obtain the trained large model and small model.
[0037] In some embodiments of the present disclosure, the method is applied to the Ambari system to support the provisioning, management, and monitoring of an Apache Hadoop cluster.
[0038] In some embodiments of the present disclosure, the components include HDFS, MapReduce, Hive, Pig, Hbase, Zookeeper, Sqoop, Hcatalog, etc.
[0039] In some embodiments of the present disclosure, time series data refers to a string of data indexed by a time dimension, and the time series monitoring data of a component describes the monitoring data of the component at each time point within a period of time.
[0040] In some embodiments of the present disclosure, the monitoring data of the component includes CPU utilization, memory utilization, disk usage, disk IO, network speed and bandwidth, etc.
[0041] In some embodiments of the present disclosure, the time series monitoring data of each component is used as a data set, and the data set is divided into a training set and a validation set. After preprocessing and feature engineering, the data is sent to a large time series prediction model and one or more small models with different architectures for model training.
[0042] In some embodiments of the present disclosure, time series forecasting is an important forecasting method, which is particularly suitable for data that needs to predict future trends. Large models usually refer to those models with a large number of parameters, which can handle complex data patterns and provide accurate forecasts. In time series forecasting, some large models can provide better forecasting performance than traditional methods, especially when dealing with nonlinear, high-dimensional or long-term dependent data.
[0043] In some embodiments of the present disclosure, the main purpose of using an interactive introspective difference learning algorithm is to enable the large model to further enhance its capabilities by learning the differences in data understanding between small models of different architectures.
[0044] In some embodiments of the present disclosure, a pre-trained large time series prediction model with rich knowledge and experience is used as a teacher model, and small models of different architectures are used as student models. The predicted output of the teacher model is used as the soft label of the student model for training, and the trust assessment scoring data obtained by the collaborative trust voting algorithm in step 102 is used to fit the soft label weights, thereby improving the accuracy of the trained large model.
[0045] Step 102, the time series monitoring data of the components to be deployed are respectively input into the trained large model and the small model to obtain the corresponding prediction results, and the prediction results corresponding to the trained large model and the small model are fused through the collaborative trust voting algorithm to obtain the fused prediction results.
[0046] In some embodiments of the present disclosure, the collaborative trust voting algorithm is a model fusion method that forms a more reliable comprehensive voting decision by fusing the results of multiple prediction models and considering the trust relationship between voters. Its principle involves key steps such as individual voting, trust evaluation, trust weighting, dynamic adjustment and anomaly detection.
[0047] In some embodiments of the present disclosure, individual voting refers to each model acting as a voter to make independent predictions or classifications for given input data. Each model provides a predicted confidence or probability for each category; trust assessment scoring refers to each model performing trust assessment scoring on other models; trust weighting includes the weight of each model being related to its trust level, and the prediction results of highly trusted models have a larger scoring proportion in the fusion; anomaly detection is used to detect situations where the model performance is unstable or the trust level fluctuates greatly.
[0048] In some embodiments of the present disclosure, the prediction results corresponding to the trained large model and the small model are fused through a collaborative trust voting algorithm to obtain the fused prediction results, including: for each model in the trained large model and the small model, trust assessment scores are performed on models other than the model based on preset metrics to obtain trust assessment scoring data; the weight of each model is determined according to the trust assessment scoring data, and the prediction results corresponding to all models are fused based on the weights to obtain the fused prediction results.
[0049] Step 103 , based on the fused prediction result, determine the optimal node from among the nodes in the data cluster, and deploy the component to be deployed on the optimal node.
[0050] In some embodiments of the present disclosure, the prediction result is the confidence of each node or the probability distribution of the optimal node, and the node with the highest confidence or the highest probability is determined as the optimal node based on the fused prediction results.
[0051] In summary, the embodiments provided by the present disclosure include reading the time series monitoring data of the components on each node of the big data cluster, inputting the time series monitoring data into the time series prediction large model and one or more small models with different architectures, using the interactive introspection difference learning algorithm to perform model training, and obtaining the trained large model and small model; inputting the time series monitoring data of the components to be deployed into the trained large model and small model respectively to obtain the corresponding prediction results, and fusing the prediction results corresponding to the trained large model and small model through the collaborative trust voting algorithm to obtain the fused prediction results; determining the optimal node from each node of the data cluster based on the fused prediction results, and deploying the components to be deployed on the optimal node. This solution generates an efficient and stable training model by collecting the monitored time series data and adopting the advanced training method of the interactive introspection difference learning algorithm, which saves the training cost and improves the stability and training efficiency of the model; when deploying components, after fusing the prediction results of multiple training models, the optimal deployment node is recommended for deployment, without the need to manually read the monitoring data for judgment, thereby improving the deployment efficiency and resource utilization.
[0052] based on Figure 2 The embodiment shown, as Figure 3 As shown, Figure 3 A flowchart of another big data cluster deployment method provided in an embodiment of the present disclosure.
[0053] The big data cluster deployment method provided in the embodiment of the present disclosure includes the following steps:
[0054] Step 201, call the Ambari indicator system interface to read the timing monitoring data of each component, the timing monitoring data includes at least one of CPU utilization, memory utilization, disk utilization, disk input / output, and network speed bandwidth.
[0055] In some embodiments of the present disclosure, the method can be applied to Figure 4 The Ambari system shown in the figure has a new training module, namely "Training Module" and a prediction module, namely "Prediction Module", compared with the Ambari system in the related art. The training module is mainly responsible for training and modeling the monitoring data of each node component at regular intervals, which is specifically used to execute the method in steps 201-203; the prediction module is mainly responsible for recommending the optimal node deployment for the components to be deployed, which is specifically used to execute the method in steps 204-205.
[0056] In some embodiments of the present disclosure, Figure 4The Ambari system shown in the figure reads the historical monitoring data of components on each node at regular intervals, and enters the model training after preprocessing and feature engineering. When deploying components, the system will enter the prediction module according to the component type, integrate the prediction results of multiple training models, and recommend the optimal deployment node for deployment. There is no need to manually read the monitoring data for judgment, which improves deployment efficiency and resource utilization.
[0057] In some embodiments of the present disclosure, the training module calls the AMS interface to obtain historical monitoring data of each component. The historical monitoring data of each component includes CPU utilization, memory utilization, disk usage, disk IO, network speed bandwidth and other timing monitoring data.
[0058] Step 202: preprocess and perform feature engineering on the time series monitoring data to obtain effective features in the time series monitoring data.
[0059] In some embodiments of the present disclosure, preprocessing includes invalid value processing, missing value and duplicate value processing, consistency processing, data subset screening and data sorting, etc., wherein invalid value processing is to use statistical analysis methods to identify extreme values and delete or replace them, missing value and duplicate value processing is to replace missing values with average values, maximum values, minimum values or probability estimates and delete duplicate data, and consistency processing is to delete data that exceeds the normal range according to variable legal rules and logic.
[0060] In some embodiments of the present disclosure, feature engineering processing includes time series feature extraction, historical feature extraction, translation feature extraction and sliding window feature extraction, etc., wherein the time series historical features include relevant hours, weeks and months; hour feature values include {0, 1, 2...23}, which correspond to 24 hours in a day; week feature values include {0, 1, 2, 3, 4, 5, 6}, which correspond to dates from Monday to Sunday; month feature values include {1, 2, 3, ..., 12}, which correspond to months from January to December; translation feature refers to the correlation feature between the corresponding resource usage and the original data after the time series is translated forward; sliding window feature is the embodiment of the same feature in different time dimensions, which is obtained by statistical analysis of the corresponding resource usage in fixed interval time periods.
[0061] Step 203, input the effective features into the time series prediction large model and one or more small models with different architectures, use an interactive introspection difference learning algorithm, perform model training with the goal of minimizing the empirical accuracy of the time series prediction large model and the small model and minimizing the prediction error of the small model, and obtain the trained large model and small model.
[0062] In some embodiments of the present disclosure, the training module mainly adopts an interactive introspective difference learning algorithm, the main purpose of which is to enable the large model to further enhance the capabilities of the large model by learning the differences in data understanding of small models with different architectures.
[0063] In some embodiments of the present disclosure, a large time series prediction model is used as a teacher model, and one or more small models with different architectures are used as student models. During model training, the output probability distribution of the teacher model is used as the target label of the student model. By using an approximate target for training, that is, the output probability distribution of the teacher model, the student model can learn more details and knowledge of the teacher model, including the relative weights of the probability distribution and the relationship between categories. This knowledge transfer helps to improve the generalization ability of the student model and achieve performance similar to or even better than the teacher model while keeping the model size small. In addition, the trust assessment scoring data generated in the prediction module is used, weighted fitting is performed with the soft label and guiding the training of the large model, which can further improve the accuracy of the training module in real prediction scenarios.
[0064] In some embodiments of the present disclosure, the goal optimization during training includes two aspects: on the one hand, minimizing the empirical accuracy difference between the student model and the teacher model; on the other hand, minimizing the prediction error of the student model on the task. By optimizing these two goals at the same time, it is possible to gradually improve its own accuracy and performance while maintaining similar performance to the teacher model.
[0065] In some embodiments of the present disclosure, the timing scheduler will trigger the training model at regular intervals, first obtaining the time series monitoring data of each component in the AMS at each node through the interface, and then performing data preprocessing, and converting it into a format that can be trained by the model through feature engineering. Each training model reads the same data after feature engineering for training, and after the training is completed, the large time series prediction model is saved for subsequent predictions.
[0066] In some embodiments of the present disclosure, the iTransformer large model is used as the teacher model, the three small models with different architectures of LightGBM, Prophet and LSTM are used as student models, and the prediction output of the iTransformer teacher model is used as the soft label of the three student models of LightGBM, Prophet and LSTM. The training module extracts the trust evaluation scoring data generated in the prediction phase to fit the soft label weights, thereby improving the accuracy of the iTransformer training large model.
[0067] In the embodiments disclosed herein, iTransformer is one of the most advanced large models for time series prediction based on the Transformer structure. The model embeds each time series as a variable tag, uses attention for multivariate correlation modeling, uses layer normalization (LayerNorm) to stabilize data drift, and uses a feedforward network for sequence encoding. The model uses reprogramming to train some parameters, and efficiently fine-tunes the model to achieve top time series prediction results without changing the original Transformer architecture and retaining its original text understanding capabilities.
[0068] In the embodiments of the present disclosure, the working principle of LightGBM mainly involves the gradient boosting algorithm and histogram learning. LightGBM aims to efficiently process large-scale data and high-dimensional features, and has achieved advantages in performance and efficiency through techniques such as gradient boosting and histogram learning; Prophet is a time series prediction tool, mainly used to process time series data with seasonal and holiday effects. It automatically detects and processes change points in the time series, which may be trend changes caused by external factors. This makes the model more flexible and able to adapt to structural changes in time series; LSTM (Long Short-Term Memory) is a variant of RNN (Re current Neural Network) designed to solve the long-term dependency problem in RNN. LS TM introduces a special memory unit and gating mechanism to more effectively capture and manage long-term dependencies in time series data.
[0069] From the above, it can be seen that the iTransformer large model and the three small models with different architectures of LightGBM, Prophet and LSTM have greater advantages in large-scale data processing, time series data prediction, and model training. Other large models and small models for time series prediction can also be used, which is not limited by the present disclosure.
[0070] In some embodiments of the present disclosure, during the large model and small model initialization stage, a pre-trained iTransformer large model with rich knowledge and experience is used as the teacher model for initialization, and three smaller and simpler student models, LightGBM, Prophet, and LSTM, are randomly initialized.
[0071] In some embodiments of the present disclosure, in the data set preparation stage, historical monitoring data of each component, such as CPU utilization, memory utilization, disk usage, disk IO, network speed bandwidth and other time-series monitoring data, are used as data sets, and the data sets are divided into training sets and validation sets. After data preprocessing and feature engineering, they are used as input for the training model.
[0072] In some embodiments of the present disclosure, Figure 5 As shown in the figure, the effective features are input into the iTransformer large model and the Light GBM model, the Prophet model, and the long short-term memory network LSTM model for processing, and the outputs of the iTransformer large model, the LightGBM model, the Prophet model, and the LSTM model are obtained; the outputs of the iTransformer large model are used as soft labels for the LightGBM model, the Prophet model, and the LSTM model respectively; based on the soft labels, the LightGBM model, the Prophet model, and the LSTM model are trained through the self-reflection prompt learning algorithm to obtain the trained iTransformer large model, the LightGBM model, the Prophet model, and the LSTM model.
[0073] In some embodiments of the present disclosure, Figure 6 As shown, the user "Client" selects the components to be deployed through the user interface of Ambari Server, and the timer scheduler "Timer" schedules the training module "Training Module" to initiate training at the scheduled time, read monitoring data, and update the saved model.
[0074] In step 204, the time series monitoring data of the components to be deployed are respectively input into the trained large model and the small model to obtain corresponding prediction results, and the prediction results corresponding to the trained large model and the small model are fused through the collaborative trust voting algorithm to obtain the fused prediction results.
[0075] In some embodiments of the present disclosure, the list of components that the user needs to deploy is read to determine whether it is a new component. If it is, after data preprocessing and feature engineering according to the current resource situation of the cluster, the model is adapted during testing, and then recommended nodes are deployed; if not, after data preprocessing and feature engineering, the component list is input into the three small models and the iTransformer large model for prediction.
[0076] Specifically, Figure 7As shown, the component to be deployed is read from the list of components to be deployed to determine whether it is a new component; if the component to be deployed is not a new component, data preprocessing and feature engineering are performed on the time series monitoring data of the component to be deployed to obtain the effective features of the component to be deployed; the effective features of the component to be deployed are respectively input into the trained iTransformer large model, LightGBM model, Prophet model and LSTM model for processing to obtain the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model; the collaborative trust voting algorithm is used to fuse the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model to obtain the fused prediction results.
[0077] In some embodiments of the present disclosure, a collaborative trust voting algorithm is used to fuse the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, and the fused prediction results include: for each of the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, trust assessment scores are performed on models other than the model based on preset metrics to obtain trust assessment scoring data; the weight of each model is determined according to the trust assessment scoring data, and the prediction results corresponding to all models are fused based on the weights to obtain the fused prediction results.
[0078] In some embodiments of the present disclosure, in the collaborative trust voting algorithm, the output of each model is the confidence or probability that each node is the optimal node, and each model performs a trust assessment on other models. This can reflect the degree of trust of a model in other models through some metrics, such as historical performance, stability, accuracy, etc. These metrics can be set a priori or learned dynamically. When fusing the prediction results, the degree of trust of each model in other models is taken into account. A scoring method can be adopted in which the weight of each model is related to its degree of trust. The prediction results of highly trusted models have a larger scoring proportion in the fusion.
[0079] In some embodiments of the present disclosure, an anomaly detection mechanism is introduced to detect situations where the model performance is unstable or the trust level fluctuates greatly. Possible actions include reducing its proportion, alerting the system administrator, or temporarily excluding the model from voting.
[0080] In some embodiments of the present disclosure, the method further includes: reading the component to be deployed from the list of components to be deployed, and determining whether the component to be deployed is a new component; if the component to be deployed is a new component, adopting a test time adaptation strategy for the new component to be deployed, obtaining a prediction result, and ending the prediction process.
[0081] Step 205 , based on the fused prediction result, determine the optimal node from among the nodes in the data cluster, and deploy the component to be deployed on the optimal node.
[0082] In some embodiments of the present disclosure, the prediction result is the confidence of each node or the probability distribution of the optimal node, and the node with the highest confidence or the highest probability is determined as the optimal node based on the fused prediction results.
[0083] In some embodiments of the present disclosure, Figure 8 As shown in the figure, the deployment component request submitted by the user is sent to the prediction module "Prediction Module" through the API Server, the prediction module obtains the recommended optimal node, and distributes the deployment tasks of the components to be deployed to each node "Agent" for deployment and installation.
[0084] In summary, the solution provided by the present disclosure has the following improvements and beneficial effects:
[0085] 1. By collecting monitored time series data and adopting the advanced training method of interactive introspection difference learning algorithm, an efficient and stable training model is generated, which saves training costs and improves the stability and training efficiency of the model.
[0086] 2. When predicting new components that lack historical experience, a test time adaptation strategy is used to reduce the accuracy fluctuations caused by cold data. For other components that have been trained historically, the prediction results of multiple models are integrated through a collaborative trust voting algorithm to improve the accuracy of the prediction.
[0087] 3. Management and operation personnel only need to enable relevant configurations. When selecting relevant components for deployment during the deployment process, the algorithm will run to automatically select nodes for deployment. No other additional operations are required, saving manpower.
[0088] 4. The user does not need to make any business modifications. The system can automatically train and predict based on historical data, and the entire process is imperceptible to the user.
[0089] The present disclosure also provides a big data cluster deployment device. Fig. 9 shown. Fig. 9 This is a schematic diagram of the structure of a big data cluster deployment device provided in an embodiment of the present disclosure. The big data cluster deployment device 300 includes:
[0090] The training module 310 is used to read the timing monitoring data of the components on each node of the big data cluster, input the timing monitoring data into the timing prediction large model and one or more small models with different architectures, use the interactive introspection difference learning algorithm to perform model training, and obtain the trained large model and small model; the prediction module 320 is used to input the timing monitoring data of the components to be deployed into the trained large model and small model respectively for processing to obtain the corresponding prediction results, and fuse the corresponding prediction results of the trained large model and small model through the collaborative trust voting algorithm to obtain the fused prediction results; the deployment module 330 is used to determine the optimal node from each node of the data cluster based on the fused prediction results, and deploy the components to be deployed on the optimal node.
[0091] In some embodiments of the present disclosure, the training module 310 is specifically used to: call the Ambari indicator system interface to read the timing monitoring data of each component, the timing monitoring data including at least one of CPU utilization, memory utilization, disk usage, disk input / output, and network speed bandwidth; perform preprocessing and feature engineering on the timing monitoring data to obtain effective features in the timing monitoring data; input the effective features into a large timing prediction model and one or more small models with different architectures, and use an interactive introspection difference learning algorithm to perform model training with the goal of minimizing the empirical accuracy of the large timing prediction model and the small model and minimizing the prediction error of the small model to obtain the trained large model and small model.
[0092] In some embodiments of the present disclosure, the training module 310 is also used to: input effective features into the iTransformer large model and the LightGBM model, the Prophet model, and the long short-term memory network LSTM model for processing, and obtain the outputs of the iTransformer large model and the LightGBM model, the Prophet model, and the LSTM model; use the outputs of the iTransformer large model as soft labels for the LightGBM model, the Prophet model, and the LSTM model respectively; based on the soft labels, the LightGBM model, the Prophet model, and the LSTM model perform model training through a self-reflection prompt learning algorithm to obtain the trained iTransformer large model, the LightGBM model, the Prophet model, and the LSTM model.
[0093] In some embodiments of the present disclosure, the prediction module 320 is specifically used to: read the component to be deployed from the list of components to be deployed, and determine whether the component to be deployed is a new component; if the component to be deployed is not a new component, perform data preprocessing and feature engineering on the time series monitoring data of the component to be deployed, and obtain the effective features of the component to be deployed; input the effective features of the component to be deployed into the trained iTransformer large model, LightGBM model, Prophet model and LSTM model for processing, and obtain the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model; use the collaborative trust voting algorithm to fuse the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model to obtain the fused prediction results.
[0094] In some embodiments of the present disclosure, the prediction module 320 is also used to: for each of the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, perform trust assessment scoring on models other than the model based on preset metrics to obtain trust assessment scoring data; determine the weight of each model based on the trust assessment scoring data, and fuse the prediction results corresponding to all models based on the weights to obtain the fused prediction results.
[0095] In some embodiments of the present disclosure, the training module 310 is also used to: for each of the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, perform trust assessment scoring on models other than the model based on preset metrics to obtain trust assessment scoring data; determine the weight of each model based on the trust assessment scoring data, and fuse the prediction results corresponding to all models based on the weights to obtain the fused prediction results.
[0096] In summary, the big data cluster deployment device provided by the present invention generates an efficient and stable training model by collecting monitored time series data and adopts an advanced training method of interactive introspection difference learning algorithm, which saves training costs and improves the stability and training efficiency of the model. When deploying components, the prediction results of multiple training models are integrated and the optimal deployment node is recommended for deployment. There is no need to manually read the monitoring data for judgment, which improves deployment efficiency and resource utilization.
[0097] The big data cluster deployment device provided in the above embodiment and a big data cluster deployment method embodiment provided in the embodiment of the present disclosure belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0098] Fig.10 A schematic diagram of the hardware structure of the communication device provided in the embodiment of the present disclosure is shown in FIG. Fig.10 As shown, the communication device 400 includes at least one processor 402; and a memory 401 that is communicatively connected to the at least one processor 402; wherein the memory 401 stores instructions that can be executed by the at least one processor 402, and the instructions are executed by the at least one processor 402 to implement the steps of the big data cluster deployment method described in the embodiment of the present disclosure; or, the instructions are executed by the at least one processor 402 to implement the steps of the big data cluster deployment method described in the embodiment of the present disclosure.
[0099] Optionally, the communication device may specifically be a control device of an embodiment of the present application, and the communication device may implement corresponding processes implemented by the control device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.
[0100] It is understood that the communication device also includes a communication interface 403. The various components in the communication device are coupled together through a bus system 404. It is understood that the bus system 404 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 404 .
[0101] It can be understood that the memory 401 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM).The memory 401 described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memories.
[0102] The method disclosed in the above embodiment of the present disclosure can be applied to the processor 402, or implemented by the processor 402. The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 402 or the instruction in the form of software. The above processor 402 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 402 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 401, and the processor 402 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.
[0103] In an exemplary embodiment, the communication device can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), FPGA, general processor, controller, MCU, microprocessor, or other electronic components to execute the aforementioned method.
[0104] The present public security embodiment also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the steps of the big data cluster deployment method described in the embodiment of the present invention when executed; or, the computer instructions are used to enable the computer to implement the steps of the big data cluster deployment method described in the embodiment of the present invention when executed.
[0105] Optionally, the computer-readable storage medium can be applied to the control device in the embodiments of the present application, and the computer instructions enable the computer to execute the corresponding processes implemented by the control device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.
[0106] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0107] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0108] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0109] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.
[0110] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0111] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A big data cluster deployment method, characterized in that: The method comprises: Read the time series monitoring data of the components on each node of the big data cluster, input the time series monitoring data into the time series prediction large model and one or more small models with different architectures, use the interactive introspection difference learning algorithm to perform model training, and obtain the trained large model and small model; Input the time series monitoring data of the components to be deployed into the trained large model and the small model respectively for processing to obtain corresponding prediction results, and fuse the prediction results corresponding to the trained large model and the small model through a collaborative trust voting algorithm to obtain a fused prediction result; An optimal node is determined from among the nodes in the data cluster based on the fused prediction result, and the component to be deployed is deployed on the optimal node.
2. The method according to claim 1, characterized in that: The method of reading the time series monitoring data of the components on each node of the big data cluster, inputting the time series monitoring data into a time series prediction large model and one or more small models with different architectures, and using an interactive introspection difference learning algorithm to perform model training, and obtaining the trained large model and small model includes: Calling the Ambari indicator system interface to read the timing monitoring data of each of the components, the timing monitoring data including at least one of CPU utilization, memory utilization, disk utilization, disk input / output, and network speed and bandwidth; Preprocessing and feature engineering the time series monitoring data to obtain effective features in the time series monitoring data; The effective features are input into a large time series prediction model and one or more small models with different architectures, and an interactive introspective difference learning algorithm is used to perform model training with the goal of minimizing the empirical accuracy of the large time series prediction model and the small model and minimizing the prediction error of the small model to obtain the trained large model and small model.
3. The method according to claim 2, characterized in that The effective features are input into the time series prediction large model and one or more small models with different architectures, and an interactive introspection difference learning algorithm is used to perform model training with the goal of minimizing the empirical accuracy of the time series prediction large model and the small model and minimizing the prediction error of the small model, to obtain the trained large model and small model, including: Input the effective features into the iTransformer large model, the LightGBM model, the Prophet model, and the long short-term memory network LSTM model for processing, and obtain the outputs of the iTransformer large model, the LightGBM model, the Prophet model, and the LSTM model; The output of the iTransformer large model is used as the soft label of the LightGBM model, the Prophet model, and the LSTM model respectively; Based on the soft labels, the LightGBM model, the Prophet model and the LSTM model are trained through a self-reflection prompt learning algorithm to obtain the trained iTransformer large model, LightGBM model, Prophet model and LSTM model.
4. The method according to claim 3, characterized in that The time series monitoring data of the components to be deployed are respectively input into the trained large model and the small model to obtain corresponding prediction results, and the prediction results corresponding to the trained large model and the small model are fused through the collaborative trust voting algorithm to obtain the fused prediction results including: Read the component to be deployed from the list of components to be deployed, and determine whether the component to be deployed is a new component; If the component to be deployed is not a new component, perform data preprocessing and feature engineering on the time series monitoring data of the component to be deployed to obtain effective features of the component to be deployed; Input the effective features of the components to be deployed into the trained iTransformer large model, LightGBM model, Prophet model and LSTM model for processing, and obtain the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model; A collaborative trust voting algorithm is used to fuse the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model to obtain a fused prediction result.
5. The method according to claim 4, characterized in that The collaborative trust voting algorithm is used to fuse the prediction results corresponding to the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, and the fused prediction results include: For each of the trained iTransformer large model, LightGBM model, Prophet model and LSTM model, trust assessment scores are performed on other models except the trained model based on preset metrics to obtain trust assessment score data; The weight of each of the models is determined according to the trust assessment scoring data, and the prediction results corresponding to all the models are fused based on the weight to obtain a fused prediction result.
6. The method according to claim 5, characterized in that The method further comprises: Performing weight fitting on the trust assessment scoring data and the soft label to obtain a weight fitting result of the soft label; The weight fitting result is used to guide the model training of the iTransformer large model.
7. A big data cluster deployment device, characterized in that: The device comprises: A training module is used to read the time series monitoring data of the components on each node of the big data cluster, input the time series monitoring data into the time series prediction large model and one or more small models with different architectures, use an interactive introspection difference learning algorithm to perform model training, and obtain the trained large model and small model; A prediction module is used to input the time series monitoring data of the components to be deployed into the trained large model and the small model respectively to obtain corresponding prediction results, and to fuse the prediction results corresponding to the trained large model and the small model through a collaborative trust voting algorithm to obtain a fused prediction result; A deployment module is used to determine the optimal node from among the nodes of the data cluster based on the fused prediction result, and deploy the to-be-deployed component on the optimal node.
8. A communication device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.
10. A computer program product, characterized in that A computer program is included which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Cited By
Differential privacy federal learning method and system based on double knowledge transfer
CN119622824A
A differential privacy federated learning method and system based on dual knowledge transfer
CN119622824B
Wave height prediction method based on memory mechanism
CN120123846A
