Cloud platform capacity early warning method and system based on large model

Abstract

The invention provides a cloud platform capacity early warning method and system based on a large model, and is applied to the technical field of cloud platforms, and the method comprises the steps: obtaining the resource use data of a to-be-predicted cloud platform in real time; extracting feature data from the resource use data; processing the feature data by applying a prediction large model, and outputting a resource capacity prediction value; acquiring historical resource use data of the cloud platform, and determining a current service demand of the cloud platform; setting an initial capacity threshold value based on the historical resource use data, and adjusting the initial capacity threshold value based on a current service demand to obtain a capacity early warning threshold value; and when the resource capacity prediction value is greater than the capacity early warning threshold value, triggering early warning. According to the method, after the resource capacity prediction value is predicted by using the prediction large model, the dynamically changing capacity early warning threshold value is determined based on the current service demand and the historical resource use data, so that the method adapts to the dynamic change of the cloud computing environment, and a more accurate early warning mode is provided.

Description

Technical Field

[0001] The present invention relates to the field of cloud platform technology, and in particular to a cloud platform capacity early warning method and system based on a large model. Background Art

[0002] With the rapid development of cloud computing technology, cloud platforms have become the key infrastructure supporting various enterprises and services. With the surge in business volume, the scale of business carried by cloud platforms has shown a trend of continuous expansion, which makes resource capacity management a core issue in cloud platform operations.

[0003] In traditional cloud platform management, resource capacity early warning methods often rely on fixed empirical thresholds. Traditional capacity early warning methods rely on manually set empirical thresholds, which are often based on statistical laws within a certain range of historical data. However, due to the dynamics and complexity of the cloud computing environment, these empirical thresholds are difficult to accurately reflect the actual situation of resource capacity changes. Therefore, this method is not very accurate in judging whether resource capacity is exceeded, and is prone to false alarms or missed alarms. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a cloud platform capacity warning method and system based on a large model. The application can dynamically set the warning threshold, which is suitable for the complex business needs of the cloud platform and improves the accuracy of the warning.

[0005] To achieve the above objectives, the present application provides the following technical solutions:

[0006] A cloud platform capacity early warning method based on a large model, comprising:

[0007] Obtain resource usage data of the cloud platform to be predicted in real time;

[0008] extracting feature data from the resource usage data;

[0009] Applying a pre-trained prediction model to process the characteristic data and outputting a resource capacity prediction value;

[0010] Obtaining historical resource usage data of the cloud platform and determining current business needs of the cloud platform;

[0011] Based on the historical resource usage data, an initial capacity threshold is set, and based on the current business demand, the initial capacity threshold is adjusted to obtain a capacity warning threshold;

[0012] When the resource capacity prediction value is greater than the capacity warning threshold, a warning is triggered.

[0013] The above method may optionally further include:

[0014] After the warning is triggered, generating warning information of the cloud platform;

[0015] For each preset notification channel, the warning information is converted into information in a data format that satisfies the notification channel, and the converted warning information is notified to the administrator through the notification channel.

[0016] The above method optionally includes the process of training a large prediction model, including:

[0017] Get sample data;

[0018] Preprocessing the sample data to obtain training data;

[0019] Extracting data characteristics of the training data, and selecting a model architecture corresponding to the data characteristics;

[0020] Based on the model architecture, construct a large prediction model;

[0021] The prediction large model is trained using the training data, and the model parameters of the prediction large model are optimized during the training process until the prediction large model meets the preset training end conditions, thereby terminating the training of the prediction large model and obtaining a trained prediction large model.

[0022] In the above method, optionally, the obtaining of sample data includes:

[0023] A preset data collection module is used to periodically execute data collection tasks, and the data of each key performance indicator of the cloud platform collected when the data collection module executes the data collection task is determined as sample data.

[0024] In the above method, optionally, the step of extracting feature data from the resource usage data includes:

[0025] Using a preset data processing tool to clean invalid data, erroneous data and duplicate data in the resource usage data to obtain request data;

[0026] Normalizing the cleaned data to obtain normalized data;

[0027] Feature data is extracted from the normalized data.

[0028] A cloud platform capacity early warning system based on a large model, comprising:

[0029] An acquisition module is used to acquire resource usage data of the cloud platform to be predicted in real time;

[0030] A data preprocessing module, used to extract feature data from the resource usage data;

[0031] A prediction module, used to process the characteristic data using a pre-trained prediction model and output a resource capacity prediction value;

[0032] The early warning judgment module is used to obtain the historical resource usage data of the cloud platform and determine the current business needs of the cloud platform; set an initial capacity threshold based on the historical resource usage data, and adjust the initial capacity threshold based on the current business needs to obtain a capacity early warning threshold; when the resource capacity prediction value is greater than the capacity early warning threshold, trigger an early warning.

[0033] The above system may optionally further include:

[0034] Model training module, used to train large prediction models;

[0035] The process of training the large prediction model by the model training module includes:

[0036] Get sample data;

[0037] Preprocessing the sample data to obtain training data;

[0038] Extracting data characteristics of the training data, and selecting a model architecture corresponding to the data characteristics;

[0039] Based on the model architecture, construct a large prediction model;

[0040] The prediction large model is trained using the training data, and the model parameters of the prediction large model are optimized during the training process until the prediction large model meets the preset training end conditions, thereby terminating the training of the prediction large model and obtaining a trained prediction large model.

[0041] The above system may optionally further include:

[0042] Early warning notification module;

[0043] The warning notification module is used to generate warning information of the cloud platform after the warning is triggered; for each preset notification channel, the warning information is converted into information in a data format that meets the data format of the notification channel, and the converted warning information is notified to the administrator through the notification channel.

[0044] In the above system, optionally, the model training module executes a process of acquiring sample data, including:

[0045] A preset data collection module is used to periodically execute data collection tasks, and the data of each key performance indicator of the cloud platform collected when the data collection module executes the data collection task is determined as sample data.

[0046] In the above system, optionally, the data preprocessing module performs a process of extracting feature data from the resource usage data, including:

[0047] Using a preset data processing tool to clean invalid data, erroneous data and duplicate data in the resource usage data to obtain request data;

[0048] Normalizing the cleaned data to obtain normalized data;

[0049] Feature data is extracted from the normalized data.

[0050] Compared with the prior art, this application has the following beneficial effects:

[0051] The present application provides a cloud platform capacity warning method and system based on a large model, including: obtaining resource usage data of the cloud platform to be predicted in real time; extracting feature data from the resource usage data; applying the prediction large model to process the feature data and outputting a resource capacity prediction value; obtaining the historical resource usage data of the cloud platform, and determining the current business needs of the cloud platform; setting an initial capacity threshold based on the historical resource usage data, and adjusting the initial capacity threshold based on the current business needs to obtain a capacity warning threshold; triggering a warning when the resource capacity prediction value is greater than the capacity warning threshold. After the present application uses the prediction large model to predict the resource capacity prediction value, it determines a dynamically changing capacity warning threshold based on the current business needs and historical resource usage data, thereby adapting to the dynamic changes of the cloud computing environment and providing a more accurate warning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A flowchart of a cloud platform capacity early warning method based on a large model provided in an embodiment of the present application;

[0054] Figure 2 A flowchart of a training prediction model provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a cloud platform capacity early warning system based on a large model provided in an embodiment of the present application;

[0056] Figure 4Example diagram of the function implementation of the data preprocessing module provided in the embodiment of the present application

[0057] Figure 5 An example diagram of the functional implementation of the real-time monitoring module provided in the embodiment of the present application;

[0058] Figure 6 This is an example diagram of the function implementation of the early warning judgment module provided in the embodiment of the present application;

[0059] Figure 7 This is an example diagram of the function implementation of the data acquisition module provided in the embodiment of the present application;

[0060] Figure 8 An example diagram of the functional implementation of the model training module provided in the embodiment of the present application;

[0061] Fig. 9 This is an example diagram of the working principle of cloud platform capacity warning based on a large model in a cloud environment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0064] Terminology explanation:

[0065] CNN (Convolutional Neural Network): A deep learning model mainly used for tasks in the fields of image recognition, image generation, video processing, natural language processing, etc. The design of CNN is inspired by the working principle of the human visual system. It can automatically learn to extract useful features from raw data, thereby realizing the recognition and understanding of complex data. Its automatic feature extraction capability can process high-dimensional data and has good generalization ability.

[0066] Backpropagation algorithm: A supervised learning algorithm used in artificial neural networks to train multi-layer feedforward neural networks. Its main purpose is to minimize the error between the network output and the actual output by adjusting the weights and biases in the network.

[0067] GRU: A variant of a recurrent neural network (RNN) for processing sequence data, proposed by Cho et al. in 2014. GRU is a variant of LSTM (Long Short-Term Memory Network) designed to solve the gradient vanishing and gradient exploding problems encountered by standard RNN when processing long sequence data.

[0068] Logstash: An open source data processing tool that is mainly used to collect, parse, and forward log data. It is part of the Elastic Stack (formerly known as the ELK Stack, which stands for Elasticsearch, Logstash, and Kibana) and is often used with Elasticsearch and Kibana to implement log management and analysis solutions.

[0069] Cron: A time-based task scheduler that is widely used in Unix-like systems to schedule tasks that need to be executed regularly or at specific times. Cron allows users to schedule tasks based on minutes, hours, days, months, and days of the week.

[0070] LSTM: is a special recurrent neural network (RNN) architecture proposed by Hochreiter and Schmidhuber in 1997. LSTM is designed to solve the gradient vanishing and gradient exploding problems encountered by traditional RNN when dealing with long-distance dependency problems.

[0071] As can be seen from the background technology, the traditional cloud platform capacity warning method relies on using a fixed empirical threshold to judge the change of resource capacity. This method is not accurate and is prone to false alarms or missed alarms.

[0072] Moreover, with the rapid changes in cloud computing services, fixed empirical thresholds are difficult to adapt to changing business needs. During business peaks, resource demand may suddenly increase, and traditional early warning methods cannot respond to such changes in a timely manner, resulting in insufficient resources and affecting the normal operation of services. On the other hand, if the threshold is set too conservatively, it may lead to excessive resource reservation, resulting in excess and waste of resources, and increasing the operating costs of the enterprise.

[0073] In view of this, the present application provides a cloud platform capacity warning solution based on a large model, which obtains the resource usage data of the cloud platform to be predicted in real time, extracts feature data from the resource usage data, applies the prediction large model to process the feature data, and outputs the resource capacity prediction value; obtains the historical resource usage data of the cloud platform, determines the current business needs of the cloud platform, and then sets the initial capacity threshold based on the historical resource usage data, and adjusts the initial capacity threshold based on the current business needs to obtain the capacity warning threshold, and triggers the warning when the resource capacity prediction value is greater than the capacity warning threshold. Therefore, in the process of warning the cloud platform capacity, the dynamically changing capacity warning threshold is used for judgment, and the capacity warning threshold can be continuously adjusted with the current business needs of the cloud platform, so as to adapt to the constantly changing cloud computing business of the cloud platform, and the warning process of the cloud platform capacity is more flexible, which improves the accuracy of the capacity warning and avoids the occurrence of false alarms or omissions.

[0074] The present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0075] The method provided in the embodiment of the present application can be applied to a processor or server of a cloud platform, referring to Figure 1 , which is a flow chart of a cloud platform capacity early warning method based on a large model provided in an embodiment of the present application, and is specifically described as follows:

[0076] S101. Obtain resource usage data of a cloud platform to be predicted in real time.

[0077] When acquiring the resource usage data of the cloud platform to be predicted in real time, the present application uses a stream processing framework to process the data stream of the data acquisition module to obtain the resource usage data of the cloud platform.

[0078] Exemplarily, the stream processing framework may be Apache Kafka or Apache Flink. The present application utilizes the stream processing framework to receive and process data streams from the data acquisition module in real time, thereby ensuring low latency and high throughput of data.

[0079] Preferably, resource usage data includes, but is not limited to, data on key performance indicators of the cloud platform, such as CPU usage, memory, disk I / O, network traffic, application server performance, database performance, system events, hardware status, etc.

[0080] Preferably, S101 is performed using a real-time detection module.

[0081] S102: Extract feature data from resource usage data.

[0082] The process of using the data preprocessing module to execute S102 and extracting feature data from the resource usage data is as follows: using a preset data processing tool to clean up invalid data, erroneous data, and duplicate data in the resource usage data to obtain request data; normalizing the cleaned data to obtain normalized data; and extracting feature data from the normalized data.

[0083] Preferably, when cleaning the data, invalid values, erroneous records, and duplicate data in the data set can be identified and removed by writing scripts or using data processing tools, such as the Pandas library, to ensure the consistency and accuracy of the data, thereby obtaining cleaned data.

[0084] When normalizing the cleaned data, normalization techniques such as minimum-maximum normalization or Z-score normalization are used to convert data of different dimensions into a unified scale to facilitate model calculation and processing, thereby obtaining normalized data.

[0085] The process of extracting feature data from normalized data can be regarded as feature work. Specifically, it can be done by performing time series analysis, statistical analysis and machine learning techniques on normalized data to extract data containing the trend, seasonality, periodicity, mean, standard deviation and other characteristics of the cloud platform data from the normalized data, thereby obtaining feature data. Using these feature data can enhance the prediction model's ability to predict the capacity of the cloud platform.

[0086] S103: Use the pre-trained prediction model to process the feature data and output the resource capacity prediction value.

[0087] The feature data is input into the trained prediction model, so that the trained prediction model processes the feature data, thereby outputting a resource capacity prediction value, where the resource capacity prediction value is a predicted value of the resource capacity of the cloud platform at a certain point in the future. Preferably, a prediction module can be used to perform step S103, so that a resource capacity prediction value of the cloud platform at a certain point in the future can be obtained.

[0088] S104: Obtain historical resource usage data of the cloud platform and determine current business requirements of the cloud platform.

[0089] When obtaining the historical resource usage data of the cloud platform, the historical resource usage data of the cloud platform can be obtained through the data collection module. Preferably, the historical resource usage data can be data obtained within a preset historical time. For example, the resource usage data of the cloud platform within the past 20 days are all determined as historical resource usage data. Specifically, if the current time is September 21, 2024, the resource usage data of the cloud platform from September 1, 2024 to September 20, 2024 are all determined as historical resource usage data.

[0090] Preferably, the current business demand can be understood from the perspectives of key business periods, resource usage peaks, business growth expectations, etc. Preferably, business demand determines the expected goals and key performance indicators of resource usage, and affects the direction and sensitivity of capacity threshold setting.

[0091] S105. An initial capacity threshold is set based on historical resource usage data, and the initial capacity threshold is adjusted based on current business needs to obtain a capacity warning threshold.

[0092] Analyze the historical resource usage data, and then set the initial capacity threshold based on the analysis results. For example, the historical resource usage data can be analyzed, and the resource usage pattern of the cloud platform can be identified based on the analysis results. Then, the capacity threshold corresponding to the resource usage pattern is determined as the initial capacity threshold. It should be noted that the resource usage pattern of the cloud platform is constantly changing, and different resource usage patterns have different capacity thresholds.

[0093] It should be noted that the results of analyzing historical resource usage data provide a historical perspective on resource usage, including average values, standard deviations, peak values, etc. Further, these are the basis for setting initial capacity thresholds.

[0094] Resource usage patterns reveal the inherent rules of resource usage. For example, resource usage peaks caused by daily business activities or usage troughs during system maintenance are both resource usage patterns. Furthermore, resource usage patterns can be determined from analysis results in terms of data trends, seasonal changes, or abnormal behaviors.

[0095] After the initial capacity threshold is determined, the initial capacity threshold is adjusted according to the current business demand to obtain the capacity warning threshold. Different current business demands have different adjustment strategies. For example, when the current business demand is a peak in resource usage, the capacity threshold can be increased to increase the capacity warning threshold, so as to meet the capacity demand during the peak period and increase the capacity value of the peak period warning. For another example, when the current business demand is within the critical business period, the capacity threshold can also be increased to ensure the capacity demand during the critical business period to increase the capacity value of the warning during the critical business period. In this way, dynamic adjustment of the capacity warning threshold is achieved. This is only an example. The process of adjusting the initial capacity threshold according to the current business demand in other cases will not be illustrated.

[0096] S106. When the resource capacity prediction value is greater than the capacity warning threshold, a warning is triggered.

[0097] The resource capacity prediction value is compared with the capacity warning threshold. When the resource capacity prediction value is greater than the capacity warning threshold, an alert is triggered.

[0098] In the method provided in the embodiment of the present application, the resource usage data of the cloud platform to be predicted is obtained in real time; feature data is extracted from the resource usage data; the feature data is processed by applying the pre-trained prediction model to output the resource capacity prediction value; the historical resource usage data of the cloud platform is obtained, and the current business needs of the cloud platform are determined; based on the historical resource usage data, the initial capacity threshold is set, and the initial capacity threshold is adjusted based on the current business needs to obtain the capacity warning threshold; when the resource capacity prediction value is greater than the capacity warning threshold, a warning is triggered. Preferably, the warning judgment module can be used to execute steps S105 and S106 to determine whether a warning is needed.

[0099] In the method provided in the embodiment of the present application, the resource usage data of the cloud platform to be predicted is obtained in real time; feature data is extracted from the resource usage data; the feature data is processed by applying a large prediction model to output a resource capacity prediction value; the historical resource usage data of the cloud platform is obtained, and the current business needs of the cloud platform are determined; based on the historical resource usage data, an initial capacity threshold is set, and the initial capacity threshold is adjusted based on the current business needs to obtain a capacity warning threshold; when the resource capacity prediction value is greater than the capacity warning threshold, an early warning is triggered. After the present application uses a large prediction model to predict the resource capacity prediction value, a dynamically changing capacity warning threshold is determined based on the current business needs and historical resource usage data, thereby adapting to the dynamic changes of the cloud computing environment and providing a more accurate early warning method.

[0100] In the method provided in the embodiment of the present application, after the warning is triggered, the warning generation module can be used to generate warning information for the cloud platform; for each preset notification channel, the warning information is converted into information in a data format that meets the notification channel, and the converted warning information is notified to the administrator through the notification channel; the warning information includes but is not limited to resource type, warning level, prediction time, recommended measures and other contents, wherein the prediction time here is the future time point corresponding to the resource capacity prediction value output by the prediction model.

[0101] Preferably, the notification channels include but are not limited to e-mail, SMS, instant messaging tools, etc., so as to ensure that the administrator can receive the warning information in time and take corresponding measures. Furthermore, different notification channels support different data formats. For each notification channel, the warning information is converted into information that meets the data format supported by the notification channel, and then the converted warning information is sent to the administrator through the notification channel, so that the administrator can perform corresponding operations in time.

[0102] The present application embodiment also provides a process for training a large prediction model. Figure 2 , is a flowchart of the training prediction model provided in the embodiment of the present application, and the relevant description is as follows:

[0103] S201. Obtain sample data.

[0104] When acquiring sample data, a preset data collection module may be used to periodically execute data collection tasks, and the data of each key performance indicator of the cloud platform collected when the data collection module executes the data collection task is determined as sample data.

[0105] Furthermore, the data collection module uses a timed task scheduler to execute data collection tasks at a fixed frequency, thereby collecting data on various key performance indicators of the cloud platform; further, when collecting data on various key performance indicators of the cloud platform, the data collection module can use an API interface or a log collection tool for collection, and the data collection module here is the same as the data collection module mentioned in S101.

[0106] S202: Preprocess the sample data to obtain training data.

[0107] The data preprocessing module can be used to preprocess the sample data to obtain training data. Preferably, the data preprocessing module performs data cleaning, data normalization, and feature extraction on the sample data to obtain training data. Preferably, the training data can be the feature data extracted last.

[0108] S203: Extract data features of the training data and select a model architecture corresponding to the data features.

[0109] Preferably, preliminary exploratory data analysis can be performed on the training data, including visualization of data, calculation of basic statistics, etc., to understand the distribution, trend, seasonality, etc. of the data, and then determine the time series characteristics of the data, such as the time interval of the data, whether it contains seasonal patterns, whether there are long-term dependencies, etc. Preferably, the time series characteristics here are the data characteristics, and then the model architecture corresponding to the data characteristics is selected.

[0110] Exemplary: Time series data: If the data has obvious long-term dependencies, you can choose LSTM or GRU because they perform well in processing long series data. Time series data with a focus on short-term patterns: If short-term patterns are more important in the data, GRU may be more suitable than LSTM because it has a simpler structure and is faster to train. Spatial data features: If the data has spatial features (for example, the spatial relationship of multiple sensor data), you can consider using CNN to extract spatial features. Multivariate time series: For data that contains multiple related time series, you can consider using a multi-input LSTM or GRU model.

[0111] The above-mentioned time series data, time series data with attention to short-term patterns, spatial data characteristics, and multi-variable time series are all time series characteristics mentioned above.

[0112] Furthermore, model architectures such as long short-term memory networks (LSTM), gated recurrent units (GRU), or convolutional neural networks (CNN) are not described here in detail.

[0113] S204. Build a large prediction model based on the model architecture.

[0114] After determining the model architecture, determine the structure of the model's input layer, hidden layer, and output layer, including the number of neurons, activation function, dropout ratio, etc., to build an efficient large prediction model.

[0115] S205. Use the training data to train the prediction large model, and optimize the model parameters of the prediction large model during the training process. When the prediction large model meets the preset training end conditions, the training of the prediction large model is terminated to obtain a trained prediction large model.

[0116] It should be noted that the training data is input into the prediction model, and the prediction model is trained through the back propagation algorithm and optimizer (such as Adam or RMSprop). At the same time, cross-validation and other technologies are used to optimize the model parameters in the prediction model to reduce overfitting, until the prediction model meets the preset training end conditions, the training of the prediction model is terminated, and the trained prediction model is obtained.

[0117] Preferably, the training end condition includes at least one of the following:

[0118] (1) The number of training iterations reaches the preset value;

[0119] (2) The convergence function in the model reaches the preset convergence condition.

[0120] This application uses data characteristics to select the model architecture, and then constructs a large prediction model. The constructed large prediction model is more suitable for processing cloud platform data, thereby improving the accuracy of the prediction and making the early warning more accurate. Preferably, the large prediction model can be constructed before each prediction is needed, and then the large prediction model can be trained, and then the trained large prediction model can be used for prediction, so as to meet the current data characteristics of the cloud platform, improve the prediction accuracy, and then improve the early warning accuracy. The large prediction model can also be constructed in advance, and then trained. After obtaining the trained large prediction model, the trained large prediction model can be used for prediction each time. Thus, there is no need to build and train the model before each prediction, which reduces the workload and improves the prediction efficiency. Preferably, in this case, the prediction results can be used to optimize the large prediction model after each prediction, so as to continuously improve the performance of the large prediction model.

[0121] The present application embodiment provides a cloud platform capacity early warning system based on a large model, which can support Figure 1 For the implementation of the method shown, refer to Figure 3 , which is a structural diagram of a cloud platform capacity warning system based on a large model provided in an embodiment of the present application.

[0122] The cloud platform capacity warning system includes: an acquisition module 301, a data preprocessing module 302, a prediction module 303, a warning judgment module 304, a warning notification module 306, a data collection module 305 and a model training module 307.

[0123] The functions of each module are described separately as follows:

[0124] The acquisition module 301 is used to acquire the resource usage data of the cloud platform to be predicted in real time.

[0125] Preferably, the module is used to obtain resource usage data of the cloud platform in real time so that the data can be used for subsequent prediction.

[0126] The data preprocessing module 302 is used to extract feature data from the resource usage data.

[0127] It should be noted that, refer to Figure 4 , is a functional implementation example diagram of the data preprocessing module provided in the embodiment of the present application. As shown in the figure, when the data preprocessing module processes the resource usage data, it processes the data from three aspects: data cleaning, data normalization, and feature engineering. Among them, feature engineering can be used to extract feature data.

[0128] a) Data cleaning: Identify and remove invalid values, erroneous records, and duplicate data in the dataset by writing scripts or using data processing tools such as the Pandas library to ensure data consistency and accuracy.

[0129] b) Data normalization: Normalization techniques, such as minimum-maximum normalization or Z-score normalization, are used to convert data of different dimensions into a unified scale to facilitate model calculation and processing.

[0130] c) Feature Engineering: Through time series analysis, statistical analysis and machine learning techniques, the characteristics of the data are extracted, such as trend, seasonality, periodicity, mean, standard deviation, minimum / maximum value, quantile, etc. These characteristics will enhance the predictive ability of the model.

[0131] As shown in the figure, time series analysis techniques can be used to analyze trend characteristics. Regarding the analysis of trend characteristics: use time series decomposition techniques, such as STL (seasonal decomposition time series) or ETS (error, trend, seasonality) models, to identify long-term trends in the data.

[0132] Seasonal characteristics can be analyzed by statistical analysis. The analysis of seasonal characteristics is to identify periodic patterns in the data, such as daily, weekly or annual periodic changes. Preferably, seasonal characteristics can also be analyzed by time series analysis.

[0133] Features such as periodicity, mean, standard value, minimum / maximum, quantiles, etc. can be analyzed using machine learning techniques.

[0134] Analysis of cyclical characteristics: For cyclical non-seasonal changes, such as economic cycles, use methods such as Fourier transform to extract cyclical characteristics. Mean: Calculate the mean value within the time window to reflect the central trend of the data. Standard deviation: Calculate the standard deviation to measure the degree of dispersion of the data. Minimum / maximum: Determine the minimum and maximum values ​​within the time window to provide range information of the data. Quantile: Calculate quantiles, such as the median, first quartile, and third quartile, to provide distribution information of the data.

[0135] Preferably, statistical analysis can be used to analyze features such as average value, standard value, minimum / maximum value, quantile, etc.

[0136] This solution combines time series analysis and machine learning techniques when extracting feature data, which can extract features more comprehensively. It has a higher degree of automation, reducing the need for manual operations. It can handle more complex patterns and relationships, such as nonlinear and interaction effects. This application combines multiple methods to extract features of more dimensions, which can comprehensively extract features and enhance the predictive ability of the model. It also reduces the time and workload of manual feature construction and improves the efficiency of feature engineering. A richer feature set helps the model capture complex relationships in the data, thereby improving the accuracy of predictions. The features extracted by this application through time series analysis and statistical analysis often have good interpretability, which helps to understand the behavior of the model, and the machine learning method can adapt to different types of data and problems, making the feature engineering process more flexible.

[0137] The prediction module 303 is used to process the feature data using a pre-trained prediction model and output a resource capacity prediction value.

[0138] Preferably, the prediction module 303 and the acquisition module 301 of the present application can form a real-time monitoring module, which can realize two functions, namely real-time data stream processing and model prediction, wherein the real-time data stream processing is realized by the acquisition module 301, and the model prediction is realized by the prediction module 303.

[0139] Reference Figure 5 , which is an example diagram of the functional implementation of the real-time monitoring module provided in the embodiment of the present application. As shown in the figure, the content of real-time data stream processing includes: using a stream processing framework such as Apache Kafka or Apache Flink to receive and process the data stream from the data acquisition module in real time to ensure low latency and high throughput of the data. The content of model prediction includes: inputting the real-time processed data into the trained deep learning model (i.e., the prediction model), performing real-time prediction of resource capacity, and outputting the predicted value of resource usage in the future (i.e., the predicted value of resource capacity).

[0140] Furthermore, the present application provides a process of using a stream processing framework to receive and process data streams from a data acquisition module in real time. The relevant process is described in steps 1-11:

[0141] Step 1: Determine the data source.

[0142] Determine which data sources will be used to collect data in real time. These data sources may be cloud platform APIs, log files, database systems, or other monitoring tools.

[0143] Step 2: Configure the data source.

[0144] Configure data sources to push data to the stream processing system. This may involve setting up log forwarding, API calls, or other forms of real-time data push.

[0145] Preferably, step 1-2 is to configure the data source, and the data source can be determined by configuring the data source.

[0146] Step 3: Deploy the stream processing framework.

[0147] Deploy Apache Kafka or Apache Flink on a server or cloud environment.

[0148] Step 4: Create a data flow.

[0149] In Kafka, create one or more topics to receive data streams.

[0150] In Flink, define data sources and data stream processing tasks.

[0151] Step 3-4 is the stream processing framework setting, which is used to process the data collected from the data source.

[0152] Step 5: Receive data.

[0153] The data source starts pushing data to the Kafka topic or Flink data source.

[0154] Step 6: Data buffering.

[0155] As a message queue, Kafka can buffer large amounts of data to ensure that data is not lost under high throughput.

[0156] Steps 5-6 are about data collection.

[0157] Step 7: Define the processing logic.

[0158] In Flink, processing functions are defined to transform, aggregate, or filter data streams.

[0159] If you use Kafka, you can combine it with other processing frameworks such as Apache Spark Streaming or Flink to process the data.

[0160] Step 8: Real-time processing.

[0161] Flink executes the defined processing logic and performs real-time operations on the incoming data.

[0162] Processing may include data cleaning, format conversion, aggregation calculation, anomaly detection, etc.

[0163] Steps 7-8 are the content of data processing.

[0164] Step 9: Output the results.

[0165] The processed data can then be pushed to a variety of target systems such as a database, a data warehouse, a monitoring dashboard, or another Kafka topic.

[0166] Step 10: Visualization and alerting.

[0167] Visualize processing results for easy monitoring and analysis.

[0168] Set up alert mechanisms to send notifications when anomalies or specific conditions are detected.

[0169] Steps 9-10 are the contents of data output.

[0170] Step 11: Monitor stream processing performance.

[0171] Monitor the performance of the stream processing system, including latency, throughput, and resource utilization. Ensure stable system operation and handle performance bottlenecks or failures in a timely manner. Step 11 implements continuous monitoring and maintenance of data, and the early warning judgment module 304 is used to obtain historical resource usage data of the cloud platform and determine the current business needs of the cloud platform; based on the historical resource usage data, set the initial capacity threshold, and adjust the initial capacity threshold based on the current business needs to obtain the capacity warning threshold; when the resource capacity prediction value is greater than the capacity warning threshold, trigger an early warning.

[0172] Reference Figure 6 , which is an example diagram of the functional implementation of the early warning judgment module provided in the embodiment of the present application. As shown in the figure, the early warning judgment module is divided into two functions: setting a dynamic threshold (i.e., a capacity early warning threshold) and early warning logic. The content of setting a dynamic threshold includes: determining an initial capacity threshold based on the statistical analysis results of historical data (i.e., historical resource usage data), and then adjusting the initial capacity threshold based on current business needs, thereby obtaining a capacity early warning threshold, and completing the setting of a dynamic early warning threshold.

[0173] The content of the early warning logic includes: comparing the prediction result of the model (i.e., the resource capacity prediction value) with the set threshold (i.e., the capacity warning threshold). If the prediction value exceeds the threshold, the early warning mechanism is triggered and the early warning notification process is started.

[0174] Warning notification module: used to generate warning information of the cloud platform after the warning is triggered; for each preset notification channel, the warning information is converted into information that meets the data format of the notification channel, and the converted warning information is notified to the administrator through the notification channel. Preferably, the warning information can be fed back to the administrator in a clear and easy-to-understand format.

[0175] The functions realized by the data acquisition module are described in detail. Figure 7 , which is an example diagram of the functional implementation of the data acquisition module provided in an embodiment of the present application. The data acquisition module in the figure includes the content of collecting data using an API interface or a log collection tool, as well as the content of the work of a scheduled task scheduler.

[0176] Related instructions include: Data collected using API interfaces or log collection tools: Through the API interfaces provided by the cloud platform, or using log collection tools such as Fluentd and Logstash, key performance indicator data such as CPU usage, memory usage, disk I / O, and network traffic are regularly extracted from the cloud platform's infrastructure and services.

[0177] The work content of the scheduled task scheduler: deploy a scheduled task scheduler, such as a cron job or a Quartz scheduling framework, to execute data collection tasks at a fixed frequency (for example, every 5 minutes or every hour) so that data can be collected using an API interface or a log collection tool. By collecting data at a fixed frequency, the continuity and real-time nature of the data can be ensured. Preferably, the scheduled task scheduler is also responsible for managing failed retries, logging, and task monitoring of collection tasks.

[0178] The model training module is used to obtain sample data; preprocess the sample data to obtain training data; extract the data characteristics of the training data and select the model architecture corresponding to the data characteristics; build a prediction model based on the model architecture; use the training data to train the prediction model, and optimize the model parameters of the prediction model during the training process until the prediction model meets the preset training end conditions, then end the training of the prediction model and obtain a trained prediction model.

[0179] Reference Figure 8 , which is an example diagram of the functional implementation of the model training module provided in an embodiment of the present application. As shown in the figure, when training the model, the model training module includes selecting a suitable deep learning model, designing the network structure of the deep learning model, and performing training content.

[0180] Selecting a suitable deep learning model (i.e., the model architecture mentioned above) includes: selecting a suitable model architecture, such as a long short-term memory network (LSTM), a gated recurrent unit (GRU), or a convolutional neural network (CNN), based on the characteristics of the resource usage data.

[0181] The content of designing the network structure includes: determining the structure of the input layer, hidden layer and output layer of the model, including the number of neurons, activation function, dropout ratio, etc., so as to build an efficient prediction model (the prediction large model mentioned above).

[0182] The content of training the model using historical data: input the preprocessed data (the training data mentioned above) into the model, and train the model through the back propagation algorithm and optimizer (such as Adam or RMSprop). At the same time, use cross-validation and other techniques to optimize model parameters and reduce overfitting.

[0183] Reference Fig. 9 , is an example diagram of the working principle of the cloud platform capacity warning based on the big model in the cloud environment provided by the embodiment of the present application. The figure can also be regarded as an example diagram of the big model security prevention and control inspection method. As shown in the figure, it includes data collection, model training, real-time monitoring and early warning judgment. Further, data collection includes using monitoring tools to collect data on key performance indicators of the cloud platform and collect logs of the cloud platform, and then preprocessing the collected data, and the preprocessing includes but is not limited to cleaning, normalization and feature extraction. Model training includes first building a deep learning model (equivalent to the prediction of the big model), and then using historical data to train the constructed deep learning model, and the parameters of the model are continuously optimized during the training process. The content of real-time monitoring includes obtaining real-time data streams, and then using the stream processing framework for processing, and then inputting the processed data into the trained deep learning model for prediction, and outputting the predicted value (equivalent to the resource capacity prediction value). The content of early warning judgment includes dynamically setting a threshold, which is the capacity warning threshold, and then comparing the capacity warning threshold with the resource smelting prediction value according to the early warning logic, and then determining whether to trigger the early warning. After the early warning is triggered, the notification process is entered.

[0184] By applying the solution provided in the embodiments of the present application, the prediction model built using deep learning technology can learn resource usage patterns from historical data, thereby more accurately predicting the changing trend of resource capacity and reducing false alarms and missed alarms. In addition, the real-time monitoring module can process real-time data streams, promptly detect changes in resource usage, and provide early warnings, so that administrators can take timely measures to ensure the stable operation of the cloud platform.

[0185] The warning threshold of this application can be dynamically adjusted according to the business characteristics and historical data of the cloud platform to meet the needs of different scenarios and improve the adaptability and robustness of the warning system. Through accurate capacity warning, the system can optimize resource allocation, improve the utilization rate of cloud platform resources and reduce operating costs.

[0186] This application uses deep learning technology to build a large prediction model to predict the resource capacity of the cloud platform, which reduces manual intervention in the process of cloud platform capacity warning. The warning method based on artificial intelligence technology reduces the need for manual intervention and improves the intelligent level of cloud platform management. By sending warning information to administrators, it helps administrators quickly locate problems, take effective measures, reduce operational risks, and ensure the stability and security of the cloud platform.

[0187] The application of the early warning system provided by this application helps prevent insufficient or excessive resources, ensures that the cloud platform can still provide stable services during peak periods or emergencies, and ensures business continuity. Through effective resource management, the cloud platform can better meet user needs, improve service quality, and thus improve user satisfaction.

[0188] This application provides a cloud platform capacity warning solution based on a large model in a cloud environment, which can realize real-time monitoring and warning of cloud platform capacity data and improve the stability of the cloud platform. The application of this application can achieve the following goals:

[0189] Intelligence: Through algorithms such as deep learning, it is possible to automatically learn patterns and rules of resource usage from historical data, thereby improving the accuracy of predictions.

[0190] Dynamicity: The early warning model can dynamically adjust the prediction results based on real-time data, respond to changes in resource capacity in a timely manner, and avoid early warning deviations caused by fixed thresholds.

[0191] Predictive: The new early warning method can not only reflect the current resource usage status, but also predict the resource demand trend in the future, providing cloud platform administrators with more forward-looking decision support.

[0192] Adaptability: The early warning system can adapt to changes in different business scenarios and needs, and improve its adaptability and robustness through continuous learning and optimization.

[0193] In summary, the cloud platform capacity warning method based on big data and artificial intelligence technology will provide strong technical support for the stable operation of the cloud platform, help improve the quality and efficiency of cloud services, reduce operational risks, and achieve optimal allocation of resources.

[0194] Although the present invention depicts operations in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0195] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0196] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned cloud platform capacity warning method based on the large model.

[0197] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards in the relevant regions.

[0198] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0199] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0200] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0201] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cloud platform capacity early warning method based on a large model, characterized in that: include: Obtain resource usage data of the cloud platform to be predicted in real time; extracting feature data from the resource usage data; Applying a pre-trained prediction model to process the characteristic data and outputting a resource capacity prediction value; Obtaining historical resource usage data of the cloud platform and determining current business needs of the cloud platform; Based on the historical resource usage data, an initial capacity threshold is set, and based on the current business demand, the initial capacity threshold is adjusted to obtain a capacity warning threshold; When the resource capacity prediction value is greater than the capacity warning threshold, a warning is triggered.

2. The method according to claim 1, characterized in that Also includes: After the warning is triggered, generating warning information of the cloud platform; For each preset notification channel, the warning information is converted into information in a data format that satisfies the notification channel, and the converted warning information is notified to the administrator through the notification channel.

3. The method according to claim 1, characterized in that: The process of training a large prediction model includes: Get sample data; Preprocessing the sample data to obtain training data; Extracting data characteristics of the training data, and selecting a model architecture corresponding to the data characteristics; Based on the model architecture, construct a large prediction model; The prediction large model is trained using the training data, and the model parameters of the prediction large model are optimized during the training process until the prediction large model meets the preset training end conditions, thereby terminating the training of the prediction large model and obtaining a trained prediction large model.

4. The method according to claim 3, characterized in that: The obtaining of sample data comprises: A preset data collection module is used to periodically execute data collection tasks, and the data of each key performance indicator of the cloud platform collected when the data collection module executes the data collection task is determined as sample data.

5. The method according to claim 1, characterized in that The extracting characteristic data from the resource usage data comprises: Using a preset data processing tool to clean invalid data, erroneous data and duplicate data in the resource usage data to obtain request data; Normalizing the cleaned data to obtain normalized data; Feature data is extracted from the normalized data.

6. A cloud platform capacity early warning system based on a large model, characterized in that: include: An acquisition module is used to acquire resource usage data of the cloud platform to be predicted in real time; A data preprocessing module, used to extract feature data from the resource usage data; A prediction module, used to process the characteristic data using a pre-trained prediction model and output a resource capacity prediction value; An early warning judgment module is used to obtain historical resource usage data of the cloud platform and determine the current business needs of the cloud platform; Based on the historical resource usage data, an initial capacity threshold is set, and based on the current business demand, the initial capacity threshold is adjusted to obtain a capacity warning threshold; When the resource capacity prediction value is greater than the capacity warning threshold, a warning is triggered.

7. The system according to claim 6, characterized in that Also includes: Model training module, used to train large prediction models; The process of training the large prediction model by the model training module includes: Get sample data; Preprocessing the sample data to obtain training data; Extracting data characteristics of the training data, and selecting a model architecture corresponding to the data characteristics; Based on the model architecture, construct a large prediction model; The prediction large model is trained using the training data, and the model parameters of the prediction large model are optimized during the training process until the prediction large model meets the preset training end conditions, thereby terminating the training of the prediction large model and obtaining a trained prediction large model.

8. The system according to claim 6, characterized in that Also includes: Early warning notification module; The warning notification module is used to generate warning information of the cloud platform after the warning is triggered; For each preset notification channel, the warning information is converted into information in a data format that satisfies the notification channel, and the converted warning information is notified to the administrator through the notification channel.

9. The system according to claim 7, characterized in that The model training module executes a process of acquiring sample data, including: A preset data collection module is used to periodically execute data collection tasks, and the data of each key performance indicator of the cloud platform collected when the data collection module executes the data collection task is determined as sample data.

10. The system according to claim 6, characterized in that The data preprocessing module performs a process of extracting feature data from the resource usage data, including: Using a preset data processing tool to clean invalid data, erroneous data and duplicate data in the resource usage data to obtain request data; Normalizing the cleaned data to obtain normalized data; Feature data is extracted from the normalized data.

Images