Data intelligent optimization method based on AI

Abstract

The invention discloses an AI-based data intelligent optimization method, which comprises the following steps of: extracting a plurality of modal features in data, and performing weighted fusion to obtain multi-modal features; constructing a performance prediction model, and inputting the multi-modal features into the constructed performance prediction model to obtain performance prediction indexes; and constructing a scheduling optimization model, and inputting the performance prediction index into the constructed performance prediction model to obtain a resource scheduling strategy. By adopting the dynamic scheduling strategy based on the AI, the load condition of the data center can be predicted and analyzed in real time, so that a more accurate resource scheduling scheme is provided. By introducing the performance prediction model and the scheduling optimization model, resource configuration can be automatically adjusted, the utilization rate of data center resources is effectively improved, and energy waste is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to an AI-based data intelligent optimization method. Background Art

[0002] With the rapid development of information technology, data centers, as the core facilities supporting high-tech applications such as cloud computing, big data, and artificial intelligence, undertake massive data processing and storage tasks. However, with the continuous expansion of the scale of data centers and the continuous change of resource requirements, traditional data scheduling and optimization strategies have gradually revealed some deficiencies. Existing technologies usually rely on static resource allocation schemes and manual scheduling mechanisms, lacking the ability to adapt to dynamic environmental changes, resulting in low resource utilization, low energy efficiency, and difficulty in dealing with complex scenarios such as hardware failures and load fluctuations. Therefore, how to achieve more intelligent and efficient data center resource scheduling in a changing environment has become a current research hotspot.

[0003] Most current scheduling optimization methods are based on traditional algorithms such as heuristic algorithms, queuing theory, and greedy algorithms. Although these methods can optimize resource allocation to a certain extent, they often show performance degradation when facing complex and changing system loads. Specifically, existing scheduling methods lack the ability to accurately predict resource loads and cannot adjust scheduling strategies in real time to adapt to different environmental changes. In addition, many methods are optimized based on a single data source, ignoring the multi-dimensional and cross-modal data characteristics in the system, resulting in large prediction errors and unsatisfactory scheduling effects. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide an AI-based data intelligent optimization method.

[0005] The present invention provides an AI-based data intelligent optimization method, including the following steps: S1. Extract several modal features from the data and perform weighted fusion to obtain multi-modal features; S2. Construct a performance prediction model, input the multi-modal features into the constructed performance prediction model to obtain performance prediction indicators; S3. Construct a scheduling optimization model, input the performance prediction indicators into the constructed performance prediction model to obtain a resource scheduling strategy.

[0006] Further, in step S1, the modal features include text features, image features, and sensor data features, and the text features, image features, and sensor data features are weighted and fused through a self-attention mechanism to obtain multi-modal features.

[0007] Further, in step S2, the formula of the performance prediction model is expressed as follows:

[0008] Wherein, is the performance prediction index at time t, which represents the predicted resource demand at this time point; N is the number of modalities, representing the number of types of input data; is the weight matrix of the i-th modality, which is used to control the influence of this modality in the final prediction. The weight matrix is optimized through learning during the training process; is the representation of the i-th modality feature at time t; is the feature of the i-th modality The function for non-linear activation; b is the bias term, which adjusts the output of the model to avoid the deviation of all outputs; is the activation function, which maps the output result to the interval [0, 1].

[0009] Further, in step S2, the formula of the scheduling optimization model is expressed as follows:

[0010] Wherein,[[]] is the optimal resource scheduling decision at time t, which is a vector indicating how to allocate various resources at this time point; M is the number of optimization decision parameters, representing the number of resource categories to be optimized; is the weight coefficient of the i-th resource scheduling, which is used to balance different resource demands; is the influence function of the resource demand according to the performance prediction index at time t; is the cost function of resource scheduling, representing the cost after a given resource allocation R; represents selecting the resource scheduling decision that minimizes the objective function through the optimization algorithm, that is, obtaining the optimal resource allocation strategy by minimizing the weighted sum of the cost function.

[0011] This method further includes step S4, which is specifically as follows: The real-time performance monitoring module comprehensively monitors the resource usage of the data center, tracks the real-time performance metrics, and compares them with the output of the performance prediction model to obtain an error value. If there is a significant deviation between the predicted resource demand and the actual demand, the subsequent error analysis and model update process will be initiated.

[0012] Further, the error analysis and model update are specifically as follows: when the error between the actual performance metric detected by the performance monitoring module and the prediction result exceeds the threshold, the error analysis mechanism is activated, the model is reconstructed, the feature extraction process is readjusted, and the weight matrix is updated , or readjust the decision variables to adapt to the new operating environment.

[0013] Furthermore, the performance metrics include the actual utilization rate of the CPU, the actual occupancy rate of the memory, and the performance prediction metrics.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention adopts an AI-based dynamic scheduling strategy, which can perform real-time prediction and analysis on the load conditions of the data center, so as to provide a more accurate resource scheduling scheme. By introducing a performance prediction model and a scheduling optimization model, it can automatically adjust the resource configuration, effectively improve the utilization rate of the data center resources, and reduce energy waste.

[0015] 2. The present invention, through a real-time monitoring and error analysis mechanism, automatically adjusts the model parameters when detecting prediction errors, and performs dynamic optimization using causal inference. This process can enhance the system's ability to cope with uncertainties and changes, and enhance the robustness and reliability of the data center scheduling system, so as to ensure that the system can still maintain an efficient and stable operating state under different loads and resource configurations.

[0016] 3. The present invention, by combining an adaptive scheduling strategy update and a model iterative optimization mechanism, can dynamically respond to environmental changes. Through continuous model updates and adaptive adjustments, the system can maintain high-efficiency scheduling performance and avoid the impact of outdated data on the scheduling results, thereby ensuring the reliability and accuracy of the scheduling scheme in various environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is the principle flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0019] As Figure 1 shown, the present invention provides an AI-based data intelligent optimization method, including the following steps: S1. Extract several modal features from the data and perform weighted fusion to obtain multi-modal features; the modal features include text features, image features, and sensor data features, and the text features, image features, and sensor data features are weighted and fused through a self-attention mechanism to obtain multi-modal features.

[0020] In a data center, the data sources are diverse and the modalities are complex, such as text logs, device images, and sensor signals. To achieve intelligent optimization, it is first necessary to perform feature extraction and unified representation on multi-modal data, which provides basic support for subsequent feature fusion and optimization strategies.

[0021] In this step, by constructing a multi-modal feature deep extraction model, specific neural networks are used for feature extraction for different types of data.

[0022] Text feature extraction: For logs and configuration files, the BERT model based on the Transformer architecture is used to extract context semantic features and generate high-dimensional semantic embedding vectors.

[0023] Image feature extraction: For device monitoring videos and images, a convolutional neural network (CNN) is used for feature extraction to obtain device status and abnormal change features.

[0024] Sensor data feature extraction: For sensor data such as temperature, humidity, and energy consumption, a long short-term memory network (LSTM) is used to extract temporal features and generate sequence embedding representations.

[0025] To ensure feature consistency, the extracted multi-modal features are weighted and fused through the self-attention mechanism to form a unified feature representation matrix. This matrix serves as the input for the subsequent optimization model. Through the feature embedding space alignment mechanism, it is ensured that different modal features have a similar distribution in the same space, thereby reducing the impact of feature heterogeneity on the optimization effect.

[0026] S2. Build a performance prediction model. Input the multi-modal features into the constructed performance prediction model to obtain performance prediction metrics. The goal of the performance prediction model is to predict future resource usage through feature fusion of multi-modal data. Assume that the input feature of each modality is X i , where i represents the modality type, such as text, image, sensor data, etc. After these modal data are subjected to feature extraction, the unified representation obtained is H i , and the final representation obtained through feature fusion is H m , which serves as the input for the prediction model.

[0027] The formula expression of the performance prediction model is as follows:

[0028] Among them, is the performance prediction metric at time t, which represents the predicted resource demand at this time point, usually performance metrics such as CPU utilization rate, memory occupancy rate, network bandwidth, and energy consumption; N is the number of modalities, representing the number of types of input data, such as text, images, sensor data, etc. Each modality data requires independent processing and feature extraction; is the weight matrix for the i-th modality, used to control the influence of this modality in the final prediction. The weight matrix is optimized through learning during the training process; is the representation of the i-th modality feature at time t, usually a feature vector extracted by a neural network, such as text features processed by BERT, image features processed by CNN, time series data processed by LSTM, etc.; is for the feature of the i-th modality is a function for non-linearly activating the feature of the i-th modality. Usually, it can be the ReLU (Rectified Linear Unit) or Sigmoid function, used to improve the non-linear expression ability of the model; b is the bias term, which adjusts the output of the model to avoid the bias of all outputs; is the activation function that maps the output result to the interval [0, 1], which is suitable for prediction problems or outputs that need to be normalized.

[0029] Through the multi-modal data fusion model, a weighted sum is calculated based on the features of each modality. After passing through the activation function, a predicted value in the interval [0, 1] is obtained. These predicted values correspond to various performance metrics of the data center, such as CPU utilization, memory occupancy, etc. During the performance prediction process, the trained weight matrix will be automatically adjusted according to the importance of different modality data to obtain more accurate prediction results.

[0030] S3. Build a scheduling optimization model, input the performance prediction metrics into the constructed performance prediction model to obtain a resource scheduling strategy; The input of the scheduling optimization model is the output of the performance prediction model , and this predicted value represents the future load demand. According to these demands, the resource allocation strategy is adjusted through reinforcement learning to ensure the efficient use of resources and optimize the overall performance of the system.

[0031] The formula expression of the scheduling optimization model is as follows:

[0032] where, is the optimal resource scheduling decision at time t. It is a vector indicating how to allocate various resources (such as CPU cores, memory, storage, etc.) at this time point. These resource configurations are determined by the reinforcement learning model; M is the number of optimization decision parameters, representing the number of resource categories to be optimized. For example, a data center may have multiple resource categories such as CPU, memory, storage, network bandwidth, etc.; is the weight coefficient for the i-th resource scheduling, used to balance different resource requirements. The weight coefficient reflects the priority or importance of resource scheduling and can be dynamically adjusted according to actual needs; is the performance prediction index at time t is the impact function of resource demand, which describes how the predicted value affects the resource allocation decision. Usually, this function can be linear or non-linear, depending specifically on the relationship between resource demand and performance; is the cost function of resource scheduling, representing the cost (such as energy consumption, latency, etc.) after a given resource allocation R. The cost function is used to measure the overhead brought by resource allocation and is usually a negative objective function, aiming to minimize the burden brought by resource use; represents the resource scheduling decision that minimizes the objective function through the optimization algorithm, that is, the optimal resource allocation strategy is obtained by minimizing the weighted sum of the cost function.

[0033] The core of this scheduling optimization model is to optimize the scheduling decisions of various resources (such as CPU, memory, storage, etc.) to achieve the goal of minimizing performance impact and cost. Specifically, the model calculates the cost of different resource configurations based on the predicted load demand (output by the performance prediction model) and finds the optimal resource scheduling strategy by minimizing the cost function. This process is based on the reinforcement learning algorithm and can continuously adaptively optimize resource scheduling in a real-time data center environment.

[0034] Through the joint application of the performance prediction model and the scheduling optimization model, it is possible to intelligently predict and adjust the resource allocation strategy, ensuring the efficient operation of the data center when load changes and emergencies occur.

[0035] The scheduling optimization model effectively reduces energy consumption and improves resource utilization by comprehensively considering the cost and performance requirements of resource scheduling.

[0036] S4. In the aforementioned scheduling optimization model, the resource configuration is adjusted through reinforcement learning to adapt to the load changes in the data center. However, in the actual application process, the environment is constantly changing, which may lead to the degradation or failure of the model performance. Therefore, in this step, we introduce an adaptive scheduling strategy update and model iterative optimization to ensure that the model can dynamically respond to environmental changes and optimize the resource scheduling strategy.

[0037] Specifically, the real-time performance monitoring module comprehensively monitors the resource usage of the data center, tracks real-time performance metrics, and compares them with the output of the performance prediction model to obtain an error value. If there is a significant deviation between the predicted resource demand and the actual demand, the subsequent error analysis and model update processes are initiated.

[0038] The goal of error analysis is to deeply identify the root causes of performance degradation. Possible causes include: Obsolescence of the model itself: Due to the time difference between historical data and the model training process, the performance prediction model may lose its sensitivity to the current system state.

[0039] External environment changes: For example, hardware device failures, new task load changes, etc., resulting in inconsistent model prediction results and actual loads.

[0040] When the performance monitoring module detects that the error between the actual performance metrics and the prediction results exceeds the threshold, the error analysis mechanism is activated to reconstruct the model, readjust the feature extraction process, and update the weight matrix , or readjust the decision variables , to adapt to the new operating environment.

[0041] To cope with the dynamic changes in the external environment and the introduction of errors, this step dynamically updates the key parameters affecting the scheduling optimization effect through a causal inference mechanism. Causal inference can analyze the potential causal relationships affecting system performance from the actual operation data and adjust the important parameters in the model according to these relationships.

[0042] Based on the results of causal inference, some key parameters in the model will be dynamically adjusted. For example, if the demand for a certain type of resource is mispredicted, the system will adjust the relevant scheduling strategy to better adapt to the new load situation.

[0043] After the model is updated, to ensure that the scheduling optimization model has stronger generalization ability and anti-interference ability, this step iteratively trains the model through a transfer learning strategy. Specifically, the latest operation data (including actual performance data and resource allocation strategy results) is used to update the model parameters, so that the model can continuously adapt to environmental changes and avoid over-reliance on historical data.

[0044] Performance metrics include the actual utilization rate of the CPU, the actual occupancy rate of memory, and performance prediction metrics.

[0045] This step effectively copes with the challenges brought by environmental changes (such as hardware failures, load changes, etc.) through adaptive updates and model iterative optimization, ensuring that the scheduling model can continuously adapt to the new operating conditions and improving its robustness and adaptability.

[0046] Meanwhile, through online learning and real-time monitoring, it is ensured that the model does not lose its sensitivity to the system state over time, preventing model degradation.

[0047] Moreover, through continuous optimization and dynamic adjustment, the scheduling strategy can provide the best resource allocation scheme at different times, thereby improving the overall performance and energy efficiency of the system and enhancing the scheduling efficiency.

[0048] The key to this step is to combine performance monitoring, error analysis, and an adaptive update mechanism to ensure that the model can always respond quickly to environmental changes and maintain long-term high efficiency through iterative training.

[0049] The above embodiments are only used to illustrate the technical method of the present invention and not to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A data intelligent optimization method based on AI, characterized in that: The following steps are involved: S1, extract several modal features from the data, and perform weighted fusion to obtain multimodal features; S2, constructing a performance prediction model, inputting the multimodal features into the constructed performance prediction model, and obtaining a performance prediction index; S3: construct a scheduling optimization model, input the performance prediction index into the constructed performance prediction model, and obtain a resource scheduling strategy.

2. The AI-based data intelligent optimization method according to claim 1, characterized in that: In step S1, the modal features include text features, image features and sensor data features, and the text features, image features and sensor data features are weightedly fused through a self-attention mechanism to obtain multimodal features.

3. The AI-based data intelligent optimization method according to claim 2, characterized in that: In step S2, the performance prediction model is expressed as follows:

4. Among them, is the performance prediction index at time t, which indicates the predicted resource demand at that time point; N is the number of modes, indicating the number of types of input data; is the weight matrix of the i-th mode, which is used to control the influence of the mode in the final prediction. The weight matrix is ​​optimized through learning during the training process; is the representation of the i-th modal feature at time t; is the characteristic of the i-th mode A function for nonlinear activation; b is a bias term that adjusts the output of the model to avoid deviations in all outputs; is an activation function that maps the output to the interval [0, 1].

5. The AI-based data intelligent optimization method according to claim 3, characterized in that: In step S2, the scheduling optimization model is expressed as follows:

6. Among them, is the optimal resource scheduling decision at time t, which is a vector indicating how various resources should be allocated at that time point; M is the number of optimization decision parameters, indicating the number of resource categories that need to be optimized; is the weight coefficient of the i-th resource scheduling, which is used to balance different resource demands; is the performance prediction index at time t Impact function on resource demand; is the cost function of resource scheduling, which represents the cost after a given resource allocation R; It means that the resource scheduling decision that minimizes the objective function is selected through the optimization algorithm, that is, the optimal resource allocation strategy is obtained by minimizing the weighted sum cost function.

7. The AI-based data intelligent optimization method according to claim 4, characterized in that: The step S4 is further included, which is as follows: The real-time performance monitoring module comprehensively monitors the resource usage of the data center, tracks real-time performance indicators, and compares them with the output of the performance prediction model to obtain the error value. If there is a significant deviation between the predicted resource demand and the actual demand, the subsequent error analysis and model update process will be initiated.

8. The AI-based data intelligent optimization method according to claim 5, characterized in that: The error analysis and model update are as follows: when the performance monitoring module detects that the error between the actual performance indicator and the predicted result exceeds the threshold, the error analysis mechanism is activated, the model is reconstructed, the feature extraction process is readjusted, and the weight matrix is ​​updated. , or rescale the decision variables , to adapt to the new operating environment.

9. The AI-based data intelligent optimization method according to claim 6, characterized in that: The performance indicators include actual CPU utilization, actual memory occupancy, and performance prediction indicators.

Images