AI-based cloud data intelligent analysis and service management system and method

Abstract

The invention relates to the technical field of data analysis, and discloses an AI-based cloud data intelligent analysis and service management system and method. The system comprises a cloud data priority division module, a resource demand prediction module, a cloud data scheduling scheme generation module and a service management dynamic adjustment module. The method comprises the steps of firstly collecting multi-source cloud data, performing data preprocessing, and then performing data priority division; secondly, introducing a hierarchical asynchronous processing architecture to construct a hierarchical neural network prediction model, and outputting a predicted value according to a data priority; establishing a resource scheduling objective function according to the predicted value, optimizing cloud data scheduling parameters in a resource scheduling process by using a multi-strategy fusion grey wolf algorithm, and generating a cloud data scheduling scheme; and finally, calculating a service management health degree, and establishing a health degree response mechanism closed-loop dynamic adjustment cloud data scheduling scheme. By analyzing and processing the cloud data, the purposes of intelligent analysis and service management are achieved, and the method is accurate and objective.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and specifically to an AI-based cloud data intelligent analysis and service management system and method. Background Art

[0002] With the widespread adoption of cloud computing and the explosive growth of IoT devices, global data volumes are expanding exponentially. Cloud platforms have become the storage and processing hub for core enterprise data. According to forecasts, global data volumes will exceed 175ZB by 2025, with over 50% of this data requiring cloud-based analysis. Against this backdrop, intelligent cloud data analysis and service management have become core requirements for enterprise digital transformation. By integrating AI technologies such as machine learning and natural language processing, cloud data analysis and service management enable real-time insights into massive amounts of heterogeneous data, predict trends, and enable automated service orchestration.

[0003] Traditional cloud data intelligent analysis and service management methods mainly rely on a layered architecture and rule-driven mechanism. That is, they use batch processing processes, combined with predefined data models (such as star and snowflake models) for offline analysis, and then trigger manual intervention based on static threshold alarms. They use pre-set fixed priority strategies to generate standardized reports, which are then manually interpreted to formulate operation and maintenance or business strategies.

[0004] Traditional cloud data intelligent analysis and service management methods are difficult to cover dynamic scenarios when conducting cloud data analysis, and cannot adapt to dynamically changing business loads and complex data patterns. At the same time, the batch processing mode leads to high analysis latency, cannot support business decisions that respond in seconds, and has inefficient resource utilization. Summary of the Invention

[0005] In response to the problems in the related technology, the present invention provides an AI-based cloud data intelligent analysis and service management method to overcome the technical problems existing in the existing related technology.

[0006] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] The present invention is an AI-based cloud data intelligent analysis and service management system, which includes the following steps: a cloud data prioritization module, a resource demand prediction module, a cloud data scheduling solution generation module, and a service management dynamic adjustment module;

[0008] The cloud data prioritization module is used to collect multi-source cloud data and perform data preprocessing, and then set a hierarchical transmission strategy to prioritize the data to obtain a hierarchical set of cloud data;

[0009] The resource demand prediction module is used to construct a hierarchical neural network prediction model based on a dual-branch neural network, and output a resource demand prediction value according to different data priorities in the cloud data hierarchical set;

[0010] The cloud data scheduling solution generation module is used to establish a resource scheduling objective function based on the resource demand forecast value, use the multi-strategy fusion gray wolf algorithm to optimize the cloud data scheduling parameters during the resource scheduling process, and generate a cloud data scheduling solution;

[0011] The service management dynamic adjustment module is used to calculate the service management health according to the cloud data scheduling plan, establish a health response mechanism, and adjust the cloud data scheduling plan through closed-loop feedback.

[0012] Preferably, the collecting multi-source cloud data and performing data preprocessing includes:

[0013] Collect multi-source cloud data and cloud data time series to obtain the initial cloud data set;

[0014] Set the sliding window, align the timestamps, calculate the standard deviation coefficient in the sliding window, mark and delete abnormal data, and obtain the processed cloud data set;

[0015] Dividing missing data in the processed cloud data set into long-term data segments and short-term data segments;

[0016] For the short-time data segment, calculating the interpolation coefficient of the missing data in the short-time data segment to obtain a short-time filling value, and filling the missing data in the short-time data segment;

[0017] For the long-term data segment, the interpolation coefficient of the missing data in the center of the long-term data segment is calculated to obtain the filling value of the missing data in the center, and the missing data in the center is filled until the long-term data segment is divided into short-term data segments, and the short-term filling value is calculated and filled again to obtain the processed cloud data set.

[0018] Preferably, the step of resetting a hierarchical transmission strategy to prioritize data includes:

[0019] Calculate the priority score of the processed cloud data set, set a hierarchical transmission strategy, and set a priority threshold. When the priority score is greater than the priority threshold, the corresponding cloud data is recorded as real-time processed cloud data, otherwise the corresponding cloud data is recorded as batch processed cloud data to form a hierarchical cloud data set.

[0020] Preferably, the constructing of a hierarchical neural network prediction model based on a dual-branch neural network includes:

[0021] A hierarchical neural network prediction model is established based on the hierarchical set of cloud data, and a layered asynchronous processing architecture is introduced. The hierarchical neural network prediction model includes a high-level data stream processing model and a low-level data stream processing model. The high-level data stream processing model is recorded as a first neural network branch, which receives and processes cloud data in real time; the low-level data stream processing model is recorded as a second neural network branch, which receives and processes cloud data in batches;

[0022] The first neural network branch adopts multi-scale LSTM to output the first branch result, and the second neural network branch adopts a feature cross pyramid structure to output the second branch result; the first branch result and the second branch result are passed through a gated fusion unit and a residual connection layer, and then through a decoder to output the model prediction value to construct a hierarchical neural network prediction model.

[0023] Preferably, outputting the resource demand prediction value according to different data priorities in the cloud data hierarchical set includes:

[0024] Obtain historical multi-source cloud data, perform data preprocessing and data prioritization, and obtain a hierarchical cloud data sample set. This is then input into a hierarchical neural network prediction model for training to obtain the final hierarchical neural network prediction model.

[0025] The cloud data hierarchical set is input into a final hierarchical neural network prediction model to output a resource demand prediction value.

[0026] Preferably, establishing a resource scheduling objective function according to the resource demand forecast value includes:

[0027] The resource demand prediction value is used as the cloud data scheduling parameter. During the resource scheduling process, the cloud data calculates the resource cost and cloud data migration overhead based on the resource demand prediction value, and establishes the resource scheduling objective function.

[0028] Preferably, the use of the multi-strategy fusion grey wolf algorithm to optimize cloud data scheduling parameters during resource scheduling includes:

[0029] The resource scheduling objective function is regarded as the fitness function, and the process of finding the optimal fitness function value is regarded as the process of finding the optimal resource scheduling objective function value. The dynamic reverse learning strategy and the adaptive dimension learning strategy are integrated to improve the gray wolf algorithm, and a multi-strategy fusion gray wolf algorithm is obtained;

[0030] Assume that there is a gray wolf population in the search space. Each gray wolf in the gray wolf population represents a set of cloud data scheduling parameters. The process of updating the positions of gray wolves is the process of optimizing cloud data scheduling parameters. Initialize the gray wolf population, use the dynamic reverse learning strategy to solve the reverse gray wolf population initial position, and generate a new gray wolf population.

[0031] The positions of individual gray wolves are updated, and an adaptive dimensional learning strategy is introduced to replace the position update strategy based on the three leading gray wolves. When the current number of iterations reaches the maximum number of iterations, the iteration is stopped to obtain the final gray wolf population. The gray wolf individual corresponding to the best fitness function value is found in the final gray wolf population to obtain the optimized cloud data scheduling parameters.

[0032] Use the optimized cloud data scheduling parameters to schedule cloud data and generate a cloud data scheduling plan.

[0033] Preferably, the calculating of the service management health according to the cloud data scheduling scheme includes:

[0034] According to the cloud data scheduling solution, the proportion of abnormal nodes, the sum of memory usage value marks and the sum of bandwidth allocation value marks are calculated to obtain the service management health.

[0035] Preferably, the establishment of a health response mechanism and closed-loop feedback adjustment of the cloud data scheduling solution includes:

[0036] Set a first health threshold, a second health threshold, and a third health threshold. When the service management health is less than the first health threshold and greater than or equal to the second health threshold, memory expansion is adopted; when the service management health is less than the second health threshold and greater than or equal to the third health threshold, cloud data is migrated to the backup node; when the service management health is less than the third health threshold, bandwidth is reallocated.

[0037] The present invention also discloses an AI-based cloud data intelligent analysis and service management method, which is characterized by specifically including:

[0038] S1. Collect multi-source cloud data to obtain an initial cloud data set, perform data preprocessing on the initial cloud data set, and then set a hierarchical transmission strategy to prioritize the data to obtain a hierarchical cloud data set;

[0039] S2. Constructing a hierarchical neural network prediction model based on a dual-branch neural network and introducing a layered asynchronous processing architecture to output resource demand prediction values ​​based on different data priorities in the hierarchical set of cloud data;

[0040] S3. Establish a resource scheduling objective function based on the predicted resource demand value, use the multi-strategy fusion gray wolf algorithm to optimize the cloud data scheduling parameters during the resource scheduling process, obtain the optimized cloud data scheduling parameters, and generate a cloud data scheduling plan;

[0041] S4. Calculate the service management health according to the cloud data scheduling plan, establish a health response mechanism using a hierarchical management strategy, and adjust the cloud data scheduling plan through closed-loop feedback to achieve cloud data service management.

[0042] The present invention has the following beneficial effects:

[0043] 1. This invention collects multi-source cloud data and performs data preprocessing, sets a hierarchical transmission strategy to prioritize data, and uses sliding window standardization processing to effectively filter transient interference and avoid data interference. Compared with traditional methods, the data is divided into short-term missing and long-term missing, and segmented filling is used to avoid the distortion of simple linear interpolation. At the same time, priority is divided to ensure priority processing of critical and urgent data, which greatly reduces data processing delay.

[0044] 2. This invention constructs a hierarchical neural network prediction model based on a dual-branch neural network, and introduces a layered asynchronous processing architecture to output resource demand prediction values. By constructing dual branches to analyze different data priorities respectively, layered asynchronous processing is applied to real-time data processing and batch data processing to ensure millisecond-level data response, improve the idle resource utilization rate of batch branches, reduce the conflict rate due to dual-branch staggered processing, and control cloud data prediction errors in complex cloud environments.

[0045] 3. This invention uses a multi-strategy fusion gray wolf algorithm to optimize cloud data scheduling parameters during the resource scheduling process and generate a cloud data scheduling plan. This algorithm can prevent the algorithm from converging prematurely and improve its global exploration capability, overcome the traditional algorithm's dependence on the leader, and enhance the diversity and exploration capability of the population. At the same time, after optimization, it saves hardware costs while meeting performance requirements, and dynamically allocates resources to avoid congestion during peak periods, which can significantly reduce the operating costs of the cloud platform while ensuring service quality.

[0046] 4. This invention quantitatively evaluates data service management through a hierarchical management strategy, establishes a health response mechanism using the hierarchical management strategy, and adjusts the cloud data scheduling plan through closed-loop feedback. It can adapt to dynamically changing business loads and complex data environments, and reduce resource waste in the cloud data analysis and service process.

[0047] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, they can also obtain drawings based on these drawings without paying any creative work.

[0049] Figure 1 The present invention provides a flow chart of an AI-based cloud data intelligent analysis and service management system;

[0050] Figure 2The present invention provides a flow chart of an AI-based cloud data intelligent analysis and service management method;

[0051] Figure 3 A schematic diagram of the process of optimizing cloud data scheduling parameters in the resource scheduling process using the multi-strategy fusion grey wolf algorithm provided by the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] Traditional cloud data intelligent analysis and service management methods are difficult to cover dynamic scenarios when conducting cloud data analysis, and cannot adapt to dynamically changing business loads and complex data patterns. At the same time, the batch processing mode leads to high analysis latency, cannot support business decisions that respond in seconds, and has inefficient resource utilization.

[0054] In order to solve the above technical problems, Figure 1 As shown, an embodiment of the present invention provides an AI-based cloud data intelligent analysis and service management system, which specifically includes: a cloud data priority division module, a resource demand prediction module, a cloud data scheduling scheme generation module and a service management dynamic adjustment module; the cloud data priority division module is used to collect multi-source cloud data and perform data preprocessing, and then set a hierarchical transmission strategy to perform data priority division to obtain a cloud data hierarchical set; the resource demand prediction module is used to construct a hierarchical neural network prediction model based on a dual-branch neural network, and output a resource demand prediction value according to different data priorities in the cloud data hierarchical set; the cloud data scheduling scheme generation module is used to establish a resource scheduling objective function according to the resource demand prediction value, and use a multi-strategy fusion gray wolf algorithm to optimize the cloud data scheduling parameters in the resource scheduling process to generate a cloud data scheduling scheme; the service management dynamic adjustment module is used to calculate the service management health according to the cloud data scheduling scheme, establish a health response mechanism, and adjust the cloud data scheduling scheme through closed-loop feedback.

[0055] A specific embodiment is that a large e-commerce cloud platform faces enormous resource scheduling pressure during a promotion period. The cloud platform has servers 1, 2, and 3, which collect cloud data once a minute for 15 minutes. The real-time data stream includes user order requests (tens of thousands of times per second) and real-time inventory queries, while the batch data stream includes historical transaction logs (TB level) and user behavior analysis data (updated every hour), providing a data intelligent analysis basis for the embodiments of the present invention.

[0056] In the specific implementation process of the above embodiment, first, multi-source cloud data such as resource utilization, API request queue, network bandwidth data flow, etc. are collected and pre-processed, and a hierarchical transmission strategy is set to prioritize data to obtain a hierarchical set of cloud data; this method can effectively filter transient interference through sliding window standardization processing to avoid data interference. Compared with traditional methods, the data is divided into short-term missing and long-term missing, and segmented filling is used to avoid the distortion of simple linear interpolation. At the same time, priority scores are calculated to classify the data to ensure priority processing of critical and urgent data, which greatly reduces data processing delay; secondly, a hierarchical neural network prediction model is constructed based on a dual-branch neural network, and a hierarchical asynchronous processing architecture is introduced to output resource demand prediction values ​​according to different data priorities in the cloud data hierarchical set; this method analyzes different data priorities separately by constructing dual branches, and applies hierarchical asynchronous processing to real-time data processing and batch data processing to ensure millisecond-level data response, improve the idle resource utilization rate of batch branches, reduce the conflict rate of dual-branch staggered processing, and control the cloud data in a complex cloud environment. According to the prediction error, resource utilization efficiency is improved; then, the resource scheduling objective function is established according to the predicted value of resource demand, and the multi-strategy fusion gray wolf algorithm is used to optimize the cloud data scheduling parameters in the resource scheduling process to generate a cloud data scheduling plan; the algorithm introduces a dynamic reverse learning strategy and an adaptive dimensional learning strategy, which can prevent the algorithm from convergence prematurely and improve its global exploration ability, overcome the traditional algorithm's dependence on the leader, and enhance the diversity and exploration ability of the population. At the same time, it optimizes memory usage, node allocation and bandwidth requirements, saves hardware costs while meeting performance requirements, and dynamically allocates to avoid peak congestion, which can significantly reduce the operating costs of the cloud platform while ensuring service quality; finally, the service management health is calculated according to the cloud data scheduling plan, and a hierarchical management strategy is used to establish a health response mechanism. The cloud data scheduling plan is adjusted through closed-loop feedback to realize cloud data service management; this method quantitatively evaluates data service management through a hierarchical management strategy, which can adapt to dynamically changing business loads and complex data environments, and reduce resource waste in the cloud data analysis and service process.

[0057] Furthermore, in order to better introduce the technical solutions of the embodiments of the present invention, Figure 2 As shown, combined with an AI-based cloud data intelligent analysis and service management method, the AI-based cloud data intelligent analysis and service management system is described in detail, including the following contents:

[0058] S1. Collect multi-source cloud data to obtain an initial cloud data set, perform data preprocessing on the initial cloud data set, and then set a hierarchical transmission strategy to prioritize the data to obtain a hierarchical cloud data set;

[0059] Said S1 comprises the following steps:

[0060] S11. Collect multi-source cloud data through distributed probes, where the multi-source cloud data includes resource utilization, API request queues, network bandwidth data flows, etc., and sequentially record the time points corresponding to the collection of the multi-source cloud data to obtain a cloud data time series. Combine the multi-source cloud data and the cloud data time series to obtain an initial cloud data set.

[0061] S12: searching for abnormal data in the initial cloud data set, deleting the abnormal data, dividing the initial cloud data set into short-term data segments and long-term data segments, and filling in missing data to obtain a processed cloud data set. The specific steps are as follows:

[0062] S121. Set a sliding window, place the sliding window in the initial cloud data set, align the cloud data time series according to the sliding window, and obtain an alignment timestamp. ,in represents the kth time point, Represents the size of the sliding window, and L represents the number of sliding window moves. At this time, the mean and standard deviation of the cloud data in the sliding window are calculated. The mean value of each cloud data in the sliding window is subtracted from the cloud data, and then divided by the cloud data standard deviation to obtain the standard deviation coefficient. A standard deviation threshold is set. When the absolute value of the standard deviation coefficient is greater than the standard deviation threshold, the cloud data corresponding to the standard deviation coefficient is marked as abnormal data and deleted. The sliding window is moved sequentially to traverse the initial cloud data set, all abnormal data are found and deleted, and the processed cloud data set is obtained.

[0063] S122. For the missing data in the processed cloud data set, set a continuous missing threshold. When the number of consecutive missing data in the processed cloud data set is greater than the continuous missing threshold, record the corresponding consecutive missing data as a long-term data segment; otherwise, record the corresponding consecutive missing data as a short-term data segment.

[0064] For the short-time data segment, record the time point of collecting the short-time data segment and calculate the interpolation coefficient of the i-th missing data in the short-time data segment ,in represents the time point adjacent to the missing data, represents the i-th missing data in the short-time data segment; then obtaining the adjacent data of the missing data, calculating the mean of the product of the adjacent data of the missing data and the interpolation coefficient of the missing data to obtain a short-time filling value, and using the short-time filling value to fill the missing data in the short-time data segment;

[0065] For the long-term data segment, selecting missing data at the center of the long-term data segment, calculating the interpolation coefficient of the missing data at the center of the long-term data segment, obtaining a filling value for the missing data at the center, and filling the missing data at the center; then selecting missing data at the long-term data segment for filling, until the long-term data segment is divided into short-term data segments, calculating the short-term filling value and filling again, to obtain a processed cloud data set;

[0066] S13. Obtain a time period for processing the processed cloud data set, and normalize the time period to obtain data complexity; set a maximum allowable delay for the cloud data in the processed cloud data set, and quantize the maximum allowable delay to the range of [0, 1] to obtain data criticality; assign weights to the data complexity and data criticality respectively, and calculate a priority score ,in and represents the weight, Indicates the data complexity, Indicates data criticality; sets a hierarchical transmission strategy and a priority threshold. When the priority score is greater than the priority threshold, the corresponding cloud data is recorded as real-time processing cloud data; otherwise, the corresponding cloud data is recorded as batch processing cloud data to form a hierarchical cloud data set;

[0067] In this embodiment, multi-source cloud data is collected and pre-processed, and a hierarchical transmission strategy is set to prioritize data to obtain a hierarchical set of cloud data. This method can effectively filter transient interference and avoid data interference through sliding window normalization processing. Compared with traditional methods, the data is divided into short-term missing and long-term missing, and segmented filling is used to avoid the distortion of simple linear interpolation. At the same time, priority scores are calculated to classify the data, ensuring that key and urgent data are processed first, which greatly reduces data processing delays. Specifically, for example, distributed probes collect data from 09:00 to 09:15 to generate an initial cloud data set (including a timestamp sequence). The sliding window is set: the window size is 3 minutes, the moving step is 1, the mean and standard deviation in the window (09:00-09:02) are calculated, and the standard deviation coefficient of 0.71 is greater than the standard deviation threshold of 0.5, and the mark Abnormal data; short-term missing data (09:02-09:03) is interpolated, resulting in difference coefficients of 0.99 and 0.99, respectively, and a calculated filling value of 2.29; long-term missing data (09:05-09:14) is prioritized for calculation at the center point 09:10, and the remaining data is segmented as short-term missing data, and interpolation and filling are continued according to the short-term missing data. It takes 50ms to process the API request queue, and the CPU allows a delay of 10ms. When the system load is high, the complexity weight can be increased, and when the business is urgent, the criticality weight can be increased. For example, setting the complexity weight to 0.6 and the criticality weight to 0.4, the calculated priority scores are 0.86 and 0.24. The midpoint of the normalized interval [0,1] is selected as the priority threshold, which effectively distinguishes data urgency. The result is 0.86 for real-time processing and 0.24 for batch processing.

[0068] S2. Constructing a hierarchical neural network prediction model based on a dual-branch neural network and introducing a layered asynchronous processing architecture to output resource demand prediction values ​​based on different data priorities in the hierarchical set of cloud data;

[0069] The S2 comprises the following steps:

[0070] S21. Establishing a hierarchical neural network prediction model based on the hierarchical set of cloud data and introducing a layered asynchronous processing architecture. The hierarchical neural network prediction model includes a high-level data stream processing model and a low-level data stream processing model. The high-level data stream processing model is recorded as a first neural network branch, which receives and processes cloud data in real time; the low-level data stream processing model is recorded as a second neural network branch, which receives and processes cloud data in batches.

[0071] The first neural network branch uses a multi-scale LSTM, which includes a bottom-level LSTM and a middle-level LSTM, and adds an attention mechanism to the top level to output the first branch result. The second neural network branch uses a feature cross-pyramid structure, in which the first layer is a fully connected layer, the second layer is a graph convolutional network, and the third layer adds a self-attention mechanism to output the second branch result. The first branch result and the second branch result are passed through a gated fusion unit and a residual connection layer, and then through a decoder to output the model prediction value, thereby constructing a hierarchical neural network prediction model.

[0072] S22. Obtain historical multi-source cloud data, perform data preprocessing and data prioritization, obtain a hierarchical cloud data sample set, train a hierarchical neural network prediction model, and obtain a final hierarchical neural network prediction model. The specific steps are as follows:

[0073] S221, dividing the cloud data hierarchical sample set into a sample training set and a sample test set, inputting the sample training set into a hierarchical neural network prediction model for training, setting a maximum number of training rounds, and stopping training when the number of training rounds of the hierarchical neural network prediction model reaches the maximum number of training rounds, thereby obtaining a trained hierarchical neural network prediction model;

[0074] S222: Input the sample test set into the trained hierarchical neural network prediction model, output the sample prediction value, set the accuracy threshold, and when the sample prediction value accuracy is greater than the accuracy threshold, obtain the final hierarchical neural network prediction model; otherwise, adjust the neural network weights until the sample prediction value accuracy is greater than the accuracy threshold;

[0075] S23, inputting the real-time processed cloud data in the cloud data hierarchical set into the first neural network branch of the final hierarchical neural network prediction model, inputting the batch processed cloud data in the cloud data hierarchical set into the second neural network branch of the final hierarchical neural network prediction model, and outputting resource demand prediction values, wherein the resource demand prediction values ​​include a memory usage prediction value, a node ID code prediction value, and a bandwidth allocation prediction value;

[0076] In this embodiment, a hierarchical neural network prediction model is constructed based on a dual-branch neural network, and a hierarchical asynchronous processing architecture is introduced to output a resource demand prediction value according to different data priorities in the hierarchical set of cloud data; this method analyzes different data priorities separately by constructing dual branches, and applies hierarchical asynchronous processing to real-time data processing and batch data processing, ensuring millisecond-level data response, improving the idle resource utilization rate of batch branches, reducing the conflict rate of dual-branch staggered processing, and controlling the cloud data prediction error in a complex cloud environment, so that resource utilization efficiency is improved; specifically, for example, high-level data stream processing (first neural network branch): processing real-time data (API queue, CPU utilization), bottom-level LSTM: processing 1-second API request fluctuations, middle-level LSTM: analyzing 10-second CPU utilization trends, and top-level attention machine The system focuses on burst request peaks. The underlying data stream processing (second neural network branch) processes batch data (bandwidth, disk IO). The first fully connected layer compresses the bandwidth data dimension. The second layer of graph convolution captures the topological relationship between server nodes. The third layer of self-attention identifies key nodes. The gated fusion unit is dynamically weighted, the residual connection retains the original features, and the decoder outputs three-way predictions. Cloud monitoring data from the past three months is obtained, with a maximum training round number of 500 and a batch size of 512 items per batch for the real-time branch and 2048 items per batch for the batch branch. Requirements include memory prediction error, node ID accuracy, and bandwidth prediction error. Attention weights are adjusted during training, and resource demand predictions are output. The node ID prediction accuracy is 92%, enabling precise pre-allocation of hotspot resources. The bandwidth prediction error is 7.3%, effectively avoiding over-provisioning.

[0077] S3. Establish a resource scheduling objective function based on the predicted resource demand value, use the multi-strategy fusion gray wolf algorithm to optimize the cloud data scheduling parameters during the resource scheduling process, obtain the optimized cloud data scheduling parameters, and generate a cloud data scheduling plan;

[0078] The S3 includes the following steps:

[0079] S31, using the resource demand prediction value as the cloud data scheduling parameter, during the resource scheduling process, the cloud data calculates the resource cost and cloud data migration overhead based on the resource demand prediction value, and sets the maximum resource capacity and maximum bandwidth to establish the resource scheduling objective function ,in and represents the weight coefficient, represents resource cost, represents the cloud data migration overhead;

[0080] S32, taking the resource scheduling objective function as the fitness function, and considering the process of finding the optimal fitness function value as the process of finding the optimal resource scheduling objective function value, the dynamic reverse learning strategy and the adaptive dimension learning strategy are integrated to improve the gray wolf algorithm, and a multi-strategy fusion gray wolf algorithm is obtained. The multi-strategy fusion gray wolf algorithm is used to optimize the cloud data scheduling parameters to obtain the optimized cloud data scheduling parameters. The specific steps are as follows, Figure 3 The multi-strategy fusion gray wolf algorithm is used to optimize cloud data scheduling parameters during resource scheduling as shown below:

[0081] S321. Assume that there is a gray wolf population in the search space, the number of gray wolf population is p, the dimension of gray wolf population is q, each gray wolf individual in the gray wolf population represents a set of cloud data scheduling parameters, and the process of updating the position of the gray wolf individual is the process of optimizing the cloud data scheduling parameters; initialize the gray wolf population to obtain the initial position of the gray wolf population ,set up and Represents a random number between the interval [0,1], and the upper bound of the search space is , the search space lower bound is , using the dynamic reverse learning strategy to solve the initial position of the reverse gray wolf population , merge the initial position of the gray wolf population and the initial position of the reverse gray wolf population, calculate the fitness function value, and select the top p gray wolf individuals in the fitness function value ranking to generate a new gray wolf population;

[0082] S322, set the current number of iterations to t, the maximum number of iterations to T, select the j-th gray wolf individual from the new gray wolf population, and the position of the j-th gray wolf individual at the t-th iteration is , the prey position at the tth iteration is ;set up and represents a random number between the interval [0,1], then the first coefficient vector , the second coefficient vector , calculate the distance from the gray wolf's individual position to the prey's position At this time, the position of the jth gray wolf at the t+1th iteration is calculated ; Now find the gray wolf individual corresponding to the current best fitness function value and obtain the current cloud data scheduling parameters;

[0083] S323, introduce the adaptive dimensional learning strategy to replace the position update strategy based on the three leading gray wolves, calculate the difference between the position of the j-th gray wolf individual at the t-th iteration and the position of the j-th gray wolf individual at the t+1-th iteration, and obtain the iteration distance; select a random gray wolf individual from the new gray wolf population, and when the difference between the position of the random gray wolf individual and the position of the j-th gray wolf individual at the t-th iteration is less than the iteration distance, the position of the j-th gray wolf individual is again updated. Update; set represents a random number between the interval [0,1], and selects the cth gray wolf individual, the dth gray wolf individual, and the eth gray wolf individual from the new gray wolf population. The positions of the gray wolf individuals are 、 and ;when At this time, the position of the j-th gray wolf at the t+1th iteration is ,when At this time, the position of the j-th gray wolf at the t+1th iteration is Complete all stages of the t+1th iteration and enter the next iteration until the current number of iterations reaches the maximum number of iterations, then stop iterating to obtain the final gray wolf population, and find the gray wolf individual corresponding to the best fitness function value in the final gray wolf population to obtain the optimized cloud data scheduling parameters;

[0084] S33. Schedule cloud data using the optimized cloud data scheduling parameters, where the optimized cloud data scheduling parameters include an optimized memory usage value, an optimized node ID code, and an optimized bandwidth allocation value, to generate a cloud data scheduling solution.

[0085] In this embodiment, a resource scheduling objective function is established, and a multi-strategy fusion gray wolf algorithm is used to optimize the cloud data scheduling parameters in the resource scheduling process to generate a cloud data scheduling plan; the algorithm introduces a dynamic reverse learning strategy and an adaptive dimensional learning strategy, which can prevent the algorithm from converging prematurely and improve its global exploration ability, overcome the traditional algorithm's dependence on the alpha wolf, and improve the diversity and exploration ability of the population. At the same time, it optimizes memory usage, node allocation, and bandwidth requirements, saves hardware costs while meeting performance requirements, and dynamically allocates to avoid peak congestion, which can significantly reduce the operating costs of the cloud platform while ensuring service quality; specifically, for example, it is assumed that the cloud platform has 3 servers with 2000 nodes. According to the prediction value, the core transaction system requires 1.2TB, and 32 GPU nodes are preferentially allocated to the real-time pricing system. The payment gateway requires 40Gbps bandwidth and the user behavior analysis requires 20Gbps bandwidth. The constraint condition is that the maximum memory of a single node is 64GB. The maximum available bandwidth is 100Gbps. The gray wolf population size is set to 50, the dimension is 3 (memory, node, bandwidth), the maximum number of iterations is 200, and the upper and lower bounds of the search space are (64, 200, 40) and (8, 1, 1), respectively. The initial position is randomized and the reverse position is generated. The population is merged to select the top 50 optimal solutions. The adaptive position update and dimension learning strategies are used to continuously update the individual positions of the gray wolves. The optimized parameters are: memory allocation (56GB×22 nodes = 1.23TB for the trading system and 56GB×14 nodes = 784GB for the recommendation system), node allocation (32 nodes for the real-time pricing system), and bandwidth allocation (38Gbp for the payment gateway and 18Gbps for user behavior analysis). The entire process improves GPU node memory, reduces the total number of nodes while meeting demand, saves 32 node resources, and achieves a resource scheduling error rate of <3%, verifying the superiority of the multi-strategy fusion gray wolf algorithm in ultra-large-scale cloud environments.

[0086] S4. Calculate the service management health according to the cloud data scheduling plan, establish a health response mechanism using a hierarchical management strategy, and adjust the cloud data scheduling plan through closed-loop feedback to achieve cloud data service management;

[0087] The S4 comprises the following steps:

[0088] S41. Obtain the number of abnormal nodes based on the optimized node ID codes in the cloud data scheduling scheme, calculate the abnormal node ratio, set the maximum memory usage value and the maximum bandwidth allocation value based on the optimized memory usage value and the optimized bandwidth allocation value in the cloud data scheduling scheme, and detect the memory usage and bandwidth allocation;

[0089] When the optimized memory usage value is greater than the maximum memory usage value, the corresponding optimized memory usage value is marked as 1, otherwise it is marked as 0, and the sum of the memory usage value marks is calculated; when the optimized bandwidth allocation value is greater than the maximum bandwidth allocation value, the corresponding optimized bandwidth allocation value is marked as 1, otherwise it is marked as 0, and the sum of the bandwidth allocation value marks is calculated; the service management health is calculated ,in Indicates the total memory usage value. Indicates the number of memory usage detections, Indicates the total value of bandwidth allocation. represents the number of bandwidth allocation detections, g represents the proportion of abnormal nodes, 、 and represents the health weight;

[0090] S42. Three health thresholds are set, namely, the first health threshold, the second health threshold, and the third health threshold. When the service management health is less than the first health threshold and greater than or equal to the second health threshold, memory expansion is performed. When the service management health is less than the second health threshold and greater than or equal to the third health threshold, cloud data is migrated to the backup node. When the service management health is less than the third health threshold, bandwidth is reallocated to complete cloud data service management.

[0091] In this embodiment, the service management health is calculated according to the cloud data scheduling plan, a health response mechanism is established using a hierarchical management strategy, and a closed-loop feedback is used to adjust the cloud data scheduling plan to achieve cloud data service management. This method quantitatively evaluates data service management through a hierarchical management strategy, which can adapt to dynamically changing business loads and complex data environments, and reduce resource waste in cloud data analysis and service processes. Specifically, for example, abnormal node detection, the total number of abnormal nodes is: 5 (transactions) + 3 (risk control) = 8, memory usage detection, the number of over-limit nodes is 12; according to historical data, in the electricity During peak hours of Shangyun data scheduling, the health weights of memory usage, bandwidth allocation, and abnormal nodes are set to 0.4, 0.3, and 0.3, respectively, and automatically calibrated based on historical data. The calculated service management health is 0.91. The threshold is automatically adjusted according to the quarterly settlement period to allow higher loads during peak hours, and a graded health response mechanism is set: threshold ≥ 0.90 indicates a healthy state and no action is required. For thresholds [0.85, 0.9), memory expansion is taken; for thresholds [0.80, 0.85), data migration is taken; for thresholds < 0.8, bandwidth reallocation is taken.

[0092] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0093] The preferred embodiments of the invention disclosed above are intended only to help illustrate the invention. These preferred embodiments do not exhaust all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. An AI-based cloud data intelligent analysis and service management system, characterized by: include: The cloud data prioritization module is used to collect multi-source cloud data and perform data preprocessing, and then set a hierarchical transmission strategy to prioritize the data to obtain a hierarchical set of cloud data; The resource demand prediction module is used to build a hierarchical neural network prediction model based on a dual-branch neural network, and output resource demand prediction values ​​according to the different data priorities in the cloud data hierarchical set; The cloud data scheduling solution generation module is used to establish a resource scheduling objective function based on the predicted resource demand value, use the multi-strategy fusion gray wolf algorithm to optimize the cloud data scheduling parameters during the resource scheduling process, and generate a cloud data scheduling solution; The service management dynamic adjustment module is used to calculate the service management health according to the cloud data scheduling plan, establish a health response mechanism, and adjust the cloud data scheduling plan through closed-loop feedback.

2. The AI-based cloud data intelligent analysis and service management system according to claim 1, characterized in that: The collecting of multi-source cloud data and performing data preprocessing includes: Collect multi-source cloud data and cloud data time series to obtain the initial cloud data set; Set the sliding window, align the timestamps, calculate the standard deviation coefficient in the sliding window, mark and delete abnormal data, and obtain the processed cloud data set; Dividing missing data in the processed cloud data set into long-term data segments and short-term data segments; For the short-time data segment, calculating the interpolation coefficient of the missing data in the short-time data segment to obtain a short-time filling value, and filling the missing data in the short-time data segment; For the long-term data segment, the interpolation coefficient of the missing data in the center of the long-term data segment is calculated to obtain the filling value of the missing data in the center, and the missing data in the center is filled until the long-term data segment is divided into short-term data segments, and the short-term filling value is calculated and filled again to obtain the processed cloud data set.

3. The AI-based cloud data intelligent analysis and service management system according to claim 2 is characterized in that: The step of setting a hierarchical transmission strategy to prioritize data includes: Calculate the priority score of the processed cloud data set, set a hierarchical transmission strategy, and set a priority threshold. When the priority score is greater than the priority threshold, the corresponding cloud data is recorded as real-time processed cloud data, otherwise the corresponding cloud data is recorded as batch processed cloud data to form a hierarchical cloud data set.

4. The AI-based cloud data intelligent analysis and service management system according to claim 3 is characterized in that: The hierarchical neural network prediction model constructed based on the dual-branch neural network includes: Establishing a hierarchical neural network prediction model based on the hierarchical set of cloud data, the hierarchical neural network prediction model includes a high-level data stream processing model and a low-level data stream processing model, wherein the high-level data stream processing model is recorded as a first neural network branch, receiving and processing cloud data in real time; and the low-level data stream processing model is recorded as a second neural network branch, receiving and processing cloud data in batches; The first neural network branch adopts multi-scale LSTM to output the first branch result, and the second neural network branch adopts a feature cross pyramid structure to output the second branch result; the first branch result and the second branch result are passed through a gated fusion unit and a residual connection layer, and then through a decoder to output the model prediction value to construct a hierarchical neural network prediction model.

5. The AI-based cloud data intelligent analysis and service management system according to claim 4 is characterized in that: Outputting resource demand prediction values ​​according to different data priorities in the cloud data classification set includes: Obtain historical multi-source cloud data, perform data preprocessing and data prioritization, and obtain a hierarchical cloud data sample set. This is then input into a hierarchical neural network prediction model for training to obtain the final hierarchical neural network prediction model. The cloud data hierarchical set is input into a final hierarchical neural network prediction model to output a resource demand prediction value.

6. The AI-based cloud data intelligent analysis and service management system according to claim 5, characterized in that: The establishment of a resource scheduling objective function according to the resource demand prediction value includes: The resource demand prediction value is used as the cloud data scheduling parameter. During the resource scheduling process, the cloud data calculates the resource cost and cloud data migration overhead based on the resource demand prediction value, and establishes the resource scheduling objective function.

7. The AI-based cloud data intelligent analysis and service management system according to claim 6, characterized in that: The method of optimizing cloud data scheduling parameters using the multi-strategy fusion grey wolf algorithm in the resource scheduling process includes: The resource scheduling objective function is regarded as the fitness function, and the process of finding the optimal fitness function value is regarded as the process of finding the optimal resource scheduling objective function value. The dynamic reverse learning strategy and the adaptive dimension learning strategy are integrated to improve the gray wolf algorithm, and a multi-strategy fusion gray wolf algorithm is obtained; Assume that there is a gray wolf population in the search space. Each gray wolf in the gray wolf population represents a set of cloud data scheduling parameters. The process of updating the positions of gray wolves is the process of optimizing cloud data scheduling parameters. Initialize the gray wolf population, use the dynamic reverse learning strategy to solve the reverse gray wolf population initial position, and generate a new gray wolf population. The positions of individual gray wolves are updated, and an adaptive dimensional learning strategy is introduced to replace the position update strategy based on the three leading gray wolves. When the current number of iterations reaches the maximum number of iterations, the iteration is stopped to obtain the final gray wolf population. The gray wolf individual corresponding to the best fitness function value is found in the final gray wolf population to obtain the optimized cloud data scheduling parameters. Use the optimized cloud data scheduling parameters to schedule cloud data and generate a cloud data scheduling plan.

8. The AI-based cloud data intelligent analysis and service management system according to claim 7, characterized in that: Calculating the service management health according to the cloud data scheduling solution includes: According to the cloud data scheduling solution, the proportion of abnormal nodes, the sum of memory usage value marks and the sum of bandwidth allocation value marks are calculated to obtain the service management health.

9. The AI-based cloud data intelligent analysis and service management system according to claim 8, characterized in that: The establishment of a health response mechanism and closed-loop feedback adjustment of the cloud data scheduling solution includes: Set a first health threshold, a second health threshold, and a third health threshold. When the service management health is less than the first health threshold and greater than or equal to the second health threshold, memory expansion is adopted; when the service management health is less than the second health threshold and greater than or equal to the third health threshold, cloud data is migrated to the backup node; when the service management health is less than the third health threshold, bandwidth is reallocated.

10. The AI-based cloud data intelligent analysis and service management method according to any one of claims 1 to 9, characterized in that: Specifically include: S1. Collect multi-source cloud data to obtain an initial cloud data set, perform data preprocessing on the initial cloud data set, and then set a hierarchical transmission strategy to prioritize the data to obtain a hierarchical cloud data set; S2. Constructing a hierarchical neural network prediction model based on a dual-branch neural network and introducing a layered asynchronous processing architecture to output resource demand prediction values ​​based on different data priorities in the hierarchical set of cloud data; S3. Establish a resource scheduling objective function based on the predicted resource demand value, use the multi-strategy fusion gray wolf algorithm to optimize the cloud data scheduling parameters during the resource scheduling process, obtain the optimized cloud data scheduling parameters, and generate a cloud data scheduling plan; S4. Calculate the service management health according to the cloud data scheduling plan, establish a health response mechanism using a hierarchical management strategy, and adjust the cloud data scheduling plan through closed-loop feedback to achieve cloud data service management.

Images