Cross-cloud resource scheduling method and system based on deep learning

Abstract

The invention discloses a cross-cloud resource scheduling system and method based on deep learning, and the system comprises a resource monitoring service module which is used for monitoring resource information applied to a plurality of cross-cloud platforms, and the output end of the resource monitoring service module is connected to a data preprocessing module; the data preprocessing module carries out preprocessing operation on monitored data and then sends the data to the CNN prediction model, the CNN prediction model processes time series data through multiple convolutional layers based on the preprocessed data and captures resource use trend and periodic changes, and the model outputs resource demand prediction in a period of time in the future. And the output end of the resource scheduling module is connected with the strategy optimization module, the strategy optimization module outputs an optimal resource scheduling strategy to the strategy implementation module through processing scheduling, and the strategy implementation module executes the resource scheduling strategy. According to the scheme, real-time dynamic resource scheduling can be realized to meet the requirement of cross-cloud platform resource scheduling.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and machine learning, and in particular to a cross-cloud resource scheduling method and system based on deep learning, which uses artificial intelligence and machine learning technologies to continuously and automatically optimize the scheduling of various resources in application systems, such as computing resources (CPU, memory), storage resources, network bandwidth, etc. Background Art

[0002] In today's technological environment, cloud computing has become an important infrastructure supporting modern enterprise operations. As enterprises' demands for high availability, flexibility, and fault tolerance continue to increase, more and more application systems are deployed across clouds. This cross-cloud deployment of applications can not only improve service stability, but also optimize the speed of access to application data around the world. However, this deployment model also brings complex resource scheduling challenges, which are often difficult to cope with by existing resource scheduling technologies.

[0003] First, traditional resource scheduling strategies are mostly designed for a single cloud environment and are difficult to apply directly to a cross-cloud environment. In cross-cloud deployments, different cloud service providers' platforms may use different infrastructure and technical standards, which requires resource management and scheduling to consider the compatibility issues between platforms. This heterogeneity makes it impossible to optimize resource allocation, which affects application performance and overall resource utilization efficiency.

[0004] Second, cross-cloud resource scheduling needs to respond to environmental changes in real time, including load fluctuations, network status, and the availability of various services. Existing static scheduling methods cannot effectively adapt to these dynamic factors and often appear clumsy and inefficient in actual operations. For example, the resource requirements of an application distributed across multiple cloud platforms may change depending on factors such as user access location and time, but traditional methods often cannot adjust resource configuration in real time to adapt to such changes.

[0005] In addition, cross-cloud deployment also involves data security and compliance issues, which also need to be considered in resource scheduling strategies. For example, some data may not be able to be transmitted across clouds due to regulatory restrictions, which requires the scheduling system to be able to intelligently identify and reasonably allocate data storage and processing locations to comply with relevant laws and regulations.

[0006] Therefore, for the resource management of cross-cloud application deployment, there is an urgent need for a more intelligent, dynamic and adaptive resource scheduling strategy, and existing technologies cannot meet the relevant requirements. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a cross-cloud resource scheduling method and system based on deep learning, which dynamically optimizes resource allocation according to application requirements and network conditions by real-time monitoring of resource usage.

[0008] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a cross-cloud resource scheduling system based on deep learning, including a resource monitoring service module, a data preprocessing module, a CNN prediction model, a policy optimization module, and a policy implementation module, wherein the resource monitoring service module is used to monitor resource information applied in multiple cross-cloud platforms, and its output end is connected to the data preprocessing module. The data preprocessing module performs preprocessing operations on the monitored data and then sends it to the CNN prediction model. The CNN prediction model processes time series data based on the preprocessed data through multiple convolutional layers to capture trends and periodic changes in resource usage. The model outputs resource demand forecasts for a period of time in the future, and its output end is connected to the policy optimization module. The policy optimization module outputs the optimal resource scheduling strategy to the policy implementation module through processing scheduling, and the policy implementation module executes based on the resource scheduling strategy.

[0009] The scheduling system also includes a performance feedback module, which is used to feedback information after the execution of the scheduling strategy and send the feedback information to the CNN prediction model and the strategy optimization module to optimize the parameters of the CNN prediction model and the strategy optimization module according to the feedback information.

[0010] The strategy optimization module uses a genetic algorithm to optimize the scheduling strategy by defining a set of initial scheduling strategies and generating new strategies through operations such as selection, crossover and mutation, and continuously evaluating and selecting the best performing strategy.

[0011] The scheduling system also includes a cloud management platform, in which resource monitoring services and resource management services are developed and set up. Based on the resource monitoring service, resource information across multiple cloud platforms is monitored, and based on the resource management service, the scheduling strategy to be executed is scheduled and controlled.

[0012] A cross-cloud resource scheduling method based on deep learning includes the following steps:

[0013] Step 1: Monitor and collect real-time and historical operational data of resources from across cloud environments;

[0014] Step 2: Preprocess the collected data;

[0015] Step 3: Use the trained CNN model to analyze the collected data and predict short-term and long-term changes in resource demand;

[0016] Step 4: Input the prediction results into the genetic algorithm optimization module to find the optimal resource scheduling strategy by simulating the evolution process;

[0017] Step 5: Implement the optimized scheduling strategy based on the resource management service in the cloud management platform, and dynamically adjust resource allocation based on the optimized scheduling strategy.

[0018] The method also includes step 6: after the scheduling is executed, feedback information of the execution results is collected, and based on the feedback information, the parameters in the CNN model and the genetic algorithm module are optimized and adjusted to achieve dynamic closed-loop adjustment of the CNN model and the genetic algorithm.

[0019] The advantages of the present invention are that it can monitor resource usage in real time and dynamically optimize resource allocation according to application requirements and network conditions. By adopting advanced machine learning and artificial intelligence technologies, a scheduling algorithm that can predict changes in application resource requirements and respond quickly can be trained, thereby greatly improving resource utilization efficiency, reducing application costs, and ensuring application performance and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0021] Figure 1 This is a schematic diagram of the architecture of the resource scheduling system of the present invention;

[0022] Figure 2 The figure is a flow chart of the resource scheduling method of the present invention.

[0023] Figure 3 The monitoring module of the present invention is implemented as a schematic diagram

[0024] Figure 4 Schematic diagram of resource management service module implementation in the present invention DETAILED DESCRIPTION

[0025] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0026] This invention uses the cross-cloud resource monitoring indicator data of applications as the source of deep learning data, the multi-cloud management platform as the resource scheduling strategy implementation tool, and the deep learning engine as the core. It continuously optimizes the resource scheduling algorithm parameters through the self-feedback of application performance indicators to achieve the optimal resource scheduling effect of application-level resources in a cross-cloud environment. The main components and innovative features of this solution are introduced below:

[0027] 1. Application resource monitoring indicator collection: The collection of resource monitoring indicators in the application dimension is the first step in cross-cloud resource scheduling of applications. Traditional applications are deployed in a single cloud environment, and the resource monitoring indicator data in the application dimension depends on the monitoring data of a single cloud platform. In the cross-cloud deployment mode, the monitoring indicator collection in the application dimension needs to extract monitoring data from multi-cloud platform resources and convert it into the same resource monitoring data model. At the same time, it is also necessary to monitor and process application-level performance indicators such as throughput, TPS, QPS, etc. to form a unified application-dimensional monitoring indicator data model.

[0028] In this solution, by monitoring resources and application performance at the application level, we can understand the resource consumption of applications in more detail. To this end, we have designed a set of lightweight, plug-in, and extensible monitoring modules that can not only collect key resource indicators required for application runtime in real time, such as CPU utilization, memory usage, network bandwidth, etc., but also uniformly collect overall application performance indicators. These indicators can not only reflect the current resource status of the application, but also be used as data support for subsequent resource scheduling decisions.

[0029] In this solution, the core implementation architecture of the monitoring module is as follows: Figure 3 As shown in the figure, its core function is to provide monitoring data for the data preprocessing module. The overall implementation plan is as follows: It is divided into four layers from top to bottom, namely the extraction and display layer, the configuration management layer, the collection and cleaning layer, and the collection plug-in layer. The collection plug-in layer is responsible for raw data collection; the collection and cleaning layer is responsible for data cleaning and format conversion based on the conversion rules for the raw monitoring data collected by the plug-in layer; the configuration management layer is used to configure and manage unified monitoring indicators and the conversion relationship with the original resource monitoring indicators; the extraction and display layer provides a data interface for the cloud management platform to display the resource performance data view, and provides a data push interface for the data preprocessing module. By configuring the conversion relationship between the unified monitoring indicators and the specific cloud resources and application monitoring indicators, the automatic conversion processing of raw data and predicted data is realized.

[0030] 2. Resource scheduling parameter setting: The setting of scheduling parameters is a very critical step. Scheduling parameters include various parameters of resource allocation, such as the number of virtual machines, CPU allocation ratio, network bandwidth limit, etc. Traditional resource scheduling methods usually set these parameters manually or rely on static configuration files. However, this method is often difficult to adapt to dynamically changing environments and needs. Therefore, this solution proposes an automated scheduling parameter management mechanism that can automatically adjust the scheduling parameter settings based on real-time monitoring data and predicted demand conditions. In this way, dynamic optimization of resource configuration can be achieved to meet the real-time needs of different applications.

[0031] 3. Scheduling algorithm setting and self-learning feedback: In this solution, the setting of the scheduling algorithm is another key link. Traditional resource scheduling algorithms are often based on static rules or simple heuristic methods, lacking flexibility and adaptability. The method we proposed uses a scheduling algorithm based on deep learning, which can achieve self-optimization and adaptation through learning and analysis of a large amount of historical data. Specifically, we designed a scheduling algorithm based on reinforcement learning, which continuously adjusts scheduling decisions through interactive learning with the environment to achieve optimal resource utilization efficiency and application performance.

[0032] In addition, this solution also introduces a self-learning feedback mechanism, which can timely evaluate the effect of the scheduling algorithm and make adjustments and improvements based on the feedback results. This self-learning ability enables our scheduling system to continuously adapt to changing environments and needs, thereby achieving more intelligent and efficient resource scheduling.

[0033] 4. Optimize the scheduling algorithm variable settings: This solution also includes a method for optimizing the scheduling algorithm variable settings. In traditional resource scheduling methods, it is often necessary to manually set the parameters and variables of various scheduling algorithms, which is not only time-consuming and labor-intensive, but also prone to errors and uncertainties. To this end, we propose an automated parameter optimization method that can automatically adjust the algorithm parameter settings based on real-time monitoring data and the performance indicators of the scheduling algorithm to achieve the optimal scheduling effect. This method can not only improve the stability and accuracy of the scheduling algorithm, but also reduce the need for manual intervention, thereby reducing the management and maintenance costs of the system.

[0034] Through the above innovations, our invention realizes an intelligent and adaptive resource scheduling method, which can effectively improve the resource utilization efficiency and application performance of the system, and provide a new solution for resource management in a cross-cloud deployment environment.

[0035] The following is based on the Figure 1 , 2 , 3, 4 are introduced as follows to the embodiments of the present application:

[0036] In a complex cloud computing environment, the efficiency and accuracy of resource scheduling are crucial. This technical solution proposes an innovative scheduling algorithm variable optimization method that uses an advanced convolutional neural network (CNN) structure and combines genetic algorithms to optimize resource scheduling decisions. This technical solution covers three key links: data preprocessing, application performance model prediction, and optimization scheduling strategy, and cooperates with self-feedback and parameter optimization technology to ensure the efficiency and accuracy of the entire system.

[0037] Convolutional neural networks are powerful tools commonly used in deep learning for image and sequence data processing. In the present invention, CNN is used to process and analyze time series data, such as historical resource usage and application performance indicators. The power of CNN lies in its excellent feature extraction, which can automatically identify and learn important patterns in the data, which is particularly critical for predicting resource requirements. At the same time, genetic algorithms are used to optimize resource scheduling decisions. Genetic algorithms are heuristic search algorithms that imitate the mechanism of natural selection and find the optimal solution through iterative evolution. In the scenario of resource scheduling, we use genetic algorithms to select and adjust parameters in the scheduling strategy to achieve optimal resource utilization efficiency and application performance.

[0038] like Figure 1 As shown, the model architecture of the present invention includes the following main parts:

[0039] (1) Cloud management platform: On this platform, resource monitoring services and resource management services are developed and implemented. The resource monitoring service is used to implement data monitoring, and the resource management service is used to implement resource scheduling and control between cloud platforms based on scheduling policies.

[0040] Monitoring services (such as Figure 3 (As shown in the figure) The monitoring indicators of applications and resources are maintained online (unified abstraction and management) through the monitoring indicator management module, and the conversion relationship between the unified application and resource monitoring indicators and the original monitoring data in the cloud platform or application monitoring system is configured through the monitoring indicator mapping module, providing the unified monitoring engine with monitoring indicator information and monitoring data conversion rules, thereby realizing flexible expansion and mapping configuration of collected indicators.

[0041] The monitoring module manages the specific collection cycle of the collection engine and issues collection indicators through the collection task module. After the start-up conditions of the task module are triggered, the unified collection engine calls the unified interface layer and, based on the indicators and conversion rules provided by the collection indicator module and mapping module, calls the interface of the specific cloud platform or application monitoring system, and organizes and writes the acquired data through the data cleaning module. At the same time, the data extraction module is used to provide standard monitoring data for the upper-level data preprocessing module.

[0042] Resource management services (such as Figure 4 As shown in the figure, it includes the view display layer, configuration management layer, unified engine layer and plug-in implementation layer from top to bottom. The configuration management layer manages the address and authentication information of the cloud platform through the cloud platform management module, and manages the overall resources in a unified manner through the resource management module and sends instructions to the unified resource management engine. The unified resource management engine selects a specific cloud platform management plug-in through the unified resource management abstract interface to call the resource management interface of the target cloud platform to implement the scheduling and control operations of specific resources.

[0043] (2) Data preprocessing module: For the cross-cloud resource scheduling system based on CNN, we need to perform a series of preprocessing operations on the data to ensure that the data meets the input requirements of the model and improve the accuracy and generalization ability of the model. The data preprocessing module of the present invention includes the following aspects:

[0044] ① Data collection: Obtain historical and real-time monitoring indicators of applications and resources from the cloud management platform and application monitoring system.

[0045] ②Data cleaning: process missing values, delete duplicate data, and unify data formats.

[0046] ③Data denoising: Use smoothing filtering or anomaly detection to remove noise and abnormal data.

[0047] ④Data normalization: Normalize or standardize numerical data to ensure that the features have the same magnitude.

[0048] ⑤Feature engineering: perform feature selection and construction, and encode category data.

[0049] ⑥Data enhancement: Generate more samples to improve the generalization ability of the model.

[0050] (3) CNN prediction model: The preprocessed data is fed into the CNN model. The model processes time series data through multiple convolutional layers to capture trends and periodic changes in resource usage. The model outputs a forecast of resource demand in the future, providing a scientific basis for scheduling decisions.

[0051] The structure of the CNN model

[0052] Input layer: The input layer receives the cloud host resource performance data and overall monitoring data of the application system. Example input: resource performance data (such as CPU, memory, disk usage, etc.), monitoring data (such as system response time, number of requests, etc.). Example specification: The input size is (N,M,K,P), where N represents the time step, M represents different application indicators, K represents different cloud hosts, and K represents multiple performance indicators of each host.

[0053] Convolutional Layer: Extracts correlation features between different time steps and different hosts. Parameters: kernel size, stride, padding, etc. Example: kernel size 3x3, stride 1, padding 'same'.

[0054] Activation function layer: Common activation function: ReLU (Rectified Linear Unit). Example: Use the ReLU activation function.

[0055] Pooling Layer: Reduce the size of the feature map by pooling while retaining the most important features. Common methods: Max Pooling or Average Pooling. Example: Pooling size 2x2, stride 2.

[0056] Fully Connected Layer: Flatten the extracted features and connect them to the fully connected layer for classification or regression. Example: The number of neurons in the fully connected layer is 1024.

[0057] Output Layer: Provides the final scheduling prediction results, which can be a classification problem (such as selecting the best host) or a regression problem (such as predicting resource consumption). Example: Use the softmax activation function for classification output.

[0058] Building a CNN model

[0059] The following is a sample code for building a model using the Keras library:

[0060] from tensorflow.keras.models import Sequential

[0061] from tensorflow.keras.layers import Conv2D,MaxPooling2D,Flatten,Dense,Activation

[0062] model=Sequential()

[0063] #Input layer

[0064] model.add(Input(shape=(N,M,K)))

[0065] #The first convolutional layer and pooling layer

[0066] model.add(Conv2D(64,(3,3),padding='same'))

[0067] model.add(Activation('relu'))

[0068] model.add(MaxPooling2D(pool_size=(2,2)))

[0069] #The second convolutional layer and pooling layer

[0070] model.add(Conv2D(128,(3,3),padding='same'))

[0071] model.add(Activation('relu'))

[0072] model.add(MaxPooling2D(pool_size=(2,2)))

[0073] # Flatten the layers

[0074] model.add(Flatten())

[0075] #Fully connected layer

[0076] model.add(Dense(1024))

[0077] model.add(Activation('relu'))

[0078] #Output layer, assuming it is a classification problem

[0079] model.add(Dense(num_classes))

[0080] model.add(Activation('softmax'))

[0081] Training process:

[0082] 1. Define the loss function: Use a loss function that is appropriate for the problem, such as cross entropy loss (classification) or mean squared error (regression). Example: categorical_crossentropy.

[0083] 2. Select an optimizer: For example, Adam, which has good convergence. Example: Adam optimizer, learning rate 0.001.

[0084] 3. Set hyperparameters: batch size, number of iterations, etc. Example: batch size 32, training 100 epochs.

[0085] 4. Model training: Use training data for training and gradually optimize model parameters.

[0086] Here’s how CNN works:

[0087] 1. Feature Extraction: The convolution layer automatically extracts the temporal and spatial features of cloud host resource performance data and monitoring data through convolution operations. The activation function layer adds nonlinear changes to enhance the model's expressiveness.

[0088] 2. Feature Compression (Feature Pooling): The pooling layer reduces the data dimension through downsampling operations while retaining important features and improving computational efficiency.

[0089] 3. Feature Flattening and Fusion: Flatten the high-dimensional feature matrix after pooling and further process it through the fully connected layer.

[0090] 4. Classification or Regression: Finally, the output layer provides specific scheduling decisions, such as selecting the optimal host or predicting resource consumption.

[0091] Based on the above introduction, this system design adopts the following design ideas:

[0092] 1. Model structure:

[0093] Input layer: Receives an input matrix of size (30, 5, 10, 5), which represents the five performance metrics on 10 cloud hosts under the five application metrics for the last 30 time steps.

[0094] First convolutional layer: uses 64 3x3 convolution kernels with a stride of 1 and padding of 'same' to extract local features. The activation function is ReLU.

[0095] First pooling layer: 2x2 max pooling with a stride of 2 to reduce the size of the feature map.

[0096] The second convolutional layer uses 128 3x3 convolution kernels with a stride of 1 and padding of 'same' to further extract features. The activation function is ReLU.

[0097] Second pooling layer: 2x2 max pooling with a stride of 2.

[0098] Flattening layer: Flattens the feature map into a one-dimensional vector to facilitate processing by the fully connected layer.

[0099] Fully connected layer: contains 1024 neurons, and the activation function is ReLU.

[0100] Output layer: softmax activation function, outputting the classification result matrix (30, 10, 5), which corresponds to the prediction of the five performance indicators of 10 cloud hosts in the next 30 time steps.

[0101] 2. Model training:

[0102] Use historical cloud host resource performance data and system monitoring data for training, data preprocessing and data enhancement.

[0103] The loss function is categorical_crossentropy, the optimizer is Adam, and the learning rate is 0.001.

[0104] The training process was performed for 100 epochs with a batch size of 32 and cross-validation was used to select the best model.

[0105] 3. Input and Output:

[0106] Input: A matrix of size (30,5,10,5) representing the five performance metrics on 10 cloud hosts for the five application metrics for the last 30 time steps.

[0107] Output: A 3D vector (30, 10, 5) representing the prediction of the five performance indicators of the application on 10 cloud hosts in the next 30 time steps.

[0108] (3) Strategy Optimization Module: The prediction results are then fed into the strategy optimization module, which uses a genetic algorithm to optimize the scheduling strategy. It defines a set of initial scheduling strategies (populations) and generates new strategies (individuals) through operations such as selection, crossover, and mutation, and continuously evaluates and selects the best performing strategy.

[0109] The strategy optimization module compares the configurations and five monitoring indicators of the 10 hosts under the current application with the predicted data of the five monitoring indicators of the 10 hosts in the next 30 steps output by the CNN module, and combines the current resource capacity and price factors of each cloud platform to form the specific configurations of the CPU, memory, disk and network bandwidth of the 10 hosts that need to be met to achieve the resource performance indicators required for the predicted application indicators.

[0110] In the present invention, the design and implementation of the genetic algorithm includes the following key steps: individual representation, initial population generation, fitness function design, selection, crossover, mutation and termination condition.

[0111] 1. Individual representation: represented by a vector of length 40 (10 hosts × 4 configuration parameters).

[0112] 2. Initial population generation: Randomly generate a set of configurations as the initial population, and each individual is a possible configuration set.

[0113] 3. Fitness function design: The fitness function is defined as the error (such as mean square error) between the predicted performance data and the target performance data. The smaller the error, the higher the fitness. In addition, the configuration cost or resource usage restriction is added as a penalty term.

[0114] 4. Selection: Use tournament selection strategy to select individuals with higher fitness for reproduction.

[0115] 5. Crossover (recombination): Perform a crossover operation on the selected individuals to generate new offspring. The crossover operation can be single-point crossover, multi-point crossover, or uniform crossover, depending on how to mix the configuration of the parent generations.

[0116] 6. Mutation: You can randomly change the value of a configuration parameter, such as fine-tuning the CPU or memory within a certain range

[0117] 7. Termination conditions: Set the termination conditions of the genetic algorithm, such as reaching the maximum number of iterations, or the fitness of individuals in the population has not significantly improved in several generations. It can be adjusted according to the actual application scenario to achieve better performance or faster convergence speed.

[0118] The genetic algorithm finally outputs the result, and the optimal configuration combination of the 10 hosts under the current application is obtained, and output to the policy implementation module for resource configuration changes.

[0119] (5) Policy implementation module: This module is used to execute the scheduling policy output by the policy optimization module to achieve dynamic scheduling and control of resources. The policy implementation module calls the resource management service of the cloud management platform (such as Figure 4 As shown in the figure, the resource target configuration change parameters are passed, and the resource management service calls the unified resource management engine through the resource management module. The management engine obtains the specific platform management plug-in through the platform information in the resource configuration parameters, and implements the configuration change request issuance operation through the management plug-in.

[0120] (6) Performance feedback module: It is used to monitor the parameter data of the system in real time after the resource scheduling strategy is executed. The parameter data is sent to the CNN and strategy optimization modules for optimizing and adjusting their internal parameters. The performance feedback module obtains the performance indicators of the application system such as the number of concurrency, throughput, and resource utilization rate (CPU, memory, network, disk) under the application system through the monitoring module, and feeds them back to the monitoring indicator parameters in the CNN model. The CNN module will adjust the specific number of convolution parameters and step size.

[0121] like Figure 2 As shown, the cross-platform resource scheduling method based on deep learning of the present application includes the following steps:

[0122] S1, system initialization and configuration step, in which the application system, cloud platform authentication information, unified monitoring indicators and mapping relationship with specific resource / application monitoring indicators are completed, and the application resource collection task is started; at the same time, the CNN model service is started.

[0123] S2, Application resource operation monitoring step: In this step, data is collected from various monitoring tools and APIs in the cloud environment, including real-time and historical data, such as the number of concurrent and throughput of the application system, and the CPU, memory, network rate, and disk utilization of the resources under the application system.

[0124] S3. Determine whether the construction of the CNN model is completed. If so, the construction of the CNN model is completed and enter step S4; if not, it is necessary to build the model. First, determine whether the amount of application historical monitoring data has reached the set threshold; if not, continue to monitor the operation of application resources, and the amount of intelligence data has reached the set threshold; then build a data set based on the monitored application resource data; then build a resource demand prediction model, and train the demand resource prediction model through the constructed data set, and then obtain the trained resource prediction model, which is the CNN model, which is used to predict resource demand.

[0125] S4. Data cleaning and preprocessing of application resource operation data monitored in real time: Formatting the collected data, including data cleaning, missing value processing, feature engineering, etc., to prepare data for model learning and prediction.

[0126] S5. Application resource demand prediction: Use the trained CNN model to analyze the collected data and predict short-term and long-term resource demand changes.

[0127] S6. Resource scheduling decision optimization: Genetic algorithm optimization module is used to realize resource scheduling decision optimization. The prediction results are input into the genetic algorithm optimization module to find the optimal resource scheduling strategy by simulating the evolution process.

[0128] S6. Scheduling strategy execution: Apply the optimized scheduling strategy to the actual resource management system and dynamically adjust resource allocation to adapt to predicted demand changes.

[0129] S7, Application Performance Feedback: After scheduling execution, the system will collect feedback on the execution results, including resource utilization efficiency and application performance, etc. This feedback information will be used to further adjust and optimize the CNN model and genetic algorithm, forming a closed loop of continuous learning and adaptation.

[0130] The resource scheduling strategy using the above solution has the following technical features:

[0131] 1. Combination of deep learning and genetic algorithms: Deep learning is used to accurately predict resource demand and combined with genetic algorithms to optimize resource scheduling decisions, which is uncommon in traditional resource management systems.

[0132] 2. Dynamically optimize scheduling strategies: Through real-time data-driven prediction and optimization, resource scheduling becomes more accurate and efficient.

[0133] 3. Adaptive learning mechanism: The system can adjust the prediction model and optimization algorithm based on real-time feedback to continuously improve the scheduling strategy.

[0134] This highly automated and intelligent resource scheduling solution can significantly improve resource utilization in cloud computing environments, reduce costs, and improve the quality and response speed of user services. This innovative technical solution provides an effective solution to complex resource scheduling problems.

[0135] When implementing the above solution, this application requires steps such as environment deployment and resource configuration, including:

[0136] (1) Deployment Environment

[0137] The embodiments of the present invention are applicable to various cloud computing environments, including public clouds, private clouds, and hybrid clouds. The system can be deployed on the infrastructure of a cloud service provider, or run in an enterprise's internal data center or hybrid cloud environment. Key system components can be containerized and managed using container orchestration tools (such as Kubernetes) to achieve highly scalable and flexible deployment.

[0138] (2) System initialization and configuration:

[0139] 1. Deploy deep learning frameworks (such as TensorFlow, PyTorch, etc.) and genetic algorithm libraries (such as DEAP).

[0140] 2. Configure data collection agents to ensure real-time collection of resource usage data and application performance data in the cloud environment.

[0141] 3. Deploy a prediction model training environment, including GPU acceleration devices (such as NVIDIA GPUs), high-performance computing clusters, etc.

[0142] (3) Data flow establishment and testing:

[0143] 1. Ensure data flow is smooth and data collected in real time from the monitoring agent can be accurately transmitted to the data preprocessing module.

[0144] 2. Test the performance of data processing and prediction models, including the accuracy of data processing and the prediction accuracy of the model.

[0145] (4) Model training and optimization:

[0146] 1. Use historical data to train prediction models, including CNN models and genetic algorithm models.

[0147] 2. Use automated hyperparameter optimization techniques to optimize prediction models and genetic algorithm models to improve model accuracy and efficiency.

[0148] (5) Real-time scheduling decision:

[0149] When the model training is completed and reaches a certain accuracy, it is deployed as a real-time service to receive real-time data from the data collection agent and generate scheduling decisions.

[0150] Based on real-time data and prediction results, the system dynamically adjusts resource allocation and scheduling strategies to cope with environmental changes and load fluctuations.

[0151] (6) Performance monitoring and feedback:

[0152] 1. Implement a performance monitoring system to continuously monitor the system’s scheduling effects and resource utilization.

[0153] 2. Collect performance feedback data, including resource utilization, response time, cost and other indicators, and feed it back into the model for continuous optimization of scheduling strategies.

[0154] Through the above implementation steps, we can effectively apply the technical solution of the present invention to various cloud computing environments to achieve efficient and intelligent resource scheduling and management. The resource scheduling optimization solution of the present invention not only has significant technical innovation, but also has strong commercial potential and broad technical prospects. By combining convolutional neural networks and genetic algorithms, our solution provides an unprecedented resource scheduling solution for cross-cloud application deployment, which can greatly improve the efficiency of application resource utilization, reduce operating costs, and enhance the response speed and service quality of the application system. It also has the following commercial advantages:

[0155] 1. Cost-effectiveness: By optimizing resource scheduling, this system can maximize the output efficiency of each computing unit, reduce resource idleness and waste, and thus directly reduce the enterprise's IT expenditure.

[0156] 2. Improve service quality: Intelligent scheduling systems can predict and respond to demand fluctuations, ensure the performance and availability of key applications, and thus improve end-user satisfaction and loyalty.

[0157] 3. Flexibility and scalability: The system design supports containerization and microservice architecture, making it easy to deploy in a variety of cloud environments, including private cloud, public cloud and hybrid cloud, to meet the needs of enterprises of different sizes.

[0158] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. A cross-cloud resource scheduling system based on deep learning, characterized by: It includes a resource monitoring service module, a data preprocessing module, a CNN prediction model, a strategy optimization module, and a strategy implementation module. The resource monitoring service module is used to monitor resource information applied on multiple cross-cloud platforms, and its output end is connected to the data preprocessing module. The data preprocessing module preprocesses the monitored data and then sends it to the CNN prediction model. The CNN prediction model processes time series data based on the preprocessed data through multiple convolutional layers to capture trends and periodic changes in resource usage. The model outputs resource demand forecasts for a period of time in the future. Its output end is connected to the strategy optimization module. The strategy optimization module outputs the optimal resource scheduling strategy to the strategy implementation module through processing scheduling. The strategy implementation module executes based on the resource scheduling strategy.

2. A cross-cloud resource scheduling system based on deep learning as claimed in claim 1, characterized in that: The scheduling system also includes a performance feedback module, which is used to feedback information after the execution of the scheduling strategy and send the feedback information to the CNN prediction model and the strategy optimization module to optimize the parameters of the CNN prediction model and the strategy optimization module according to the feedback information.

3. A cross-cloud resource scheduling system based on deep learning as claimed in claim 1, characterized in that: The strategy optimization module uses a genetic algorithm to optimize the scheduling strategy by defining a set of initial scheduling strategies and generating new strategies through operations such as selection, crossover and mutation, and continuously evaluating and selecting the best performing strategy.

4. A cross-cloud resource scheduling system based on deep learning as described in any one of claims 1 to 3, characterized in that: The scheduling system also includes a cloud management platform, in which resource monitoring services and resource management services are developed and set up. Based on the resource monitoring service, resource information across multiple cloud platforms is monitored, and based on the resource management service, the scheduling strategy to be executed is scheduled and controlled.

5. A cross-cloud resource scheduling method based on deep learning, characterized in that: The steps include: Step 1: Monitor and collect real-time and historical operational data from across cloud environments; Step 2: Preprocess the collected data; Step 3: Use the trained CNN model to analyze the collected data and predict short-term and long-term changes in resource demand; Step 4: Input the prediction results into the genetic algorithm optimization module to find the optimal resource scheduling strategy by simulating the evolution process; Step 5: Implement the optimized scheduling strategy based on the resource management service in the cloud management platform, and dynamically adjust resource allocation based on the optimized scheduling strategy.

6. A cross-cloud resource scheduling method based on deep learning as claimed in claim 5, characterized in that: The method also includes step 6: after the scheduling is executed, feedback information of the execution results is collected, and based on the feedback information, the parameters in the CNN model and the genetic algorithm module are optimized and adjusted to achieve dynamic closed-loop adjustment of the CNN model and the genetic algorithm.

7. A cross-cloud resource scheduling method based on deep learning as claimed in claim 5, characterized in that: The data monitoring module is used to implement data monitoring. The data monitoring module includes an extraction and display layer, a configuration management layer, a collection and cleaning layer, and a collection plug-in layer. The collection plug-in layer is responsible for collecting raw data; the collection and cleaning layer is responsible for data cleaning and format conversion of the raw monitoring data collected by the plug-in layer based on conversion rules; The configuration management layer is used to configure and manage unified monitoring indicators and their conversion relationship with original resource monitoring indicators. The extraction and display layer provides a data interface for the cloud management platform to display resource performance data views, and a data push interface for the data preprocessing module. Data monitoring and processing are achieved by configuring the conversion relationship between unified monitoring indicators and specific cloud resource and application monitoring indicators.

8. The cross-cloud resource scheduling method based on deep learning according to claim 5, characterized in that: The resource management service in the cloud management platform is used to implement the scheduling control of resources between cloud platforms based on scheduling strategies. It includes a view display layer, a configuration management layer, a unified engine layer, and a plug-in implementation layer. The configuration management layer manages the address and authentication information of the cloud platform through the cloud platform management module, and uniformly manages the overall resources through the resource management module and issues instructions to the unified resource management engine. The unified resource management engine uses management instructions and the unified resource management abstract interface to select a specific cloud platform management plug-in to call the resource management interface of the target cloud platform to implement the scheduling and control operations of specific resources.

9. The cross-cloud resource scheduling method based on deep learning according to claim 5, characterized in that: By developing and implementing resource monitoring services on the cloud management platform, data monitoring is achieved through resource monitoring services. The monitoring service maintains the monitoring indicators of applications and resources online through the monitoring indicator management module. The conversion relationship between unified application and resource monitoring indicators and the original monitoring data in the cloud platform or application monitoring system is configured through the monitoring indicator mapping module, providing monitoring indicator information and monitoring data conversion rules for the unified monitoring engine, thereby achieving flexible expansion and mapping configuration of collected indicators.

Images