Big data driven intelligent cloud service scheduling and deployment system

Abstract

The invention discloses a cloud service scheduling system and method combining big data and an AI technology. The system comprises a data acquisition and preprocessing module which is responsible for collecting and preprocessing various data of a cloud service platform; the big data analysis module is used for analyzing data by using data mining and machine learning technologies and analyzing a user resource use mode by using a clustering algorithm; the AI prediction model module is used for constructing a prediction model based on big data analysis, training by utilizing a TensorFlow or PyTorch framework and adopting algorithms such as stochastic gradient descent and the like, and predicting a user request quantity and a resource demand; the cloud service scheduling decision module is used for generating a scheduling scheme by applying intelligent algorithms such as a genetic algorithm and the like in combination with the prediction result and the resource state, and calculating a scheduling priority score through a formula; and the scheduling execution and feedback module is used for executing the scheme through an API interface, collecting performance index data and feeding back an optimization model and the scheme. The cloud service resource utilization rate can be improved, the service quality is improved, the system adaptability is enhanced, and the operation cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of big data driven intelligent cloud service scheduling and deployment system, and in particular to a big data driven intelligent cloud service scheduling and deployment system. Background Art

[0002] With the rapid development of cloud computing technology, cloud services are being used more and more widely. In a cloud service environment, how to efficiently schedule resources to meet the diverse needs of users while improving resource utilization and reducing costs has become an urgent problem to be solved. Traditional cloud service scheduling methods are often based on simple rules or experience and are difficult to adapt to the complex and changing cloud service environment. Big data technology can collect, store and analyze massive amounts of cloud service operation data, while AI technology has powerful learning and prediction capabilities. Combining the two and applying them to cloud service scheduling has great potential.

[0003] This case is proposed to solve or improve the shortcomings or deficiencies of the prior art. Summary of the invention

[0004] The present invention is achieved by adopting the following technical solutions:

[0005] Data collection and preprocessing module: collects data such as the usage of various resources in the cloud service platform, user request information, service response time, etc., and cleans, denoises and normalizes these data to provide high-quality data for subsequent data analysis and model training.

[0006] Big data analysis module: Use data mining, machine learning and other technologies to conduct in-depth analysis of pre-processed data to explore potential patterns and associations in the data. For example, analyze the resource usage patterns of different time periods and different types of users, as well as the relationship between resource usage and service quality.

[0007] AI prediction model module: Based on the results of big data analysis, AI prediction models are built, such as time series prediction models and neural network models, to predict user request volumes and resource requirements in the future. The accuracy of predictions is improved through continuous training and optimization.

[0008] Cloud service scheduling decision module: Based on the prediction results of the AI prediction model and the resource status of the current cloud service platform, the optimal cloud service scheduling plan is generated using intelligent algorithms (such as genetic algorithms, simulated annealing algorithms, etc.). The scheduling plan includes decisions such as resource allocation and task deployment.

[0009] Scheduling Execution and Feedback Module: Send the generated scheduling plan to the cloud service platform for execution and monitor the execution process in real time. Collect feedback data during the execution process, such as actual resource usage, service quality indicators, etc., and feed this data back to the data collection and preprocessing module for further optimizing the model and scheduling plan.

[0010] The advantages and positive effects of the present invention are:

[0011] 1. Improve resource utilization rate: Through accurate prediction and intelligent scheduling decisions, resources can be reasonably allocated according to actual needs, avoiding waste and over-allocation of resources.

[0012] 2. Enhance service quality: Ensure that user requests can be responded to in a timely and efficient manner, improving user satisfaction.

[0013] 3. Strengthen system adaptability: It can quickly adapt to the dynamic changes in the cloud service environment and adjust the scheduling strategy in a timely manner.

[0014] 4. Reduce operating costs: Optimize resource allocation, reduce unnecessary resource investment, and lower the operating costs of cloud service providers. Specific Embodiment

[0015] A big data-driven intelligent cloud service scheduling and deployment system described in the present invention

[0016] Big data analysis: Rely on rich data analysis libraries in Python. For example, Pandas with its powerful data processing and analysis capabilities, and NumPy providing efficient numerical computing support, conduct in-depth analysis on the data after preprocessing such as cleaning and denoising. During this process, clustering algorithms such as K-Means are selected to accurately classify users. By carefully analyzing the usage patterns of different types of users in terms of computing resources, storage resources, and network resources, potential laws are mined to provide key feature data for the subsequent training of the AI prediction model.

[0017] AI prediction model training: Build a complex neural network prediction model based on deep learning frameworks such as TensorFlow or PyTorch. Use the historical data accumulated over a long period in the past as the training set, and this data covers user behavior, resource usage, and various business scenario information. During the training process, optimization algorithms such as stochastic gradient descent are used to repeatedly adjust parameters such as weights and biases in the model. At the same time, means such as cross-validation are used to avoid overfitting, and the prediction accuracy of the model for future resource requirements and user behavior is continuously improved through multiple rounds of iterative training.

[0018] Cloud service scheduling decision: The output results of the AI prediction model optimized through training are accurately input into a carefully designed scheduling decision algorithm. This algorithm comprehensively considers the real-time availability of resources, including the idle status of servers in each data center, the remaining capacity of storage devices, and the remaining network bandwidth, as well as the pre-set user priorities. For example, for an emergency service request from a high-priority user, the system quickly invokes the resource allocation strategy and preferentially allocates excellent computing resources and sufficient network bandwidth for it to ensure timely response and efficient execution of tasks.

[0019] Cloud service scheduling decision: The output results of the AI prediction model optimized through training are accurately input into a carefully designed scheduling decision algorithm. This algorithm comprehensively considers the real-time availability of resources, including the idle status of servers in each data center, the remaining capacity of storage devices, and the remaining network bandwidth, as well as the pre-set user priorities. The specific decision condition formula is as follows:

[0020] PriorityScore = α × UserPriority + β × ResourceAvailability

[0021] Among them, PriorityScore

[0022] is the final scheduling priority score, and α and β are weight coefficients, which are adjusted according to actual business needs to balance the importance of user priorities and resource availability.

[0023] UserPriority represents the pre-set priority of the user, with a value range of 1 - 10, and the larger the value, the higher the priority. ResourceAvailability represents the real-time availability of resources, which is calculated by the following formula:

[0024] ResourceAvailability = γ × ServerAvailability + δ × StorageAvailability + ∈ × NetworkBandwidthAvailability;

[0025] γ, δ, ∈

[0026] are also weight coefficients, representing the weights of server, storage device, and network bandwidth availability respectively. ServerAvailability: calculated by the proportion of the number of idle servers to the total number of servers, StorageAvailability: calculated by the proportion of the remaining capacity of the storage device to the total capacity,

[0027] NetworkBandwidthAvailability: calculated by the proportion of the remaining network bandwidth to the total bandwidth.

[0028] For example, for an emergency service request from a high-priority user, the system quickly invokes the resource allocation policy and preferentially allocates excellent computing resources and sufficient network bandwidth for it to ensure timely response and efficient execution of the task. Scheduling execution and feedback: By means of the standardized API interface provided by the cloud service platform, the generated scheduling plan is accurately sent to each resource node, and each node executes the task in an orderly manner according to the instructions. During the execution process, a series of performance index data are collected in real time. For example, the resource utilization rate reflects the usage degree of resources such as the server CPU and memory, and the task completion time reflects the efficiency of business processing. These data are timely fed back to the data acquisition module through a specific data transmission channel, providing real and effective data support for further optimizing the structure and parameters of the AI prediction model and adjusting the cloud service scheduling policy in the future, forming a virtuous cycle and continuously improving the overall performance of the cloud service scheduling system.

[0029] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the embodiments described in the specific implementation manners. Any other implementation manners obtained by those skilled in the art according to the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A cloud service scheduling system combining big data and AI technologies, characterized in that, Including: A data collection and preprocessing module, which is used to collect data such as the usage of various resources, user request information, and service response time in the cloud service platform, and perform cleaning, denoising, and normalization processing.

2. The cloud service scheduling system according to claim 1, wherein The big data analysis module uses data mining and machine learning techniques to deeply analyze the preprocessed data, including classifying users using clustering algorithms and analyzing the resource usage patterns of different types of users.

3. The cloud service scheduling system according to claim 2, wherein The AI prediction model module constructs an AI prediction model based on the big data analysis results, uses the TensorFlow or PyTorch framework, uses historical data as the training set, and uses optimization algorithms such as stochastic gradient descent to train and optimize the model, which is used to predict the user request volume and resource requirements in the future for a period of time.

4. The cloud service scheduling system according to claim 3, wherein The cloud service scheduling decision module generates an optimal cloud service scheduling plan according to the prediction results of the AI prediction model and combines the resource status of the current cloud service platform. The intelligent algorithms include genetic algorithms, simulated annealing algorithms, etc. The scheduling decision algorithm comprehensively considers the real-time availability of resources and user priorities, and calculates the scheduling priority score through a specific formula.

5. The cloud service scheduling system according to claim 4, wherein The formula for calculating the scheduling priority score is: PriorityScore = α × UserPriority + β × ResourceAvailability where α and β are weight coefficients adjusted according to actual business requirements. UserPriority represents the priority preset by the user. ResourceAvailability represents the real-time availability of resources.

6. The cloud service scheduling system according to claim 5, wherein, The real-time availability of the resources is calculated through the formula ResourceAvailability = γ × ServerAvailability + δ × StorageAvailab ility + ∈ × NetworkBandwidthAvailability where: γ, δ, ∈ are weight coefficients representing the availability of the server, storage device, and network bandwidth respectively. ServerAvailability, StorageAvailability, NetworkBandwidthAvailability are calculated through the proportion of the number of idle servers, the proportion of the remaining capacity of the storage device, and the proportion of the network bandwidth margin respectively.

7. The cloud service scheduling system according to claim 1, wherein, The scheduling execution and feedback module sends the scheduling plan to each resource node for execution through the API interface of the cloud service platform, monitors the execution process in real time and collects performance index data such as resource utilization rate and task completion time, and feeds it back to the data collection and preprocessing module for optimizing the model and scheduling plan.

8. A cloud service scheduling method combining big data and AI technologies, characterized in that, Including the following steps: Collect cloud service platform data and perform preprocessing; Use data mining and machine learning techniques for big data analysis; Construct an AI prediction model and train and optimize it; Generate a scheduling plan using intelligent algorithms according to the prediction results and resource status; Execute the scheduling plan through the API interface and collect feedback data for optimization.