Decision management method and device for multi-cloud management platform, and storage medium

Abstract

The invention provides a decision management method and device for a multi-cloud management platform and a storage medium, and the method comprises the steps: collecting all real-time data of the multi-cloud management platform, and carrying out the preprocessing of all real-time data; according to the processed real-time data, SARS structure data is constructed; the method comprises the following steps: constructing an Actor-Critic model; according to the SARS structure data, an Actor-Critic model is trained, and a trained Actor-Critic model is obtained; and deploying the trained Actor-Critic model on a multi-cloud management platform, and performing decision management on a plurality of cloud services. The method has the beneficial effects that the resource allocation in the multi-cloud environment can be dynamically adjusted according to the real-time environment state and the performance index, so that more efficient resource utilization and performance optimization are realized, and the like.

Description

Technical Field

[0001] The present application relates to the technical field of a multi-cloud management platform, and in particular to a decision management method, device and storage medium for a multi-cloud management platform. Background Art

[0002] With the development of cloud computing, enterprises have more and more operational data. "Cloud computing" has become not only a must, but also a multiple option. Hybrid cloud and multi-cloud strategies are favored by the market and have gradually become the basic path for enterprises to go to the cloud. In order to solve the difficulties brought by multi-cloud management to enterprises, multi-cloud management platforms have emerged.

[0003] According to the survey, many multi-cloud management platforms face challenges such as the complexity of resource allocation and scheduling, load balancing challenges, and difficulties in automated expansion and reduction. However, due to the dynamic and uncertain nature of these challenges that cannot adapt to the environment, traditional multi-cloud management methods often find it difficult to effectively address these challenges. Summary of the invention

[0004] The embodiment of the present application provides a decision management method, device and storage medium of a multi-cloud management platform to solve the problems existing in the related technologies. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a decision management method for a multi-cloud management platform, including:

[0006] Collect various real-time data of the multi-cloud management platform and pre-process the various real-time data;

[0007] Constructing SARS structure data according to the real-time data after each processing, wherein the SARS structure data includes the current system state, execution action, reward obtained according to the execution action, and new system state transferred to according to the execution action corresponding to the multi-cloud management platform on the real-time data after each processing;

[0008] Constructing an Actor-Critic model, wherein the Actor network in the Actor-Critic model is used to predict action probabilities, and the Critic network in the Actor-Critic model is used to evaluate state values;

[0009] According to the SARS structure data, the Actor-Critic model is trained to obtain a trained Actor-Critic model;

[0010] The trained Actor-Critic model is deployed on the multi-cloud management platform to perform decision-making management on multiple cloud services.

[0011] In one implementation, preprocessing the various items of real-time data includes:

[0012] The missing values, abnormal values ​​and noise data are processed for each item of real-time data.

[0013] In one embodiment, constructing SARS structure data according to various processed real-time data includes:

[0014] Determine the data type of each item of processed real-time data;

[0015] According to the data type of each processed real-time data, the processed real-time data is subjected to state coding, action coding and reward processing;

[0016] According to the state codes, action codes and rewards of the processed real-time data, the processed real-time data are converted into four-tuple data to construct the SARS structure data.

[0017] In one embodiment, the Actor-Critic model is trained according to the SARS structure data to obtain a trained Actor-Critic model including:

[0018] Saving the SARS structure data as a TensorFlow dataset;

[0019] Divide the TensorFlow dataset into a training set, a validation set, and a test set;

[0020] The Actor-Critic model is trained using the training set, and the Actor network and the Critic network are updated according to the prediction result of the Actor network, the evaluation result of the Critic network and the true value of the training set to obtain a trained Actor-Critic model;

[0021] During the training process, the validation set is regularly used to evaluate the performance of the trained Actor-Critic model, and relevant parameters of the trained Actor-Critic model are saved;

[0022] After the training is completed, the test set is used to evaluate the final performance of the trained Actor-Critic model. When the final performance reaches a preset performance, the trained Actor-Critic model is used as the trained Actor-Critic model.

[0023] In one implementation, the Actor-Critic model is trained using the training set, and the Actor network and the Critic network are updated according to the prediction result of the Actor network, the evaluation result of the Critic network, and the true value of the training set, so as to obtain the trained Actor-Critic model, including:

[0024] Determine the input parameters of the Actor-Critic model, wherein the input parameters include the number of iterations T, the state feature dimension N, and the action set A;

[0025] Initializing the state environment of the multi-cloud management platform, and using the initialization state as the current system state of the multi-cloud management platform;

[0026] Randomly initialize the parameters θ of the Actor network and the parameters w of the Critic network;

[0027] According to the training set, randomly obtain the first state S of the current state sequence, and obtain the state feature vector φ(S) corresponding to the first state S;

[0028] Input the state feature vector φ(S) into the Actor network to predict the action probability distribution;

[0029] According to the action probability distribution, a target execution action is selected and executed to obtain a new second state S', a state feature vector φ(S') corresponding to the second state S', and a reward R;

[0030] Input the state feature vector φ(S) and the state feature vector φ(S') into the Critic network to obtain the evaluation state value V(S,w) corresponding to the state feature vector φ(S) and the evaluation state value V(S',w) corresponding to the state feature vector φ(S');

[0031] Calculate a TD error value according to the evaluation state value V(S, w), the evaluation state value V(S', w) and the reward R, and update the parameter θ and the parameter w according to the TD error value;

[0032] Return to execute, randomly obtain the first state S of the current state sequence according to the training set, and obtain the state feature vector φ(S) corresponding to the first state S, until the number of iterations T is reached, to obtain the trained Actor-Critic model.

[0033] In one implementation, updating the parameter θ and the parameter w according to the TD error value includes:

[0034] Calculating the loss function of the Actor-Critic model according to the TD error value;

[0035] According to the loss function, the parameter θ and the parameter w are updated.

[0036] In one implementation, deploying the trained Actor-Critic model on the multi-cloud management platform and performing decision management on multiple cloud services includes:

[0037] Packing the trained Actor-Critic model into a Docker container to obtain a containerized Actor-Critic model;

[0038] An instance is created on the multi-cloud management platform, Docker is installed on the instance, and a containerized Actor-Critic model is deployed on the instance, and decision management is performed on multiple cloud services through the containerized Actor-Critic model.

[0039] In a second aspect, an embodiment of the present application further provides a decision management device for a multi-cloud management platform, including:

[0040] A collection unit, used to collect various real-time data of the multi-cloud management platform and pre-process the various real-time data;

[0041] A construction unit is used to construct SARS structure data according to the real-time data after each processing, wherein the SARS structure data includes the current system state, execution action, reward obtained according to the execution action, and new system state transferred according to the execution action corresponding to the multi-cloud management platform on the real-time data after each processing; construct an Actor-Critic model, wherein the Actor network in the Actor-Critic model is used to predict the action probability, and the Critic network in the Actor-Critic model is used to evaluate the state value;

[0042] A management unit is used to train the Actor-Critic model according to the SARS structure data to obtain a trained Actor-Critic model; deploy the trained Actor-Critic model on the multi-cloud management platform to perform decision-making management on multiple cloud services.

[0043] In a third aspect, an embodiment of the present application further provides a computer device, comprising: a memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned aspects, wherein the memory and the processor communicate with each other through an internal connection path.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is implemented.

[0045] The advantages or beneficial effects of the above technical solution include at least:

[0046] This application applies the Actor-Ctitic model to the multi-cloud management platform, and through real-time learning and optimization of decision-making strategies, the multi-cloud management platform can dynamically adjust the resource allocation in the multi-cloud environment according to the real-time environment status and performance indicators, thereby achieving more efficient resource utilization and performance optimization. At the same time, it can also optimize resource configuration and scheduling strategies to achieve effective management and optimization of resources in the multi-cloud environment, thereby improving system performance and efficiency and optimizing resource utilization and cost allocation, achieving effective resource utilization and cost savings, and reducing the operating costs of enterprises, and can provide enterprises with smarter and more efficient multi-cloud management services.

[0047] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0049] Figure 1 An example flow chart of a decision management method for a multi-cloud management platform provided in an embodiment of the present application;

[0050] Figure 2 A structural block diagram of a decision management device for a multi-cloud management platform provided in an embodiment of the present application;

[0051] Figure 3 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0053] Figure 1 A flowchart of a decision management method for a multi-cloud management platform according to an embodiment of the present application is shown. Figure 1 As shown, the method may include the following steps:

[0054] S110. Collect various real-time data of the multi-cloud management platform and pre-process the various real-time data.

[0055] In one embodiment, the multi-cloud management platform refers to a platform that can simultaneously manage multiple cloud services (including public cloud, private cloud, and hybrid cloud services, etc.) and the resources of each cloud service.

[0056] In one embodiment, various real-time data may include, but are not limited to: CPU utilization, memory utilization, storage space usage, network bandwidth usage, system logs (operation records, service status) and costs of different cloud service providers.

[0057] In one embodiment, the implementation process of step S110 may include: calling the API interface provided by the monitoring platform to collect various real-time data of the multi-cloud management platform. In this way, the real-time data obtained can be more real-time and comprehensive, and the real-time data returned by the API are all structured data with clear data formats and fields, which are more convenient for analysis and processing.

[0058] In one implementation, missing values, abnormal values, and noise data may be processed for each item of real-time data.

[0059] Among them, the missing value processing of each real-time data may include: when the missing value ratio in any real-time data is within 5% and the overall impact on any real-time data is small, you can consider deleting the row or column where the missing value is located in any real-time data. If the missing value ratio of the data is greater than 5%, or deletion will cause serious information loss of any real-time data, you can choose to fill the missing value. Among them, the mean, median or mode can be used to fill the missing values ​​in any real-time data, or the most appropriate statistics can be selected according to the data distribution and business scenario to fill the missing values ​​in any real-time data.

[0060] Among them, outlier processing for each real-time data may include: defining an outlier as exceeding the mean ± 3 times the standard deviation. If the proportion of outliers in any real-time data is within 5%, the abnormal record can be directly deleted, that is, the real-time data can be deleted; if the proportion is greater than 5%, the outlier in any real-time data can be replaced with the mean.

[0061] The noise data processing for each item of real-time data may include: using the existing popular K-Means clustering method to find out the noise data of any item of real-time data, and removing the noise data from the any item of real-time data to retain clean data.

[0062] In the embodiment of the present application, the quality of each item of real-time data can be ensured by preprocessing the collected items of real-time data.

[0063] S120. Construct SARS structure data based on the real-time data after various processing.

[0064] In one embodiment, the SARS structure data may include: the current system state (state) corresponding to each processed real-time data of the multi-cloud management platform, the execution action (action), the reward (reward) obtained according to the execution action, and the new system state (next_state) transferred according to the execution action. That is, SARS refers to the combination of state, action, reward and next_state.

[0065] In one implementation, each item of processed real-time data can be sorted out to determine the data type of each item of processed real-time data; then, according to the data type of each item of processed real-time data, each item of processed real-time data can be state-coded, action-coded, and reward-processed; finally, according to the state coding, action coding, and reward of each item of processed real-time data, each item of processed real-time data can be converted into four-tuple data to construct SARS structure data.

[0066] In specific implementations, the data types of various processed real-time data may include but are not limited to: security logs, operation logs, system logs, exception logs, prices and fee structures of different cloud service providers, and application requirements and behavior patterns.

[0067] Among them, by encoding the state of the real-time data after each processing, it is convenient to describe the current system state. By encoding the action of the real-time data after each processing, it can be used to indicate the possible operation or strategy of the system. By performing reward processing on the real-time data after each processing, it can be used to quantify the effect of each execution action.

[0068] In the embodiment of the present application, by constructing SARS structure data according to the real-time data after various processes, it is possible to facilitate the training and optimization process of the subsequent Actor-Critic model.

[0069] S130. Construct an Actor-Critic model.

[0070] In one embodiment, the Actor network in the Actor-Critic model is used to predict action probabilities, and the Critic network in the Actor-Critic model is used to evaluate state values. The Actor-Critic model is a powerful reinforcement learning algorithm framework that can effectively solve reinforcement learning problems in continuous action spaces and high-dimensional state spaces.

[0071] S140. According to the SARS structure data, the Actor-Critic model is trained to obtain a trained Actor-Critic model.

[0072] In one implementation, the implementation process of step S140 may include the following steps:

[0073] S141. Save the SARS structure data as a TensorFlow dataset.

[0074] In the embodiment of the present application, the SARS structure data is saved as a TensorFlow data set, which can provide a unified and efficient interface to process various forms of data.

[0075] S142. Divide the TensorFlow dataset into a training set, a validation set, and a test set.

[0076] As an example, the TensorFlow dataset can be divided into training set, validation set, and test set in a ratio of 70%, 15%, and 15%.

[0077] S143. The Actor-Critic model is trained using the training set. According to the prediction results of the Actor network, the evaluation results of the Critic network and the true value of the training set, the Actor network and the Critic network are updated to obtain a trained Actor-Critic model.

[0078] In one implementation, the implementation process of step S143 may include the following steps:

[0079] S1431. Determine input parameters of the Actor-Critic model.

[0080] In a specific implementation, the input parameters may include, but are not limited to: the number of iterations T, the state feature dimension N, and the action set A. For example, the step size α, the step size β, the attenuation factor γ, and the exploration rate ∈ may also be included.

[0081] As an example, the action set A can be represented as A={increase CPU, decrease CPU, increase memory, decrease memory}.

[0082] S1432. Initialize the state environment of the multi-cloud management platform, and use the initialization state as the current system state of the multi-cloud management platform.

[0083] S1433. Randomly initialize the parameters θ of the Actor network and the parameters w of the Critic network.

[0084] As an example, the parameters of the Actor network and the Critic network may be randomly initialized using a normal distribution.

[0085] S1434. According to the training set, randomly obtain the first state S of the current state sequence, and obtain the state feature vector φ(S) corresponding to the first state S.

[0086] S1435. Input the state feature vector φ(S) into the Actor network and predict the action probability distribution.

[0087] By executing step S1435, the probability of each possible action can be predicted.

[0088] S1436. According to the action probability distribution, select and execute the target execution action to obtain a new second state S', a state feature vector φ(S') corresponding to the second state S', and a reward R.

[0089] For example, according to the action probability distribution, a target execution action is selected and executed from a training set.

[0090] S1437. Input the state feature vector φ(S) and the state feature vector φ(S') into the Critic network to obtain the evaluation state value V(S,w) corresponding to the state feature vector φ(S) and the evaluation state value V(S',w) corresponding to the state feature vector φ(S').

[0091] S1438. Calculate the TD error value based on the evaluation state value V(S,w), the evaluation state value V(S',w) and the reward R, and update the parameters θ and w based on the TD error value.

[0092] As an example, the calculation formula of the TD error value can be expressed as the following formula (1):

[0093] TD=R+γV(S',w)-V(S,w) (1)

[0094] Here, γ is a discount factor that controls the importance of future rewards.

[0095] In specific implementation, TD error can be used to represent the difference between the actual return of the current state and the return estimated by the Critic network.

[0096] In one implementation, updating the parameter θ and the parameter w according to the TD error value may include: calculating a loss function of the Actor-Critic model according to the TD error value; and then updating the parameter θ and the parameter w according to the loss function.

[0097] As an example, the loss function of the Actor-Critic model can be calculated according to the TD error value according to the following formula (2). The loss function can be used to measure the size of the TD error.

[0098] L(w)=∑(R+γV(S′,w)-V(S,w)) 2 (2)

[0099] In one embodiment, according to the loss function, updating the parameter w may include: calculating the gradient of the loss function L(w) with respect to the parameter w The parameter w of the Critic network is updated using the gradient descent method. That is, the parameter w can be updated using the gradient descent method using the following formula (3).

[0100]

[0101] Among them, w' is the updated parameter w, and β is the learning rate, which controls the step size of each update.

[0102] In one embodiment, updating the parameter θ according to the loss function may include: updating the parameter θ using a gradient ascent method according to the loss function in combination with the following formula (4) and formula (5).

[0103]

[0104] Among them, θ' is the updated parameter θ, πθ(a|s; θ) is the strategy probability of selecting action a given state s, is the policy probability gradient with respect to θ, A(s,a) is the advantage function, which represents the advantage of action a under state s. The TD error is usually used to approximate the advantage function: A(s,a)≈TD.

[0105] S1439, return to execute step S1434 until the number of iterations T is reached, and the trained Actor-Critic model is obtained.

[0106] S144. During the training process, the validation set is regularly used to evaluate the performance of the trained Actor-Critic model, and the relevant parameters of the trained Actor-Critic model are saved.

[0107] In an embodiment of the present application, by regularly using a validation set to evaluate the performance of a trained Actor-Critic model during the training process, the trained Actor-Critic model can be adjusted and optimized as needed, and by saving relevant parameters of the trained Actor-Critic model, training interruptions or unexpected situations can be prevented.

[0108] S145. After the training is completed, the final performance of the trained Actor-Critic model is evaluated using the test set. When the final performance reaches the preset performance, the trained Actor-Critic model is used as the trained Actor-Critic model.

[0109] In the embodiment of the present application, in step S145, the trained Actor-Critic model may also be adjusted and optimized as needed.

[0110] S150. Deploy the trained Actor-Critic model on the multi-cloud management platform to perform decision management on multiple cloud services.

[0111] In one implementation, the implementation process of step S150 may include the following steps:

[0112] S151. Package the trained Actor-Critic model into a Docker container to obtain a containerized Actor-Critic model.

[0113] As an example, the trained Actor-Critic model and related preprocessing scripts, dependencies, etc. can be packaged into a Docker container.

[0114] S152. Create an instance on the multi-cloud management platform, install Docker on the instance, and deploy the containerized Actor-Critic model on the instance, and perform decision management on multiple cloud services through the containerized Actor-Critic model.

[0115] In the specific implementation, an instance is created on the multi-cloud management platform, which can serve as the host for deploying the containerized Actor-Critic model.

[0116] In the specific implementation, when deploying the containerized Actor-Critic model on the instance, you can also configure security group rules on the multi-cloud management platform, add inbound rules to allow HTTP / HTTPS access (such as port 80 and 443) and model API access (such as port 5000), and at the same time, you can also configure SSL certificates on the multi-cloud management platform to ensure the security of data transmission, and you can also configure CloudWatch monitoring and logging services on the multi-cloud management platform to monitor the running status and performance of the containerized Actor-Critic model, and record logs for troubleshooting and performance optimization.

[0117] As an example, after completing the deployment of the containerized Actor-Critic model, you can use some test data sets to test the performance and accuracy of the containerized Actor-Critic model. Then, based on the test results, adjust the hyperparameters of the containerized Actor-Critic model, optimize the structure of the containerized Actor-Critic model, etc., so as to further improve the performance of the containerized Actor-Critic model.

[0118] As an example, the process of decision management for multiple cloud services through a containerized Actor-Critic model can be as follows:

[0119] Determine the current state feature vector: φ(S) = [CPU utilization, memory utilization, response time, current cost], determine the action set A = {increase CPU, reduce CPU, increase memory, reduce memory}, and determine the reward R = -(response time) + λ × (cost savings) where λ is a weight parameter used to balance the relationship between response time and cost. Decision execution is performed through the containerized Actor-Critic model: Under state S, the Actor network predicts the action probability πθ(a|s; θ), selects action A and executes it, obtains the new state S′ and reward R, uses the Critic network to evaluate the state value, calculates the TD error, and updates the parameters w of the Critic network. After updating the parameters w of the Critic network, the TD error is calculated using the updated parameters w of the critic network, and the parameters θ of the actor network are updated. The actor network predicts the next action based on this parameter θ. If the CPU utilization and response time are high, the Actor network will choose the action of increasing CPU resources.

[0120] In an embodiment of the present application, by executing step S150, the Actor-Critic framework in the reinforcement learning algorithm can be introduced into the multi-cloud management platform as an intelligent decision-making technology, which has the advantages of handling uncertainty, adapting to dynamic environments, and optimizing complex decision-making problems, thereby realizing intelligent resource optimization and automated decision-making of the platform, which can not only improve resource utilization and optimize system performance, but also reduce costs and improve reliability.

[0121] In summary, in the decision-making management method of the multi-cloud management platform provided in the embodiment of the present application, by applying the Actor-Ctitic model in the multi-cloud management platform, through real-time learning and optimization of decision-making strategies, the multi-cloud management platform can dynamically adjust the resource allocation in the multi-cloud environment according to the real-time environmental status and performance indicators, thereby achieving more efficient resource utilization and performance optimization. At the same time, it can also optimize resource configuration and scheduling strategies to achieve effective management and optimization of resources in the multi-cloud environment, thereby improving system performance and efficiency and optimizing resource utilization and cost allocation, achieving effective utilization of resources and cost savings, and reducing the operating costs of enterprises, and can provide enterprises with smarter and more efficient multi-cloud management services.

[0122] Figure 2 FIG. 1 is a block diagram showing a decision management device of a multi-cloud management platform according to an embodiment of the present application. Figure 2 As shown, the device may include:

[0123] The collection unit 210 is used to collect various real-time data of the multi-cloud management platform and pre-process the various real-time data;

[0124] A construction unit 220 is used to construct SARS structure data according to the real-time data after each processing, wherein the SARS structure data includes the current system state, the execution action, the reward obtained according to the execution action, and the new system state transferred according to the execution action corresponding to the multi-cloud management platform on the real-time data after each processing; construct an Actor-Critic model, wherein the Actor network in the Actor-Critic model is used to predict the action probability, and the Critic network in the Actor-Critic model is used to evaluate the state value;

[0125] The management unit 230 is used to train the Actor-Critic model according to the SARS structure data to obtain a trained Actor-Critic model; deploy the trained Actor-Critic model on the multi-cloud management platform to perform decision management on multiple cloud services.

[0126] The functions of each unit in the decision management device of the multi-cloud management platform in the embodiment of the present application can be referred to the corresponding description in the above method, and will not be repeated here.

[0127] Figure 3 FIG. 2 shows a block diagram of a computer device according to an embodiment of the present application. Figure 3 As shown, the computer device includes: a memory 310 and a processor 320. The memory 310 stores instructions, and the instructions are loaded and executed by the processor 320 to implement the decision management method of the multi-cloud management platform in the above embodiment. The number of the memory 310 and the processor 320 can be one or more.

[0128] The computer device also includes:

[0129] The communication interface 330 is used to communicate with external devices and perform data exchange transmission.

[0130] If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0131] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.

[0132] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method provided in the embodiment of the present application is implemented.

[0133] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0134] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0135] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0136] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0137] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0138] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0139] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0140] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0142] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0143] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A decision management method for a multi-cloud management platform, characterized in that: include: Collect various real-time data of the multi-cloud management platform and pre-process the various real-time data; Constructing SARS structure data according to the real-time data after each processing, wherein the SARS structure data includes the current system state, execution action, reward obtained according to the execution action, and new system state transferred to according to the execution action corresponding to the multi-cloud management platform on the real-time data after each processing; Constructing an Actor-Critic model, wherein the Actor network in the Actor-Critic model is used to predict action probabilities, and the Critic network in the Actor-Critic model is used to evaluate state values; According to the SARS structure data, the Actor-Critic model is trained to obtain a trained Actor-Critic model; The trained Actor-Critic model is deployed on the multi-cloud management platform to perform decision-making management on multiple cloud services.

2. The method according to claim 1, characterized in that: Preprocessing the various real-time data includes: The missing values, abnormal values ​​and noise data are processed for each item of real-time data.

3. The method according to claim 1, characterized in that According to the real-time data after various processing, the SARS structure data is constructed including: Determine the data type of each item of processed real-time data; According to the data type of each processed real-time data, the processed real-time data is subjected to state coding, action coding and reward processing; According to the state codes, action codes and rewards of the processed real-time data, the processed real-time data are converted into four-tuple data to construct the SARS structure data.

4. The method according to claim 1, characterized in that: The Actor-Critic model is trained according to the SARS structure data, and the trained Actor-Critic model includes: Saving the SARS structure data as a TensorFlow dataset; Divide the TensorFlow dataset into a training set, a validation set, and a test set; The Actor-Critic model is trained using the training set, and the Actor network and the Critic network are updated according to the prediction result of the Actor network, the evaluation result of the Critic network and the true value of the training set to obtain a trained Actor-Critic model; During the training process, the validation set is regularly used to evaluate the performance of the trained Actor-Critic model, and relevant parameters of the trained Actor-Critic model are saved; After the training is completed, the test set is used to evaluate the final performance of the trained Actor-Critic model. When the final performance reaches a preset performance, the trained Actor-Critic model is used as the trained Actor-Critic model.

5. The method according to claim 4, characterized in that The Actor-Critic model is trained using the training set, and the Actor network and the Critic network are updated according to the prediction result of the Actor network, the evaluation result of the Critic network and the true value of the training set, so as to obtain the trained Actor-Critic model, including: Determine the input parameters of the Actor-Critic model, wherein the input parameters include the number of iterations T, the state feature dimension N, and the action set A; Initializing the state environment of the multi-cloud management platform, and using the initialization state as the current system state of the multi-cloud management platform; Randomly initialize the parameters θ of the Actor network and the parameters w of the Critic network; According to the training set, randomly obtain the first state S of the current state sequence, and obtain the state feature vector φ(S) corresponding to the first state S; Input the state feature vector φ(S) into the Actor network to predict the action probability distribution; According to the action probability distribution, select and execute the target execution action to obtain a new second state S', a state feature vector φ(S') corresponding to the second state S', and a reward R; Input the state feature vector φ(S) and the state feature vector φ(S') into the Critic network to obtain the evaluation state value V(S,w) corresponding to the state feature vector φ(S) and the evaluation state value V(S',w) corresponding to the state feature vector φ(S'); Calculate a TD error value according to the evaluation state value V(S, w), the evaluation state value V(S', w) and the reward R, and update the parameter θ and the parameter w according to the TD error value; Return to execute, randomly obtain the first state S of the current state sequence according to the training set, and obtain the state feature vector φ(S) corresponding to the first state S, until the number of iterations T is reached, to obtain the trained Actor-Critic model.

6. The method according to claim 5, characterized in that Updating the parameter θ and the parameter w according to the TD error value includes: Calculating the loss function of the Actor-Critic model according to the TD error value; According to the loss function, the parameter θ and the parameter w are updated.

7. The method according to any one of claims 1 to 6, characterized in that Deploying the trained Actor-Critic model on the multi-cloud management platform to perform decision management on multiple cloud services includes: Packing the trained Actor-Critic model into a Docker container to obtain a containerized Actor-Critic model; An instance is created on the multi-cloud management platform, Docker is installed on the instance, and a containerized Actor-Critic model is deployed on the instance, and decision management is performed on multiple cloud services through the containerized Actor-Critic model.

8. A decision management device for a multi-cloud management platform, characterized in that: include: A collection unit, used to collect various real-time data of the multi-cloud management platform and pre-process the various real-time data; A construction unit is used to construct SARS structure data according to the real-time data after each processing, wherein the SARS structure data includes the current system state, execution action, reward obtained according to the execution action, and new system state transferred according to the execution action corresponding to the multi-cloud management platform on the real-time data after each processing; construct an Actor-Critic model, wherein the Actor network in the Actor-Critic model is used to predict the action probability, and the Critic network in the Actor-Critic model is used to evaluate the state value; A management unit is used to train the Actor-Critic model according to the SARS structure data to obtain a trained Actor-Critic model; deploy the trained Actor-Critic model on the multi-cloud management platform to perform decision-making management on multiple cloud services.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the method according to any one of claims 1 to 7 is implemented.

Images