A cloud manufacturing service composition method based on multi-strategy deep reinforcement learning

Abstract

The present invention relates to a cloud manufacturing service composition method based on multi-strategy deep reinforcement learning, which combines the basic DQN algorithm, competition structure, dual estimator and priority replay mechanism, and also adds some strategies to the model, such as reward strategy, ε‑greedy strategy and heuristic strategy, to improve the performance of the algorithm. The heuristic strategy can shield unavailable services, overcoming the shortcoming that DLR cannot be restored to a stable state due to unreasonable hyperparameter and reward design. The reward allows DLR to effectively adjust parameters according to the real reward value, and the ε‑greedy strategy gives the algorithm the opportunity to jump out of the local optimum. Experiments show that the model not only has the advantage of strong adaptability to dynamic environments, but also is better than other deep reinforcement learning methods.

Description

Technical Field

[0001] The present invention relates to the field of cloud manufacturing service groups, and in particular to a cloud manufacturing service combination method based on multi-strategy deep reinforcement learning. Background Art

[0002] As the quality of life continues to improve, users are increasingly demanding personalized products. However, due to limited resources, the manufacturing resources and capabilities of a single enterprise can no longer meet the needs of users. To solve this problem, enterprises need to collaborate effectively by sharing manufacturing resources and capabilities. A new service-oriented intelligent manufacturing model, cloud manufacturing (CMfg), has emerged. In the cloud manufacturing platform, the manufacturing resources and capabilities shared by enterprises are encapsulated into services and provided to users for selection through the Internet. Service composition and optimization selection (SCOS) is considered to be the key technology for cloud manufacturing platforms to achieve resource and capability sharing. According to different combination structures, SCOS integrates various fine-grained services with different functions into coarse-grained services with comprehensive functions to handle complex manufacturing tasks and meet user needs. The problem of quickly and effectively handling the optimal combination strategy has received widespread attention.

[0003] So far, many researchers have proposed many heuristic methods to solve the SCOS problem. Although these methods have promoted the research work of SCOS, they lack adaptability to dynamic environments. For example, when the environment changes, the algorithm may need to be redesigned. Therefore, the SCOS problem requires new methods. Based on this, some researchers believe that reinforcement learning (RL) is adaptable to dynamic environments and try to use reinforcement learning to solve the SCOS environment change problem. For example, Wang et al. used deep reinforcement learning (DRL) to solve the SCOS problem. He divided the state set into two state subsets and selected behavior strategies according to different types of states. Although the deep reinforcement learning method can adaptively readjust the system to a stable state when the environment changes suddenly. However, if the hyper-parameters and rewards in the algorithm are not designed reasonably, it is difficult to return to a stable state. Summary of the invention

[0004] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is that the existing algorithms are not suitable for solving complex and dynamic service combination problems and are difficult to return to a stable state when the hyperparameters and reward designs are unreasonable.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solution: a cloud manufacturing service combination method based on multi-strategy deep reinforcement learning, comprising the following steps:

[0006] S1: Obtain a virtual dataset through existing methods, which consists of multiple services;

[0007] Assume that a task T consists of n subtasks, one subtask corresponds to a service set, one service in the service set corresponds to an action, and each service has multiple QoS attribute values, and the QoS attribute value is any data value between [0.7, 0.95];

[0008] definition:

[0009] T={t1,t2,...t i ...t n}, t i represents the i-th subtask of T, i=1, 2, ...n;

[0010] Represents subtask t i The corresponding μth service, μ=1, 2, ...c;

[0011] Represents subtask t i The action corresponding to the μth service;

[0012] Represents subtask t i a corresponding μth service QoS attribute value, where the μth service QoS attribute value is obtained by normalizing multiple QoS attribute values ​​corresponding to the μth service;

[0013] m1 is the arithmetic mean of all service QoS values ​​in the service chain obtained by the current task T;

[0014] m2 is the arithmetic mean of all service QoS values ​​in the service chain obtained by the current task T after the interval setting time;

[0015] S2: Construct a Multi-D3QN model, which modifies the network structure of the DQN algorithm into a competitive structure, modifies the random sampling mechanism of the DQN algorithm into a priority replay mechanism, and modifies the single estimator into a dual estimator in the learning phase;

[0016] The Multi-D3QN model also includes a Q-network model and a Q-target model;

[0017] S3: Initialize the parameters of the Multi-D3QN model, initialize m1=m2=0, initialize the experience pool D, set the priority, initialize the action value function Q of the model Q-network, and initialize the action value function Q of the model Q-target * ;

[0018] S4: Randomly select a task T from the virtual dataset;

[0019] S5: let k=1;

[0020] S6: let i=1;

[0021] S7: If m1>m2, execute S8-1; otherwise execute S8-4;

[0022] S8-1: Select t i The best action The steps are as follows: Determine t i Corresponding service set QoS attribute values ​​of all services in Is it less than 0?

[0023] If t i The corresponding QoS attribute value of the μth service Less than 0, it means that there is an unavailable service and the next step is executed; if t i The corresponding QoS attribute value of the μth service If it is greater than or equal to 0, execute S8-4;

[0024] S8-2: Randomly generate a probability x1, ε1 represents the preset probability, if x1 < ε1, then the quasi-optimal action And execute the next step, where θ represents the parameters of the action value function Q;

[0025] Otherwise Randomly select an action As a quasi-optimal action And execute the next step;

[0026] S8-3: Determine the quasi-optimal action Whether the QoS attribute value of the corresponding service is less than 0:

[0027] If the quasi-optimal action If the QoS attribute value of the corresponding service is less than 0, it will be blocked The corresponding service and randomly select an action as

[0028] Otherwise, the action As and execute S8;

[0029] S8-4: Randomly generate a probability x2, ε2 represents the preset probability:

[0030] If x2<ε2, then the optimal action is And execute the next step, where θ represents the parameters of the action value function Q;

[0031] Otherwise Randomly select an action As a quasi-optimal action

[0032] S9: Execution And t i Corresponding service set t i QoS attribute values ​​of all services in the corresponding service set and t i The next service set As a sample Stored in D, the sample is represented as

[0033] S10: Determine whether the samples in D have reached the maximum capacity. If not, proceed to the next step. Otherwise, proceed to S12.

[0034] S11: Determine whether i is greater than n, if yes, execute the next step, otherwise set i=1+1 and return to S8;

[0035] S12: Determine whether j is greater than the maximum number of iterations. If so, output Each corresponding service is taken as the optimal service and output and a service chain consisting of optimal services;

[0036] Otherwise, let k=k+1, calculate m1 and m2, update m1 and m2 and return to S6;

[0037] S13: Use the samples obtained in S8 to train the model Q-network and return to S10.

[0038] As an improvement, the process of training the model Q-network in S13 is as follows:

[0039] S13-1: Set the initial priority of all samples in D

[0040] S13-2: Let c=1;

[0041] S13-3: Sample b samples in a small batch from D according to the sampling probability. The sampling probability of each sample is calculated using the following formula:

[0042]

[0043] Among them, p(h j ) represents the probability of sampling the jth sample in a total of b samples obtained by mini-batch sampling, ph j Indicates the priority of the jth sample in a total of b samples obtained by mini-batch sampling;

[0044] S13-4: Input b samples into Q-network and Q-target, and train Q-network and Q-target;

[0045] S13-5: let j=1;

[0046] S13-6: Calculate the weight w(h of the jth sample j );

[0047] S13-7: If j>b, let y j =r j , calculate the loss loss of the jth sample j , according to the loss loss j Update the parameters θ of the Q-network and proceed to the next step;

[0048] Otherwise, y is calculated by the dual estimator j , calculate the loss loss of the jth sample j , according to the loss loss j Update the parameter θ of the Q-network, set j=j+1 and return to S13-6;

[0049] y j Represents the estimated function value obtained by inputting the j-th sample into Q-target;

[0050] r j represents the QoS attribute value of the service corresponding to the optimal action in the jth sample;

[0051] The loss of the jth sample is calculated using the following formula: j ,

[0052]

[0053] Among them, Q(S j , a j , θ) represents the function value obtained by inputting the j-th sample into the Q-network, and the function value is taken as the true value of the j-th sample, S j represents the service set corresponding to the optimal action in the jth sample, a jThe optimal action in the jth sample, θ represents the parameters of the Q-network;

[0054] S13-8: Calculate the loss of the jth sample in the cth iteration The formula is as follows:

[0055]

[0056] According to the loss Update the priority of the jth sample, the formula is as follows:

[0057] ph j ←|δ j | (10);

[0058] S13-9: If c>c max , update the parameters θ in the Q-target network with the parameters in the Q-network * ; Otherwise, let c=c+1 and return to S13-2.

[0059] As an improvement, the process of calculating the weight of each sample in the b samples in S13-6 is as follows:

[0060]

[0061] Among them, w(h j ) represents the weight of the jth sample, w i represents the weight of the i-th sample, max i (w i ) represents the maximum sample weight β represents an exponential factor.

[0062] As an improvement, in S13-7, y is calculated by a dual estimator j The process is as follows:

[0063]

[0064] Among them, a means "s j+1 ” corresponds to the action set, and γ represents the discount rate.

[0065] As an improvement, the process of normalizing the multiple QoS attribute values ​​corresponding to the μth service in S1 to obtain the μth service QoS attribute value is as follows:

[0066] The QoS attribute value corresponding to the μth service includes a positive attribute value and a negative attribute value;

[0067] The positive attribute value corresponding to the μth service The normalized calculation formula is as follows:

[0068]

[0069] The negative attribute value corresponding to the μth service The normalized calculation formula is as follows:

[0070]

[0071] in, and They represent the minimum QoS attribute value and the maximum QoS attribute value among multiple QoS attribute values ​​of the μth service, respectively. μ It is the aggregate value of multiple QoS attribute values ​​of the μth service.

[0072] Compared with the prior art, the present invention has at least the following advantages:

[0073] 1. A better solution can be obtained when dealing with complex design processes in dynamic environments. The Multi-D3QN method proposed in this invention can combine DQN, competitive architecture, dual estimator and priority replay mechanism, and integrate their advantages to generate an algorithm that can obtain better solutions in dynamic environments.

[0074] 2. Avoid the situation where it is difficult to recover to a stable state due to unreasonable parameter value design. In view of the disadvantage that DRL is difficult to recover to a stable state when the service is unavailable due to unreasonable parameter value design, a strategy is developed based on heuristic rules to shield the unavailable service.

[0075] 3. According to the different characteristics of key issues, the impacts of many different strategies are compared to improve the wide applicability of the method of the present invention as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Schematic diagram of the process of the Multi-D3QN method of the present invention.

[0077] Figure 2 Composition flow chart for dynamic services.

[0078] Figure 3 Comparison of different learning rates for the D3QN model used in the present invention.

[0079] Figure 4 Training curves for D3QN, DQN, dueling-DQN, and double-DQN.

[0080] Figure 5 The impact of different strategies on service attribute values.

[0081] Figure 6Adaptability for Multi-D3QN, double DQN, DQN, and dueling DQN.

[0082] Figure 3-Figure 6 The vertical axis Epoch represents the number of iterations. DETAILED DESCRIPTION

[0083] The present invention is described in further detail below.

[0084] See also Figure 1-Figure 6 , a cloud manufacturing service composition method based on multi-strategy deep reinforcement learning, comprising the following steps:

[0085] S1: Obtain a virtual dataset through existing methods, which consists of multiple services;

[0086] Assume that a task T consists of n subtasks, one subtask corresponds to a service set, one service in the service set corresponds to an action, and each service has multiple QoS attribute values, and the QoS attribute value is any data value between [0.7, 0.95];

[0087] definition:

[0088] T={t1,t2,...t i ...t n}, t i represents the i-th subtask, i=1, 2, ...n;

[0089] Represents subtask t i The corresponding μth service, μ=1, 2, ...c;

[0090] Represents subtask t i The action corresponding to the μth service;

[0091] Represents subtask t i a corresponding μth service QoS attribute value, where the μth service QoS attribute value is obtained by normalizing multiple QoS attribute values ​​corresponding to the μth service;

[0092] m1 is the arithmetic mean of all service QoS values ​​in the service chain obtained by the current task T;

[0093] m2 is the arithmetic mean of all service QoS values ​​in the service chain obtained by the current task T after the interval setting time;

[0094] S2: Construct a Multi-D3QN model. The Multi-D3QN model is an improved DQN algorithm. The network structure of the DQN algorithm is modified to the dueling architecture, the random sampling mechanism of the DQN algorithm is modified to the prioritized replay mechanism, and the single estimator is modified to the double estimator in the learning phase.

[0095] The Multi-D3QN model also includes a Q-network model and a Q-target model;

[0096] S3: Initialize the parameters of the Multi-D3QN model, initialize m1=m2=0, initialize the experience pool D, set the priority, initialize the action value function Q of the model Q-network, and initialize the action value function Q of the model Q-target * ;

[0097] S4: Randomly select a task T from the virtual dataset;

[0098] S5: let k=1;

[0099] S6: let i=1;

[0100] S7: If m1>m2, execute S8-1; otherwise execute S8-4;

[0101] S8-1: Select t i The best action The steps are as follows: Determine t i Corresponding service set QoS attribute values ​​of all services in Is it less than 0?

[0102] If t i The corresponding QoS attribute value of the μth service Less than 0, it means that there is an unavailable service and the next step is executed; if t i The corresponding QoS attribute value of the μth service If it is greater than or equal to 0, execute S8-4;

[0103] S8-2: Randomly generate a probability x1, ε1 represents the preset probability, if x1 < ε1, then the quasi-optimal action And execute the next step, where θ represents the parameters of the action value function Q;

[0104] Otherwise Randomly select an action As a quasi-optimal action And execute the next step;

[0105] S8-3: Determine the quasi-optimal action Whether the QoS attribute value of the corresponding service is less than 0:

[0106] If the quasi-optimal action If the QoS attribute value of the corresponding service is less than 0, it will be blocked The corresponding service and randomly select an action as

[0107] Otherwise, the action As and execute S8;

[0108] S8-4: Randomly generate a probability x2, ε2 represents the preset probability:

[0109] If x2<ε2, then the optimal action is And execute the next step, where θ represents the parameters of the action value function Q;

[0110] Otherwise Randomly select an action As a quasi-optimal action

[0111] S9: Execution And t i Corresponding service set t i QoS attribute values ​​of all services in the corresponding service set and t i The next service set As a sample Stored in D, the sample is represented as

[0112] S10: Determine whether the samples in D have reached the maximum capacity. If not, proceed to the next step. Otherwise, proceed to S12.

[0113] S11: Determine whether i is greater than n, if yes, execute the next step, otherwise set i=1+1 and return to S8;

[0114] S12: Determine whether j is greater than the maximum number of iterations. If so, output Each corresponding service is taken as the optimal service and output and a service chain consisting of optimal services;

[0115] Otherwise, let k=k+1, calculate m1 and m2, update m1 and m2 and return to S6;

[0116] S13: Use the samples obtained in S8 to train the model Q-network and return to S10.

[0117] Specifically, the process of training the model Q-network in S13 is as follows:

[0118] S13-1: Set the initial priority of all samples in D

[0119] S13-2: Let c=1;

[0120] S13-3: Sample b samples in a small batch from D according to the sampling probability. The sampling probability of each sample is calculated using the following formula:

[0121]

[0122] Among them, p(h j ) represents the probability of sampling the jth sample in a total of b samples obtained by mini-batch sampling, ph j Indicates the priority of the i-th sample in a total of b samples obtained by mini-batch sampling;

[0123] S13-4: Input b samples into Q-network and Q-target, and train Q-network and Q-target;

[0124] S13-5: let j=1;

[0125] S13-6: Calculate the weight w(h of the jth sample j );

[0126] S13-7: If j>b, let y j =r j , calculate the loss of the jth sample j , according to the loss loss j Update the parameters θ of the Q-network and proceed to the next step;

[0127] Otherwise, y is calculated by the dual estimator j , calculate the loss of the jth sample j , according to the loss loss j Update the parameter θ of the Q-network, set j=j+1 and return to S13-6;

[0128] y j Represents the estimated function value obtained by inputting the j-th sample into Q-target;

[0129] r j represents the QoS attribute value of the service corresponding to the optimal action in the jth sample;

[0130] The loss of the jth sample is calculated using the following formula: j ,

[0131]

[0132] Among them, Q(S j , a j , θ) represents the function value obtained by inputting the j-th sample into the Q-network, and the function value is taken as the true value of the j-th sample, S j represents the service set corresponding to the optimal action in the jth sample, a j The optimal action in the jth sample, θ represents the parameters of the Q-network;

[0133] S13-8: Calculate the loss of the jth sample in the cth iteration The formula is as follows:

[0134]

[0135] According to the loss Update the priority of the jth sample, the formula is as follows:

[0136] ph j ←|δ j | (10);

[0137] S13-9: If c>c max , update the parameters θ in the Q-target network with the parameters in the Q-network * ; Otherwise, let c=c+1 and return to S13-2.

[0138] Specifically, the process of calculating the weight of each sample in the b samples in S13-6 is as follows:

[0139]

[0140] Among them, w(h j ) represents the weight of the jth sample, w i represents the weight of the i-th sample, max i (w i ) represents the maximum sample weight β represents an exponential factor.

[0141] Specifically, in S13-7, y is calculated by a dual estimator j The process is as follows:

[0142] yj =r j +γQ * (s j+1 , arg max a Q(s j+1 ,a,θ),θ * ) (12);

[0143] Among them, a means "s j+1 ” corresponds to the action set, and γ represents the discount rate.

[0144] Specifically, the process of normalizing multiple QoS attribute values ​​corresponding to the μth service in S1 to obtain the μth service QoS attribute value is as follows:

[0145] The QoS attribute value corresponding to the μth service includes a positive attribute value and a negative attribute value;

[0146] The positive attribute value corresponding to the μth service The normalized calculation formula is as follows:

[0147]

[0148] The negative attribute value corresponding to the μth service The normalized calculation formula is as follows:

[0149]

[0150] in, and They represent the minimum QoS attribute value and the maximum QoS attribute value among multiple QoS attribute values ​​of the μth service, respectively. μ It is the aggregate value of multiple QoS attribute values ​​of the μth service.

[0151] In a cloud manufacturing environment, a task T is usually decomposed into several subtasks t, where T = {t1, t2, ..., t i , ..., t n}, where n is the number of subtasks of T, t i is the i-th subtask of T, and the service composition process of T can be modeled as an MDP.

[0152] State: In MDP, for each subtask t i (i=1, 2, ..., n) selects an optimal (or nearly optimal) service, where the subtasks and the selected service correspond to different states. As the service combination proceeds, the states of the subtasks and services will change.

[0153] Action: Each t iThere is an action set, an action is equivalent to t i Select a service from the service set. The number of actions in the action set is equal to the number of services in the service set. The size of the action set of each subtask is equal to the size of the service set corresponding to the subtask.

[0154] Reward Function: When a service is selected from the service set in the current state, after executing the action, a reward r can be obtained from the environment. Since the measurement standards and units of each QoS attribute are different, the QoS values ​​of each indicator need to be normalized before calculating the reward. The reward is calculated based on the normalized QoS value. The larger the QoS value of the positive attribute indicator, the better, such as reliability, security, and availability; on the contrary, the smaller the QoS value of the negative attribute indicator, the better; after normalization, the larger the value, the better the performance of all indicators. Through a simple weighting method, the normalized QoS values ​​of different attributes can be integrated into a single QoS value, that is, the overall QoS value.

[0155] The method of the present invention is referred to as Multi-D3QN, which combines the basic DQN algorithm, competition structure, dual estimator and priority replay mechanism. Specifically, the competition structure can shield some actions that have little effect on the optimal value function, so that the algorithm can quickly obtain the optimal value function and improve the convergence speed of the algorithm. The actions found by the competition structure may be over-estimated, and the dual estimator can avoid this situation. The priority replay mechanism can effectively find the sample data that needs to be learned and improve the learning speed of the algorithm. In addition, the method of the present invention also adds a heuristic strategy to shield unavailable services to adapt to dynamic environments.

[0156] (1) DQN algorithm

[0157] The DQN algorithm is the most basic algorithm in deep reinforcement learning. It uses a deep neural network to approximate a value function instead of the Q-table in Q-leaning, overcoming the shortcoming of Q-leaning that cannot handle multiple data. In DQN, two deep neural networks are constructed: Q-network and Q-target. DQN uses Q-network to estimate the current value function and Q-target to generate the target Q value. During the learning process, DQN generally extracts samples from the experience pool D by random sampling, and then updates the parameters in the Q-network according to the loss value, and finally maximizes the future reward.

[0158] (2) Competition structure

[0159] The competitive structure is based on DQN, which improves the deep learning network to make the algorithm learn faster. Its core idea is to generate two value functions in the last layer of the network: the state value function V(s) and the action advantage function A(s, a), namely:

[0160] Q(s,a)=V(s)+A(s,a) (3);

[0161] This means that A(s, a) is just an increment that indicates how much extra reward can be brought when action a is selected. The division of state value function V(s) and action advantage function A(s, a) improves the stability and convergence speed of the algorithm.

[0162] (3) Dual Estimator:

[0163] The core idea of ​​the dual estimator method can be simply summarized as follows: in the traditional RL learning phase, two different network parameters are used to estimate the value function, as shown in the following formula:

[0164] y j =r j +γQ(s t+1 , argmax a Q(s t+1 ,a,θ),θ * ) (4);

[0165] Among them, r j represents the jth reward obtained from the experience pool D, s t+1 is the next state of iteration t+1, θ, θ * are the parameters of Q-network and Q-target, and γ is the discount rate.

[0166] The loss function is calculated as follows:

[0167]

[0168] Q(s t , a t ,θ) is the output of Q-network, s t , a t is the state and action of the tth iteration, b is the sample mini-batch size, and w j is the sample weight.

[0169] (4) Priority playback mechanism

[0170] DQN uses random sampling to learn, that is, randomly extracting samples from the experience pool, which ignores the importance of conversion. The core idea of ​​the priority replay mechanism is that the RL agent can learn from experience more efficiently, that is, more high-expectation experiences calculated by TD-error will be selected.

[0171] TD-error formula:

[0172] δ j =y j -Q(s t , a t ,θ) (6);

[0173] Among them, y j is the Q value at the t+1th iteration.

[0174] (5) Adaptive strategy

[0175] In the process of agent interaction with the environment, given a state, the agent may get an action set with different values. We set each action value to the QoS value of a single service in the service set.

[0176] When a service is unavailable, the QoS value of the entire service chain will become smaller, so we give the following definition:

[0177] Unavailable service: In a state, when the QoS value of some actions becomes abnormal (according to multiple experiments, it is observed that the abnormal value is a negative QoS value of an action in the present invention), the service corresponding to the action can be judged as an unavailable service.

[0178] If a service chain contains unavailable services, the impact on this service chain is very large. It not only reduces the overall QoS value but also affects the execution of tasks. Therefore, in order to ensure the smooth progress of tasks, it is necessary to use heuristic strategies to shield these unavailable services. The heuristic strategies are as follows:

[0179] When the program starts, the average QoS value m1 is calculated. If the value of an action is less than zero, the service corresponding to the action can be judged as unavailable; after a preset interval, the average QoS value m2 is calculated. If m2 is less than m1, it is judged that the program has not adaptively returned to a stable state and the heuristic is started.

[0180] Experiment and analysis

[0181] We experimentally evaluate the proposed multi-strategy deep reinforcement learning algorithm (Multi-D3QN) from three aspects: effectiveness, adaptability, and robustness. The experimental results of the Multi-D3QN algorithm are compared with those of DQN, dueling-DQN, and double-DQN.

[0182] 1. Experimental Setup

[0183] Dataset: In the experiment, we mainly considered five QoS attributes, including reliability, security, availability, time and cost. Given that there is no publicly available dataset for cloud manufacturing services, without loss of generality, all QoS attribute values ​​in the present invention are randomly generated between [0.7, 0.95]. Since the aggregated QoS value obtained after the data in the dataset is calculated by formulas (1) and (2) is too small, the comparison of the results between the algorithms is not obvious. Therefore, we set the weights of the QoS attributes to 1 instead of the commonly used decimals. In addition, each task is decomposed into 30 subtasks, and each subtask corresponds to 30 candidate services.

[0184] Network structure: The network structure of the Multi-D3QN model includes an input layer, an LSTM layer, and two fully connected layers. Except for the LSTM layer, which has 64 neurons, the other layers have 30 neurons. The hyperparameters used in Multi-D3QN training are shown in Table 1.

[0185] Table 1 Hyperparameter settings for Multi-D3QN training

[0186]

[0187] 2. Results Analysis

[0188] (1) Learning rate

[0189] The value of the learning rate plays an important role in the performance of the algorithm. A learning rate that is too high will cause the model to not converge, and a learning rate that is too low will cause the model to converge slowly or fail to learn. Therefore, it is important to choose an optimal learning rate. Figure 3 As shown in Figure 2, it can be clearly observed that D3QN with α = 0.01 performs better than α = 0.001 and α = 0.1 because it has a higher QoS value and faster convergence. Specifically, α = 0.01 converges to 46000. th , while α=0.001 and α=0.1 converge to 50000 th and 65000 th . Moreover, α=0.01 and α=0.001 have almost the same vibration amplitude, but α=0.01 can obtain a higher QoS value. This is because when α=0.001, the learning rate is too small, resulting in a slower learning algorithm. When α=0.1, the fluctuation amplitude is larger than that of α=0.01 and α=0.001, because the learning rate is too large, which makes it difficult for the model to converge. Therefore, we choose α=0.01 as the learning rate for the next experiment.

[0190] (2) Effectiveness and efficiency

[0191] In order to verify the effectiveness of the method of the present invention, D3QN is compared with DQN, dueling-DQN and double-DQN. The experimental results of the four different algorithms are shown in Figure 4 As shown in the figure, the initial values ​​of each algorithm are not much different, but as the number of iterations increases, D3QN obtains a larger QoS value than other algorithms. In terms of algorithm convergence, DQN converges at the 55000th iteration. th , double-DQN converges at the 48000th th Dueling-DQN and D3QN converge to 45000 respectively. th and 46000 th This shows that D3QN has the advantage of fast convergence speed of dueling-DQN. Compared with other algorithms, D3QN not only converges quickly, but also obtains better solutions. This is because the competition structure can improve the convergence of the algorithm, the dual estimator can overcome the disadvantage of over-estimation, and the priority replay mechanism can improve the learning speed of samples and enable the algorithm to learn quickly.

[0192] (3) Impact of strategies

[0193] In order to verify that the algorithm we proposed is suitable for dynamic environments, we simulated dynamic environments by randomly disabling a certain proportion of services. And in order to make the algorithm achieve the best performance, we studied the impact of different strategies on the algorithm. They are D3QN (no reward, ε-greedy strategy and heuristic strategy), NS-D3QN (no ε-greedy strategy, but with rewards and heuristic strategies), and Multi-D3QN (with rewards, ε-greedy strategy and heuristic strategies). In particular, a reward means that the reward value is equal to the QoS value of the selected service after performing an action, and no reward means that the reward value is equal to zero. The changes in these strategies start from disabling the service. Specific approach: When the algorithm reaches a stable state, 10% of the services are randomly disabled in a certain period of time. The present invention sets this stage to 60,000. th -80000 th .

[0194] The experimental results are as follows Figure 5 The results show that Multi-D3QN performs best in this experiment. Similarly, NS-D3QN and D3QN have the second and worst results, respectively. A closer look reveals that the QoS values ​​of all algorithms decrease when the service is unavailable. However, at 80,000 thWhen the number of nodes is 60000, Multi-D3QN and NS-D3QN return to a stable state, while D3QN does not. This indicates that the heuristic strategies in Multi-D3QN and NS-D3QN algorithms shield unavailable services, making the selected service chain free of unavailable services, thereby improving the QoS value. It can also be observed that at 60000 th Multi-D3QN obtains a higher QoS value than NS-D3QN because the ε-greedy strategy can give the algorithm a chance to jump out of the local optimum. th --80000 th ), the algorithms with rewards (Multi-D3QN and NS-D3QN) can find higher solutions than those without rewards (D3QN), because DLR can effectively adjust parameters according to the true reward value. Through experiments, it can be concluded that Multi-D3QN with multiple strategies has better performance.

[0195] (4) Adaptability and robustness

[0196] In order to verify the adaptability and robustness of the proposed algorithm, Multi-D3QN is compared with other algorithms. In this experiment, the experimental setting is the same as that of experiment (3). Figure 6 The experimental results of Multi-D3QN, double-DQN, DQN and dueling-DQN are shown in Figure 2. th Before), DQN has the best QoS value, while Multi-D3QN, double-DQN and dueling-DQN get the second, third and worst QoS values ​​respectively. However, when the service is unavailable (60000 th -80000 th ), Multi-D3QN has a higher QoS value than the other algorithms. This is because the reward strategy and ε-greedy strategy in Multi-D3QN enable the algorithm to find a better solution. Dueling-DQN obtains the worst QoS value in the whole process, indicating that dueling-DQN has an over-estimation problem. Double-DQN obtains the second highest QoS value, which confirms that the dual estimator can overcome the over-estimation problem and find a better solution. The QoS value of DQN decreases the most, indicating that DQN is very unstable when the environment changes. After a period of time, Multi-D3QN returns to a stable state, while the other three algorithms do not have this change. The reason is that Multi-D3QN triggers a heuristic strategy to block unavailable services in the service chain. In addition, from Figure 6As can be seen, the fluctuation range of Multi-D3QN is also smaller than that of double-DQN, DQN and dueling-DQN, which proves that our method is robust.

[0197] In this invention, we proposed a Multi-D3QN model by combining the basic DQN algorithm, a competitive structure, a dual estimator, and a priority replay mechanism. Most importantly, we added some strategies to the model, such as reward strategy, ε-greedy strategy, and heuristic strategy, to improve the performance of the algorithm. Specifically, the heuristic strategy can shield unavailable services, overcoming the shortcoming that DLR cannot recover to a stable state due to unreasonable hyperparameter and reward design. Rewards allow DLR to effectively adjust parameters according to the true reward value, and the ε-greedy strategy gives the algorithm a chance to jump out of the local optimum. Experiments show that the model not only has the advantage of strong adaptability to dynamic environments, but also finds better solutions than other deep reinforcement learning methods such as DQN, duelingDQN, and double DQN.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A cloud manufacturing service composition method based on multi-strategy deep reinforcement learning, characterized by: The steps include: S1: Obtain a virtual dataset through existing methods, which consists of multiple services; Assume that a task T consists of n subtasks, one subtask corresponds to a service set, one service in the service set corresponds to an action, and each service has multiple QoS attribute values, and the QoS attribute value is any data value between [0.7, 0.95]; definition: T={t1,t2,…t i …t n }, t i represents the i-th subtask of T, i = 1, 2, ... n; Represents subtask t i The corresponding μth service, μ=1,2,…c; Represents subtask t i The action corresponding to the μth service; Represents subtask t i a corresponding μth service QoS attribute value, where the μth service QoS attribute value is obtained by normalizing multiple QoS attribute values ​​corresponding to the μth service; m1 is the arithmetic mean of all service QoS values ​​in the service chain obtained by the current task T; m2 is the arithmetic mean of all service QoS values ​​in the service chain obtained by the current task T after the interval setting time; The process of normalizing multiple QoS attribute values ​​corresponding to the μth service to obtain the QoS attribute value of the μth service is as follows: The QoS attribute value corresponding to the μth service includes a positive attribute value and a negative attribute value; The positive attribute value corresponding to the μth service The normalized calculation formula is as follows: The negative attribute value corresponding to the μth service The normalized calculation formula is as follows: in, and They represent the minimum QoS attribute value and the maximum QoS attribute value among multiple QoS attribute values ​​of the μth service, respectively. μ is the aggregate value of multiple QoS attribute values ​​of the μth service; S2: Construct a Multi-D3QN model, modify the network structure of the DQN algorithm to a competitive structure, modify the random sampling mechanism of the DQN algorithm to a priority replay mechanism, and modify the single estimator to a dual estimator in the learning phase; The Multi-D3QN model also includes a Q-network model and a Q-target model; S3: Initialize the parameters of the Multi-D3QN model, initialize m1=m2=0, initialize the experience pool D, set the priority, initialize the action value function Q of the model Q-network, and initialize the action value function Q of the model Q-target * ; S4: Randomly select a task T from the virtual dataset; S5: let k=1; S6: let i=1; S7: If m1>m2, execute S8-1; otherwise execute S8-4; S8-1: Select t i The best action The steps are as follows: Determine t i Corresponding service set QoS attribute values ​​of all services in Is it less than 0? If t i The corresponding QoS attribute value of the μth service Less than 0, it means that there is an unavailable service and the next step is executed; if t i The corresponding QoS attribute value of the μth service If it is greater than or equal to 0, execute S8-4; S8-2: Randomly generate a probability x1, ε1 represents the preset probability, if x1<ε1, then the quasi-optimal action And execute the next step, where θ represents the parameters of the action value function Q; Otherwise Randomly select an action As a quasi-optimal action And execute the next step; S8-3: Determine the quasi-optimal action Whether the QoS attribute value of the corresponding service is less than 0: If the quasi-optimal action If the QoS attribute value of the corresponding service is less than 0, it will be blocked The corresponding service and randomly select an action as Otherwise, the action As and execute S8; S8-4: Randomly generate a probability x2, ε2 represents the preset probability: If x2<ε2, then the optimal action is And execute the next step, where θ represents the parameters of the action value function Q; Otherwise Randomly select an action As a quasi-optimal action S9: Execution And t i Corresponding service set t i QoS attribute values ​​of all services in the corresponding service set and t i The next service set As a sample Stored in D, the sample is represented as S10: Determine whether the samples in D have reached the maximum capacity. If not, proceed to the next step. Otherwise, proceed to S12. S11: Determine whether i is greater than n, if yes, execute the next step, otherwise set i=1+1 and return to S8; S12: Determine whether j is greater than the maximum number of iterations. If so, output Each corresponding service is taken as the optimal service and output and a service chain consisting of optimal services; Otherwise, let k=k+1, calculate m1 and m2, update m1 and m2 and return to S6; S13: Use the samples obtained in S8 to train the model Q-network and return to S10.

2. The cloud manufacturing service composition method based on multi-strategy deep reinforcement learning as claimed in claim 1, characterized in that: The process of training the model Q-network in S13 is as follows: S13-1: Set the initial priority of all samples in D S13-2: Let c=1; S13-3: Sample b samples in a small batch from D according to the sampling probability. The sampling probability of each sample is calculated using the following formula: Among them, p(h j ) represents the probability of sampling the jth sample in a total of b samples obtained by mini-batch sampling, p_h j Indicates the priority of the jth sample in a total of b samples obtained by mini-batch sampling; S13-4: Input b samples into Q-network and Q-target, and train Q-network and Q-target; S13-5: let j=1; S13-6: Calculate the weight w(h of the jth sample j ); S13-7: If j>b, let y j =r j , calculate the loss loss of the jth sample j , according to the loss loss j Update the parameters θ of the Q-network and proceed to the next step; Otherwise, y is calculated by the dual estimator j , calculate the loss loss of the jth sample j , according to the loss loss j Update the parameter θ of the Q-network, set j=j+1 and return to S13-6; y j Represents the estimated function value obtained by inputting the j-th sample into Q-target; r j represents the QoS attribute value of the service corresponding to the optimal action in the jth sample; The loss of the jth sample is calculated using the following formula: j , Among them, Q(S j ,a j ,θ) represents the function value obtained by inputting the jth sample into Q-network, and the function value is taken as the true value of the jth sample, S j represents the service set corresponding to the optimal action in the jth sample, a j The optimal action in the jth sample, θ represents the parameters of the Q-network; S13-8: Calculate the loss of the jth sample in the cth iteration The formula is as follows: According to the loss Update the priority of the jth sample, the formula is as follows: p_h j ←|d j | (10); S13-9: If c>c max , update the parameters θ in the Q-target network with the parameters in the Q-network * ; Otherwise, let c=c+1 and return to S13-2.

3. The cloud manufacturing service composition method based on multi-strategy deep reinforcement learning as claimed in claim 2, characterized in that: The process of calculating the weight of each sample in the b samples in S13-6 is as follows: Among them, w(h j ) represents the weight of the jth sample, w i represents the weight of the i-th sample, max i (w i ) represents the maximum sample weight β represents an exponential factor.

4. The cloud manufacturing service composition method based on multi-strategy deep reinforcement learning as claimed in claim 3, characterized in that: In S13-7, y is calculated by a dual estimator. j The process is as follows: y j =r j +γQ * (s j+1 ,argmax a Q(s j+1 ,a,θ),θ * ) (12); Among them, a represents "s j+1 ” corresponds to the action set, and γ represents the discount rate.

Images