A component service assembly integration method and system based on reinforcement learning

Abstract

The present invention provides a reinforcement learning component service assembly integration method and system, the method comprising the following steps: obtaining a target workflow, constructing a dynamic service composition model according to subtasks in the workflow; determining a first quality indicator according to the current service quality of a candidate service corresponding to the subtask, the first quality indicator comprising availability, response time and cost value; constructing a reward function; determining a target service composition through the dynamic service composition model and the reward function; predicting the expected service quality of the target service composition in a dynamic environment through a time series prediction model and reinforcement learning; optimizing and adjusting the target service composition based on the predicted expected service quality, and outputting a target action selection strategy according to the optimized and adjusted target service composition; the scheme takes into account the service quality in a dynamic environment, improves the accuracy of service selection and improves the adaptability, and can be widely used in the field of computer technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a component service assembly integration method and system for reinforcement learning. Background Art

[0002] Manufacturing service collaboration based on the industrial Internet is oriented to service-oriented manufacturing, and has the characteristics of hardware and software "interoperability". It is of great significance to promote enterprise digital transformation, management innovation and technological innovation. Network technology has also had a huge impact on software development in many application fields. More and more enterprises and organizations deliver their software, systems, computing resources, storage resources, etc. in the form of Web services for other users. Therefore, how to effectively integrate various services has become a fundamental research issue. Service composition mainly studies how to produce a system that can meet complex requirements by combining existing services. It is of great significance in the development of modern software systems. In addition, services with the same functions can be provided by different service providers with different service qualities (QoS). Service quality is mainly used to represent the non-functional attributes of services, including price, availability, reliability, response time, reputation, throughput, etc. Due to the dynamic changes in the network environment, the fluctuations in the performance of the business itself, and the changes in user access methods, the service quality may also change dynamically over time. Therefore, the service composition method needs to adapt to the dynamically changing environment. The service composition method plays an important role in reducing the cost and risk of providing new Web service applications in large-scale or small-scale IoT environments. Furthermore, service composition is specified as NP-hard (i.e., non-deterministic polynomial time), so the problem of finding the best service composition becomes a significant challenge in manufacturing service collaboration. In addition, cloud applications and smart devices contain many candidate components and resources to provide the requested services. Therefore, when the number of resources and IoT devices increases, the set of available composite services also grows exponentially. Therefore, it is impractical to select and recommend the best candidate service composition in the state space to support the service level agreement (SLA) in the service composition. Summary of the invention

[0003] In view of this, in order to at least partially solve the above-mentioned technical problems or one of the defects, the purpose of an embodiment of the present invention is to provide a component service assembly integration method of reinforcement learning, which can be applicable to a larger state space, has strong adaptability, and solves the service combination problem in a large-scale dynamic environment.

[0004] On the one hand, the technical solution of the present application provides a component service assembly integration method for reinforcement learning, comprising the following steps:

[0005] Obtaining a target workflow, and constructing a dynamic service composition model according to subtasks in the workflow;

[0006] Determine a first quality indicator according to the current service quality of the candidate service corresponding to the subtask, wherein the first quality indicator includes availability, response time and cost value;

[0007] constructing a reward function based on the availability, the response time, and the cost value;

[0008] Determine the target service combination by using the dynamic service combination model and the reward function;

[0009] Predicting the expected service quality of the target service combination in a dynamic environment by means of a time series prediction model and reinforcement learning;

[0010] The predicted expected service quality is used to optimize and adjust the target service combination, and a target action selection strategy is output according to the optimized and adjusted target service combination.

[0011] In a feasible embodiment of the solution of the present application, the step of obtaining the target workflow and constructing a dynamic service composition model according to the subtasks in the workflow includes:

[0012] A candidate service set obtained by grouping the candidate services according to the functional categories of the candidate services;

[0013] A first sequence of subtasks is obtained by screening the candidate service set according to an abstract description of the target business rule, and the target workflow is determined according to the first sequence.

[0014] In a feasible embodiment of the solution of the present application, determining the first quality indicator according to the current service quality of the candidate service corresponding to the subtask includes:

[0015] Obtaining a maximum value of the current service quality, a minimum value of the current service quality, and an initial value of the current service quality corresponding to the candidate service;

[0016] The first quality indicator is obtained by performing standardization processing according to the maximum value, the minimum value and the initial value.

[0017] In a feasible embodiment of the present application, the training process of the time series prediction model includes:

[0018] Acquire a number of historical service combinations, normalize the historical service combinations, and divide the normalized results to obtain a number of training data sets;

[0019] Constructing an LSTM neural network through the initialized particle swarm parameters, inputting the training data set into the LSTM neural network, and calculating the fitness function value of the particle swarm individuals;

[0020] Iteratively updating the individual optimal position and the global optimal position of the particle swarm according to the fitness function value;

[0021] The individual optimal position and the global optimal position are mapped to the parameters of the LSTM neural network to obtain the trained time series prediction model.

[0022] In a feasible embodiment of the present application, the LSTM neural network is constructed by using the initialized particle swarm parameters, the training data set is input into the LSTM neural network, and the fitness function value of the particle swarm individuals is calculated, including:

[0023] Determine the mean square error between the actual value of the data in the training data set and the predicted value output by the LSTM neural network;

[0024] The fitness function of the particle swarm optimization algorithm is constructed according to the mean square error, and the fitness function value of the particle swarm individual is calculated according to the fitness function.

[0025] In a feasible embodiment of the present application, the calculation formula of the fitness function is as follows:

[0026]

[0027] Among them, L represents the fitness function value, x t represents the actual value at time t, x t ′ represents the predicted value at time t, and N is the number of data points in the training set.

[0028] In a feasible embodiment of the solution of the present application, the expected service quality of the target service combination in a dynamic environment is predicted by a time series prediction model and reinforcement learning, including:

[0029] Obtaining a first predicted value of a target service in the target service combination under a target state through prediction by the time series prediction model;

[0030] The reward value of the target service is linearly superimposed according to the target state and the first predicted value to obtain the expected service quality; the reward value is calculated and determined by the reward function.

[0031] In a feasible embodiment of the present application, the training process of the time series prediction model further includes:

[0032] Associating the expected service quality with the target service combination, and updating the associated data to the historical data set;

[0033] It is determined that the updated data in the historical data set meets a preset update threshold, and the time series prediction model is retrained.

[0034] In a feasible embodiment of the solution of the present application, the step of obtaining the target workflow and constructing a dynamic service composition model according to the subtasks in the workflow includes:

[0035] Obtaining a finite state set, an initial state, and a terminal state of the target workflow;

[0036] Obtaining a finite combination of behaviors in each state in the finite state set;

[0037] A sextuple is constructed according to the probability distribution function, the reward function, the finite state set, the initial state, the terminal state and the finite behavior combination, and a dynamic service composition model is determined according to the sextuple.

[0038] On the other hand, the technical solution of the present application also provides a component service assembly integration system for reinforcement learning, the system comprising:

[0039] A service acquisition unit, used to acquire a target workflow and construct a dynamic service composition model according to subtasks in the workflow;

[0040] An indicator calculation unit, configured to determine a first quality indicator according to a current service quality of the candidate service corresponding to the subtask, wherein the first quality indicator includes availability, response time and cost value;

[0041] A reward function unit, configured to construct a reward function according to the availability, the response time and the cost value;

[0042] A service composition unit, configured to determine a target service composition through the dynamic service composition model and the reward function;

[0043] A quality prediction unit, used to predict the expected service quality of the target service combination in a dynamic environment by means of a time series prediction model and reinforcement learning;

[0044] The target output unit is used to optimize and adjust the target service combination according to the predicted expected service quality, and output a target action selection strategy according to the optimized and adjusted target service combination.

[0045] The advantages and beneficial effects of the present invention will be partially given in the following description, and the other parts can be understood through the specific embodiments of the present invention:

[0046] The technical solution of the present application provides a reinforcement learning component service assembly integration method, which adopts a dynamic service composition model and simultaneously adopts a neural network and reinforcement learning. On the one hand, the dynamic service composition process is modeled as a decision process (MDP), which is regarded as a decision process optimization problem under an uncertain decision environment; on the other hand, a recursive neural network algorithm (Long Short Term Memory, LSTM) network is introduced to predict the QoS value of the service, and finally the reinforcement learning method Q-learning is used to select the best service composition strategy, which takes into account the service quality in a dynamic environment, improves the accuracy of service selection and improves the adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flowchart of the steps of a component service assembly integration method for reinforcement learning provided in the technical solution of this application;

[0049] Figure 2 This is a schematic diagram of the time prediction sequence architecture based on LSTM in the technical solution of this application;

[0050] Figure 3 This is the flow chart of the LSTM-PSO algorithm in the technical solution of this application;

[0051] Figure 4 A schematic diagram of the process of solving the optimal service combination in the technical solution of this application;

[0052] Figure 5 This is a simple example of service combination in the technical solution of this application;

[0053] Figure 6 A schematic diagram of a possible service combination in the technical solution of this application;

[0054] Figure 7 This is a schematic diagram of another possible service combination in the technical solution of this application. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] As pointed out in the background technology, the related technical solutions use existing neural networks and machine learning algorithms to find the best service combination. However, there are also some challenging problems with these methods, such as state space explosion, maximizing reachability, and avoiding deadlock conditions. The commonly used particle swarm optimization (PSO) algorithm also has some problems, such as falling into local optimality and premature convergence. For example, a service combination model based on an exact subgroup particle swarm algorithm is proposed in the related technical solution. This method improves the accuracy of the standard particle swarm algorithm to find the optimal solution for the service combination; however, the method is determined by premature convergence and low diversity of the particle swarm algorithm. For another example, the related technical solution reduces the number of candidate services by using the skyline technology, thereby greatly reducing the difficulty of integer programming, and then uses integer programming to globally optimize the service combination; however, this method is only applicable to small-scale problems and lacks adaptive capabilities; it cannot solve the service combination problem in a large-scale dynamic environment.

[0059] In view of the shortcomings of the existing technology, firstly, Figure 1 As shown, the technical solution of the present application provides a component service assembly integration method of reinforcement learning, the method comprising steps S01-S06:

[0060] S01. Obtain a target workflow, and construct a dynamic service composition model according to subtasks in the workflow.

[0061] Specifically in the technical solution of the present application, the workflow of service composition, that is, the selection of each specific service, can be regarded as a Markov decision process. The input of reinforcement learning is a Markov decision process (MDP). The input of the reinforcement learning algorithm includes a set of possible states, executable actions in each state, obtainable reward values, and which new state can be transferred to after the action is executed. The output is a strategy for selecting actions based on the state. The goal is to maximize the expected value of the total reward value. Because the quality of service may change dynamically over time, the dynamic service composition problem (DSC) can be modeled as a Markov decision process, namely DSC-MDP.

[0062] More specifically, the dynamic service composition process in the embodiment can be regarded as a multi-stage decision optimization problem in an uncertain decision environment. Markov decision process (MDP), as a model of reinforcement learning, is a theoretical tool for studying the optimization problem of multi-stage decision processes in random environments, and is therefore particularly suitable for solving service composition problems. Accordingly, the embodiment models the dynamic service composition problem as an MDP, namely Dynamic Service Composition MDP (DSC-MDP). In the embodiment, a service is a functionally complete and self-managed resource that can be published, located and accessed. A service can be formalized as a triple (ID, Function, Record); it includes:

[0063] 1. ID: ID is the identifier of a service. ID uniquely identifies a service.

[0064] 2.Function: represents the functional class of the service. It is a triple (F, I, O), where F is the functional description of the service, I represents the input of the service, and O represents the output of the service.

[0065] 3. Record: A container that records the historical values ​​of service QoS.

[0066] Based on the fact that the services in the embodiments can be described in the form of triples, in some optional embodiments, a target workflow is obtained, and a dynamic service composition model is constructed according to the subtasks in the workflow. This step S01 may include steps S011-S012:

[0067] S011. Grouping the candidate services according to their functional categories to obtain a set of candidate services.

[0068] Specifically in the embodiment, the candidate service set is a set of candidate services that provide the same function but different QoS. In the embodiment, the services in the set have the same function but different QoS values. The candidate service set can be formalized as a set {cs1,cs2,…,cs m}, where m is the number of services with the desired Function, and cs i (i=1, 2, ..., m) is the id of the candidate service.

[0069] S012: Filter and obtain a first sequence of subtasks from the candidate service set according to the abstract description of the target business rule, and determine the target workflow according to the first sequence.

[0070] Specifically in the embodiment, the workflow is an abstract description of the business rules. A sequentially executed workflow can be formalized as a sequence (t1, t2, ..., t n ), where t i (i=1,2,...,n) is the i-th abstract task, and n is the total number of tasks.

[0071] Furthermore, the dynamic service composition model DWS-MDP in the embodiment can be regarded as a six-tuple. In the step S01 of acquiring the target workflow and constructing the dynamic service composition model according to the subtasks in the workflow, the following steps can be included:

[0072] S013, obtaining a finite state set, an initial state, and a terminal state of the target workflow;

[0073] S014, obtaining a finite behavior combination in each state in the finite state set;

[0074] S015. Construct a sextuple according to the probability distribution function, the reward function, the finite state set, the initial state, the terminal state and the finite behavior combination, and determine a dynamic service composition model according to the sextuple.

[0075] Specifically in the embodiment, a DWS-MDP is a 6-tuple DWS-MDP= <S,S0,S t ,A(·),P,R>, where S represents a finite state set; S0 represents the initial state, that is, the workflow of the service composition starts from S0; S tRepresents the terminal state. When a terminal state is reached, the process of a service composition is terminated; A(·) represents a finite combination of behaviors, A(s) represents the set of behaviors that can be executed in state s∈S; P represents the probability distribution function. When a behavior is executed, the probability of transitioning from s to s' is marked as P(s'|s,a); R is the immediate reward function. In the current state s, and a behavior a is selected, then after executing a behavior, an immediate reward r=R(s,a) can be obtained from the environment.

[0076] S02. Determine a first quality indicator according to the current service quality of the candidate service corresponding to the subtask; wherein the first quality indicator includes availability, response time and cost value.

[0077] Specifically in the embodiment, in the service composition, the satisfaction of calling the service is usually determined by QoS. Therefore, the reward function in this embodiment will be determined by the QoS attributes of the service. Since different QoS attributes may have different value ranges, different attributes need to be normalized in the embodiment; first, a QoS indicator based on a normalization method is proposed, that is, the first quality indicator is defined, and the final candidate service combination is identified based on the aggregation of QoS indicators. Three representative QoS indicators are selected, including availability, response time and cost. In order to calculate the total QoS indicator of each candidate combination service, the QoS indicator is normalized.

[0078] Regarding the normalization process of the QoS indicator, in some feasible implementations, the step S02 of determining the first quality indicator according to the current service quality of the candidate service corresponding to the subtask may include steps S021-S022:

[0079] S021. Obtaining a maximum value of the current service quality, a minimum value of the current service quality, and an initial value of the current service quality corresponding to the candidate service;

[0080] S022. Perform standardization processing according to the maximum value, the minimum value and the initial value to obtain the first quality indicator.

[0081] Specifically in the embodiment, the definitions of various parameters are shown in Table 1:

[0082] Table 1

[0083]

[0084]

[0085] Among them, the QoS indicators are standardized, referring to the following calculation formula:

[0086]

[0087]

[0088]

[0089] in, is the maximum value of each QoS metric among all candidate services, is the minimum value of each QoS metric among all candidate services, It is shown as the initial value of the attribute Att of candidate service j in task i, and n represents the total number of tasks in the service composition. is the normalized value of each QoS metric of candidate service j for task i.

[0090] S03. Construct a reward function according to the availability, the response time and the cost value.

[0091] Specifically in the embodiment, as shown in formula (4) and formula (5), the total QoS value of the candidate service combination is defined as the sum of the normalized QoS metrics of the candidate services selected by each task. In addition, formula (5) expresses the fitness function of the final candidate service, i.e., the reward value:

[0092]

[0093] R QoS =∑V(Av)W+∑V(Rt)W+∑V(Pr)W (5)

[0094] Where W represents the assigned weights of different service quality metrics; V is the QoS metric (reward value) of each selected candidate service in task i.

[0095] S04: Determine a target service combination through the dynamic service combination model and the reward function.

[0096] Specifically in the embodiment, in order to meet some complex functional requirements, according to certain business logic, services with different functions are integrated into a scalable, loosely coupled application, namely, service composition. Service composition can be formalized as a sequence (ws1, ws2, ..., ws n )Where ws i (i=1, 2, ..., n) are respectively the service IDs constituting the combined service, and n is the number of services constituting the combined service.

[0097] For a workflow composed of services, the embodiment can use the DSC-MDP model to fully describe it. After the workflow is determined, the service composition process starts from the initial state, selects specific services for each state, and then enters a new state. When a terminal state is reached, the workflow composed of services is completed. This workflow composed of multiple specific services is the result of service composition. A preferred service composition result should be able to make the total reward value as large as possible.

[0098] S05. Predicting the expected service quality of the target service combination in a dynamic environment by means of a time series prediction model and reinforcement learning.

[0099] Specifically in the embodiment, a time series prediction model based on a neural network can be used to predict future service quality. In a dynamic network environment, the service quality often changes over time, and the reinforcement learning algorithm cannot estimate the changing trend of the service quality. The embodiment is based on the method of LSTM time series prediction and particle swarm optimization algorithm to help estimate the service quality in a dynamic environment.

[0100] In the embodiment, the LSTM neural network structure includes:

[0101] (1) Input layer: This layer is responsible for the input of external data. During the data processing stage, the data needs to be converted into data of corresponding dimensions.

[0102] (2) Hidden layer: Perform multiple transformations on the features of the input data.

[0103] (3) Output layer: This layer serves as the output layer of the data, that is, it outputs the prediction result of the Q value.

[0104] The basic idea of ​​time series prediction is to use past data to predict future service quality. This embodiment uses LSTM neural network to predict time series, which can be expressed as:

[0105] x t =LSTM(x t-1 ,x t-2 ,x t-3 ,…,x0;θ) (6)

[0106] Among them, x t Represents the predicted data for the next time step, LSTM(x t-1 ,x t-2 ,x t-3,…,x0;θ) represents an LSTM neural network with parameter θ. The input of the neural network is all past data. In the embodiment, it is often difficult to use all past data for prediction. Therefore, the input of the LSTM neural network is actually the data of the first n time steps. In addition, the LSTM neural network can predict more than one time step. Therefore, the prediction of the LSTM neural network can be expressed by formula (7):

[0107] x t+m-1 ,…x t+1 ,x t =LSTM(x t-1 ,x t-2 ,x t-3 ,...,x t-n ;θ) (7)

[0108] Where m is the number of time steps for prediction and n is the number of time steps for input. Figure 2 As shown, after determining the input and output of the neural network, the LSTM neural network can be constructed; Figure 2 shows the expansion of an LSTM neural network in time; that is, Figure 2 The neural network represented by module A is the same neural network, and data is passed from left to right in chronological order.

[0109] Specifically in the embodiment, Figure 3 As shown, the training process of the time series prediction model in the embodiment includes steps S051-S054:

[0110] S051. Acquire several historical service combinations, normalize the historical service combinations, and divide the normalized results to obtain several training data sets;

[0111] S052, constructing an LSTM neural network through the initialized particle swarm parameters, inputting the training data set into the LSTM neural network, and calculating the fitness function value of the particle swarm individual;

[0112] S053, iteratively updating the individual optimal position and the global optimal position of the particle swarm according to the fitness function value;

[0113] S054. Map the individual optimal position and the global optimal position to the parameters of the LSTM neural network to obtain the trained time series prediction model.

[0114] Specifically in the embodiment, during the training process of the model, the LSTM model has some important parameters that need to be determined, including but not limited to: learning rate, number of iterations and number of neurons in the hidden layer. Among them, the learning rate and number of iterations have a greater impact on the training effect of the model, and the number of neurons in the hidden layer plays a decisive role in the degree of fit of the model. These parameters are generally difficult to determine, and are parameters that must be determined before the model starts training, rather than parameters determined by the quality of training. The model designer needs to decide based on experience, which has greater randomness, and the prediction accuracy of the model is not high. Therefore, a global search algorithm can be used to determine the model parameters, because the PSO algorithm has a good global search capability, so the PSO algorithm can be used to optimize the parameters of the LSTM model. The following is the search process of the PSO algorithm for the parameters of the LSTM neural network.

[0115] The step of constructing an LSTM neural network through the initialized particle swarm parameters, inputting the training data set into the LSTM neural network, and calculating the fitness function value of the particle swarm individuals may include steps S0521-S0522:

[0116] S0521. Determine the mean square error between the actual value of the data in the training data set and the predicted value output by the LSTM neural network;

[0117] S0522. Construct a fitness function of the particle swarm optimization algorithm according to the mean square error, and calculate the fitness function value of the particle swarm individual according to the fitness function.

[0118] Specifically in the embodiment, it is necessary to initialize the particle swarm parameters first; determine the size of the population, the number of iterations, the search dimension, the initial inertia weight, and the learning factor. Then, initialize the position and speed of the particles; that is, the algorithm is initialized to a randomly generated solution. After that, determine other parameters of the LSTM, for example: the time step is the relationship between each input data and the number of previously input data in the LSTM neural network, which characterizes the correlation of the data. After that, the embodiment needs to determine the fitness function of the particle swarm algorithm; construct the LSTM neural network with the initialized particle swarm parameters, and use the mean square error (MSE) of the measured value and the predicted value of the training set as the fitness function of the PSO algorithm, and the calculation formula is as follows:

[0119]

[0120] Among them, L represents the fitness function value, x t represents the actual value at time t, x t ′ represents the predicted value at time t, and N is the number of data points in the training set.

[0121] After constructing the fitness function, iterate according to the fitness function value, calculate the position of the particle in each iteration and calculate the fitness value. By comparing with the fitness value of the initial position, determine the individual optimal position pbest, and then determine the group optimal position gbest. When the iteration stops, map the global optimal particle position to the number of iterations, learning rate, and number of neurons in the hidden layer of the LSTM neural network to obtain the optimal parameters. Finally, use the model for prediction; use the optimal parameters determined by the PSO algorithm to train the LSTM neural network. After reaching the maximum number of iterations, input the data into the trained model for prediction and output the final prediction result.

[0122] In some feasible implementations, in the embodiment, in step S05 of predicting the expected service quality of the target service combination in a dynamic environment by means of a time series prediction model and reinforcement learning, the reinforcement learning algorithm may adopt a Q-learning algorithm. Based on this, step S05 in the technical solution of the present application may include steps S055-S056:

[0123] S055. Predicting by the time series prediction model to obtain a first predicted value of the target service in the target service combination under the target state;

[0124] S056. Linearly superimpose the reward value of the target service according to the target state and the first predicted value to obtain the expected service quality; the reward value is calculated and determined by the reward function.

[0125] Specifically in the embodiment, the basic idea of ​​the Q-learning algorithm is to use a Q-value table to estimate the reward generated by an action, where the Q value is an estimate of the actual reward. When the actual value and the estimated value are different after an action is performed, the Q-value table will be updated according to the difference between the two values. The Q value can gradually approach the actual value. After a certain number of iterations, the Q-value table can have a more accurate estimate of the true reward value. First of all, in the service combination, the quality of the candidate service is not only related to the QoS attributes of the service itself, but also to the subsequent workflow generated after the service is selected. For example, although a service has good QoS attributes, in the subsequent process of selecting this service, the QoS may be poor, resulting in poor quality of the overall service composition. Therefore, when estimating QoS, we use the Q-learning algorithm and use Q value to represent the QoS in the service composition. According to the Q value formula:

[0126] Q real (s,a)=R(s′|s,a)+γmaxQ(s′,a′) (9)

[0127] Among them, γ represents the decay rate, γ∈[0,1], and the decay rate is used to discount the future reward value, so that the short-term reward is greater than the long-term reward, and the influence of the recent reward plays a more important role in the action selection. In the embodiment, γ is a hyperparameter. The Q value is actually the linear superposition of each service reward value, so it can also be estimated by the time series prediction method. In addition, if the Q value is estimated by the LSTM neural network, formula (9) needs to be rewritten as follows:

[0128] Q real (s,a)=R(s′|s,a)+γmaxLSTM(s′,a′) (10)

[0129] Among them, LSTM(s′,a′) represents the predicted value of the LSTM neural network corresponding to the state s′ and service a′, which is the Q-value estimation of the state-action pair (s′,a′).

[0130] S06: Optimize and adjust the target service combination based on the predicted expected service quality, and output a target action selection strategy according to the optimized and adjusted target service combination.

[0131] Specifically in the embodiment, an adaptive service composition method combined with QoS prediction is proposed. First, a typical service composition scenario is described to help understand the problem of service composition. In the service composition, appropriate candidate services are selected through reinforcement learning to maximize the total QoS value of the composite service. Considering the dynamics and complexity of the network environment, QoS is always changing, so the embodiment introduces a QoS prediction method based on LSTM. Finally, we combine QoS prediction and PSO algorithm to solve the service composition problem, which can improve the quality of composite services in dynamic environments.

[0132] In some feasible implementations, the training process of the time series prediction model in the embodiment may further include steps S057-S058:

[0133] S057, associating the expected service quality with the target service combination, and updating the associated data to the historical data set;

[0134] S058. Determine whether the updated data in the historical data set meets a preset update threshold, and retrain the time series prediction model.

[0135] Specifically in the embodiment, Figure 4As shown in the figure, as the service composition algorithm runs, new data will be continuously added to the historical data set. Generally speaking, neural networks are trained in a fixed training set and then applied to new data. However, during the service composition process, historical data will gradually accumulate, so incremental neural network training is required. If a certain amount of new data is added, neural network training should be performed. In addition, as historical data accumulates, this data may take up a lot of space and also affect the efficiency of neural network training. Therefore, it is necessary to limit the size of the historical data set and discard early historical data, which can improve the efficiency of neural network training.

[0136] The complete implementation process of the technical solution of this application is described below in conjunction with the accompanying drawings of the specification:

[0137] The technical solution of the present application proposes an adaptive service composition method combined with QoS prediction. First, a typical service composition scenario is described to help understand the problem of service composition. In the service composition, appropriate candidate services are selected through reinforcement learning to maximize the total QoS value of the composite service. Considering the dynamics and complexity of the network environment, QoS is always changing, so the embodiment introduces a QoS prediction method based on LSTM. Finally, the embodiment combines QoS prediction and PSO algorithm to solve the service composition problem, which can improve the quality of composite services in a dynamic environment.

[0138] Reference Figure 5 For each abstract service, we need to determine its specific service and finally develop a specific service composition workflow. Figure 5 The service composition workflow, possible service composition results are as follows Figure 6 and Figure 7 As shown in Figure 2, it is straightforward to see that the number of possible service composition results increases dramatically as the number of candidate services increases and the complexity of the service composition workflow increases. The challenge of service composition is to examine a large candidate space and get the best result that maximizes the reward value.

[0139] For a workflow composed of services, the DSC-MDP model can be used to fully describe it. Figure 5Take the simple workflow in as an example. This workflow can be described by the DSC-MDP model, where the state set is S = {S0, S1, S2, S3, S4}, the initial state is S0, and the terminal state is S4. The services that can be called in different states include A(S0) = {a0, a1} and A(S1) = {a2, a3, a4, a5}, etc. The probability distribution function is P(S1|S0, a0) = 1 (after executing action a0, the probability of transitioning from state S0 to state S1 is 1), P(S2|S1, a2) = 1, P(S3|S1, a4) = 1, etc. The reward value is calculated using the QoS obtained by calling the service.

[0140] There are two QoS estimation problems in service composition that need to be solved. First, in service composition, the quality of candidate services is not only related to the QoS attributes of the service itself, but also to the subsequent workflow generated after the service is selected. In addition, as the service composition algorithm runs, new data will be continuously added to the historical data set. Generally speaking, neural networks are trained in a fixed training set and then applied to new data. However, during the service composition process, historical data will gradually accumulate, so incremental neural network training is required. If a certain amount of new data is added, neural network training should be performed. In addition, as historical data accumulates, these data may take up a lot of space and also affect the efficiency of neural network training. Therefore, it is necessary to limit the size of the historical data set and discard the early historical data, which can improve the efficiency of neural network training.

[0141] More specifically, the code implementation of the LSTM-PSO-Qlearning algorithm in the embodiment is as follows:

[0142]

[0143] Algorithm termination (the predicted QoS value of each service can determine the best candidate service in each state to complete the optimal service combination)

[0144] On the other hand, the technical solution of the present application also provides a component service assembly integration system for reinforcement learning, the system comprising:

[0145] A service acquisition unit, used to acquire a target workflow and construct a dynamic service composition model according to subtasks in the workflow;

[0146] An indicator calculation unit, configured to determine a first quality indicator according to a current service quality of the candidate service corresponding to the subtask, wherein the first quality indicator includes availability, response time and cost value;

[0147] A reward function unit, configured to construct a reward function according to the availability, the response time and the cost value;

[0148] A service composition unit, configured to determine a target service composition through the dynamic service composition model and the reward function;

[0149] A quality prediction unit, used to predict the expected service quality of the target service combination in a dynamic environment by means of a time series prediction model and reinforcement learning;

[0150] The target output unit is used to optimize and adjust the target service combination according to the predicted expected service quality, and output a target action selection strategy according to the optimized and adjusted target service combination.

[0151] From the above specific implementation process, it can be concluded that the technical solution provided by the present invention has the following advantages or strengths compared with the prior art:

[0152] First, the technical solution of this application applies the time series prediction method to the field of service computing. A service quality prediction method based on recursive neural network is proposed. Secondly, the combination of reinforcement learning and service quality prediction can adapt to the dynamic network environment and is a self-optimizing and adaptive service combination method.

[0153] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0155] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0156] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0157] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A component service assembly integration method based on reinforcement learning, characterized in that: The following steps are involved: Obtaining a target workflow, and constructing a dynamic service composition model according to subtasks in the workflow; Determine a first quality indicator according to the current service quality of the candidate service corresponding to the subtask, wherein the first quality indicator includes availability, response time and cost value; constructing a reward function based on the availability, the response time, and the cost value; Determine the target service combination by using the dynamic service combination model and the reward function; Obtaining a first predicted value of a target service in the target service combination under a target state by predicting using a time series prediction model; According to the target state and the first predicted value, the reward value of the target service is linearly superimposed to obtain the expected service quality; the reward value is calculated and determined by the reward function; Optimizing and adjusting the target service combination based on the predicted expected service quality, and outputting a target action selection strategy according to the optimized and adjusted target service combination; The training process of the time series prediction model includes: Acquire a number of historical service combinations, normalize the historical service combinations, and divide the normalized results to obtain a number of training data sets; Constructing an LSTM neural network through the initialized particle swarm parameters, inputting the training data set into the LSTM neural network, and calculating the fitness function value of the particle swarm individuals; Iteratively updating the individual optimal position and the global optimal position of the particle swarm according to the fitness function value; Mapping the individual optimal position and the global optimal position to the parameters of the LSTM neural network to obtain the trained time series prediction model; The LSTM neural network is constructed by using the initialized particle swarm parameters, the training data set is input into the LSTM neural network, and the fitness function value of the particle swarm individuals is calculated, including: Determine the mean square error between the actual value of the data in the training data set and the predicted value output by the LSTM neural network; Constructing a fitness function of a particle swarm optimization algorithm according to the mean square error, and calculating the fitness function value of an individual particle swarm according to the fitness function; The calculation formula of the fitness function is as follows: in, represents the fitness function value, express The actual value of the moment, express The predicted value at time, is the number of data points in the training set; The step of acquiring the target workflow and constructing a dynamic service composition model according to the subtasks in the workflow includes: Obtaining a finite state set, an initial state, and a terminal state of the target workflow; Obtaining a finite combination of behaviors in each state in the finite state set; A sextuple is constructed according to the probability distribution function, the reward function, the finite state set, the initial state, the terminal state and the finite behavior combination, and a dynamic service composition model is determined according to the sextuple.

2. A component service assembly integration method for reinforcement learning according to claim 1, characterized in that: The step of obtaining a target workflow and constructing a dynamic service composition model according to subtasks in the workflow includes: A candidate service set obtained by grouping the candidate services according to the functional categories of the candidate services; A first sequence of subtasks is obtained by screening the candidate service set according to an abstract description of the target business rule, and the target workflow is determined according to the first sequence.

3. The component service assembly integration method of reinforcement learning according to claim 1, characterized in that: The determining the first quality indicator according to the current service quality of the candidate service corresponding to the subtask includes: Obtaining a maximum value of the current service quality, a minimum value of the current service quality, and an initial value of the current service quality corresponding to the candidate service; The first quality indicator is obtained by performing standardization processing according to the maximum value, the minimum value and the initial value.

4. The component service assembly integration method of reinforcement learning according to claim 1, characterized in that: The training process of the time series prediction model also includes: Associating the expected service quality with the target service combination, and updating the associated data to the historical data set; It is determined that the updated data in the historical data set meets a preset update threshold, and the time series prediction model is retrained.

5. A reinforcement learning component service assembly integration system, characterized in that: include: A service acquisition unit, used to acquire a target workflow and construct a dynamic service composition model according to subtasks in the workflow; An indicator calculation unit, configured to determine a first quality indicator according to a current service quality of the candidate service corresponding to the subtask, wherein the first quality indicator includes availability, response time and cost value; A reward function unit, configured to construct a reward function according to the availability, the response time and the cost value; A service composition unit, configured to determine a target service composition through the dynamic service composition model and the reward function; A quality prediction unit, configured to predict a first predicted value of a target service in the target service combination under a target state by using a time series prediction model; And according to the target state and the first predicted value, the reward value of the target service is linearly superimposed to obtain the expected service quality; the reward value is calculated and determined by the reward function; wherein, the training process of the time series prediction model includes: obtaining a number of historical service combinations, normalizing the historical service combinations, and dividing the normalized results to obtain a number of training data sets; constructing an LSTM neural network through the initialized particle swarm parameters, inputting the training data set into the LSTM neural network, and calculating the fitness function value of the individual particle swarm; iteratively updating the individual optimal position and the global optimal position of the particle swarm according to the fitness function value; mapping the individual optimal position and the global optimal position to the parameters of the LSTM neural network to obtain the trained time series prediction model; the quality prediction unit is also used to determine the mean square error between the actual value of the data in the training data set and the predicted value output by the LSTM neural network; constructing the fitness function of the particle swarm optimization algorithm according to the mean square error, and calculating the fitness function value of the individual particle swarm according to the fitness function; The calculation formula of the fitness function is as follows: in, represents the fitness function value, express The actual value of the moment, express The predicted value at time, is the number of data points in the training set; A target output unit, configured to optimize and adjust the target service combination according to the predicted expected service quality, and output a target action selection strategy according to the optimized and adjusted target service combination; The service acquisition unit is also used to obtain the finite state set, initial state and terminal state of the target workflow; obtain the finite behavior combination under each state in the finite state set; and construct a sextuple based on the probability distribution function, the reward function, the finite state set, the initial state, the terminal state and the finite behavior combination, and determine the dynamic service composition model based on the sextuple.

Images