Combined service dynamic reconstruction method and system based on deep reinforcement learning in cloud network convergence environment

Abstract

The invention discloses a combined service dynamic reconstruction method and system based on deep reinforcement learning in a cloud network convergence environment. According to the method, a cloud network service data set is constructed by collecting position information and edge server distribution information of cloud network users and fusing QoS data and server distribution of combined services. A user moving track is generated by using historical position information of a user accessing a base station, and a space-time diagram convolutional network model is constructed to predict a future position of the user. According to a user position, a candidate service set capable of being reconstructed is determined, a proper reconstruction decision is made for the combined service according to user requirements, and in consideration of the mobility of the user, the service of the reconstruction request needs to be unloaded to an edge server covering the future position range of the user to be executed. Due to the fact that the service selected by the reconstruction decision is executed locally or unloaded on different air-ground collaborative edge servers, the time delay cost, the energy consumption cost and the like are different, a deep reinforcement learning algorithm is designed and used for carrying out the reconstruction decision and the unloading decision on the combined service, and the influence brought by coupling between the two decisions is reduced. According to the combined service dynamic reconstruction method provided by the invention, cloud network service reconstruction decision and unloading decision can be balanced, and the service unloading cost is minimized while the user service quality is ensured.

Description

Technical Field

[0001] The present invention relates to a method and system for dynamically reconstructing combined services based on deep reinforcement learning in a cloud-network fusion environment, and belongs to the field of information technology. Background Art

[0002] In recent years, cloud-network integration has become a core feature of information infrastructure. Through technological innovation and breakthroughs, it provides ubiquitous connectivity and massive data storage and multi-modal, high-performance computing that integrates air, space, land, and sea, and also provides a large number of services for users in different scenarios of air, space, land, and sea. With the increasing maturity of cloud computing and mobile edge computing, various advanced application scenarios have increasing requirements for basic services. Relying on the powerful computing power of cloud computing and scalable network architecture, cloud-network integration can provide more flexible and adaptable services to support these diverse application needs. In order to meet the growing complex service needs of users, service composition technology has made great progress. By decomposing the complex service quality (QoS) requirements of users into simple sub-requirements, service composition technology is used to select qualified services from a large number of candidate services based on these service sub-requirements, so as to aggregate and build a more powerful composite service. However, user needs are dynamically changing, and frequent reorganization of composite services to meet their needs will consume a lot of time and resources. In most cases, only local modifications to the composite service are required. Therefore, a dynamic reconstruction method of composite services is proposed to ensure the continuity of services and avoid re-combining services.

[0003] As an extension of service composition, the research on dynamic service reconstruction is mainly aimed at avoiding frequent recombinations and saving time and resources. In addition to traditional SOA environments, existing dynamic reconstruction work has also provided different solutions to actual problems in cloud manufacturing, fog computing, e-commerce and other cloud-edge collaborative environments. Lin et al. searched for replaceable reconstruction areas through an iterative algorithm for failures that occurred during the runtime of SOA service processes, and replaced the faulty services and adjacent services in the area using one-to-one, one-to-many and other methods to ensure that the entire service process still maintains end-to-end constraints. In order to solve the problem of composite service failure handling in SOA environments, Shrivastava et al. continuously expanded the area of ​​faulty services, reconfigured the services in the area, and replaced the faulty services using the optimal path selection strategy. However, this method only targets single failures in composite services. When there are multiple failures, this method is not applicable. In the field of cloud manufacturing, Wang et al. focused on actual constraints such as the occupied time and dynamic changes in service quality generated by business abnormalities, proposed a dynamic service composition reconstruction model, and optimized the reconstruction algorithm using the Harris Hawk Optimizer HHO. In fog computing, He et al. modularized the basic operations of services from the perspective of resource conservation, thereby realizing the dynamic recycling of resources. Combining the content-aware capabilities of fog computing systems, service processing strategies for different contents are formulated, and the concept of weighted graphs is introduced to further optimize resource consumption during the reconstruction process. In the field of e-commerce, Gao et al. proposed a service selection method for service workflow reconstruction, generating one-to-one, one-to-many, and many-to-many reconstruction solutions from the two aspects of functional interface matching and QoS matching.

[0004] There have been many works on dynamic reconstruction of composite services, but they do not meet the needs of cloud-network convergence environments. In cloud-network convergence environments, user mobility and heterogeneity between edge servers bring new challenges to dynamic reconstruction of composite services. Summary of the invention

[0005] Purpose of the invention: The traditional dynamic reconstruction method of combined services does not take into account the mobility of users. As users move, they may not be within the coverage of the edge server, resulting in the inability to return the calculation results from the original server after the service selected by the reconstruction decision is unloaded to the edge server, thereby affecting the reconstruction effect of the combined service and reducing the user experience. At the same time, in a cloud-network fusion environment, edge servers are no longer limited to ground deployment. By installing edge servers on drones and aerial platforms, the delay time of the overall combined service can be more effectively reduced. Therefore, when dynamically reconstructing the combined service, it is necessary to consider the coordination of air-ground edge servers. The present invention proposes a dynamic reconstruction method for combined services based on deep reinforcement learning in a cloud-network fusion environment. The method can dynamically reconstruct the combined service in a cloud-network fusion environment to ensure that the service quality of the reconstructed combined service can still meet user needs and minimize the cost of service unloading.

[0006] Technical solution: To achieve the above invention objectives, the present invention provides a method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment. The technical solution is as follows:

[0007] Step 1: Collect cloud network user location information, edge server distribution, cloud network combined service QoS data, and server log files;

[0008] Step 2: Obtain the cloud network service set required for reconstruction by integrating the service input and output data volume and the combined service QoS data in the service log file;

[0009] Step 3: Obtain available resource information of edge servers and mobile devices, and build a cloud-network fusion environment system model and a composite service model. The cloud-network fusion environment system model includes an available edge server model that defines air-ground collaboration, and a cost model that includes latency and energy consumption; the composite service model defines the structure of the composite service, the input and output data size of each sub-service, the service type, and the execution time;

[0010] Step 4: Generate user dynamic trajectories using user historical location information and build an STGCN model to predict user future locations;

[0011] Step 5: Obtain a set of candidate edge servers for service offloading based on the predicted user location;

[0012] Step 6: Based on the reconstruction information requested by the user, the cloud network service set, and the mobile-aware candidate edge server set, the deep reinforcement learning DDPG algorithm is used to generate the composite service reconstruction decision and offloading decision to ensure that the QoS of the reconstructed composite service can still meet user needs and minimize the service offloading cost.

[0013] 2. Preferably, the collection of user location information, edge server distribution, combined service QoS data and server log files in step 1 specifically includes:

[0014] Step 11: Extract the geographical location information of the mode of transportation, date, time, longitude and latitude from the user's mobile trajectory data set, classify them according to different travel modes, and eliminate invalid user data that is at the same longitude and latitude position for most of the time;

[0015] Step 12: Collect the base station geographic location data set, extract the longitude, latitude, coverage and other information of the base station, and use it for the distribution of traditional edge servers on the ground;

[0016] Step 13: Collect the QoS data of the combined service based on the QWS dataset, which contains 9 types of QoS data of 2507 real services. Collect QoS data such as reliability, success rate, and throughput.

[0017] Step 14: Collect a server data set that contains 500 services deployed on 10 servers, and extract information about the service input and output data volumes.

[0018] 3. Preferably, in step 2, a service data set for dynamic reconstruction of combined services in a cloud-network fusion environment is constructed by fusing data sets, and corresponding QoS data such as reliability, success rate, and throughput are associated with each service according to the amount of input and output data of each service:

[0019] 4. Preferably, in step 3, obtaining the available resource information of the edge server and the mobile device, and constructing the cloud-network fusion environment system model and the combined service model include the following steps:

[0020] Step 31: The system model of the cloud-network fusion environment considered in the present invention mainly includes a mobile user set EU = {{eu 1 ,eu 2 ...,eu k}}, ground edge server set and EG = {eg 1 , e.g. 2 ..., e.g. n}, drone edge server set ES = {es 1 ,es 2 ...,es m}; define eg n =(fre n g , cap n g , core n g , pwn g , ban n g , c0v n g ), where fre n g Represents CPU frequency, cap n g Represents the storage capacity of the server, core n g Represents the number of available CPU cores, pw n g Represents the energy consumption coefficient, ban n g represents the bandwidth allocated to the edge server, cov n g Represents the coverage of the edge server, defining es m =(fre n m , cap n m , pw n m ,loc n m ), where loc n m Represents the three-dimensional coordinates of the drone.

[0021] Step 32: The composite service model considered in the present invention is defined as a triplet of CS = {ST, RT, S}, where ST represents the set of all abstract sub-services of the composite service, RT represents the set of relationships between the abstract sub-services, which usually include order, merge, branch and other relationships, and S represents the set of specific candidate services corresponding to each abstract sub-service. The candidate service set S of the i-th abstract sub-service is i = {I i n , O i n , Loc i n}, where I i n represents the input data size of the candidate service with index n, O i n Indicates the service output data size.

[0022] Step 33: The cost model of service offloading is mainly composed of local execution cost, ground edge server offloading cost and drone offloading cost.

[0023] Calculate user mobile device EU kTo calculate the local execution cost, we first need to calculate the local unloading delay of the service. The calculation formula is as follows:

[0024]

[0025] in, represents the computing power of the user's mobile device, which is usually related to the CPU frequency and the number of available cores. Therefore, the energy consumption of computing services offloaded locally is i local The formula is as follows:

[0026]

[0027] in, represents the energy consumption coefficient of the user's mobile device. Therefore, the calculation formula of the local execution cost is as follows:

[0028]

[0029] where σ 1 , σ 2 are the weight factors of delay and energy consumption respectively, and their sum is 1.

[0030] Computing ground servers n To calculate the offloading cost, we first need to calculate the delay time of offloading to the edge server, which mainly includes two parts: one is the time for service data to be transmitted to the edge server, and the other is the execution time of the computing task after offloading to the edge server. The calculation formula is as follows:

[0031]

[0032] in, Indicates the data transmission rate. Then we need to calculate the execution time of the task after offloading execution. The calculation formula is as follows:

[0033]

[0034] in, Represents the edge server eg n The proportion of resources that can be allocated to the service, Indicates eg n The computing power of the service is large, so the calculation formula for the delay cost of unloading the service to the ground server is as follows:

[0035] t i edge =t i trans +t i com

[0036] Therefore, the energy consumption cost of unloading services on the ground server is calculated as follows:

[0037]

[0038] in, Represents the edge server eg n The energy consumption coefficient of the ground server is n The calculation formula of the unloading cost is as follows:

[0039]

[0040] Computing drone edge servers m The unloading cost is similar to that of the ground edge server, and its latency cost is calculated as follows:

[0041]

[0042] Different from the way that ground edge servers calculate energy consumption costs, drones themselves have limited resources. In addition to considering the energy consumption generated by task calculations after offloading and the energy consumption generated during data transmission, the energy consumption generated by the drone’s own flight also needs to be considered. Therefore, the calculation formula for the energy consumption cost of drone edge server offloading is as follows:

[0043] e i uav =e i trans +t i com +e i fly

[0044] Next, the offloading cost of the drone edge server is calculated as follows:

[0045]

[0046] Finally, the cost model of service offloading is to accumulate the local execution costs of all sub-services of the combined service, the ground edge server offloading costs, and the drone offloading costs. Since the smaller the offloading cost, the more consistent with the expected result, it should be set to a negative reward value when designing the reward function in the DDPG model below.

[0047] 5. Preferably, step 4 comprises the following steps:

[0048] Step 41: Use the time convolution layer to extract the temporal features of the user location information, the graph convolution layer to capture the spatial relationship, and use the proximity relationship of the geographic location to obtain the adjacency matrix, so as to build the STGCN model, and perform model training and debugging.

[0049] Step 42: Through continuous debugging and comparison, the historical trajectory data of the user at the previous four moments is input into the STGCN model for prediction to obtain the user's position at the next moment.

[0050] 6. Preferably, step 5 comprises the following:

[0051] Step 51: According to step 32, the geographical location information of the mobile user at time t+1 is obtained. The longitude and latitude of the user are first projected into a plane rectangular coordinate system using the Universal Transverse Mercator Projection (UTM) to reduce the error of direct coordinate calculation. Then, the distance between the user and the edge node is calculated.

[0052] Step 52: According to the coverage of the edge node, determine the candidate offloading edge server set. Considering that there are not only traditional ground-deployed edge servers in the cloud-network fusion environment, but also aerial platforms such as drones to assist in offloading, the candidate edge server sets are divided into two types: one ground edge server set EG′={eg 1 , e.g. 2 ..., e.g. n}, another set of drone edge servers ES′={es 1 ,es 2 ...,es n};

[0053] 7. Preferably, the DDPG model is used to describe the two core decision-making processes of composite service reconstruction and uninstallation by using the state and action of the current time slot t system. Step 6 includes the following steps:

[0054] Step 61: According to the composite service model defined in the step, when the composite service is dynamically reconstructed, there are three reconstruction decision strategies according to the reconstruction area. The first is to replace the atomic service that needs to be reconstructed one-to-one, the second is to replace the reconstructed atomic service one-to-many, and the third is to combine local services in the reconstruction area. The corresponding reconstruction decision strategy is expressed as DR i (t) = {a i , b i , c i}, where a i Represents the index set of specific candidate services corresponding to each abstract sub-service of the composite service after selecting the one-to-one reconstruction decision;

[0055] Step 62: According to the cloud-network fusion environment system model defined in step 6, the decision strategy for uninstalling the i-th service of the composite service is expressed as SL i (t) = {αi , β i ,θ i}, α i It consists of two variables, 0 and 1, indicating whether it is executed locally. i Indicates which ground edge server is offloaded to, θ i Indicates which drone edge server to offload to;

[0056] Step 63: Reconstruction decision DR according to step 61 i (t), get the specific candidate service index corresponding to the current composite service, and then calculate the reward value of the reconstruction action. The calculation formula is as follows:

[0057]

[0058] Where n represents the number of atomic services contained in the composite service, and VQoS(i) represents the i-th attribute of each dimensional service;

[0059] According to the service offloading decision SL of step 62 i (t), calculate the reward value of the service uninstallation action, the calculation formula is as follows:

[0060]

[0061] Finally, the overall reward value of the algorithm is obtained, and the calculation formula is as follows:

[0062] r(s t ,a t )=ω 1 ·r 1 (s t ,a t )+ω 2 ·r 2 (s t ,a t )

[0063] Among them, ω 1 +ω 2 =1, representing the weights corresponding to the reward values ​​of reconstruction decision and uninstallation decision respectively.

[0064] Step 64: The current state, current action, corresponding reward value, and the state at the next moment after the state transfer are stored in the experience replay buffer pool, from which batch data is sampled to continuously train the network, and the Q value and gradient strategy are used to continuously update the network parameters, so as to achieve the purpose of optimizing the DDPG model, and finally obtain the maximum reward value, and obtain the best trade-off between the combined service reconstruction decision and the offloading decision, so as to ensure that the user QoS requirements are met while minimizing the offloading cost.

[0065] Based on the same inventive concept, the present invention provides a combined service dynamic reconstruction system based on deep reinforcement learning in a cloud-network fusion environment, comprising:

[0066] Data collection module: used to collect cloud network user location information, edge server distribution, cloud network combined service QoS data, and server log files.

[0067] Data fusion module: used to fuse the service input and output data volume and combined service QoS data in the service log file to obtain the cloud network service set required for reconstruction.

[0068] Cloud-network system model and composite service building module: used to obtain available resource information of edge servers and mobile devices, and build cloud-network fusion environment system model and composite service model.

[0069] Mobile trajectory prediction module: Generate user dynamic trajectory using user historical location information, and build STGCN model to predict user future location; and obtain a set of candidate edge servers for service offloading based on the predicted user location.

[0070] Composite service reconstruction decision and offloading decision module: used to obtain the optimal trade-off between composite service dynamic reconstruction decision and offloading decision based on the reconstruction information requested by the user, the cloud network service set, and the mobile-aware candidate edge server set.

[0071] Based on the same inventive concept, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the dynamic reconstruction method of combined services based on deep reinforcement learning in a cloud-network fusion environment.

[0072] Beneficial effects: Compared with the existing technology, the present invention has the following advantages: The present invention fully considers the user mobility, obtains the user's candidate edge server set by predicting the user's future position, and avoids the situation where the user may exceed the coverage of the edge server as the user moves, resulting in the service selected by the reconstruction decision being unloaded on the edge server, and the result cannot be obtained from the original server, thereby affecting the reconstruction effect of the combined service. In addition, the present invention combines the air-ground collaboration between edge servers in the cloud-network fusion environment, expands the range of edge servers for service unloading to the aerial platform, and uses the DDPG model to balance the reconstruction decision and unloading decision of the combined service, ensuring that the reconstructed combined service meets the user's needs while minimizing the unloading delay and energy consumption cost, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 1 is an overall step diagram of an embodiment of the present invention.

[0074] Figure 2 The following is a detailed flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The present invention is further illustrated below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0076] like Figure 1 As shown, the embodiment of the present invention discloses a method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment, which mainly includes the following steps:

[0077] Step 1: Collect cloud network user location information, edge server distribution, cloud network combined service QoS data, and server log files;

[0078] Step 2: Obtain the cloud network service set required for reconstruction by integrating the service input and output data volume and the combined service QoS data in the service log file;

[0079] Step 3: Obtain available resource information of edge servers and mobile devices, and build a cloud-network fusion environment system model and a composite service model. The cloud-network fusion environment system model includes an available edge server model that defines air-ground collaboration, and a cost model that includes latency and energy consumption; the composite service model defines the structure of the composite service, the input and output data size of each sub-service, the service type, and the execution time;

[0080] Step 4: Generate user dynamic trajectories using user historical location information and build an STGCN model to predict user future locations;

[0081] Step 5: Obtain a set of candidate edge servers for service offloading based on the predicted user location;

[0082] Step 6: Based on the reconstruction information requested by the user, the cloud network service set, and the mobile-aware candidate edge server set, the deep reinforcement learning DDPG algorithm is used to generate the composite service reconstruction decision and offloading decision to ensure that the QoS of the reconstructed composite service can still meet user needs and minimize the service offloading cost.

[0083] The specific steps of the embodiment of the present invention are further described below with reference to specific examples. Figure 2As shown, an embodiment of the present invention provides a method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment. The data collected in step 1 mainly includes four aspects: first, the user movement trajectory data set, which extracts geographical location information such as longitude and latitude according to different travel modes, which can be used to extract real user trajectory data; second, the base station geographical location data set, which is mainly used to simulate the distribution of ground edge servers; then the QWS data set, which provides a QoS data source for combined services; and finally, the server log data set, which is used to obtain the actual service input and output data volume. The specific steps are as follows:

[0084] Step 11: Extract the geographical location information of the mode of transportation, date, time, longitude and latitude from the user's mobile trajectory data set, classify them according to different travel modes, and eliminate invalid user data that is at the same longitude and latitude position for most of the time;

[0085] Step 12: Collect the base station geographic location data set, extract the longitude, latitude, coverage and other information of the base station, and use it for the distribution of traditional edge servers on the ground;

[0086] Step 13: Collect the QoS data of the combined service based on the QWS dataset, which contains 9 types of QoS data of 2507 real services. Collect QoS data such as reliability, success rate, and throughput.

[0087] Step 14: Collect a server data set that contains 500 services deployed on 10 servers, and extract information about the service input and output data volumes.

[0088] Based on the above-obtained data set, in step 2, the data set is fused to construct a service data set for dynamic reconstruction of composite services in a cloud-network fusion environment. According to the input and output data volume of each service, the corresponding QoS data such as reliability, success rate, and throughput are associated with it:

[0089] Step 3 obtains the available resource information of edge servers and mobile devices, and builds a cloud-network fusion environment system model and a combined service model. The specific steps are as follows:

[0090] Step 31: The system model of the cloud-network fusion environment considered in the present invention mainly includes a mobile user set EU = {{eu 1 ,eu 2 ...,eu k}}, ground edge server set and EG = {e g1 , e g2 ..., e.g. n}, drone edge server set ES = {es 1 ,es2 ...,es m}; define eg n =(fre n g , cap n g , core n g , pw n g , ban n g , cov n g ), where fre n g Represents CPU frequency, cap n g Represents the storage capacity of the server, core n g Represents the number of available CPU cores, pw n g Represents the energy consumption coefficient, ban n g represents the bandwidth allocated to the edge server, cov n g Represents the coverage of the edge server, defining es m =(fre n m , cap n m , pw n m ， loc n m ), where loc n m Represents the three-dimensional coordinates of the drone.

[0091] Step 32: The composite service model considered in the present invention is defined as a triplet of CS = {ST, RT, S}, where ST represents the set of all abstract sub-services of the composite service, RT represents the set of relationships between the abstract sub-services, which usually include order, merge, branch and other relationships, and S represents the set of specific candidate services corresponding to each abstract sub-service. The candidate service set S of the i-th abstract sub-service is i = {I i n , O i n , Loc i n}, where I i n represents the input data size of the candidate service with index n, O i nIndicates the service output data size.

[0092] Step 33: The cost model of service offloading is mainly composed of local execution cost, ground edge server offloading cost and drone offloading cost.

[0093] Calculate user mobile device EU k To calculate the local execution cost, we first need to calculate the local unloading delay of the service. The calculation formula is as follows:

[0094]

[0095] in, represents the computing power of the user's mobile device, which is usually related to the CPU frequency and the number of available cores. Therefore, the energy consumption of computing services offloaded locally is i local The formula is as follows:

[0096]

[0097] in, represents the energy consumption coefficient of the user's mobile device. Therefore, the calculation formula of the local execution cost is as follows:

[0098]

[0099] where σ 1 , σ 2 are the weight factors of delay and energy consumption respectively, and their sum is 1.

[0100] Computing ground servers n To calculate the offloading cost, we first need to calculate the delay time of offloading to the edge server, which mainly includes two parts: one is the time for service data to be transmitted to the edge server, and the other is the execution time of the computing task after offloading to the edge server. The calculation formula is as follows:

[0101]

[0102] in, Indicates the data transmission rate. Then we need to calculate the execution time of the task after offloading execution. The calculation formula is as follows:

[0103]

[0104] in, Represents the edge server eg n The proportion of resources that can be allocated to the service, Indicates eg n The computing power of the service is large, so the calculation formula for the delay cost of unloading the service to the ground server is as follows:

[0105] t i edge =t i trans +t i com

[0106] Therefore, the energy consumption cost of unloading services on the ground server is calculated as follows:

[0107]

[0108] in, Represents the edge server eg n The energy consumption coefficient of the ground server is n The calculation formula of the unloading cost is as follows:

[0109]

[0110] Computing drone edge servers m The unloading cost is similar to that of the ground edge server, and its latency cost is calculated as follows:

[0111]

[0112] Different from the way that ground edge servers calculate energy consumption costs, drones themselves have limited resources. In addition to considering the energy consumption generated by task calculations after offloading and the energy consumption generated during data transmission, the energy consumption generated by the drone’s own flight also needs to be considered. Therefore, the calculation formula for the energy consumption cost of drone edge server offloading is as follows:

[0113] e i uav =e i trans +e i com +e i fly

[0114] Next, the offloading cost of the drone edge server is calculated as follows:

[0115]

[0116] Finally, the cost model of service offloading is to accumulate the local execution costs of all sub-services of the combined service, the ground edge server offloading costs, and the drone offloading costs. Since the smaller the offloading cost, the more consistent with the expected result, it should be set to a negative reward value when designing the reward function in the DDPG model below.

[0117] Step 4 uses the user's historical location information in the above dataset to generate the user's dynamic trajectory, and builds the STGCN model to predict the user's future location. The specific steps are as follows:

[0118] Step 41: Predict the user's movement trajectory based on the STGCN network model, and predict the location at future moments by learning the user's historical trajectory data. The input layer takes the user's historical trajectory as input and passes it to the spatial graph convolution layer and the temporal convolution layer to capture the spatial and temporal correlations in the trajectory data. Finally, the predicted user location is output through the activation layer and the fully connected layer.

[0119] Step 42: Represent the user's trajectory data as a three-dimensional tensor X∈R N×T×F , input into the STGCN model, N represents the number of location points, T represents the time step, here it is 4, F represents the number of features, here it is 2, represents longitude and latitude, specifically expressed as follows:

[0120] X=[HP(1),HP(2),…,HP(T)]

[0121] Among them, each HP(t) represents the user location information at the tth moment, which is specifically expressed as follows:

[0122] GP(t)={lat t ,lng t}

[0123] Step 43: In STGCN, the spatial graph convolution layer is used to capture the spatial dependencies in the user trajectory. The user's trajectory is regarded as a graph structure, where the location points are the nodes of the graph, and the adjacency relationship between nodes is determined by geographical proximity. The adjacency matrix A∈R is defined as N×N , where A ij Represents the spatial correlation between position i and position j.

[0124] Then we need to normalize the adjacency matrix to get The calculation formula is as follows:

[0125]

[0126] Then, the graph convolution operation is performed on the trajectory features of each position point. The calculation formula is as follows:

[0127] Step 44: The STGCN model uses a temporal convolutional layer to capture temporal correlations. Using a one-dimensional convolution operation, the historical trajectory features are convolved in the time dimension through a sliding window to extract temporal patterns.

[0128] Step 45: By stacking spatial convolution and temporal convolution in one module, the temporal and spatial features obtained in steps 43 and 44 are combined. In order to prevent gradient vanishing and enhance feature transfer, a residual connection needs to be added to each spatiotemporal convolution block to add the input features to the convolution results.

[0129] Step 46: Pass the output of the spatiotemporal convolutional block to the fully connected layer to learn the nonlinear mapping relationship between trajectory features and future positions. Apply the activation function ReLU to the output of the fully connected layer to introduce nonlinearity and generate the predicted value of the two-dimensional coordinates, and finally output the user's position HP(t+1) at the next moment.

[0130] Step 5: According to the STGCN mobile trajectory prediction model in step 4, predict the location of the user's next time slot and obtain the set of candidate edge servers for service offloading. The specific steps are as follows:

[0131] Step 51: According to step 32, the geographical location information of the mobile user at time t+1 is obtained. The longitude and latitude of the user are first projected into a plane rectangular coordinate system using the Universal Transverse Mercator Projection (UTM) to reduce the error of direct coordinate calculation. Then, the distance between the user and the edge node is calculated.

[0132] Step 52: According to the coverage of the edge node, determine the candidate offloading edge server set. Considering that there are not only traditional ground-deployed edge servers in the cloud-network fusion environment, but also aerial platforms such as drones to assist in offloading, the candidate edge server sets are divided into two types: one ground edge server set EG′={eg 1 , e.g. 2 ..., e.g. n}, another set of drone edge servers ES′={es 1 ,es 2 ...,es n};

[0133] Step 6: Generate composite service reconstruction decisions and offloading decisions through the deep reinforcement learning DDPG algorithm to ensure that the QoS of the reconstructed composite service can still meet user needs and minimize the service offloading cost. The specific steps are as follows:

[0134] Step 61: According to the composite service model defined in the step, when the composite service is dynamically reconstructed, there are three reconstruction decision strategies according to the reconstruction area. The first is to replace the atomic service that needs to be reconstructed one-to-one, the second is to replace the reconstructed atomic service one-to-many, and the third is to combine local services in the reconstruction area. The corresponding reconstruction decision strategy is expressed as DRi (t) = {a i , b i , c i}, where a i Represents the index set of specific candidate services corresponding to each abstract sub-service of the composite service after selecting the one-to-one reconstruction decision;

[0135] Step 62: According to the cloud-network fusion environment system model defined in step 6, the decision strategy for uninstalling the i-th service of the composite service is expressed as SL i (t) = {α i , β i ,θ i}, α i It consists of two variables, 0 and 1, indicating whether it is executed locally. i Indicates which ground edge server is offloaded to, θ i Indicates which drone edge server to offload to;

[0136] Step 63: Reconstruction decision DR according to step 61 i (t), get the specific candidate service index corresponding to the current composite service, and then calculate the reward value of the reconstruction action. The calculation formula is as follows:

[0137]

[0138] Where n represents the number of atomic services contained in the composite service, and VQoS(i) represents the i-th attribute of each dimensional service;

[0139] According to the service offloading decision SL of step 62 i (t), calculate the reward value of the service uninstallation action, the calculation formula is as follows:

[0140]

[0141] Finally, the overall reward value of the algorithm is obtained, and the calculation formula is as follows:

[0142] r(s t ,a t )=ω 1 ·r 1 (s t ,a t )+ω 2 ·r 2 (s t ,a t )

[0143] Among them, ω 1 +ω 2 =1, representing the weights corresponding to the reward values ​​of reconstruction decision and uninstallation decision respectively.

[0144] Step 64: The current state, current action, corresponding reward value, and the state at the next moment after the state transfer are stored in the experience replay buffer pool, from which batch data is sampled to continuously train the network, and the Q value and gradient strategy are used to continuously update the network parameters, so as to achieve the purpose of optimizing the DDPG model, and finally obtain the maximum reward value, and obtain the best trade-off between the combined service reconstruction decision and the offloading decision, so as to ensure that the user QoS requirements are met while minimizing the offloading cost.

[0145] Based on the same inventive concept, the present invention provides a combined service dynamic reconstruction system based on deep reinforcement learning in a cloud-network fusion environment, including: a data collection module: used to collect cloud-network user location information, edge server distribution, cloud-network combined service QoS data, and server log files. Data fusion module: used to fuse the service input and output data volume and combined service QoS data in the service log file, so as to obtain the cloud-network service set required for reconstruction. Cloud-network system model and combined service construction module: used to obtain the available resource information of edge servers and mobile devices, and construct a cloud-network fusion environment system model and a combined service model. Mobile trajectory prediction module: uses user historical location information to generate user dynamic trajectories, and constructs an STGCN model to predict the user's future location; and according to the predicted user location, obtains a set of candidate edge servers for service unloading. Combined service reconstruction decision and unloading decision module: used to obtain the optimal trade-off between the dynamic reconstruction decision and the unloading decision of the combined service according to the reconstruction information requested by the user, the cloud-network service set, and the mobile-aware candidate edge server set.

[0146] For the convenience and simplicity of description, the specific working process of each module described above can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. The division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system.

[0147] Based on the same inventive concept, the present invention provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment is implemented.

[0148] Based on the same inventive concept, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the dynamic reconstruction method of combined services based on deep reinforcement learning in a cloud-network fusion environment.

Claims

1. A method for dynamic reconstruction of composite services based on deep reinforcement learning in a cloud-network fusion environment, characterized in that: The steps include: Step 1: Collect cloud network user location information, edge server distribution, cloud network combined service QoS data, and server log files; Step 2: Obtain the cloud network service set required for reconstruction by integrating the service input and output data volume and the combined service QoS data in the service log file; Step 3: Obtain available resource information of edge servers and mobile devices, and build a cloud-network fusion environment system model and a composite service model. The cloud-network fusion environment system model includes an available edge server model that defines air-ground collaboration, and a cost model that includes latency and energy consumption; the composite service model defines the structure of the composite service, the input and output data size of each sub-service, the service type, and the execution time; Step 4: Generate user dynamic trajectories using user historical location information and build an STGCN model to predict user future locations; Step 5: Obtain a set of candidate edge servers for service offloading based on the predicted user location; Step 6: Based on the reconstruction information requested by the user, the cloud network service set, and the mobile-aware candidate edge server set, the deep reinforcement learning DDPG algorithm is used to generate the composite service reconstruction decision and offloading decision to ensure that the QoS of the reconstructed composite service can still meet user needs and minimize the service offloading cost.

2. According to claim 1, the method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment is characterized in that: The collection of user location information, edge server distribution, combined service QoS data and server log files in step 1 specifically includes the following steps: Step 11: Extract the geographical location information of the mode of transportation, date, time, longitude and latitude from the user's mobile trajectory data set, classify them according to different travel modes, and eliminate invalid user data that is at the same longitude and latitude position for most of the time; Step 12: Collect the base station geographic location data set, extract the longitude, latitude, coverage and other information of the base station, and use it for the distribution of traditional edge servers on the ground; Step 13: Collect the QoS data of the combined service based on the QWS dataset, which contains 9 types of QoS data of 2507 real services. Collect the QoS data on reliability, success rate, throughput, etc. Step 14: Collect a server data set that contains 500 services deployed on 10 servers, and extract information about the service input and output data volumes.

3. According to claim 1, the method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment is characterized in that: In step 2, a service data set for dynamic reconstruction of combined services in a cloud-network fusion environment is constructed by fusing data sets, and corresponding QoS data such as reliability, success rate, and throughput are associated with each service based on the amount of input and output data of each service.

4. The method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment according to claim 1 is characterized in that: In step 3, the available resource information of the edge server and the mobile device is obtained, and the cloud-network fusion environment system model and the combined service model are constructed, including the following steps: Step 31: The system model of the cloud-network fusion environment considered in the present invention mainly includes a mobile user set EU = {eu1,eu2…,eu k }、Ground edge server set EG={eg1,eg2…,eg n } and the drone edge server set ES = {es1, es2…, es m }. Define eg n =(fre n g ,cap n g ,core n g ,pw n g ,ban n g ,cov n g ), where fre n g Represents CPU frequency, cap n g Represents the storage capacity of the server, core n g Represents the number of available CPU cores, pw n g Represents the energy consumption coefficient, ban n g represents the bandwidth allocated to the edge server, cov n g Represents the coverage of the edge server, defining es m =(fre n m ,cap n m ,pw n m ,loc n m ), where loc n m Represents the three-dimensional coordinates of the drone; Step 32: The composite service model considered in the present invention is defined as a triplet of CS = {ST, RT, S}, where ST represents the set of all abstract sub-services of the composite service, RT represents the set of relationships between the abstract sub-services, which usually include order, merge, branch and other relationships, and S represents the set of specific candidate services corresponding to each abstract sub-service. The candidate service set S of the i-th abstract sub-service is i = {I i n ,O i n ,Loc i n }, where I i n represents the input data size of the candidate service with index n, O i n Indicates the service output data size; Step 33: The cost model of service offloading is mainly composed of local execution cost, ground edge server offloading cost and drone offloading cost. Calculate the user mobile device eu k To calculate the local execution cost, we first need to calculate the local unloading delay of the service. The calculation formula is as follows: in, represents the computing power of the user's mobile device, which is usually related to the CPU frequency and the number of available cores. Therefore, the energy consumption of computing services offloaded locally is i local The formula is as follows: in, represents the energy consumption coefficient of the user's mobile device. Therefore, the calculation formula of the local execution cost is as follows: Among them, σ1 and σ2 are the weight factors of delay and energy consumption respectively, and the sum of the two is 1; Computing ground servers n To calculate the offloading cost, we first need to calculate the delay time of offloading to the edge server, which mainly includes two parts: one is the time for service data to be transmitted to the edge server, and the other is the execution time of the computing task after offloading to the edge server. The calculation formula is as follows: in, Indicates the data transmission rate. Then we need to calculate the execution time of the task after offloading execution. The calculation formula is as follows: in, Represents the edge server eg n The proportion of resources that can be allocated to the service, Indicates eg n The computing power of the service is large, so the calculation formula for the delay cost of unloading the service to the ground server is as follows: t i edge =t i trans +t i com Therefore, the energy consumption cost of unloading services on the ground server is calculated as follows: in, Represents the edge server eg n The energy consumption coefficient of the ground server is n The calculation formula of the unloading cost is as follows: Computing drone edge servers m The unloading cost is similar to that of the ground edge server, and its delay cost is calculated as follows: t i uav =t i trans +t i com Different from the way that ground edge servers calculate energy consumption costs, drones themselves have limited resources. In addition to considering the energy consumption generated by task calculations after offloading and the energy consumption generated during data transmission, the energy consumption generated by the drone’s own flight also needs to be considered. Therefore, the calculation formula for the energy consumption cost of drone edge server offloading is as follows: And i uav =and i trans +e i com +e i fly Next, the offloading cost of the drone edge server is calculated as follows: Finally, the cost model of service offloading is to accumulate the local execution costs of all sub-services of the combined service, the ground edge server offloading costs, and the drone offloading costs. Since the smaller the offloading cost, the more consistent with the expected result, it should be set to a negative reward value when designing the reward function in the DDPG model below.

5. The method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment according to claim 1 is characterized in that: The step 4 comprises the following steps: Step 41: Use the time convolution layer to extract the temporal features of the user location information, the graph convolution layer to capture the spatial relationship, and use the proximity relationship of the geographic location to obtain the adjacency matrix, so as to build the STGCN model, and perform model training and debugging; Step 42: Through continuous debugging and comparison, the historical trajectory data of the user at the previous four moments is input into the STGCN model for prediction to obtain the user's position at the next moment.

6. The method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment according to claim 1 is characterized in that: The step 5 comprises the following steps: Step 51: According to step 42, the geographical location information of the mobile user at time t+1 is obtained. The longitude and latitude of the user are first projected into a plane rectangular coordinate system using the Universal Transverse Mercator Projection (UTM) to reduce the error of direct coordinate calculation. Then, the distance between the user and the edge node is calculated. Step 52: According to the coverage of the edge node, determine the candidate offloading edge server set. Considering that there are not only traditional ground-deployed edge servers in the cloud-network fusion environment, but also aerial platforms such as drones to assist in offloading, the candidate edge server set is divided into two types: a ground edge server set EG′={eg1,eg2…,eg n }, another set of drone edge servers ES′={es1,es2…,es n }.

7. According to the method for dynamic reconstruction of composite services based on deep reinforcement learning in a cloud-network fusion environment of claim 1, the DDPG model is used to describe the two core decision-making processes of composite service reconstruction and uninstallation by using the state and action of the current time slot t system, and is characterized in that: The step 6 comprises the following steps: Step 61: According to the composite service model defined in step 32, when the composite service is dynamically reconstructed, there are three reconstruction decision strategies according to the reconstruction area: one is to replace the atomic service that needs to be reconstructed one-to-one, the second is to replace the reconstructed atomic service one-to-many, and the third is to combine local services in the reconstruction area. The corresponding reconstruction decision strategy is expressed as DR i (t) = {a i ,b i ,c i }, where a i Represents the index set of specific candidate services corresponding to each abstract sub-service of the composite service after selecting the one-to-one reconstruction decision; Step 62: Based on the cloud-network fusion environment system model defined in step 31, the decision strategy for uninstalling the i-th service of the composite service is expressed as SL i (t) = {α i ,β i ,θ i }, α i It consists of two variables, 0 and 1, indicating whether it is executed locally. i Indicates which ground edge server is offloaded to, θ i Indicates which drone edge server to offload to; Step 63: Reconstruction decision DR according to step 61 i (t), get the specific candidate service index corresponding to the current composite service, and then calculate the reward value of the reconstruction action. The calculation formula is as follows: Where n represents the number of atomic services contained in the composite service, and VQoS(i) represents the i-th attribute of each dimensional service; According to the service offloading decision SL of step 62 i (t), calculate the reward value of the service uninstallation action, the calculation formula is as follows: Finally, the overall reward value of the algorithm is obtained, and the calculation formula is as follows: r(s t ,a t )=ω1·r1(s t ,a t )+ω2·r2(s t ,a t ) Among them, ω1+ω2=1, which represents the weights corresponding to the reward values ​​of the reconstruction decision and the uninstallation decision respectively; Step 64: The current state, current action, corresponding reward value, and the state at the next moment after the state transfer are stored in the experience replay buffer pool, from which batch data is sampled to continuously train the network, and the Q value and gradient strategy are used to continuously update the network parameters, so as to achieve the purpose of optimizing the DDPG model, and finally obtain the maximum reward value, and obtain the best trade-off between the combined service reconstruction decision and the offloading decision, so as to ensure that the user QoS requirements are met while minimizing the offloading cost.

8. A dynamic reconstruction system for combined services based on deep reinforcement learning in a cloud-network fusion environment, characterized in that: include: Data collection module: used to collect cloud network user location information, edge server distribution, cloud network combined service QoS data, and server log files; Data fusion module: used to fuse the service input and output data volume and combined service QoS data in the service log file to obtain the cloud network service set required for reconstruction; Cloud-network system model and composite service building module: used to obtain available resource information of edge servers and mobile devices, and build cloud-network fusion environment system model and composite service model; Mobile trajectory prediction module: Generate user dynamic trajectory using user historical location information, and build STGCN model to predict user future location; and obtain the set of candidate edge servers for service offloading based on the predicted user location; Composite service reconstruction decision and offloading decision module: used to obtain the optimal trade-off between composite service dynamic reconstruction decision and offloading decision based on the reconstruction information requested by the user, the cloud network service set, and the mobile-aware candidate edge server set.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into the processor, the main steps of the method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the method for dynamic reconstruction of combined services based on deep reinforcement learning in a cloud-network fusion environment according to any one of claims 1 to 7.

Images