Service migration method and system based on semi-distributed multi-target reinforcement learning algorithm

By adopting a semi-distributed multi-objective reinforcement learning algorithm in the IVEC network and combining with the Actor-Critic network, service migration decisions are optimized, and multi-objective problems of latency and energy consumption are solved, and service quality and user experience are improved.

CN119071853APending Publication Date: 2024-12-03NORTHEASTERN UNIV AT QINHUANGDAO
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411143652.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Service migration solutions in existing IVEC networks are difficult to take into account the diversified needs of latency and energy consumption, especially under dynamic vehicle movement and environmental changes, which lead to a service quality.

Method used

The service migration method based on semi-distributed multi-objective reinforcement learning algorithm is adopted to design the Markov decision-making process framework, and the multi-objective decision-making of delay and energy consumption is optimized to realize the dynamic service migration strategy.

Benefits of technology

It improves the efficiency and accuracy of service migration decisions, adapts to dynamic environments, reduces system delay and energy consumption, and improves service quality and user experience in the Internet of Vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071853A_ABST
    Figure CN119071853A_ABST
Patent Text Reader

Abstract

The invention provides a service migration method based on semi-distributed multi-objective reinforcement learning, and aims to optimize system time delay and energy consumption in the Internet of Vehicles. A three-layer IVEC system comprising a micro base station and a macro base station is constructed, a vehicle runs in the area of the micro base station, and the macro base station serves as an information collection and control center. In order to solve the problem that the preference of the vehicle-mounted application is unknown, a weight vector conversion optimization problem is adopted, and the quality of the approximate Pareto frontier is evaluated by using the hyperlome measurement. Meanwhile, a multi-target Markov decision process framework is introduced, and the Pareto optimal solution is researched to represent the performance of the multi-target service migration strategy. The method effectively reduces system time delay and energy consumption, adapts to dynamics and uncertainty in an IVEC network environment, and remarkably improves service quality and user experience in the Internet of Vehicles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of service migration, and in particular to a service migration method and system based on a semi-distributed multi-objective reinforcement learning algorithm. Background Art

[0002] The introduction of the mobile edge computing (MEC) architecture has enabled the rapid integration of MEC and the Internet of Vehicles (IoV) to form an intelligent vehicle edge computing (IVEC) network. The computing and storage capabilities of vehicle intelligent terminals have been enhanced to meet the needs of various vehicle applications. However, combining the Internet of Vehicles with edge computing also brings new challenges. First, although the auxiliary edge server provides computing resources, its limited computing power still takes a long time to process complex tasks. Second, the edge computing server is usually configured in the base station and together with the base station constitutes a roadside service unit (RSU), but the base station coverage is limited. For a smart vehicle moving quickly on the road, the task it requested has not yet been calculated, and the vehicle may have left the road section covered by the base station. If the vehicle continues to communicate with the previous server at this time, multi-hop delivery is required, which will bring greater latency and energy consumption. Therefore, it is necessary to use service migration technology to dynamically migrate service instances according to the mobility of the vehicle to better ensure the quality of service.

[0003] Some existing works use a central controller to make service migration decisions. However, in a dynamic IVEC network, it may be difficult and time-consuming to collect complete system-level information. This affects the actual performance of service migration in the Internet of Vehicles scenario. Some studies also use distributed control algorithms to make service migration decisions, and establish the service migration process as a partial Markov decision process (POMDP). However, due to the partial observability of POMDP, the distributed control algorithm can only observe local information of the system, making it difficult to achieve global indicator optimization. In order to improve the efficiency and accuracy of service migration decisions, a service migration strategy based on deep reinforcement learning (DRL) has also been proposed in the prior art. The centralized control strategy uses a central controller to make service migration decisions, while the distributed control strategy uses a multi-agent deep reinforcement learning (MADRL) algorithm to achieve collaborative service migration and optimization of tasks between different edge servers.

[0004] However, there are still some problems with the existing service migration scheme in IVEC networks. Existing technologies often consider a single optimization objective in service migration decisions, such as minimizing latency or minimizing energy consumption. This single-objective optimization method cannot meet the diverse needs of in-vehicle applications for latency and energy consumption. In particular, for applications with extremely high latency requirements such as autonomous driving, real-time monitoring, and advanced driver assistance, as well as applications with high energy consumption requirements such as multimedia entertainment and electric power steering systems, the existing single-objective method is difficult to take into account at the same time. Another problem is that in the dynamic IVEC network, the high-speed movement of vehicles and the constant changes in road conditions lead to the dynamics and uncertainty of the environment, making service migration decisions more complicated. Therefore, a multi-objective optimization framework that can comprehensively consider latency and energy consumption is needed to achieve the optimal service migration strategy and reduce the impact of vehicle mobility on service quality. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present invention provides a service migration method and system based on a semi-distributed multi-objective reinforcement learning algorithm, which solves the problems of dynamics and uncertainty in the IVEC network environment.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a service migration method based on a semi-distributed multi-objective reinforcement learning algorithm, comprising the following steps:

[0007] S1. Design a service migration decision framework based on Markov decision process (MDP), which is defined by a five-tuple: {S, A, R, T, γ}, where the state space S contains multiple vehicle information vectors, the action space A represents the migration of services to a certain edge computing (MEC) server, and the reward function R contains the energy consumption reward r E and delayed reward r T , the transition probability T describes the state transition, and γ is the discount factor;

[0008] S2: When the vehicle moves out of the coverage of the current roadside unit (RSU), the vehicle transmits this information to the MEC central on the macro base station;

[0009] S3 and MEC central use the semi-distributed multi-objective reinforcement learning (SDMORL) algorithm to make service migration decisions, including:

[0010] S301, the Actor network selects an action at, which is determined by the parameterized probability distribution π(a|s);

[0011] S302, the Critic network evaluates the action value selected by the Actor and guides the Actor on how to improve;

[0012] S4, update the Actor and Critic network parameters, calculate the Pareto frontier PF(Π), and for each preference w, train the strategy π(a|s) to maximize the reward Rw and approximate the Pareto frontier PF(Π)

[0013]

[0014] r w (s t ,a t )=w T ×(r E (s t ,a t ), r T (s t ,a t ))

[0015] Among them, w T is the weight vector.

[0016] Preferably, in the state space S, the vehicle information vector is defined by a sextuple:

[0017]

[0018] Among them, u t Represents the MEC server running the vehicle service;

[0019] u' t Indicates the MEC server corresponding to the vehicle's current location;

[0020] v t Indicates wireless network upload speed;

[0021] c t Indicates the number of CPU cycles required to run the current service;

[0022] data t Indicates the amount of uninstall service data

[0023] n represents the number of edge servers covered by the server on the macro base station.

[0024] Preferably, the reward function R includes an energy consumption reward and a delay reward, and the energy consumption reward r E and delayed reward r T Respectively expressed as:

[0025] r E (s t , a t )=-E

[0026] r T (s t , a t)=-T

[0027] Among them, E represents system energy consumption and T represents system delay.

[0028] Preferably, the system total reward R w for:

[0029]

[0030] Preferably, the Actor network is updated by the following objective function:

[0031]

[0032] in, is the behavior strategy of the sampling trajectory, To optimize the target strategy, is the important sampling ratio, which is used to correct the distribution error caused by the difference between the behavioral strategy and the target strategy.

[0033] Preferably, the objective function of the Critic network is defined as the mean square error between the predicted value and the actual return, specifically:

[0034]

[0035] Where N is the number of samples, V i,θ (s t ) indicates state S t The value of .

[0036] Preferably, the service migration strategy is implemented by the following steps:

[0037] Initialize the experience replay pool D for each preference w w , Actor network parameters θ w ;

[0038] Initialize the learning rate, learning rate α and the number of training episodes T epi ;

[0039] Set the policy set Π to an empty set;

[0040] For each step t do the following:

[0041] 1) According to the state s t From strategy π θw Sampling action a t ;

[0042] 2) will (s t , a t , r w (s t , a t), s t+1 )Add to D w ;

[0043] 3) Update Actor network parameters θ w , Critic network parameter Φ w ;

[0044] After all episodes are completed, the Pareto front PF(Π) is calculated.

[0045] Preferably, the network model comprises a convolutional layer and a multi-layer perceptron (MLP) layer, wherein the convolutional layer is used to encode the input state and convert the information vector of each vehicle into a feature vector F, and the MLP layer is used to encode the total feature vector to obtain the output.

[0046] Preferably, the output of the Actor network is the probability distribution π of each action θw (a t ∣s t ), the output of the Critic network is the state value V Φ (s t ).

[0047] The service migration system based on semi-distributed multi-objective reinforcement learning algorithm includes:

[0048] Multiple micro base stations (MCS), each MCS contains a MEC server;

[0049] Multiple macro base stations (MBS), each MBS contains a MEC server (MEC central) with stronger computing power, MCS is in the MBS coverage area, and vehicles move within the coverage areas of different MCSs;

[0050] The base station and MEC server together form the road side unit (RSU);

[0051] A control unit for executing the service migration method according to any one of claims 1 to 9

[0052] The present invention provides a service migration method and system based on a semi-distributed multi-objective reinforcement learning algorithm. It has the following beneficial effects:

[0053] 1. The service migration method based on the semi-distributed multi-objective reinforcement learning (SDMORL) algorithm of the present invention realizes efficient service migration decision-making by combining the advantages of centralized and distributed control strategies. MEC central on the macro base station serves as the information collection and control center, dynamically adjusting the migration target of the service instance when the vehicle moves, effectively reducing system delay and energy consumption.

[0054] 2. The present invention introduces a multi-objective optimization framework, which comprehensively considers the balance between latency and energy consumption through weight vectors. This method can adapt to vehicle-mounted applications with different preferences for latency and energy consumption. It can simultaneously meet the needs of applications with extremely high latency requirements and applications with high energy consumption requirements, improving the overall system performance and user experience.

[0055] 3. The present invention adapts to the dynamic and uncertain problems in the IVEC network environment by designing a framework of multi-objective Markov decision process. Through the SDMORL algorithm, it can quickly respond and adjust the service migration strategy in a dynamically changing environment, ensuring the efficient operation of the system and the stability of service quality.

[0056] 4. The SDMORL algorithm based on the Actor-Critic (AC) policy iteration method proposed in this invention achieves an approximate solution of the Pareto frontier by learning a set of policies under different preferences. The hypervolume metric is used to evaluate the quality of the approximate Pareto frontier, optimize the service migration decision process, and enable the system to approach the Pareto optimal solution in multi-objective optimization, thereby improving service quality.

[0057] 5. The present invention dynamically migrates service instances and performs base station switching. The method of the present application reduces unnecessary multi-hop communications and waste of computing resources, improves the computing resource utilization of edge servers, reduces the total energy consumption of the system while ensuring the quality of service, and improves the energy efficiency of the overall system.

[0058] 6. The present invention adopts a semi-distributed control strategy. The method of the present application retains the accuracy of centralized control while taking into account the flexibility and scalability of distributed control. The method can adapt to complex application scenarios in large-scale IVEC networks and improve the system's scalability and scope of application.

[0059] The present invention improves the efficiency and accuracy of service migration by comprehensively optimizing latency and energy consumption, adapts to the dynamics and uncertainty in the IVEC network environment, enhances the scalability of the system, and achieves the Pareto optimal solution under multi-objective optimization, thereby significantly improving the service quality and user experience in the Internet of Vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The three-layer VEC system architecture diagram of the present invention;

[0061] Figure 2 A schematic diagram of base station switching time of the present invention;

[0062] Figure 3 It is a schematic diagram of the rising simplex of the present invention;

[0063] Figure 4A neural network model diagram of the present invention;

[0064] Figure 5 Schematic diagram of the SDMORL algorithm of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Example:

[0067] Please see attached Figure 1 The embodiment of the present invention provides a service migration system based on a semi-distributed multi-objective reinforcement learning algorithm, including:

[0068] First layer: terminal layer

[0069] The terminal layer nodes are a set of intelligent vehicles, denoted as V = {v1,v2,...,v i}. Among them, v i Represents any smart vehicle in the set. Smart vehicles communicate with roadside units (RSUs) via V2R (vehicle to RSU) communication.

[0070] Layer 2: Distributed Edge Layer

[0071] The nodes of the distributed edge layer are composed of RSUs, each of which contains a set of edge computing (MEC) servers and a micro base station (MCS). Therefore, the edge node set can be expressed as R = {MCS, M}. Among them, MCS = {mcs1, mcs2, ..., mcs i} represents a micro base station set, mcs i is any micro base station in the set; M = {m1,m2,...,m i} represents the MEC server set, m i is any MEC server in the set. mcs and m belonging to the same RSU have the same subscript number. Vehicles move within different RSU coverage areas in the distributed edge layer and communicate with RSUs through V2R links.

[0072] The third layer: core edge layer

[0073] The nodes of the core edge layer are also composed of RSUs, but unlike the distributed edge layer, each RSU in the core edge layer contains a macro base station (MBS) and a group of MEC servers with stronger computing power. The edge node set of the core edge layer is represented as Rc = {MBS, MC}, where MBS = {mbs1, mbs2, ..., mbs i} represents a macro base station set, mbs i is any macro base station in the set. MC={mc1,mc2,...,mc i} represents the MEC server set, mc i It is any MEC server in the set. This MEC server has stronger computing power and is called MEC central. It is the central point of system information and is responsible for collecting information from all MEC servers, MCSs and smart vehicles within the coverage area and storing it in the central database. MBs and MCs belonging to the same RSU have the same subscript number. RSUs communicate with each other through R2R links.

[0074] In the VEC system architecture of the present invention, smart vehicles communicate with RSUs through V2R links. When the vehicle moves to different RSU coverage areas, MEC central can dynamically adjust the migration target of the service instance according to the location of the vehicle to ensure service quality and efficient operation of the system.

[0075] Through the design of this three-layer structure, the present invention can effectively improve the computing and storage capabilities of the system, optimize the latency and energy consumption during service migration, and adapt to the high-speed movement of vehicles on the road and complex environmental changes.

[0076] At time slot t1, the vehicle offloads the task to the MEC server of the RSU node through the V2R link. This MEC server is called the source MEC server and is denoted by u t The MEC server creates a dedicated service instance for the vehicle, which encapsulates the runtime data of the task requested by the vehicle and the user context information. However, before the task is calculated, the vehicle moves from the current RSU coverage area to another RSU coverage area. At this time, if the vehicle continues to communicate with the previous RSU, it needs to be forwarded through the intermediate RSU, which will bring greater communication delay and energy consumption.

[0077] In order to optimize the service quality, the MEC central of the core edge layer will notify u t Migrate the service instance to the destination MEC server (denoted as d t ). However, migrating service instances will also consume energy. The longer the migration distance, the greater the migration energy consumption. At the same time, during the migration process, not only the service instance is moved from u tMigrate to d t , and also need to switch the base station, that is, switch the vehicle's V2R link from the original base station to the d t Migrating services and switching base stations will cause service downtime. Therefore, it is necessary to consider which MEC server to migrate the service to (which MEC server is d t ) brings the least total service delay and system energy consumption. At the same time, MEC server processing tasks will also generate computing delay and energy consumption, so this article will comprehensively consider the delay and energy consumption caused by communication, migration and computing.

[0078] During the service migration process, the latency mainly consists of three parts: communication latency, computing latency, and service downtime.

[0079] Communication delay: Communication delay is the delay of the vehicle offloading tasks to the MEC server on the RSU node through the V2R link. The main offload service data volume is data t and the average upload rate v of the wireless channel t Decide.

[0080]

[0081] Calculate the delay: In each time slot, the MEC server responds to the vehicle's service request. In time slot t, using symbol c t To represent the sum of CPU cycles required to run the service, is the computing power allocated by the service node to the current service, measured in CPU cycles / second. Therefore, the computing latency when running the migration service in time slot t can be calculated as follows:

[0082]

[0083] Service downtime: Migrating service instances and switching base stations will cause service downtime. mig Indicates the migration delay, which is determined by u t With d t The jump distance y t (measured by Manhattan distance) and the migration delay factor m t c The migration delay is:

[0084] t mig =m t c y t

[0085] Please refer to Appendix 2 to consider when to perform a base station switch with the least service downtime. han represents the delay caused by switching base stations, t sdIndicates service downtime. Handover option 1-3 are switching before migration, switching after migration, and switching during migration. It can be seen that the service downtime caused by Handover option 3, that is, switching during migration, is the smallest. Service downtime is the maximum value of migration delay and base station switching time.

[0086] t sd =max(t mig , t han )

[0087] The total system delay is the sum of communication delay, computing delay, and service downtime.

[0088] T=t comm +t comp +t sd

[0089] Energy consumption:

[0090] Corresponding to the latency, during the entire service migration process, the system energy consumption mainly includes three aspects: first, the communication energy consumption of the vehicle offloading tasks to the MEC server on the RSU node through the V2R link. Second, the computing energy consumption generated by the service node running the service, which is related to the amount of computing tasks processed and the allocation of computing resources. Third, part of the energy consumed by users in a waiting state when the service is down.

[0091] e comm =t comm *p1

[0092] e comp =t comp *p2

[0093] e sd =t sd *p3

[0094] E=e comm +e comp +e sd

[0095] p1 represents the wireless transmission power during communication. p2 represents the service node calculation power. p3 represents the power consumed per second by the user in the waiting state.

[0096] Since the preferences of vehicle applications for latency and energy consumption may be different and this preference is unknown when designing the solution, a weight vector w = (w t ,w e ), satisfying w t +w e =1.

[0097] When a weight vector is given, the goal of the service migration problem is to make a service migration decision a according to the service migration strategy π, so that t1-t n Within this time, the system overhead consisting of the normalized weighted sum of system delay and energy consumption is minimized.

[0098]

[0099] st

[0100] T<=T max

[0101] E<=E max

[0102] Weight vector set W: Since there are many possible weight vectors, we introduce the weight vector set W = {w1,w2,...,w n}, the weight vector set includes n weight vectors.

[0103] Strategy set Π: For each weight vector, find the strategy π that satisfies the objective function, and there are n corresponding strategies. Thus, we get the strategy set Π = {π1,π2,...,π n}.

[0104] Performance indicator set Y: Use y to represent the performance indicator, as shown in the following formula:

[0105]

[0106] The performance indicator of each strategy is y π . There are n indicators corresponding to n strategies

[0107] Pareto frontier PF(Π): This optimization problem is a multi-objective optimization problem. There is no single optimal solution, but a set of optimal solutions. The set of all Pareto optimal solutions is called the Pareto frontier. For the strategy set Π, PF(Π) is as follows:

[0108]

[0109] Among them, >p is a Pareto advantage relationship, satisfying:

[0110]

[0111] Hypervolume indicator: The Hypervolume indicator evaluation method is a Pareto-consistent evaluation method that defines the hypervolume metric as:

[0112]

[0113] in, is a reference performance point. Function is an indicator function, when z∈V h Its value is 1 when (PF(Π)') is true, and 0 otherwise.

[0114] Since the natural network environment of VEC is full of dynamics and uncertainty, the preferences of MEC systems are unknown. The computational complexity of traditional optimization methods is high because the goal is to obtain a vector reward rather than a reward value. The above problems prompted the design of a SDMORL scheme to find the optimal service migration decision.

[0115] MEC central on the macro base station is the information collection center and control center. It collects information from all MEC servers, MCSs, and Vs in its coverage area. When a vehicle leaves the current RSU coverage, the vehicle passes this information to MEC central, and MEC central makes a service migration decision, deciding whether to migrate the service and which edge server to migrate the service instance to.

[0116] A service migration decision framework based on MDP theory is defined by a five-tuple, {S, A, R, T, γ}. γ∈[0,1] is the discount factor, and the other details are as follows:

[0117] State space S: Consider vehicle i, so the state s at step t is t ∈S is a fixed-length set containing i vehicle information vectors. t Represented as s t ={s t,vi |v i ∈V}. The information vector of vehicle vi at step t is defined by a six-tuple:

[0118]

[0119] They represent the MEC servers u that run vehicle services. t , the MEC server u corresponding to the vehicle’s current location t ', wireless network upload speed v t 、The number of CPU cycles required to run the current service c t , uninstall service data volume t , the number of edge servers n covered by the server on the macro base station.

[0120] Action space A: action a t ∈A indicates which MEC server the service is migrated to. So an action can be defined as:

[0121] a t =m i mi ∈M

[0122] a t =m i Indicates migrating the service to MEC server mi, where M is a collection of servers.

[0123] R: R represents the reward. Unlike the traditional MDP setting, where each reward is a scalar, the setting requires a vector. Therefore, the reward function is Denote energy consumption and delayed reward as r E and r t MECcentral makes a service migration decision at step t, then the state s t and action a t The energy consumption reward is: E (s t ,a t )=-E, the delayed reward is r T (s t ,a t )=-T, given the weight vector w, the system reward value is:

[0124] r w (s t ,a t )=w T ×(r E (s t ,a t ), r T (s t ,a t ))

[0125] The total rewards for one episode are:

[0126]

[0127] T: T stands for transition probability, which describes the action as a t When the state changes from s t to t+1 The transformation of P(s t+1 |s t ,a t )express.

[0128] SDMORL algorithm design

[0129] We chose to design an algorithm based on the Actor-Critic (AC) policy iteration method. The advantage of the AC policy iteration algorithm is that it combines the advantages of policy-based reinforcement learning methods and value-based reinforcement learning methods, and improves learning efficiency and performance by introducing the estimation of the value function in the process of policy iteration. In the AC algorithm, the Actor is responsible for selecting actions, and the network parameter is θ, while the Critic evaluates the quality of these actions and guides the Actor on how to improve. The network parameter is For each preference ω in the set Ω, train the policy to maximize the reward R w And approximate the Pareto front PF(Π). The details are as follows:

[0130] Actor network: Based on a random strategy, migration decisions are selected through parameterized probability distribution. The input is the state s t , the output is the probability distribution of migration decisions. The goal of the Actor network is to learn a strategy π(a|s) to maximize long-term rewards, which are usually calculated by the discounted cumulative sum of rewards. The off-policy strategy can achieve efficient sample utilization, can use pre-collected data for policy updates, and can find the optimal strategy faster in some cases. The off-policy strategy is introduced into the AC policy iteration method. The Actor network objective function is as follows:

[0131]

[0132] in, is the behavior policy for sampling trajectories, which does not participate in gradient updates. Optimized target strategy. is the important sampling ratio, which is used to correct the distribution error caused by the difference between the behavior strategy and the target strategy. h H(π t ) as a regularization term to further encourage exploration during training, where H(π t ) represents the entropy of the strategy, c h is the coefficient. Off-policy methods are known for being unstable and difficult to cover. To address this problem, the clip function is introduced to limit the value of the important sampling ratio, which limits the value of the important sampling ratio to the range [1-∈, 1+∈], where ∈ is the clip hyperparameter. The clip function can prevent very large policy updates and stabilize training. In order to balance the trade-off between the variance and bias of the training objective, the GAE advantage function is used The variance of the target can be significantly reduced while maintaining a tolerable level of deviation, w Tis the weight vector. In order to achieve the goal of obtaining the Pareto optimal solution between latency and energy consumption, two GAE advantage functions are introduced as shown in the formula. Where λ is the GAE discount factor in the range of [0,1]. Indicates state s t The value of .

[0133]

[0134] The gradient corresponding to i∈{T,E} is:

[0135]

[0136] Given the gradient directions of the two objectives, the policy can reach the Pareto front along one of the directions in the ascending simplex. The ascending simplex is defined by a convex combination of the gradients of the single objectives. As shown in the figure, the green arrow and the blue arrow represent the gradient directions of the delay objective and the energy consumption objective, respectively. The light yellow area represents the ascending simplex. For the reward function r ω (·), the gradient direction when the preference is ω is

[0137]

[0138] The Actor network can use the gradient ascent method to update the network parameters by maximizing the above objective function. α is the learning rate, which controls the pace of parameter update.

[0139]

[0140] Critic network: evaluates the action value selected by the Actor, that is, measures the value of taking a specific action in a given state. The input of the Critic can be the state, and the output is the state value. The objective function of the Critic network is defined as the mean square error between the predicted value and the actual return. As shown in the formula:

[0141]

[0142] N is the number of samples, V i,θ (s t ) indicates state s t The Critic network is updated using stochastic gradient descent.

[0143]

[0144] Please see attached Figure 4 , introduced a well-designed encoding method to construct the features of the multi-edge system. The neural network contains convolutional layers and multi-layer perceptron (MLP) layers. The convolutional layer encodes the input state with a point-by-point convolution kernel and converts the information vector s of each vehicle into tTransformed into feature vector F. Reshape all feature vectors and concatenate them to get the total feature vector. The MLP layer encodes the total feature vector to get the output. For the actor network, the output is the probability π of each action θ (a t ,s t ). For the critic network, the output is w when the preference is ω T [V T,θ (s t ),V E,θ (s t )] value.

[0145] Please see attached Figure 5 , is the proposed SDMORL algorithm, which initializes the experience replay pool, policy parameters, learning rate, etc. for each weight vector, and then iterates in each preference and episode, sampling actions and updating the reward and next state according to the state, and finally updates the policy parameters and calculates the Pareto frontier to optimize the service migration decision process.

[0146] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Service migration method based on semi-distributed multi-objective reinforcement learning algorithm, It is characterized in that The following steps are involved: S1. Design a service migration decision framework based on Markov decision process (MDP), which is defined by a five-tuple: {S, A, R, T, γ}, where the state space S contains multiple vehicle information vectors, the action space A represents the migration of services to a certain edge computing (MEC) server, and the reward function R contains the energy consumption reward r E and delayed reward r T , the transition probability T describes the state transition, and γ is the discount factor; S2: When the vehicle moves out of the coverage of the current roadside unit (RSU), the vehicle transmits this information to the MEC central on the macro base station; S3 and MEC central use the semi-distributed multi-objective reinforcement learning (SDMORL) algorithm to make service migration decisions, including: S301, the Actor network selects an action at, which is determined by the parameterized probability distribution π(a|s); S302, the Critic network evaluates the action value selected by the Actor and guides the Actor on how to improve; S4, update the Actor and Critic network parameters, calculate the Pareto frontier PF(Π), and for each preference w, train the strategy π(a|s) to maximize the reward Rw and approximate the Pareto frontier PF(Π) r w (s t ,a t )=w T ×(r E (s t ,a t ),r T (s t ,a t )) Among them, w T is the weight vector.

2. The service migration method based on the semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that In the state space S, the vehicle information vector is defined by a six-tuple: Among them, u t Represents the MEC server running the vehicle service; U t 'Indicates the MEC server corresponding to the vehicle's current location; v t Indicates wireless network upload speed; c t Indicates the number of CPU cycles required to run the current service; data t Indicates the amount of uninstall service data n represents the number of edge servers covered by the server on the macro base station.

3. The service migration method based on the semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The reward function R includes energy consumption reward and delay reward. The energy consumption reward r E and delayed reward r T Respectively expressed as: r E (s t ,a t )=-E r T (s t ,a t )=-T Among them, E represents system energy consumption and T represents system delay.

4. The service migration method based on the semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The total reward of the system is R w for:

5. The service migration method based on semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The Actor network is updated by the following objective function: in, is the behavior strategy of the sampling trajectory, To optimize the target strategy, is the important sampling ratio, which is used to correct the distribution error caused by the difference between the behavioral strategy and the target strategy.

6. The service migration method based on semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The objective function of the Critic network is defined as the mean square error between the predicted value and the actual return, specifically: Where N is the number of samples, V i,θ (s t ) indicates state S t The value of .

7. The service migration method based on semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The service migration strategy is implemented through the following steps: Initialize the experience replay pool D for each preference w w , Actor network parameters θ w ; Initialize the learning rate, learning rate α and the number of training episodes T epi ; Set the policy set Π to an empty set; For each step t do the following: 1) According to the state s t From strategy π θw Sampling action a t ; 2) will (s t , a t , r w (s t , a t ), s t+1 )Add to D w ; 3) Update Actor network parameters θ w , Critic network parameter Φ w ; After all episodes are completed, the Pareto front PF(Π) is calculated.

8. The service migration method based on semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The network model comprises a convolutional layer and a multi-layer perceptron (MLP) layer, wherein the convolutional layer is used to encode the input state and convert the information vector of each vehicle into a feature vector F, and the MLP layer is used to encode the total feature vector to obtain the output.

9. The service migration method based on semi-distributed multi-objective reinforcement learning algorithm according to claim 1, It is characterized in that The output of the Actor network is the probability distribution π for each action θw (a t ∣s t ), the output of the Critic network is the state value V Φ (s t ).

10. A service migration system based on a semi-distributed multi-objective reinforcement learning algorithm, according to a service migration method based on a semi-distributed multi-objective reinforcement learning algorithm as described in any one of claims 1 to 9, It is characterized in that include: Multiple micro base stations (MCS), each MCS contains a MEC server; Multiple macro base stations (MBS), each MBS contains a MEC server (MEC central) with stronger computing power, MCS is in the MBS coverage area, and vehicles move within the coverage areas of different MCSs; The base station and MEC server together form the road side unit (RSU); A control unit for executing the service migration method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Electromechanical system reliability evaluation method based on deep reinforcement learning and important sampling

    CN119885759A