A Distributed Service Migration Method for Vehicle Edge Computing

Abstract

This invention belongs to the field of vehicular edge computing and discloses a distributed service migration method for vehicular edge computing environments. This method, based on contrastive role representation and multi-agent deep reinforcement learning, optimizes service migration decisions by constructing a three-layer heterogeneous vehicular edge network model, including a cloud center layer, an edge layer, and a user layer. This invention proposes a multi-agent deep reinforcement learning method based on an improved QMIX. It constructs agent behavior state trajectory embeddings for local Q-networks and hybrid networks based on agent historical trajectory information, capturing time-related behavioral patterns and solving some observability problems. Furthermore, it uses a contrastive learning-based role encoder to extract unique behavioral patterns of different agents to generate role representations, capturing long-term features of agent behavior. A multi-head attention mechanism is introduced into the hybrid network to dynamically allocate attention weights. This method exhibits superior performance.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted edge computing technology, and in particular to a distributed service migration method for vehicle-mounted edge computing. Background Technology

[0002] In recent years, with the development of 5G networks, autonomous driving, and intelligent transportation systems, modern vehicles are equipped with a large number of sensors and electronic devices, generating massive amounts of data for real-time applications such as obstacle detection, path planning, and driver assistance. However, due to limitations such as data transmission latency, traditional cloud computing architectures struggle to meet the real-time demands of vehicle users. Therefore, Vehicle Edge Computing (VEC) has emerged. VEC pushes computing resources and data storage down to edge servers closer to the data source, allowing vehicle users to deploy services on edge nodes to process task requests, thereby reducing service response latency. However, VEC still faces many challenges. One critical technology is dynamic service migration. Edge server coverage is limited, and the mobility of vehicle users increases the distance between vehicles and the servers hosting services. Communication between users and services may involve multiple hops. To ensure service quality, services need to be dynamically migrated to more suitable edge servers, but frequent migrations can increase costs. Balancing migration costs and transmission latency to make optimal decisions, considering user mobility and the dynamic nature of task requests, is a challenge.

[0003] Early research modeled service migration decisions as Markov decision processes and employed centralized algorithms based on deep reinforcement learning. These approaches require a central decision controller (often a cloud center) to obtain complete system-level information. However, vehicular edge computing is a distributed environment, and the latency, bandwidth, and reliability of edge networks are insufficient to meet the demands of centralized collection of overall information for migration algorithms. In practical applications, each server often only obtains information about a portion of the current service users. Recent research has begun to utilize multi-agent deep reinforcement learning for distributed decision-making. However, due to the partial observability of environmental information, decision-making agents lack a global view, often making suboptimal decisions. Furthermore, the state information involved in policy formulation includes multiple dimensions and a large number of users, resulting in high dimensionality and complexity that makes it difficult for algorithms to extract key features. Simultaneously, in a VEC environment, the load, deployed users, and service capabilities of each base station are different. However, in the service migration problem, agents often share the same rewards, leading to agents across all base stations potentially learning similar policies, resulting in unreasonable migration decisions and low efficiency. In response to these problems, this invention proposes a distributed service migration method for vehicle-mounted edge computing. By introducing recurrent neural networks, multi-head attention mechanisms, and role representations, it optimizes deep reinforcement learning models, extracts key state features, and solves the problem of agent homogenization. Summary of the Invention

[0004] The main content of this invention is to propose a distributed service migration method for vehicular edge computing, applicable to partially observable vehicular edge computing environments. It is based on an attention-guided contrastive role representation method and is a distributed service migration decision algorithm for vehicular edge computing based on contrastive role representation and multi-agent deep reinforcement learning.

[0005] The technical solution of the present invention is as follows: a distributed service migration method for vehicle edge computing, which establishes a three-layer heterogeneous vehicle edge computing environment, and obtains a distributed service migration scheme based on an improved multi-agent deep reinforcement learning algorithm in the three-layer heterogeneous vehicle edge computing environment.

[0006] The three-layer heterogeneous vehicle edge computing environment consists of three entities: a cloud center layer, an edge layer, and a user layer. The user layer is responsible for generating tasks and prioritizing their offloading to nearby edge nodes to reduce latency. The edge node layer is responsible for processing tasks locally. The cloud center layer provides global optimization support to ensure the efficiency and reliability of the entire system.

[0007] The improved multi-agent deep reinforcement learning algorithm models the optimization objective as a decentralized partially observable Markov decision process. This algorithm, based on an improved QMIX algorithm, constructs a distributed service migration strategy, enabling each agent to make service migration decisions based on local information and shared role information. It constructs agent behavior state trajectory embeddings for the local Q-network and hybrid network based on agent historical trajectory information, capturing time-related behavioral patterns and addressing some observability issues. Furthermore, a contrastive learning-based role encoder extracts unique behavioral patterns from different agents to generate role representations, capturing long-term features of agent behavior. A multi-head attention mechanism is introduced into the hybrid network to dynamically allocate attention weights.

[0008] Furthermore, the improved multi-agent deep reinforcement learning algorithm specifically includes:

[0009] (1) Construct decision variables and optimize target model: Adopt a distributed decision-making approach, with each edge node as a decision-making body, and make migration decisions for the service instances it carries in each time slot, i.e., the target server for migrating services, to minimize the user service response cost.

[0010] 1) Decision variables: Define variable α u (t)∈{0,…,|M|} represents the target server for the service instance of user u to migrate in time slot t; the set of target servers for all users at time t is shown in the following formula;

[0011] α(t)={α1(t),…,α |U| (t)} (1)

[0012] α u (t) = m∈M, indicating that at time slot t, the target server for the service instance migration of user u is m, and the service deployment variable p is updated simultaneously. u,m (t) = 1;

[0013] 2) Optimization objective: For the service migration problem in a three-layer heterogeneous vehicle edge environment, the ultimate goal is to minimize the weighted sum of latency and energy consumption of all tasks, i.e., the cost; the optimization objective function is defined as follows;

[0014]

[0015] Formula (2) represents minimizing the long-term response cost for all users based on service migration decisions, including the cost of uploading service requests. Cost of forwarding service requests Cost of access request calculation Cost of service migration α(t) is the migration decision made by the edge node for all vehicle users in time slot t; the migration target of each user service can only be the edge server in the environment; each user can only connect to one base station in a certain time slot, and its service instance can only be deployed on one edge server; the computing resources allocated to users cannot exceed the total resource limit of the server.

[0016] Furthermore, in the improved QMIX algorithm, the agent is an edge node, the observation is all task information received by the agent in each time slot, the user's current location, and the number of services carried by the server, the state is the joint observation state of all agents, and the action is the migration decision made by the edge node for the user service in each time slot. The local Q network and the hybrid network in the QMIX algorithm are improved as follows to obtain the improved local Q network and the improved hybrid network.

[0017] Furthermore, the improved local Q-network is a local Q-network of the QMIX algorithm reconstructed by constructing an agent behavior trajectory embedding module and an agent role representation module;

[0018] In each time slot t, each agent m will record the current observation O m (t) and the action A in the previous time slot m (t-1) is input into a fully connected neural network for preliminary feature extraction. The extracted features, along with the agent's behavioral trajectory from the previous time slot, are embedded in e. m (t-1) is input to the agent behavior trajectory embedding module to generate the agent behavior trajectory embedding e for the current time slot. m (t); embed the agent's behavioral trajectory into e m (t) Input into the agent role representation module to generate the current agent's role representation z. m (t); Finally, embed the role representation with the previously generated agent behavior trajectory into e. m (t) Input a fully connected layer to obtain the Q value of the agent's local Q network based on the current observation.

[0019] Furthermore, the agent behavior trajectory embedding module specifically involves: in the local Q-network of each agent m, the agent behavior trajectory {(O1,A0),…, ... |T| A |T|-1 Encoding is performed to obtain the agent's behavior trajectory embedding; after receiving observations and actions, agent m combines the previous hidden state e generated by the GRU network in the agent behavior trajectory embedding module. m (t-1) Generate the current hidden state e m (t); e m (t) refers to the embedding of the behavioral trajectory of agent m; e mThe formula for calculating (t) is as follows;

[0020] e m (t)=f φ (O m (t),A m (t-1),e m (t-1)) (3)

[0021] Where φ represents the shared GRU encoder of the agent, O m (t) represents the current observation of agent m, A m (t-1) represents the last action of agent m, e m (t-1) is the previous hidden state of GRU.

[0022] Furthermore, the intelligent agent role representation module specifically comprises:

[0023] a) Role Representation Definition: In collaborative multi-agent tasks, each agent m∈M is associated with a role. m ∈Role, which is associated with a role used to describe the behavioral patterns of an agent; role m The behavioral pattern is represented by a role z m ∈Z is used for quantization, which means it is obtained by training a role encoder f, as shown in the following formula;

[0024] z m =f(ω) m (4)

[0025] Where, ω m The behavioral trajectory ω is the sequence of observations and actions of agent m over the entire time period. m This includes the agent's decision-making behavior under different states and its interaction with the environment; the distance between the agent's behavior trajectory embeddings obtained by the agent behavior trajectory embedding module is used as an indicator to quantify the agent's role representation; and the individual policy of agent m is further improved by introducing role representation Z through the agent's local trajectory. As shown in the following formula;

[0026]

[0027] Where O represents the agent's observation; A represents the agent's action; Z represents the agent's role representation; and in this context, the individual strategy... It not only considers the agent's current local observations and actions, but also incorporates the role representation z. m Provided behavioral pattern information;

[0028] b) Role Representation Discrimination Method: A mutual information maximization mechanism is introduced for formal role representation learning, and a contrastive learning objective is derived through InfoNCE; the contrastive learning objective optimizes the role encoder of each agent in a self-supervised manner, and the role encoder embeds the agent's behavioral trajectory into {e1,…,e...} M The encoding process ultimately yields the appropriate role representation {z1,…,z} for each agent. M};

[0029] Specifically, based on the embedding of agent behavior trajectories, all agents are periodically divided into K clusters. For agent m, a role encoder θ is used. q Generate its role representation z m And set it as the query vector q; through a momentum encoder θ k Generate role representations for other intelligent agents, as keys Agent roles that are in the same cluster as agent m are considered positive keys {k}. + Other intelligent agent roles are considered as negative keys. A similarity score function is constructed using bilinear product; the similarity between the query and the key is calculated as follows: Where W is a learnable parameter matrix, the final InfoNCE loss function is as shown in the following equation, and the contrastive learning objective is to minimize this loss function;

[0030]

[0031] During training, the character encoder is updated via backpropagation; the momentum encoder, which needs to generate stable samples, is updated using momentum, as shown in the formula below.

[0032] θ k ←βθ k +(1-β)θ q (7)

[0033] Where β∈[0,1) is the momentum coefficient.

[0034] Furthermore, the improved hybrid network is a hybrid network that reconstructs the QMIX algorithm by constructing an agent state trajectory embedding module and a multi-head attention module. In each time slot t, the agent's global state S(t) is processed through a fully connected layer and then input together with the agent state trajectory embedding τ(t-1) from the previous time slot into the agent state trajectory embedding module to obtain the state trajectory embedding τ(t) for the current time slot. The state trajectory embedding τ(t) for the current time slot and the current role representation z(t) of all agents are input into the multi-head attention module to obtain the hybrid output τ of the multi-head attention module. mha (t); will output τ mhaThe Q(t) is concatenated with the global state S(t) and input into the supernetwork of the original QMIX hybrid network along with the local Q values ​​of all agents. A set of dynamic weights is then calculated to perform a weighted summation of the local Q values ​​of each agent, ultimately yielding the global Q value.

[0035] Furthermore, the agent state trajectory embedding module, based on a gated recurrent unit (GRU), embeds the historical global state {S0,…,S} containing observations from all agents. T The process involves encoding to obtain the agent's state trajectory embedding. After obtaining the global state at the current moment, the agent's state trajectory embedding module combines the hidden state τ(t-1) generated by the GRU network in the module at the previous moment with the hidden state τ(t) at the current moment as input to generate the hidden state τ(t) at the current moment. τ(t) is the agent's state trajectory embedding. The calculation formula for τ(t) is as follows.

[0036] τ(t)=f φ (S(t),τ(t-1)) (8)

[0037] Where φ represents the global state GRU encoder, S(t) represents the current global state of all agents, and τ(t-1) is the previous hidden state of GRU.

[0038] Furthermore, the multi-head attention module specifically comprises: setting the state trajectory embedding τ as the query of the multi-head attention mechanism, setting the role representation z as the key and value of the multi-head attention mechanism, and the weighted combination τ of the role representations. atten As shown in the following formula;

[0039]

[0040] Where, μ m Through the shared parameter matrix W V For z m Attention weights are obtained through a linear transformation. Calculate the state trajectory embedding τ and the role representation z of the m-th agent. m The correlation between them is used to obtain the weights using a softmax function. As shown in the following formula;

[0041]

[0042] Among them, W Q and W K It is a shared parameter matrix used for querying key-pair linear transformations. It is a factor for scaling dot product attention; multi-head attention is used to enable the hybrid network to simultaneously pay attention to positional information from different representation subspaces, and the aggregated output is represented as follows;

[0043]

[0044] in, Is using W h Q W h K and W h V The attention output calculated from the projection, h∈H={1,…,|H} represents the number of heads in the multi-head attention mechanism, W O It is a parameter matrix that combines the outputs of all heads; finally, the multi-head attention outputs are combined with the global state to generate the weights of the hybrid network.

[0045] The beneficial effects of this invention are as follows: Compared with existing distributed service migration algorithms in vehicular edge computing systems, this invention models edge nodes as agents and makes service migration decisions for vehicle users based on an improved QMIX algorithm. To improve decision quality, this invention improves the QMIX algorithm by introducing an agent behavior trajectory embedding module and a role representation module in the local Q-network, and an agent state trajectory embedding module and a multi-head attention mechanism in the hybrid network. The agent behavior trajectory embedding module and the agent state trajectory embedding module effectively capture long-term dependencies in service migration decisions by utilizing the agent's local observations and historical information of the global state, alleviating the problem of insufficient multi-agent cooperation caused by partial observability. The role representation module learns the agent's compact role representation, enabling the agent to make more flexible decisions based on its own resource conditions and historical behavior, thereby avoiding the problem of decision homogenization in multi-agent systems. In addition, the multi-head attention mechanism in the hybrid network can filter out the most critical information when facing a high-dimensional complex state space, reducing the interference of irrelevant information, thereby achieving accurate state space compression. This mechanism effectively integrates global and local information from QMIX, further enhancing the collaborative capabilities of multi-agent systems. Comparative experiments with baseline algorithms show that this algorithm outperforms the baseline in both convergence and reward performance, and successfully reduces user service response costs. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a three-layer heterogeneous vehicle edge computing environment;

[0047] Figure 2 A diagram illustrating the migration process of user service instances;

[0048] Figure 3 A diagram of the improved QMIX algorithm framework;

[0049] Figure 4(a) is a schematic diagram of the intelligent agent behavior trajectory embedding module;

[0050] Figure 4(b) is a schematic diagram of the agent state trajectory embedding module;

[0051] Figure 5 A schematic diagram of the agent role representation module;

[0052] Figure 6 A schematic diagram of a hybrid network for attention guidance;

[0053] Figure 7 A block diagram of a distributed service migration algorithm for vehicle-mounted edge computing;

[0054] Figure 8 This is a schematic diagram illustrating the convergence of model training.

[0055] Figure 9(a) Comparison of average service costs of the algorithms;

[0056] Figure 9(b) shows a comparison of the average service latency of the algorithms;

[0057] Figure 9(c) shows the comparison of average service energy consumption of the algorithms. Detailed Implementation

[0058] (a) Construction of system model and optimization objectives;

[0059] (1) Constructing a three-layer heterogeneous vehicle edge computing environment: Considering the characteristics and functions of each entity in the VEC scenario, this invention designs a three-layer distributed VEC network architecture that includes a large number of vehicle users, namely the cloud center layer, the edge layer and the user layer.

[0060] 1) Cloud Central Layer: The cloud central layer assists in the centralized training of algorithm models. Considering that mobile users with limited resources cannot afford model training, the cloud central layer undertakes the training work. The trained agent is then deployed in a distributed manner on each edge server to help it formulate service migration strategies.

[0061] 2) Edge Layer: The edge layer consists of a group of edge servers and base stations. Each base station is integrated with an edge server in the same location, and they can also be collectively referred to as edge nodes m∈M={1,2,…,|M|}. Each edge node provides heterogeneous computing and communication resources. Servers can transmit data with other servers and mobile users, and multiple services can be deployed on each server. Wireless backhaul links are used for communication between base stations, and cellular networks are used for communication between base stations and users.

[0062] 3) User Layer: The user layer consists of a group of vehicle mobile users u∈{1,2,…,|U|}. Each user can continuously send task requests (such as requests to update autonomous driving service maps) to the edge nodes they are connected to. Vehicle users move within the coverage area of ​​the edge nodes and connect to the base station with the strongest signal in their area. Adjacent base stations can communicate with each other through backhaul links to receive and forward task requests from vehicle users, and migrate services deployed by users when necessary.

[0063] This invention discretizes time into multiple equal time slots t∈T={1,2,…,|T|}, and sets the vehicle user u to remain in the same position in each time slot, but to a new position in the next time slot. The position in each time slot is (x u (t),y u (t)). The server location is set initially and will not change over time; the location is Loc. m Define variable c. u,m (t)∈{0,1} indicates whether user u is directly connected to server m at time slot t (i.e., the user uploads the task request to the server). When a user is within the coverage area of ​​multiple overlapping base stations, the default is to connect to the nearest base station.

[0064] (2) Constructing a user service and task request model: In this invention, the entities that connect vehicle users and edge nodes include user services and user-uploaded task requests.

[0065] 1) User Services: This invention sets up an Operating System (OS) on each edge server as the underlying support environment to support task processing for different intelligent services and operation of upper-layer software. Various edge services required by vehicle users are deployed on the OS, with each vehicle user corresponding to one service. Initially, these services are randomly deployed on an edge server. One edge server can deploy multiple services (such as navigation services) and create corresponding Service Instances (SIs) for connected users. These SIs are specifically designed to handle user computational tasks and store and manage user-specific state context data, such as real-time task processing status and private data. Define variable p. u,m (t)∈{0,1} represents the service instance SI of user u at time slot t. u Whether to deploy on server m. Subsequent user-generated task requests (such as updating the current map) need to be offloaded to the instance created for them by the server for calculation. The instance size and other attributes vary for different service types and remain fixed after creation.

[0066] 2) Task Request: For each vehicle user, a new task request for that user's service is generated at the beginning of each time slot in a Poisson distribution. This task request needs to be processed and responded to on the service instance corresponding to that user. At the beginning of each time slot, the task request e generated by the mobile user... u (t) is defined as the following tuple.

[0067] e u (t)=(ι u (t),s u (t),v u (t),λ u (t),d u (t)) (1)

[0068] Among them, ι u (t) represents the index of the task, s u (t) represents the size of the service instance requested by the task (in bits, corresponding to one service type), v u (t) represents the task request size (in bits), λ u (t) represents the computational intensity required for this task (in CPU circles / bit), d u (t) represents the maximum tolerable delay. Task requests are real-time and heterogeneous; different users and different time slots have different data sizes, computational resource density requirements, and varying deadlines.

[0069] (3) Constructing an uplink transmission cost model: Uplink transmission cost refers to the channel link latency and energy consumption incurred by the user during the process of uploading task requests to the edge node, which is an important factor affecting service response performance. Due to the need to consider the heterogeneity of wireless resources, it is necessary to consider not only the decay of path loss, but also the real-time changing channel gain G, as shown below.

[0070] G(t)=[g u,m (t), u∈U, m∈M] U×M (2)

[0071] Specifically, the rate at which user u uploads data to edge node m It can be calculated using Shannon's formula, as shown in the following equation.

[0072]

[0073] Among them, B u,m This represents the bandwidth used for communication between user u and edge node m; g u,m (t) represents the real-time channel gain, P u This indicates the wireless transmission power of user u. To mitigate interference noise. After calculating the uplink transmission rate, user u requests task l in time slot t. u (t) Wireless uplink transmission delay from mobile user u to server m It can be expressed as the following formula.

[0074]

[0075] Among them, v u (t) represents the task request size. This is based on the user's wireless transmit power P. u The energy consumption generated by the upload task can be obtained as follows: In summary, the uplink transmission cost of the task is shown in the following formula.

[0076]

[0077] in, d represents the weighting constant. u (t) represents the maximum tolerable delay, Ω up This represents the timeout penalty incurred during task upload (which is a constant and much larger than the transmission cost). Since the data returned after task processing is usually much smaller than the size of the uplink task, this invention ignores the return cost.

[0078] (4) Construct a backhaul forwarding cost model: When user u's task request l u (t) After being uploaded to edge node m, if the service instance SI can respond to service requests from user u. u Deployed on another edge node m′, the task needs to be forwarded to server m′ using the wireless backhaul link between base stations. This introduces latency and energy consumption during service request forwarding in the backhaul link. The transmission link between base stations is relatively stable. The average wireless backhaul transmission rate from edge node m to m′ is... It is represented as follows.

[0079]

[0080] Similarly, B u,m This represents the bandwidth for communication between edge nodes m and m′; g m,m′ P represents the channel gain. m This represents the wireless transmission power of base station m. To mitigate interference noise. After calculating the backhaul transmission rate between edge nodes, user u requests task l in time slot t. u (t) Backhaul forwarding delay from edge node m to edge node m′ It can be represented as follows.

[0081]

[0082] Where, σ m (t) is a positive coefficient, h m,m′ Indicates the number of forwarding hops, v u (t) represents the task request size. When the edge node where user u's service instance resides in time slot t is the same as the edge node directly connected to the user, the backhaul forwarding cost is 0. This is based on the wireless transmit power P of base station m. m The energy consumption for task forwarding can be obtained as follows: In summary, the backhaul forwarding cost of the task is as follows.

[0083]

[0084] in, d represents the weighting constant. u (t) represents the maximum tolerable delay, Ω tran This represents the penalty caused by a timeout in the task return forwarding (which is a constant much larger than the forwarding cost).

[0085] (5) Construct a task computation cost model: Task l generated by vehicle user u in time slot t u (t) After being uploaded to the edge node m directly connected to it, the task arrives at the edge server m′ where its service instance is deployed after necessary forwarding. Server m′ then begins processing (computing) the task request from user u and returns the result (since the result is often small, the return transmission is ignored). Similarly, the cost of computing the task request on the edge server includes computation latency and energy consumption, which mainly depends on the size of the task data unloaded by the user and the amount of computing resources allocated to the mobile user by the server. When the server needs to process tasks from multiple users, it allocates computing resources to each user as fairly as possible; therefore, the computing resources allocated to user u are l. u,m' (t) is shown in the following formula.

[0086]

[0087] Among them, L m′ C represents the total available computing resources of server m′. m′ (t) represents the number of user service instances it carries in time slot t. From this, we can obtain the task request l. u The computation time (t) on server m' is as follows.

[0088]

[0089] Among them, v u (t) represents the task request size, λ u (t) represents the computational demand intensity of the task. ξ represents the energy consumption per unit time generated by the edge server processing the task. Similarly, the energy consumption of the computation task can be obtained as follows: In summary, the computational cost of the task on the server as follows.

[0090]

[0091] in, d represents the weighting constant. u (t) represents the maximum tolerable delay, Ω comp This represents the penalty (which is a constant and much larger than the cost) incurred due to task computation timeout.

[0092] (6) Constructing a service migration cost model: At the beginning of each time slot, each edge node (server) acts as the decision-making body to make migration decisions for the service instances deployed for each user. When a decision is made to migrate a service, the source server and the destination server need to perform shutdown packaging and recovery work on the user service instances to be migrated. The time required is determined by the type, size, and maximum computing resources required by the service instance. The downtime loss of the service from the source server m is shown in the following formula.

[0093]

[0094] Among them, s u The size of a user service instance (in bits) is determined by the service type; This indicates the computational effort required to package the service requested by the user, which is determined by the service type; f u,m (t) represents the computing resources allocated to user u by time slot server m in time slot t; I u This represents the proportion of computing resources used for migration, determined by the service type. Similarly, the service recovery startup time on the target server m is shown in the following formula.

[0095]

[0096] The migration service requires synchronizing and forwarding the user's entire stateful service instance from the source server m to the target server m′. The forwarding time required for service migration is shown in the following formula.

[0097]

[0098] Taking into account the service downtime, forwarding, and restart processes, the service instance SI for user u is migrated. u Service migration delays caused As shown in the following formula.

[0099]

[0100] According to the base station's wireless transmission power P mBy calculating the energy consumption ξ per unit time for the server, the energy consumption for service migration can be obtained as follows: In summary, the user's task request l u The service migration cost incurred during the processing of (t) is shown in the following formula.

[0101]

[0102] in, d is the weighting constant. u (t) represents the maximum tolerable delay, Ω mig Penalties for service migration timeouts (far exceeding the constant cost).

[0103] This invention adopts a distributed decision-making approach, with each VEC server (edge ​​node) acting as a decision-making body. In each time slot, it makes migration decisions for the service instances (users) it carries, i.e., the target server for migrating services, thereby minimizing user service response costs.

[0104] 1) Decision Variables: Due to the high-speed mobility of vehicles, as users move further away from the server, to provide them with the lowest possible latency and highest quality service experience, it is necessary to migrate the user's service instance (SI) to a more suitable server. However, the service migration process requires downtime and data transfer operations, necessitating a decision on whether to migrate the SI and determining the target server for the migration. Define variable α. u (t)∈{0,…,|M|} represents the target server for the service instance of user u to migrate in time slot t. The set of target servers for all users at time t is shown in the following formula.

[0105] α(t)={α1(t),…,α |U| (t)} (17)

[0106] If α u (t) = m∈M, indicating that at time slot t, the target server for the service instance migration of user u is m, and the service deployment variable p is updated simultaneously. u,m (t) = 1.

[0107] 2) Optimization Objective: For the service migration problem in a three-layer heterogeneous vehicular edge system, the ultimate goal is to minimize the weighted sum of latency and energy consumption of all tasks, i.e., the cost. The optimization objective function is defined as follows.

[0108]

[0109]

[0110] Where α(t) is the migration decision made by the edge node for all vehicle users in time slot t; Formula (19) indicates that the migration target of each user service can only be the edge server in the environment; Formulas (20) to (23) indicate that each user can only connect to one base station in a certain time slot, and its service instance can only be deployed on one edge server; Formula (24) indicates that the computing resources allocated to the user cannot exceed the total resource limit of the server.

[0111] The distributed decision-making scheme proposed in this invention requires each edge server to make efficient service migration decisions based on the real-time acquisition of the location information and task request information of the terminal devices it deploys (with the environment being partially observable). This determines which edge server the service will migrate to, thereby reducing response latency and energy consumption on the device side and improving the user experience.

[0112] The distributed solution to the above system objective optimization problem can be viewed as a multi-agent collaborative process. Therefore, this invention models the above problem as a decentralized partially observable Markov Decision Process (Dec-POMDP) ​​and proposes a multi-agent deep reinforcement learning method based on improved QMIX (Q-Value Mixing, QMIX) to obtain service migration decisions, which are defined in detail below.

[0113] 1) Intelligent Agent: In the distributed solution proposed in this invention, the decision-making body, i.e., the edge nodes, is regarded as the intelligent agent. The algorithm model is centrally trained by the cloud center layer during the training phase, and the trained intelligent agent model is deployed to each edge node during the execution phase, so that the edge nodes make service migration decisions based on their own observations.

[0114] 2) Observation: The agent's observations in each time slot include all task information received by the agent, the user's current location, the number of services carried by the server, etc., represented as O. m (t)={l u (t),x u (t),y u (t),C m (t)}, and needs to satisfy That is, the service instance of user u is deployed on edge node m in time slot t.

[0115] 3) State: Represented as the joint observation state S(t) of all agents = {O1(t),…,O1(t),…,O2(t)} |M| (t)}.

[0116] 4) Action: The migration action performed by the edge node for user service in each time slot is represented as A.m (t)={α1(t),…,α |U| (t)}, where if p u,m If (t) = 1, then the action α u (t) is valid, otherwise it is invalid. The joint action of all agents can be represented as A(t)={A1(t),…,A1(t),…,A1(t)…} |M| (t)}.

[0117] 5) Reward: In this invention, since all agents have the same goal, they share the same reward function. The reward for each time slot is calculated using this function.

[0118] (2) Constructing a Multi-Agent Deep Reinforcement Learning Algorithm Framework: In the process of service migration decisions within a dynamic VEC environment, agents cannot fully observe the global state of the environment and need to rely on local observations for decision-making. Furthermore, the generated actions are discrete and agents share the same reward function. This invention is based on the QMIX algorithm, employing centralized training and distributed execution of trained agents to solve the agent's decision for each time slot. QMIX is an algorithm for multi-agent deep reinforcement learning. Its design goal is to solve the credit allocation problem in multi-agent cooperation by decomposing the global action-state value function (Q-value), while ensuring the independence of individual policies. The core structure of QMIX mainly consists of two parts: a local Q-network and a hybrid network.

[0119] 1) Local Q-Network: A local Q-network is trained independently for each agent using a local Q-value function Q. m (O m A m ), m∈M, is used to estimate the behavioral value of the agent in the local observation and action space. The input is the policy learned by each agent based on local information, ensuring a certain degree of independence for the agents. To improve training efficiency, all agents share the same set of local Q-network parameters. The agent's local Q-network relies only on local observations and does not consider global information, which facilitates distributed execution.

[0120] 2) Hybrid Networks: Hybrid networks are used to non-linearly combine the local Q-values ​​of all agents into a global Q-value, representing the joint behavioral value of the entire multi-agent system. This enables global cooperation and optimization, while utilizing additional information provided by the global state during intensive training to guide the hybridization process. Hybrid networks are designed to satisfy the monotonicity constraint of the global Q-value with respect to the local Q-values ​​of each agent, ensuring that an increase in local value does not lead to a decrease in global value. This is achieved through the non-negativity of the weight network. Hybrid networks can capture the non-linear relationships between local Q-values, enhancing the system's representational capabilities.

[0121] (3) This invention constructs an agent behavior trajectory embedding module and an agent state trajectory embedding module in QMIX based on a Gated Recurrent Unit (GRU). The GRU can compensate for the inability to handle partial observability in dynamic VEC environments when agents make decisions based on current observations in distributed decision-making schemes by encoding the agent's historical trajectory information (including observation and action sequences). Furthermore, since service migration in vehicular edge computing requires both independent agent decision-making and overall system coordination, this invention constructs behavior trajectory embedding modules and state trajectory embedding modules in the agent's local Q-network and hybrid network in QMIX, respectively, maximizing the ability to model local and global time dependencies. During training, agent m extracts tuples containing observation states, rewards, and actions from the experience replay buffer, and simultaneously combines these with the hidden state e generated by the GRU network in the behavior trajectory embedding module. m (t) Generates the Q value. m (t) represents the embedding of the local behavioral trajectory of agent m. m The formula for calculating (t) is as follows.

[0122] e m (t)=f φ (O m (t),A m (t-1),e m (t-1)) (25)

[0123] Where φ represents the shared GRU encoder of the agent, O m (t) represents the current observation of agent m, A m (t-1) represents the last action of agent m, e m (t-1) is the previous hidden state of the GRU. Meanwhile, in the hybrid network, this invention uses GRU to encode the historical global state {S0,…,S}. T}, thus obtaining the state embedding τ={τ0,…,τ T In QMIX, the goal of the hybrid network is to aggregate the local Q-values ​​of all agents into a single global Q-value for overall optimization. Aggregation based solely on the current global state may miss important information relevant to long-term policies. By encoding historical state information, the hybrid network can more accurately predict the global Q-value, thereby generating a more reasonable trust value allocation.

[0124] (4) Constructing an Agent Role Representation Module: Due to the sharing of local Q-network parameters among agents in QMIX, multiple agents tend to learn the same or highly similar behavioral policies in similar environments, thus affecting the overall performance and diversity of the system. This invention simultaneously sets up GRU modules in both the agent's local network and hybrid network. Agents share the same GRU network structure and parameters, resulting in highly similar extracted hidden states under similar input observations and action histories, further leading to convergent policy behaviors. To address this homogenization problem of agent policies, this invention designs an agent role representation module. The aim is to learn a compact role representation that can characterize the complex behavioral patterns of agents and utilize role information to promote individual policy learning and guide agent collaboration. Agents with similar roles can achieve higher learning efficiency through more proactive knowledge transfer, while ensuring the heterogeneity of agents by distinguishing diverse roles.

[0125] 1) Role Representation Definition: In the cooperative multi-agent task (distributed decision-making scheme) proposed in this invention, each agent m∈M is associated with a role. m The role is associated with an agent and describes its behavioral patterns. Defining roles allows for more effective coordination among agents, avoids homogenization issues, and improves task efficiency. m The behavioral pattern is represented by a role z m ∈Z is used for quantization, and this representation is obtained by training a complex function f, as shown in the following equation.

[0126] z m =f(ω) m (26)

[0127] Where, ω m ω represents the behavioral trajectory of agent m, indicating the sequence of observations and actions performed by the agent over the entire time period. m It provides rich information, including the agent's decision-making behavior in different states and its interaction with the environment. Naturally, the distance between agent embeddings obtained by the agent behavior trajectory embedding module can serve as an indicator for quantifying the agent's role representation. After introducing the role representation Z through the agent's local trajectory, the individual policy of agent m can be further improved. As shown in the following formula.

[0128]

[0129] Individual strategies It not only considers the agent's current local observations, but also incorporates the role representation z. mThis mechanism provides behavioral pattern information, enabling agents to make differentiated decisions based on their roles, thus achieving more efficient collaboration. The introduction of role representation effectively alleviates the homogenization problem, allowing agents with the same parameter-sharing mechanism to exhibit personalized behaviors through role-based division of labor, thereby improving the overall collaborative efficiency and flexibility of the task.

[0130] 2) Role Representation Discrimination Method: An ideal role representation is an abstract generalization that focuses on the agent's behavioral patterns while ignoring the influence of the agent's individual identity. Mutual information can effectively measure the interdependence between a role and its representation. Therefore, this invention introduces a mutual information maximization mechanism to formalize the learning of role representations and derives a contrastive learning objective. This objective optimizes the agent's behavioral trajectory embedding {e1,…,e...} in a self-supervised manner. |M| Finally, the role representation of each agent is obtained as {z1,…,z}. |M| Specifically, this invention trains a role encoder θ, which can be mathematically formalized as a probabilistic encoder, i.e., z(t) ~ f. θ (z(t)|e(t)). The role follows the distribution P(Role), and the distribution of the agent's behavioral trajectory embedding e is determined by its role. The learning objective of the role encoder is as follows (maximizing the role representation and the mutual information between roles).

[0131]

[0132] In practice, directly optimizing mutual information is not feasible. Therefore, this invention introduces Information Noise Contrastive Estimation (InfoNCE) from the field of contrastive learning to indirectly maximize mutual information. Specifically, InfoNCE provides a lower bound on mutual information; by optimizing this lower bound, mutual information can be approximately maximized, as shown in the following equation.

[0133]

[0134] Where K = |Role| represents the number of roles, and h(e,z) = p(z|e) / p(z). For each role in the set of roles... * Each of these will generate a corresponding embedded e *It captures the character's behavioral patterns. Since the joint probability distribution of the character representation and the embedding of the agent's behavioral trajectory, as well as their conditional probability distributions p(z|e) and p(z), are usually not directly computable, this invention employs Noise Contrastive Estimation (NCE) and importance sampling techniques. By comparing the target value (positive sample) with a set of randomly sampled negative samples, a sampled version of the tractable lower bound is derived and used as the learning target of the character encoder, as shown in the following equation.

[0135]

[0136] Where S(z,z) * ) is a similarity scoring function that represents a measure of similarity between the latent codes of two samples. This invention approximates h using the exponent of this function. e and e′ are derived from the role... m Given two agent embedding instances sampled from a dataset, z and z′ are latent representations of e and e′. For any role * ∈RoleRrole m , z * It is a role * The corresponding intelligent agent embedding e * The representation of [the concept]. This invention denotes two embedded instances (e, e′) of the same role as a positive sample pair, and [denotes] the set of embedded instances of other different roles {(e, e′)}. * The negative sample pairs (set) are denoted as . The improved lower bound objective is to optimize a K-classification loss to correctly classify positive sample pairs from all sample pairs. Minimize the InfoNCE loss. This is equivalent to maximizing the lower bound of mutual information, and this lower bound becomes tighter as K increases. The role encoder extracts shared features from the embeddings of agents with the same role to maximize the similarity score of positive sample pairs, while capturing key differences between different roles to reduce the similarity score of negative sample pairs. This invention periodically divides all agents into K clusters based on their embeddings. For agent m, its role representation is z. m The query vector q is set as the query vector, and the roles of other agents are represented as keys. Agent roles that are in the same cluster as agent m are considered positive keys {k}. + Other intelligent agent roles are considered as negative keys. This invention uses bilinear product to construct a similarity scoring function, and the similarity between the query and the key is calculated as follows: Where W is a learnable parameter matrix, the final InfoNCE loss function can be obtained as shown in the following equation.

[0137]

[0138] During the training of the character encoder, this invention maintains a character encoder θ. q and a momentum encoder θ k Furthermore, it promotes consistency in bond representation through a momentum update mechanism, the formula of which is shown below.

[0139] θ k ←βθ k +(1-β)θ q (32)

[0140] Where β∈[0,1) is the momentum coefficient, and only the parameter θ of the query encoder is considered. q It will be updated through backpropagation.

[0141] (5) Constructing an improved hybrid network module: In VEC environments with a large number of vehicle users, existing distributed service migration schemes all face the problems of an excessively large joint observation space and interference from invalid information from other agents. Existing methods directly use the joint observations of all agents as a single input vector in the hybrid network of QMIX, lacking consideration of the above problems, resulting in poor performance. At the same time, the scheme of this invention needs to concatenate the learned role representations and global states together to generate the weights of the hybrid network. Direct concatenation cannot fully utilize the internal structure to effectively extract the correlations in the role space. Therefore, this invention introduces a multi-head attention module into the hybrid network, which can prompt the global state to pay attention to the learned role patterns, thereby providing more expressive credit allocation in value decomposition. At the same time, the attention mechanism helps the hybrid network pay attention to key information in the global state, indirectly compressing the state space and reducing interference from invalid information between agents. This invention constructs a GRU-based embedding module in the hybrid network of QMIX, encoding historical states as state trajectory embeddings {τ0,…,τ T This facilitates information matching between state and role representations. Subsequently, the invention sets the state trajectory embedding τ as the query for the multi-head attention mechanism, sets the role representation z as the key and value for the multi-head attention mechanism, and weighted combinations of role representations τ. atten As shown in the following formula.

[0142]

[0143] Where, μ m Through the shared parameter matrix W V For z m The attention weights are obtained through a linear transformation. Compute the state embedding τ and the role representation z of the m-th agent. m To determine the correlation between the two factors, this invention uses a softmax function to obtain the weights. As shown in the following formula.

[0144]

[0145] Among them, W Q and W K It is a shared parameter matrix used for querying key-pair linear transformations. This is a factor for scaling the dot product attention. This invention uses multi-head attention to enable the model to simultaneously focus on positional information from different representation subspaces, and the aggregated output is represented as follows.

[0146]

[0147] in, Is using W h Q W h K and W h V The attention output calculated from the projection, h∈H={1,…,|H} represents the number of heads in the multi-head attention module, W O This is a parameter matrix combining the outputs of all heads. Finally, the multi-head attention output is combined with the global state to generate the weights of the hybrid network. This invention implicitly guides the coordination of agents in cooperative tasks through a multi-head attention mechanism, thereby enabling the hybrid network to achieve more expressive credit allocation and reducing interference from invalid information among agents.

[0148] Algorithm 1 is the pseudocode for training the distributed service migration algorithm proposed in this invention.

[0149]

[0150]

[0151] The hardware and software environment for this implementation plan is shown in Table 1.

[0152] Table 1 Hardware and software configuration environment

[0153] name Configuration operating system Windows 11 CPU Intel Core i7 Memory 16G harddisk 1T Programming Language Python Development framework PyTorch

[0154] The following detailed description of the specific implementation of the present invention, in conjunction with the accompanying drawings, illustrates this experiment.

[0155] Step 1: Construct a three-tier VEC system: such as Figure 1As shown, this invention divides entities in the VEC into a cloud center layer, an edge layer, and a user layer. The cloud center layer includes a cloud server responsible for centralized model training. The edge layer comprises eight base stations evenly distributed within a 1 km × 1 km area. This invention sets up an edge server on each base station, collectively referred to as an edge node, responsible for carrying service deployments and unloading task requests from vehicle users, and performing service migration decisions (selecting a target server) for the user service instances it carries at the beginning of each time slot. The user layer contains 100 vehicle users, randomly distributed in the environment, moving in random directions each time slot. Each user has a service instance deployed on an edge node, generating task requests in a Poisson distribution each time slot. Each user communicates with the nearest edge node in each time slot and uploads task requests to the corresponding service instance for processing via uplink and backhaul links.

[0156] Step 2: Constructing the service response cost model in the VEC system: This invention uses the weighted sum of latency and energy consumption in processing task requests uploaded by vehicle users as the service response cost. This includes the task request upload cost, task request forwarding cost, and task request computation cost. If a service migration decision is made based on the service instance of a user at the edge node, requiring the service instance to be migrated, the service response cost will also need to be added to the service migration cost. The service migration process is as follows... Figure 2 As shown. Service migration costs include latency and energy consumption during downtime, transmission, and restart processes. The optimization objective of the algorithm in this invention is to ensure that all users continuously make reasonable service migration decisions, thereby minimizing the long-term service response costs for all users.

[0157] Step 3: Construct a distributed service migration decision algorithm based on QMIX: This invention constructs a distributed service migration scheme based on an improved QMIX algorithm. The algorithm structure is as follows: Figure 3 As shown in Figure 4, this invention constructs an agent behavior trajectory embedding module and a state trajectory embedding module in the local Q-network and hybrid network of QMIX, respectively. This enables the model to make greater use of historical information and solves the accuracy problem caused by the observability of the environment in agent decision-making. The specific structures are shown in Figure 4. Simultaneously, this invention constructs a role representation module in the local Q-network, using agent behavior trajectory embedding and contrastive learning to obtain the role representation of each agent, thereby solving the homogenization problem of agent decision-making. The specific structure is shown in Figure 4. Figure 5 As shown. Finally, in order to enable the hybrid network to better focus on key information between agents (especially role representations) and reduce interference from irrelevant information, this invention utilizes a multi-head attention mechanism to guide the hybrid network, resulting in better credit allocation. The specific structure is as follows: Figure 6 As shown. Ultimately, Figure 7 The algorithm flow for distributed service migration for vehicle-mounted edge computing is demonstrated.

[0158] Step 4, Experimental Analysis: Figure 8 The convergence of the proposed algorithm is demonstrated in Figure 9(a), which shows a comparison between the average service cost of the proposed algorithm and the baseline; Figure 9(b) shows a comparison between the average service latency of the proposed algorithm and the baseline; and Figure 9(c) shows a comparison between the average service energy consumption of the proposed algorithm and the baseline. Because this invention makes greater use of historical information and focuses on enhancing the flexibility of the agent's role and the ability to extract key information, the algorithm demonstrates good performance.

[0159] In summary, the distributed service migration scheme for vehicle-mounted edge computing designed in this invention has been implemented, and the superior performance of each proposed module has been verified.

Claims

1. A distributed service migration method for vehicle-mounted edge computing, characterized in that, A three-layer heterogeneous vehicle edge computing environment is established, and a distributed service migration scheme is obtained in the three-layer heterogeneous vehicle edge computing environment by using an improved multi-agent deep reinforcement learning algorithm. The three-layer heterogeneous vehicle edge computing environment consists of three entities: a cloud center layer, an edge layer, and a user layer. The user layer is responsible for generating tasks and prioritizing their offloading to nearby edge nodes to reduce latency. The edge node layer is responsible for processing tasks locally. The cloud center layer provides global optimization support to ensure the efficiency and reliability of the entire system. The improved multi-agent deep reinforcement learning algorithm models the optimization objective as a decentralized partially observable Markov decision process. The improved multi-agent deep reinforcement learning algorithm is based on the improved QMIX algorithm, which constructs a distributed service migration strategy, enabling each agent to make service migration decisions based on local information and shared role information. Based on the agent's historical trajectory information, the algorithm constructs agent behavior state trajectory embeddings for the local Q network and the hybrid network to capture time-related behavior patterns and solve some observability problems. Furthermore, a role encoder based on contrastive learning is used to extract the unique behavior patterns of different agents to generate role representations, thereby capturing the long-term features of agent behavior. Introduce a multi-head attention mechanism into the hybrid network to dynamically allocate attention weights; The improved local Q-network is a local Q-network of the QMIX algorithm reconstructed by building an agent behavior trajectory embedding module and an agent role representation module; The improved hybrid network is a hybrid network that reconstructs the QMIX algorithm by constructing an agent state trajectory embedding module and a multi-head attention module; The multi-head attention module specifically embeds the state trajectory. Queries configured with a multi-head attention mechanism will represent roles. Set the keys and values ​​for the multi-head attention mechanism.

2. The distributed service migration method for vehicle-mounted edge computing according to claim 1, characterized in that, The improved multi-agent deep reinforcement learning algorithm is specifically as follows: (1) Construct decision variables and optimize target model: Adopt a distributed decision-making approach, with each edge node as the decision-making body, and make migration decisions for the service instances it carries in each time slot, i.e., the target server for the migration service, to minimize the user service response cost; 1) Decision variables: Define the variables Indicates user Service instances in time slots The target server for migration; all users The target server set at any given time is shown in the following formula; （1）； , indicating time slot Time user The target server for the service instance migration is At the same time, update the service deployment variables. ; 2) Optimization objective: For the service migration problem in a three-layer heterogeneous vehicle edge environment, the ultimate goal is to minimize the weighted sum of latency and energy consumption of all tasks, i.e., the cost; the optimization objective function is defined as follows; （2）； Formula (2) represents minimizing the long-term response cost for all users based on service migration decisions, including the cost of uploading service requests. The cost of forwarding service requests Cost of access request calculation Cost of service migration ; It is an edge node for all vehicle users The migration decision is made in the time slot; the migration target of each user service can only be the edge server in the environment; each user can only connect to one base station in a certain time slot, and its service instance can only be deployed on one edge server; the computing resources allocated to users cannot exceed the total resource limit of the server.

3. The distributed service migration method for vehicle-mounted edge computing according to claim 1, characterized in that, In the improved QMIX algorithm, the agent is the edge node, the observation is all task information received by the agent in each time slot, the user's current location, and the number of services carried by the server, the state is the joint observation state of all agents, and the action is the migration decision made by the edge node for the user service in each time slot. The local Q network and the hybrid network in the QMIX algorithm are improved as follows to obtain the improved local Q network and the improved hybrid network.

4. The distributed service migration method for vehicle-mounted edge computing according to claim 3, characterized in that: For the improved local Q network in each time slot Each intelligent agent Current observation Actions in the previous time slot The input is fed into a fully connected neural network for initial feature extraction. The extracted features are then embedded along with the agent's behavioral trajectory from the previous time slot. The input is fed into the agent behavior trajectory embedding module to generate the agent behavior trajectory embedding for the current time slot. ; Embed the agent's behavior trajectory The input is fed into the agent role representation module to generate the role representation of the current agent. Finally, the role representation is embedded with the previously generated agent behavior trajectory. Input a fully connected layer to obtain the Q-value of the agent's local Q-network based on the current observation.

5. The distributed service migration method for vehicle-mounted edge computing according to claim 4, characterized in that, The intelligent agent behavior trajectory embedding module specifically involves: in each intelligent agent... In a local Q-network, the behavioral trajectory of the agent is based on the gated recurrent unit (GRU). Encode the agent's behavioral trajectory embedding; the agent After obtaining the observations and actions, the hidden state of the previous step is generated by the GRU network in the agent behavior trajectory embedding module. Generate the current hidden state ; That is, intelligent agents behavioral trajectory embedding; The calculation formula is as follows; （3）； in, The shared GRU encoder representing the agent, Represents intelligent agents Current observations Represents intelligent agents The last action, This is the previous hidden state of GRU.

6. The distributed service migration method for vehicle-mounted edge computing according to claim 4, characterized in that, The intelligent agent role representation module is specifically as follows: a) Role Representation Definition: In collaborative multi-agent tasks, each agent... All related to one character Relatedly, the role is used to describe the behavioral patterns of the agent; role Behavioral patterns are represented by a role To perform quantization, meaning through a trained character encoder The result is shown in the following formula; （4）； in, It is an intelligent agent The behavioral trajectory, i.e., the sequence of observations and actions of the agent over the entire time period; behavioral trajectory This includes the agent's decision-making behavior under different states and its interaction with the environment; the distance between the agent's behavior trajectory embeddings obtained by the agent behavior trajectory embedding module is used as an indicator to quantify the agent's role representation; and the introduction of role representation through the agent's local trajectory. Subsequently, further improvements were made to the intelligent agent. Individual strategies As shown in the following formula; （5）； in, Represents the observations of an intelligent agent; Represents the actions of the intelligent agent; This represents the role representation of the agent; in this context, the individual policy. It not only considers the agent's current local observations, but also combines role representations. Provided behavioral pattern information; b) Role Representation Discrimination Method: A mutual information maximization mechanism is introduced for learning formal role representations, and a contrastive learning objective is derived through InfoNCE; the contrastive learning objective optimizes the role encoder of each agent in a self-supervised manner, and the role encoder embeds the agent's behavioral trajectory. The encoding process ultimately yields a suitable role representation for each agent. ; The specific approach involves periodically dividing all agents into groups based on their behavioral trajectories. A cluster, for an intelligent agent Through a character encoder Generate its role representation And set it as the query vector. ; via a momentum encoder Generate role representations for other intelligent agents, as keys ; with intelligent agents Agents in the same cluster are considered positive keys. Other intelligent agent roles are considered as negative keys. A similarity score function is constructed using bilinear product; the similarity between the query and the key is calculated as follows: ,in It is a learnable parameter matrix, and the final InfoNCE loss function is shown in the following formula. The contrastive learning objective is to minimize this loss function. （6）； During training, the character encoder is updated via backpropagation; the momentum encoder, which needs to generate stable samples, is updated using momentum, as shown in the formula below. （7）； in, It is the momentum coefficient.

7. The distributed service migration method for vehicle-mounted edge computing according to claim 6, characterized in that: For the improved hybrid network in each time slot The global state of the agent After processing by a fully connected layer, it is embedded with the agent's state trajectory from the previous time slot. The data is input together into the agent's state trajectory embedding module to obtain the state trajectory embedding for this time slot. Embed the state trajectory of this time slot and the current roles of all intelligent agents The input is fed into the multi-head attention module, and the mixed output of the multi-head attention module is obtained. ; Output With global state The concatenation, along with the local Q-values ​​of all agents, is input into the supernetwork of the original QMIX hybrid network. A set of dynamic weights is then calculated to perform a weighted summation of the local Q-values ​​of each agent, ultimately yielding the global Q-value.

8. The distributed service migration method for vehicle-mounted edge computing according to claim 7, characterized in that, The agent state trajectory embedding module is based on a gated recurrent unit (GRU) to embed the historical global state containing observations of all agents. Encoding is performed to obtain the agent's state trajectory embedding; after obtaining the global state at the current moment, the agent's state trajectory embedding module combines the hidden state from the previous moment generated by the GRU network in this module. As input, generate the hidden state at the current time step. ; That is, the embedding of the agent's state trajectory; The calculation formula is as follows; （8）； in, The GRU encoder representing the global state. This represents the current global state of all agents. This is the previous hidden state of GRU.

9. The distributed service migration method for vehicle-mounted edge computing according to claim 7, characterized in that: For multi-head attention modules, the weighted combination of role representations As shown in the following formula; （9）； in, Through shared parameter matrix right Attention weights are obtained through a linear transformation. Computational state trajectory embedding and the Role representation of each agent The correlation between them is used to obtain the weights using a softmax function. As shown in the following formula; （10）； in, and It is a shared parameter matrix used for querying key-pair linear transformations. It is a factor for scaling dot product attention; multi-head attention is used to enable the hybrid network to simultaneously pay attention to positional information from different representation subspaces, and the aggregated output is represented as follows; （11）； in, Is using , and The attention output calculated from the projection. This indicates the number of heads in the multi-head attention mechanism. It is a parameter matrix that combines the outputs of all heads; finally, the multi-head attention outputs are combined with the global state to generate the weights of the hybrid network.

Images