Distributed service migration method for vehicle-mounted edge computing

Abstract

The invention belongs to the field of vehicle-mounted edge computing, and discloses a distributed service migration method oriented to a vehicle-mounted edge computing environment. According to the method, a service migration decision is optimized by constructing a three-layer heterogeneous vehicle-mounted edge network model which comprises a cloud center layer, an edge layer and a user layer based on comparison role representation and multi-agent deep reinforcement learning. The invention provides a multi-agent deep reinforcement learning method based on improved QMIX, and the method comprises the steps: constructing agent behavior state track embedding for a local Q network and a hybrid network according to the historical track information of an agent, capturing a time-related behavior mode, solving a part of observability problems, and improving the robustness of the multi-agent deep reinforcement learning. A character encoder based on comparative learning is further used for extracting unique behavior patterns of different agents to generate character representation so as to capture long-term characteristics of behaviors of the agents; a multi-head attention mechanism is introduced into the hybrid network, and attention weights are dynamically distributed. The method has excellent performance.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted edge computing technology, and in particular to a distributed service migration method for vehicle-mounted edge computing. Background Art

[0002] In recent years, with the development of 5G networks, autonomous driving, and intelligent transportation systems, modern vehicles are equipped with a large number of sensors and electronic devices, generating massive amounts of data for real-time applications such as obstacle detection, path planning, and driving assistance. However, due to limitations such as data transmission delay, traditional cloud computing architectures are difficult to meet the real-time needs of vehicle users, so vehicle edge computing (VEC) has emerged. VEC sinks computing resources and data storage to edge servers close to data sources, and vehicle users can deploy services on edge nodes to process task requests, thereby reducing service response delays. However, VEC still faces many challenges. One of the more critical technologies is dynamic service migration. The coverage of edge servers is limited, and the mobility of vehicle users causes the distance between vehicles and service servers to gradually increase. The communication between users and user services may go through multiple hops. In order to ensure service quality, services need to be dynamically migrated to more suitable edge servers, but frequent migration of services may increase migration costs. Considering the mobility of users and the dynamic nature of task requests, it is a challenge to balance migration costs and transmission delays and make the best decision.

[0003] Early research work modeled service migration decisions as Markov decision processes and used centralized algorithms based on deep reinforcement learning to solve them. These solutions require a central decision controller (often a cloud center) to obtain complete system-level information. However, vehicle-mounted edge computing is a distributed environment. The latency, bandwidth, and reliability conditions of the edge network are difficult to meet the needs of centralized collection of overall information required by the migration algorithm. In practical applications, each server can only obtain information about some of the current service users. Recent research has begun to use multi-agent deep reinforcement learning for distributed decision-making. However, due to the partial observability of environmental information, decision-making agents lack a global view and often make suboptimal decisions. In addition, the state information involved in strategy formulation includes multiple dimensions and a large number of users. The high dimensionality and complexity of the state information make it difficult for the algorithm to extract key features. At the same time, in the VEC environment, the load of each base station, the deployed users, and the service capabilities it can provide are different. However, in the service migration problem, agents often share the same rewards, resulting in the possibility that agents of all base stations may learn similar strategies, resulting in unreasonable migration decisions and inefficiency. Due to these problems, the present invention proposes a distributed service migration method for vehicle-mounted edge computing. By introducing recurrent neural networks, multi-head attention mechanisms and role representation, the deep reinforcement learning model is optimized, key state features are extracted and the problem of agent homogeneity is solved. Summary of the invention

[0004] The main content of this invention is to propose a distributed service migration method for vehicle-mounted edge computing, which is suitable for partially observable vehicle-mounted edge computing environments. It is a comparative role representation method based on attention guidance. It is a distributed service migration decision algorithm for vehicle-mounted edge computing, based on comparative role representation and multi-agent deep reinforcement learning.

[0005] The technical solution of the present invention is as follows: a distributed service migration method for vehicle-mounted edge computing, establishing a three-layer heterogeneous vehicle-mounted edge computing environment, and obtaining a distributed service migration solution under the three-layer heterogeneous vehicle-mounted edge computing environment by using an improved multi-agent deep reinforcement learning algorithm;

[0006] The entities of the three-layer heterogeneous vehicle-mounted edge computing environment are divided into a cloud center layer, an edge layer, and a user layer; the user layer is responsible for generating tasks and offloading them to nearby edge nodes first to reduce latency; the edge node layer is responsible for processing tasks locally; the cloud center layer provides global optimization support to ensure the efficiency and reliability of the entire system;

[0007] The improved multi-agent deep reinforcement learning algorithm models the optimization objective as a decentralized partially observable Markov decision process; the improved multi-agent deep reinforcement learning algorithm is based on the improved QMIX algorithm, constructs a distributed service migration strategy, so that each agent completes the service migration decision based on local information and shared role information; constructs agent behavior state trajectory embedding for the local Q network and the hybrid network according to the agent's historical trajectory information, captures time-related behavior patterns, solves the partial observability problem, and further uses a role encoder based on contrastive learning to extract the unique behavior patterns of different agents to generate role representations, so as to capture the long-term characteristics of the agent's behavior; introduces a multi-head attention mechanism in the hybrid network to dynamically allocate attention weights.

[0008] Furthermore, the improved multi-agent deep reinforcement learning algorithm is specifically:

[0009] (1) Constructing decision variables and optimizing target models: A distributed decision-making approach is adopted, with each edge node as the decision-making body. Each edge node makes a migration decision for the service instance it carries in each time slot, that is, the target server of the migration service, thereby minimizing the user service response cost;

[0010] 1) Decision variables: define variable α u (t)∈{0,…,|M|} represents the target server to which the service instance of user u migrates at time slot t; the set of target servers for all users at time t is as follows;

[0011] α(t)={α 1 (t),…,α |U| (t)} (1)

[0012] α u (t) = m∈M, indicating that the target server for the migration of the service instance of user u at time slot t is m, and the service deployment variable p is updated at the same time. u,m (t) = 1;

[0013] 2) Optimization goal: For the service migration problem in the three-layer heterogeneous vehicle edge environment, the ultimate goal to be achieved is to minimize the weighted sum of latency and energy consumption of all tasks, that is, the cost sum; the optimization objective function is defined as shown in the following formula;

[0014]

[0015] Formula (2) represents the minimization of the long-term response cost of all users based on service migration decisions, including the cost of uploading service requests Cost of service request forwarding Cost of access request calculation and service migration costs α(t) is the migration decision made by the edge node for all vehicle users in time slot t; the migration target of each user service can only be the edge server in the environment; each user can only connect to one base station in a certain time slot, and its service instance can only be deployed on one edge server; the computing resources allocated to the user cannot exceed the total resource limit of the server.

[0016] Furthermore, in the improved QMIX algorithm, the agent is an edge node, the observation is all task information received by the agent in each time slot, the current location of the user, and the number of services carried by the server, the state is the joint observation state of all agents, and the action is the migration decision made by the edge node for user services in each time slot; the following improvements are made to the local Q network and hybrid network in the QMIX algorithm to obtain an improved local Q network and an improved hybrid network.

[0017] Furthermore, the improved local Q network is a local Q network of the QMIX algorithm reconstructed by constructing an agent behavior trajectory embedding module and an agent role representation module;

[0018] At each time slot t, each agent m sends the current observation O m (t) and the action A in the previous time slot m (t-1) is input into a fully connected neural network for preliminary feature extraction. The extracted features and the agent behavior trajectory of the previous time slot are embedded in e m (t-1) is input to the agent behavior trajectory embedding module to generate the agent behavior trajectory embedding e for the current time slot m (t); embed the agent behavior trajectory into e m (t) Input to the agent role representation module to generate the current agent role representation z m (t); Finally, the role representation and the previously generated agent behavior trajectory are embedded in e m (t) Input a fully connected layer to obtain the Q value of the agent's local Q network based on the current observation.

[0019] Furthermore, the agent behavior trajectory embedding module is specifically as follows: in the local Q network of each agent m, the agent behavior trajectory {(O 1 ,A 0 ),…,(O |T| ,A |T|-1 )} to encode and obtain the agent behavior trajectory embedding; after the agent m obtains the observation and action, it combines the hidden state e of the previous step generated by the GRU network in the agent behavior trajectory embedding module m (t-1) Generate the current hidden state e m (t); e m(t) is the behavioral trajectory embedding of agent m; e m The calculation formula of (t) is as follows;

[0020] e m (t) = f φ (O m (t),A m (t-1),e m (t-1)) (3)

[0021] Where φ represents the shared GRU encoder of the agent, O m (t) represents the current observation of agent m, A m (t-1) represents the last action of agent m, e m (t-1) is the previous hidden state of GRU.

[0022] Furthermore, the agent role representation module is specifically:

[0023] a) Role representation definition: In a collaborative multi-agent task, each agent m∈M is associated with a role m ∈Role, the role is used to describe the behavior mode of the intelligent agent; role role m The behavior pattern is represented by a role m ∈Z for quantization, which is obtained by training a role encoder f, as shown in the following formula;

[0024] z m =f(ω m ) (4)

[0025] Among them, ω m is the behavior trajectory of agent m, that is, the observation and action sequence of the agent in the entire time period; the behavior trajectory ω m It includes the decision-making behavior of the agent in different states and its interaction with the environment; the distance between the agent behavior trajectory embeddings obtained by the agent behavior trajectory embedding module is used as an indicator to quantify the agent role representation; after introducing the role representation Z through the agent local trajectory, the individual strategy of the agent m is further improved As shown in the following formula;

[0026]

[0027] Among them, O represents the observation of the agent; A represents the action of the agent; Z represents the role representation of the agent; in this context, the individual strategy Not only the current local observation and action of the agent is considered, but also the role representation z m Information on behavioral patterns provided;

[0028] b) Role representation discrimination method: The mutual information maximization mechanism is introduced to formalize the learning of role representation, and a contrastive learning objective is derived through InfoNCE; the contrastive learning objective optimizes the role encoder of each agent in a self-supervised manner, and the role encoder embeds the agent behavior trajectory into {e 1 ,…,e M} encoding, and finally obtain the appropriate role representation for each agent {z 1 ,…,z M};

[0029] The specific approach is to periodically divide all agents into K clusters based on the agent behavior trajectory embedding. For agent m, a role encoder θ q Generate its role representation z m , and set it as the query vector q; through a momentum encoder θ k Generate role representations of other agents as keys The agent role representations in the same cluster as agent m are considered as positive keys {k +}, other agent roles are considered as negative keys Using the bilinear product to construct the similarity score function, the similarity between the query and the key is calculated as Where W is a learnable parameter matrix, and the final InfoNCE loss function is as shown below. The comparative learning goal is to minimize the loss function;

[0030]

[0031] During training, the role encoder is updated through back propagation; the momentum encoder needs to generate stable samples, so it is updated using momentum update. The formula is as follows:

[0032] θ k ←βθ k +(1-β)θ q (7)

[0033] where β∈[0,1) is the momentum coefficient.

[0034] Furthermore, the improved hybrid network is a hybrid network that reconstructs the QMIX algorithm by constructing an agent state trajectory embedding module and a multi-head attention module; in each time slot t, the agent global state S(t) is processed by a fully connected layer and then input into the agent state trajectory embedding module together with the agent state trajectory embedding τ(t-1) of the previous time slot to obtain the state trajectory embedding τ(t) of this time slot; the state trajectory embedding τ(t) of this time slot and the current role representation z(t) of all agents are input into the multi-head attention module to obtain the hybrid output τ of the multi-head attention modulemha (t); Output τ mha (t) is concatenated with the global state S(t) and input into the super-network of the QMIX original hybrid network together with the local Q values ​​of all agents. A set of dynamic weights is calculated to perform weighted summation of the local Q values ​​of each agent and finally obtain the global Q value.

[0035] Furthermore, the agent state trajectory embedding module embeds the historical global state {S 0 ,…,S T} is encoded to obtain the state trajectory embedding of the agent; after obtaining the global state at the current moment, the agent state trajectory embedding module combines the hidden state τ(t-1) of the previous moment generated by the GRU network in the module as input to generate the hidden state τ(t) at the current moment; τ(t) is the state trajectory embedding of the agent; the calculation formula of τ(t) is as follows;

[0036] τ(t)=f φ (S(t),τ(t-1)) (8)

[0037] Among them, φ represents the GRU encoder of the global state, S(t) represents the current global state of all agents, and τ(t-1) is the previous hidden state of GRU.

[0038] Furthermore, the multi-head attention module is specifically: setting the state trajectory embedding τ as the query of the multi-head attention mechanism, setting the role representation z as the key and value of the multi-head attention mechanism, and the weighted combination of the role representation τ atten As shown in the following formula;

[0039]

[0040] Among them, μ m is the shared parameter matrix W V Right m The attention weight is obtained by linear transformation Compute the state trajectory embedding τ and the role representation z of the mth agent m The correlation between them, applying a softmax function to obtain the weight As shown in the following formula;

[0041]

[0042] Among them, W Q and W K is the shared parameter matrix for the linear transformation of query-key pairs, is the factor that scales the dot product attention; multi-head attention is used to make the hybrid network simultaneously focus on the position information from different representation subspaces, and the aggregated output is expressed as follows;

[0043]

[0044] in, Is to use W h Q , W h K and W h V The attention output calculated by the projection of , h∈H={1,…,|H} represents the number of heads of the multi-head attention mechanism, W O is the parameter matrix that combines the outputs of all heads; finally, the multi-head attention output is combined with the global state to generate the weights of the hybrid network.

[0045] Beneficial effects of the present invention: Compared with the distributed service migration algorithm in the existing vehicle-mounted edge computing system, the present invention models the edge node as an intelligent agent, and makes service migration decisions for vehicle users based on the improved QMIX algorithm. In order to improve the quality of decision-making, the present invention improves the QMIX algorithm: an intelligent agent behavior trajectory embedding module and a role representation module are introduced in the local Q network, and an intelligent agent state trajectory embedding module and a multi-head attention mechanism are introduced in the hybrid network. The intelligent agent behavior trajectory embedding module and the intelligent agent state trajectory embedding module effectively capture the long-term dependencies in service migration decisions by utilizing the historical information of the local observation and global state of the intelligent agent, and alleviate the problem of insufficient multi-agent collaboration caused by partial observability. The role representation module enables the intelligent agent to make more flexible decisions based on its own resource conditions and historical behaviors by learning the compact role representation of the intelligent agent, thereby avoiding the problem of decision homogeneity in the multi-agent system. In addition, the multi-head attention mechanism in the hybrid network can filter out the most critical information and reduce the interference of irrelevant information when facing a high-dimensional complex state space, thereby achieving accurate state space compression. This mechanism effectively integrates the global and local information in QMIX, further improving the collaborative ability of the multi-agent system. After comparative experiments with the baseline algorithm, this algorithm outperforms the baseline in terms of convergence and reward performance, and successfully reduces the user service response cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a three-layer heterogeneous vehicle-mounted edge computing environment;

[0047] Figure 2 A schematic diagram of the migration process of user service instances;

[0048] Figure 3 This is the improved QMIX algorithm framework diagram;

[0049] Figure 4(a) is a schematic diagram of the agent behavior trajectory embedding module;

[0050] Figure 4(b) is a schematic diagram of the agent state trajectory embedding module;

[0051] Figure 5 Schematic diagram of the agent role representation module;

[0052] Figure 6 Schematic diagram of the attention-guided hybrid network;

[0053] Figure 7 This is a module diagram of the distributed service migration algorithm for vehicle-mounted edge computing;

[0054] Figure 8 This is a schematic diagram of model training convergence;

[0055] Figure 9 (a) Comparison of average service costs of algorithms;

[0056] Figure 9(b) Comparison of average service delays of algorithms;

[0057] Fig. 9(c) Comparison of average service energy consumption of algorithms. DETAILED DESCRIPTION

[0058] (a) Construction of system model and optimization objectives;

[0059] (1) Constructing a three-layer heterogeneous vehicle edge computing environment: Taking into account the characteristics and functions of each entity in the VEC scenario, the present invention designs a three-layer distributed VEC network architecture that includes a large number of vehicle users, namely the cloud center layer, edge layer, and user layer.

[0060] 1) Cloud Center Layer: The cloud center layer assists in the centralized training of the algorithm model. Considering that mobile users with limited resources cannot afford model training, the cloud center layer undertakes the training work. The trained agent will then be deployed in a distributed manner on each edge server to help it make service migration strategies.

[0061] 2) Edge layer: The edge layer consists of a group of edge servers and base stations. Each base station is integrated with an edge server at the same location, which can also be collectively referred to as edge nodes m∈M={1,2,…,|M|}. Each edge node provides heterogeneous computing resources and communication resources. The server can transmit data with other servers and mobile users, and multiple services can be deployed on each server. Among them, wireless backhaul links are used for transmission between base stations, and cellular networks are used for communication between base stations and users.

[0062] 3) User layer: The user layer consists of a group of vehicle mobile users u∈{1,2,…,|U|}. Each user can continuously send task requests to the edge node to which it is connected (for example, requesting to update the map of the autonomous driving service). Vehicle users move within the coverage of the edge node and connect to the base station with the strongest signal in their own area. Adjacent base stations can communicate with each other through the backhaul link, thereby receiving and forwarding the task requests issued by the vehicle users, and migrating the services deployed by the users when necessary.

[0063] The present invention discretizes time into multiple equal time slots t∈T={1,2,…,|T|}, and sets the position of vehicle user u unchanged in each time slot, but in a new position in the next time slot. The position of each time slot is (x u (t),y u (t)). The server location is set at the initial time and will not change over time. The location is Loc m . Define the variable c u,m (t)∈{0,1} indicates whether user u is directly connected to server m at time slot t (i.e., the user uploads the task request to the server). When the user is in the overlapping coverage of multiple base stations, it is connected to the nearest base station by default.

[0064] (2) Constructing a user service and task request model: The entities that connect the vehicle user and the edge node in the present invention include the user's services and the task requests uploaded by the user.

[0065] 1) User service: The present invention sets an operating system (OS) as the underlying support environment on each edge server to provide support for task processing of different intelligent services and operation of upper-layer software. On the OS, various edge services required by vehicle users are deployed. Each vehicle user corresponds to a service, which is randomly deployed on an edge server at the beginning. An edge server can deploy multiple services (such as navigation services) and create corresponding instances (Service Instance, SI) for connected users. The instance is specifically used to process the user's computing tasks and store and manage some user-specific state context data, such as real-time task processing status, private data, etc. Define the variable p u,m (t)∈{0,1} represents the service instance SI of user u at time slot t u Whether it is deployed on server m. Subsequent task requests generated by users (such as updating the current map) need to be offloaded to the instance created by the server for calculation. The instance size and other properties of different types of services are different and are fixed after creation.

[0066] 2) Task request: For each vehicle user, a new task request for the user service is generated in the form of Poisson distribution at the beginning of each time slot. The task request needs to be transmitted to the service instance corresponding to the user for processing and response. At the beginning of each time slot, the task request e generated by the mobile user u (t) is defined as the following tuple.

[0067] e u (t) = (ι u (t),s u (t),v u (t),λ u (t),d u (t)) (1)

[0068] Among them, ι u (t) represents the index of the task, s u (t) represents the size of the service instance requested by the task (in bits, corresponding to a service type), v u (t) represents the task request size (in bits), λ u (t) represents the computing intensity required for the task processing (in CPU circles / bit), d u (t) is the maximum tolerable delay. Task requests are real-time and heterogeneous. Task requests from different users and time slots have different data sizes, computing resource density requirements, and varying deadlines.

[0069] (3) Constructing an uplink transmission cost model: The uplink transmission cost refers to the channel link delay and energy consumption incurred by users when uploading task requests to edge nodes. It is an important factor affecting service response performance. Since the heterogeneity of wireless resources needs to be considered, it is necessary to consider not only the decay of path loss but also the real-time changing channel gain G, as shown below.

[0070] G(t)=[g u,m (t),u∈U,m∈M] U×M (2)

[0071] Specifically, the rate at which user u uploads data to edge node m It can be calculated by Shannon's formula, as shown below.

[0072]

[0073] Among them, B u,m represents the bandwidth of communication between user u and edge node m; g u,m (t) is the real-time channel gain, P u represents the wireless transmission power of user u, is the interference noise. After calculating the uplink transmission rate, user u requests l in time slot t. u (t) Wireless uplink transmission delay from mobile user u to server m It can be expressed as the following formula.

[0074]

[0075] Among them, v u (t) represents the task request size. According to the user's wireless transmission power P u , we can get the energy consumption of the upload task as In summary, the mission uplink transmission cost is shown in the following formula.

[0076]

[0077] in, represents the weight constant, d u (t) represents the maximum tolerable delay, Ω up Indicates the timeout penalty caused by task uploading (much larger than the constant of transmission cost). Since the data transmitted back after task processing is usually much smaller than the uplink task size, the present invention ignores the transmission cost.

[0078] (4) Constructing a return forwarding cost model: When user u requests a task l u (t) After being uploaded to the edge node m, if the service instance SI can respond to the service request of user u u If it is deployed on another edge node m', it needs to use the wireless backhaul link between base stations to continue forwarding the task to server m'. At this time, there is a delay and energy consumption in forwarding the service request in the backhaul link. The transmission link between base stations is more stable. The average wireless backhaul transmission rate from edge node m to m' It is expressed as follows.

[0079]

[0080] Similarly, B u,m represents the bandwidth of communication between edge nodes m and m′; g m,m′ represents the channel gain, P m represents the wireless transmission power of base station m, is the interference noise. After calculating the backhaul transmission rate between edge nodes, user u requests l in time slot t. u (t) Return forwarding delay from edge node m to edge node m′ It can be expressed as follows.

[0081]

[0082] Among them, σ m (t) is a positive coefficient, h m,m′ Indicates the forwarding hop count, v u (t) represents the task request size. When the edge node where the service instance of user u is located in time slot t is the same as the edge node directly connected to the user, the backhaul forwarding cost is 0. According to the wireless transmission power P of base station m m , we can get the task forwarding energy consumption as In summary, the backhaul link forwarding cost of the task is as follows.

[0083]

[0084] in, represents the weight constant, d u (t) represents the maximum tolerable delay, Ω tran Represents the penalty caused by task return forwarding timeout (much larger than the constant of forwarding cost).

[0085] (5) Constructing a task calculation cost model: the task l generated by vehicle user u in time slot t u (t) After being uploaded to the edge node m to which it is directly connected, it is forwarded as necessary to reach the edge server m′ where its service instance is deployed. After that, the server m′ starts to process (calculate) the task request of user u and returns the result (the result is often very small, so the return is ignored). Similarly, the cost of computing the task request on the edge server includes computing delay and energy consumption, which mainly depends on the size of the task data unloaded by the user and the size of the computing resources allocated by the server to the mobile user. When the server needs to process tasks for multiple users, it tries to allocate computing resources fairly to each user as much as possible, so the computing resources allocated to user u are l u,m' (t) is shown in the following formula.

[0086]

[0087] Among them, L m′ represents the total amount of available computing resources of server m′, C m′ (t) represents the number of user service instances carried at time slot t. Thus, we can get the task request l u The computation time of (t) on server m' is as follows.

[0088]

[0089] Among them, v u (t) represents the task request size, λ u (t) represents the computing demand intensity of the task. ξ represents the energy consumption per unit time generated by the edge server processing the task. Similarly, the energy consumption of the computing task can be obtained as In summary, the computational cost of the task on the server as follows.

[0090]

[0091] in, represents the weight constant, d u (t) represents the maximum tolerable delay, Ω comp Represents the penalty caused by task calculation timeout (a constant much larger than the cost).

[0092] (6) Constructing a service migration cost model: At the beginning of each time slot, each edge node (server) acts as a decision-making body to make migration decisions for each user's service instance. When a decision is made to migrate a service, the source server and the destination server need to shut down, package, and restart the user service instance to be migrated. The time required is determined by the type, size, and maximum computing resources required of the service instance. The downtime loss time of the service from the source server m is shown in the following formula.

[0093]

[0094] Among them, s u Indicates the size of the user service instance (in bits), which is determined by the service type; Indicates the computing intensity required for the service package requested by the user, which is determined by the service type; f u,m (t) represents the computing resources allocated by server m to user u in time slot t; u It represents the proportion of computing resources used for migration, which is determined by the service type. Similarly, the service recovery startup time on the target server m is shown in the following formula.

[0095]

[0096] Migrating services requires synchronously forwarding the user's entire stateful service instance from the source server m to the target server m'. The forwarding time required for service migration is shown in the following formula.

[0097]

[0098] Considering the service downtime, forwarding and restart process, migrate the service instance SI of user u u Delays in service migration caused As shown in the following formula.

[0099]

[0100] According to the wireless transmission power P of the base station mCalculate the energy consumption per unit time ξ with the server, and the service migration energy consumption can be obtained as In summary, the user's task request l u (t) The service migration cost incurred during the processing is shown as follows.

[0101]

[0102] in, is the weight constant, d u (t) represents the maximum tolerable delay, Ω mig Penalty for service migration timeout (constant much larger than cost)

[0103] The present invention adopts a distributed decision-making method, taking each VEC server (edge ​​node) as a decision-making body, making migration decisions for the service instances (users) it carries in each time slot, that is, migrating the target server of the service, thereby minimizing the user service response cost.

[0104] 1) Decision variables: Due to the high-speed mobility of vehicles, when users are farther and farther away from the server, in order to provide vehicle users with the lowest possible latency and high-quality service experience, it is necessary to migrate the user's service instance SI to a more suitable server. However, the service migration process requires downtime and transmission operations, and it is necessary to decide whether to migrate the SI and the target server for migration. Define the variable α u (t)∈{0,…,|M|} represents the target server to which the service instance of user u migrates at time slot t. The set of target servers for all users at time t is shown in the following formula.

[0105] α(t)={α 1 (t),…,α |U| (t)} (17)

[0106] If α u (t) = m∈M, indicating that the target server for the migration of the service instance of user u at time slot t is m, and the service deployment variable p is updated at the same time. u,m (t)=1.

[0107] 2) Optimization goal: For the service migration problem in the three-layer heterogeneous vehicle edge system, the ultimate goal is to minimize the weighted sum of latency and energy consumption of all tasks, that is, the cost sum. The optimization objective function is defined as shown in the following formula.

[0108]

[0109]

[0110] Where α(t) is the migration decision made by the edge node for all vehicle users in time slot t; Formula (19) indicates that the migration target of each user service can only be the edge server in the environment; Formula (20) to Formula (23) indicate that each user can only connect to one base station in a certain time slot, and its service instance can only be deployed on one edge server; Formula (24) indicates that the computing resources allocated to the user cannot exceed the total resource limit of the server.

[0111] The distributed decision-making scheme proposed in the present invention requires each edge server to make efficient service migration decisions based on the real-time acquisition of the terminal device location information and task request information deployed by itself (the environmental part is observable), and decide which edge server the service should be migrated to, so as to reduce the response delay and energy consumption on the device side, thereby improving the user experience.

[0112] The distributed solution to the above system objective optimization problem can be regarded as a multi-agent collaborative process. Therefore, the present invention models the above problem as a decentralized partially observable Markov decision process (DecentralizedPartiallyObservableMarkovDecisionProcess,Dec-POMDP), and proposes a multi-agent deep reinforcement learning method based on improved QMIX (Q-Value Mixing, QMIX) to obtain service migration decisions, which are defined in detail as follows.

[0113] 1) Agent: In the distributed solution proposed in this invention, the decision-making body, that is, the edge node, is used as the agent. The algorithm model is centrally trained by the cloud center layer in the training phase, and the trained agent model is deployed to each edge node in the execution phase. The edge node makes service migration decisions based on its own observations.

[0114] 2) Observation: The observation of the agent in each time slot includes all the task information accepted by the agent, the current location of the user, the number of services carried by the server, etc., which is expressed as O m (t) = {l u (t),x u (t),y u (t),C m (t)}, and needs to satisfy That is, the service instance of user u is deployed on the edge node m at time slot t.

[0115] 3) State: It is represented by the joint observation state of all agents S(t) = {O 1 (t),…,O |M| (t)}.

[0116] 4) Action: The migration action made by the edge node for user service in each time slot is represented as A m (t) = {α 1 (t),…,α |U| (t)}, where if p u,m (t) = 1, then action α u (t) is valid, otherwise it is invalid. The joint action of all agents can be expressed as A(t) = {A 1 (t),…,A |M| (t)}.

[0117] 5) Reward: In this invention, since all agents have the same goal, they share the same reward function And the reward for each time slot is calculated through this function.

[0118] (2) Constructing a multi-agent deep reinforcement learning algorithm framework: Since the agent cannot fully observe the global state of the environment during service migration decision-making in a dynamic VEC environment, it needs to rely on local observations to make decisions. At the same time, the generated actions are in discrete form and the agents share the same reward function. Based on the QMIX algorithm, the present invention uses centralized training and distributed execution of training agents to solve the agent decision of each time slot. QMIX is an algorithm for multi-agent deep reinforcement learning. Its design goal is to solve the credit allocation problem in multi-agent collaboration by decomposing the global action-state value function (Q value) while ensuring the independence of individual strategies. The core structure of QMIX is mainly divided into two parts: local Q network and hybrid network.

[0119] 1) Local Q network: The local Q network independently trains a local Q value function Q for each agent m (O m ,A m ), m∈M, is used to estimate the behavior value of the agent in the local observation and action space. The input is that each agent learns its strategy based on local information to ensure that the agent has a certain degree of independence. In order to improve training efficiency, all agents share the same set of local Q network parameters. The local Q network of the agent only relies on local observations and does not consider global information, which is convenient for distributed execution.

[0120] 2) Hybrid network: The hybrid network is used to nonlinearly combine the local Q values ​​of all agents into a global Q value, representing the joint behavior value of the entire multi-agent system, achieving global collaboration and optimization, and using the additional information provided by the global state during centralized training to guide the hybrid process. The hybrid network is designed to meet the monotonicity constraint of the global Q value for the local Q value of each agent, ensuring that the increase in local value does not lead to a decrease in global value, which is achieved through the non-negativity of the weight network. The hybrid network can capture the nonlinear relationship between local Q values ​​and enhance the representation ability of the system.

[0121] (3) The present invention constructs the agent's behavior trajectory embedding module and agent's state trajectory embedding module in QMIX based on the gated recurrent unit (GRU). GRU can encode the agent's historical trajectory information (including observations and action sequences) to make up for the lack of partial observability of the dynamic VEC environment caused by the agent ignoring potential historical information and long-term dependence of the environment when making decisions through current observations in the distributed decision-making scheme. At the same time, since service migration in vehicle-mounted edge computing requires both independent decision-making by the agent and overall coordination of the system, the present invention constructs a behavior trajectory embedding module and a state trajectory embedding module in the agent's local Q network and hybrid network of QMIX, respectively, to maximize the local and global time dependency modeling capabilities. During training, the agent m extracts a tuple containing observation status, reward, and action from the experience replay buffer, and combines it with the hidden state e generated by the GRU network in the behavior trajectory embedding module. m (t) Generate Q value. m (t) is the local behavior trajectory embedding of agent m. m The calculation formula of (t) is as follows.

[0122] e m (t) = f φ (O m (t),A m (t-1),e m (t-1)) (25)

[0123] Where φ represents the shared GRU encoder of the agent, O m (t) represents the current observation of agent m, A m (t-1) represents the last action of agent m, e m (t-1) is the last hidden state of GRU. At the same time, in the hybrid network, the present invention uses GRU to encode the historical global state {S 0 ,…,S T}, and get the state embedding τ = {τ 0 ,…,τ T}. The goal of the hybrid network in QMIX is to aggregate the local Q values ​​of all agents into a global Q value for overall optimization. Aggregation based only on the current global state may miss some important information related to long-term strategies. By encoding historical state information, the hybrid network can more accurately predict the global Q value, thereby generating a more reasonable trust value distribution.

[0124] (4) Constructing an agent role representation module: Since the local Q network parameters of each agent in QMIX are shared, multiple agents tend to learn the same or highly similar behavior strategies in similar environments, thus affecting the overall performance and diversity of the system. The present invention sets a GRU module in both the agent local network and the hybrid network. The agents share the same GRU network structure and parameters, resulting in very similar extracted hidden states under similar input observations and action histories, further making the strategy behaviors converge. In order to solve this problem of agent strategy homogeneity, the present invention designs an agent role representation module, the purpose of which is to learn a compact role representation that can characterize the complex behavior patterns of the agent, and use role information to promote individual strategy learning and guide agent collaboration. Agents with similar roles can achieve higher learning efficiency through more active knowledge transfer, while ensuring the heterogeneity of agents by distinguishing diverse roles.

[0125] 1) Role representation definition: In the collaborative multi-agent task (distributed decision-making scheme) proposed in this invention, each agent m∈M is associated with a role role m ∈Role, which is used to describe the behavior pattern of the agent. By defining roles, we can more effectively coordinate the collaboration between agents, avoid homogeneity problems, and improve task efficiency. m The behavior pattern is represented by a role m ∈Z for quantization, and the representation is obtained by training a complex function f, as shown in the following formula.

[0126] z m =f(ω m ) (26)

[0127] Among them, ω m is the behavior trajectory of agent m, which represents the observation and action sequence of the agent in the entire time period. Behavior trajectory ω m It provides rich information, including the decision-making behavior of the agent in different states and its interaction with the environment. Naturally, the distance between the agent embeddings obtained by the agent behavior trajectory embedding module can be used as an indicator to quantify the agent role representation. After introducing the role representation Z through the agent local trajectory, the individual strategy of agent m can be further improved. As shown in the following formula.

[0128]

[0129] Individual strategies It not only considers the current local observation of the agent, but also combines the role representation z m The behavior pattern information provided by the agent. This mechanism enables the agent to make differentiated decisions according to its role, thereby achieving more efficient collaboration. The introduction of role representation can effectively alleviate the homogeneity problem, allowing agents with the same parameter sharing mechanism to show personalized behavior through role division of labor, thereby improving the overall task collaboration efficiency and flexibility.

[0130] 2) Role representation discrimination method: The ideal role representation is an abstract summary that can focus on the agent's behavior pattern while ignoring the influence of the agent's individual identity. The mutual information can be used to measure the interdependence between the role and its representation. Therefore, the present invention introduces a mutual information maximization mechanism to formalize the learning of role representation and derives a contrastive learning objective that optimizes the embedding of the agent's behavior trajectory {e 1 ,…,e |M|}, and finally obtain the role representation of each agent {z 1 ,…,z |M| Specifically, the present invention trains a role encoder θ, which can be mathematically formalized as a probability encoder, i.e. z(t)~f θ (z(t)|e(t)). The role Role follows the distribution of roles P(Role), and the distribution of the agent's behavior trajectory embedding e is determined by its role. The learning objective of the role encoder is as follows (maximizing the mutual information between role representation and role).

[0131]

[0132] In actual operation, it is not feasible to directly optimize the mutual information. Therefore, the present invention introduces the information noise contrast estimation (InfoNCE) in the field of contrastive learning to indirectly maximize the mutual information. Specifically, InfoNCE provides a lower bound of the mutual information. By optimizing this lower bound, the mutual information can be approximately maximized, as shown in the following formula.

[0133]

[0134] Where K = |Role| represents the number of roles, and h(e,z) = p(z|e) / p(z). For each role in the role set * , will generate a corresponding embedding e* , which captures the behavior pattern of the character. Since the joint probability distribution of the character representation and the embedding of the intelligent agent's behavior trajectory and their conditional probability distributions p(z|e) and p(z) are usually not directly calculable, the present invention uses noise contrast estimation (NCE) and importance sampling techniques to derive a sampled version of the processable lower bound by comparing the target value (positive sample) with a set of randomly sampled negative samples, and uses it as the learning objective of the character encoder, as shown in the following formula.

[0135]

[0136] Among them, S(z,z * ) is a similarity score function, which represents the similarity measure between the potential codes of two samples. The present invention approximates h by the exponent of this function. e and e′ are derived from the role role. m Two agent embedding instances sampled from the dataset of , z and z′ are the potential representations of e and e′. For any role * ∈RoleRrole m , z * It is a role * The corresponding agent embedding e * The present invention denotes two embedding instances (e, e′) of the same character as a positive sample pair, and denotes the embedding instance set {(e, e * )} is recorded as a negative sample pair (set). The improved lower bound goal is to optimize a K classification loss to correctly classify the positive sample pairs from all sample pairs. Minimize InfoNCE loss It is equivalent to maximizing the lower bound of mutual information, and the lower bound becomes tighter as K increases. The role encoder extracts the shared features in the embedding of agents of the same role to maximize the similarity score of positive sample pairs, while capturing the key differences between different roles to reduce the similarity score of negative sample pairs. The present invention divides all agents into K clusters regularly according to the agent embedding. For agent m, its role representation z m is set to the query vector q, with the role representations of other agents as keys The agent role representations in the same cluster as agent m are considered as positive keys {k +}, other agent roles are considered as negative keys The present invention uses bilinear product to construct a similarity score function, and the similarity between the query and the key is calculated as Where W is a learnable parameter matrix, and the final InfoNCE loss function is as shown in the following formula.

[0137]

[0138] When training the role encoder, the present invention maintains a role encoder θ q and a momentum encoder θ k , and the consistency of the bond representation is promoted through a momentum update mechanism, which is formulated as follows.

[0139] θ k ←βθ k +(1-β)θ q (32)

[0140] where β∈[0,1) is the momentum coefficient and only the parameters θ of the query encoder are q It will be updated via back propagation.

[0141] (5) Constructing an improved hybrid network module: In a VEC environment with a large number of vehicle users, existing distributed service migration solutions all face the problem of too large a joint observation space and interference from invalid information from other agents. Existing methods directly use the joint observations of all agents as a single input vector of the hybrid network in QMIX, lacking consideration of the above-mentioned problems and resulting in poor performance. At the same time, the solution of the present invention requires splicing the learned role representation and the global state together to generate the weights of the hybrid network. The direct splicing method cannot fully utilize the internal structure to effectively extract the correlation in the role space. Therefore, the present invention introduces a multi-head attention module in the hybrid network, which can prompt the global state to pay attention to the learned role model, thereby providing a more expressive credit allocation in the value decomposition. At the same time, the attention mechanism helps the hybrid network to focus on key information in the global state, indirectly compresses the state space, and reduces the interference of invalid information between agents. The present invention constructs a GRU-based embedding module in the hybrid network of QMIX, encoding the historical state as a state trajectory embedding {τ 0 ,…,τ T} to promote information matching between state and role representation. Afterwards, the present invention sets the state trajectory embedding τ as the query of the multi-head attention mechanism, sets the role representation z as the key and value of the multi-head attention mechanism, and the weighted combination of the role representation τ atten As shown in the following formula.

[0142]

[0143] Among them, μ m is the shared parameter matrix W V Right m The attention weight is obtained by linear transformation. Compute the state embedding τ and the role representation z of the mth agent mThe correlation between the two, the present invention applies a softmax function to obtain the weight As shown in the following formula.

[0144]

[0145] Among them, W Q and W K is the shared parameter matrix for the linear transformation of query-key pairs, is the factor that scales the dot product attention. The present invention uses multi-head attention to enable the model to simultaneously focus on position information from different representation subspaces, and the aggregated output is represented as follows.

[0146]

[0147] in, Is to use W h Q , W h K and W h V The attention output calculated by the projection of , h∈H={1,…,|H} represents the number of heads of the multi-head attention module, W O is the parameter matrix combining all head outputs. Finally, the multi-head attention output is combined with the global state to generate the weights of the hybrid network. The present invention implicitly guides the coordination of agents in collaborative tasks through the multi-head attention mechanism, so that the hybrid network can achieve more expressive credit allocation and reduce the interference of invalid information between agents.

[0148] Algorithm 1 is the pseudo code of the distributed service migration algorithm training proposed in this invention.

[0149]

[0150]

[0151] The hardware and software environment of this implementation plan is shown in Table 1.

[0152] Table 1 Software and hardware configuration environment

[0153] name Configuration operating system Windows 11 CPU Intel Core i7 Memory 16G harddisk 1T Development Language Python Development Framework PyTorch

[0154] This experiment uses the following detailed description of the specific implementation of the present invention in conjunction with the accompanying drawings.

[0155] Step 1: Build a three-layer VEC system: Figure 1As shown, the present invention divides the entities in VEC into a cloud center layer, an edge layer, and a user layer. The cloud center layer includes a cloud server, which is responsible for the centralized training of the model. The edge layer includes 8 base stations evenly distributed within a range of 1 km × 1 km. The present invention sets an edge server on each base station, which is collectively referred to as an edge node together with the base station, and is responsible for carrying the services deployed by vehicle users and unloading task requests, and making service migration decisions (selecting the target server) for the user service instances it carries at the beginning of each time slot. The user layer includes 100 vehicle users, which are randomly distributed in the environment and move in random directions in each time slot. Each user has a service instance deployed on an edge node, and generates task requests in the form of Poisson distribution in each time slot. The user communicates with the edge node closest to it in each time slot, and uploads the task request to the corresponding service instance for processing through the uplink and return links.

[0156] Step 2: Construct a service response cost model in the VEC system: The present invention uses the weighted sum of the delay and energy consumption of processing the task request uploaded by the vehicle user as the service response cost. It includes the task request upload cost, task request forwarding cost, and task request calculation cost. If the service instance needs to be migrated according to the service migration decision made by the edge node for a certain user's service instance, the service response cost also needs to be added with the service migration cost. The service migration process is as follows: Figure 2 The service migration cost includes the delay and energy consumption of downtime, transmission and restart process. The optimization goal of the algorithm of the present invention is to minimize the long-term service response cost of all users by making reasonable service migration decisions continuously for all users.

[0157] Step 3: Construct a distributed service migration decision algorithm based on QMIX: The present invention constructs a distributed service migration solution based on the improved QMIX algorithm. The algorithm structure is as follows: Figure 3 The present invention constructs an agent behavior trajectory embedding module and a state trajectory embedding module in the local Q network and hybrid network of QMIX, respectively, so that the model can make more use of historical information and solve the accuracy problem of the agent decision-making caused by the observable part of the environment. The specific structures are shown in Figure 4. At the same time, the present invention constructs a role representation module in the local Q network, and uses the agent's behavior trajectory embedding and comparative learning to obtain the role representation of each agent to solve the homogeneity problem of the agent's decision-making. The specific structure is shown in Figure 5 Finally, in order to enable the hybrid network to better focus on the key information between agents (especially role representation) and reduce the interference of irrelevant information, the present invention uses a multi-head attention mechanism to guide the hybrid network to obtain better credit allocation. The specific structure is as follows Figure 6 As shown. Finally, Figure 7 The distributed service migration algorithm process for vehicle-mounted edge computing is demonstrated.

[0158] Step 4: Experimental analysis: Figure 8 The convergence of the algorithm proposed in the present invention is demonstrated. Figure 9(a) shows the comparison between the average service cost of the algorithm of the present invention and the baseline, Figure 9(b) shows the comparison between the average service delay of the algorithm of the present invention and the baseline, and Figure 9(c) shows the comparison between the average service energy consumption of the algorithm of the present invention and the baseline. Since the present invention makes more use of historical information and focuses on strengthening the role flexibility of the intelligent agent and the ability to extract key information, the algorithms all show good performance.

[0159] In summary, the distributed service migration solution for vehicle-mounted edge computing designed in the present invention has been realized, and the superiority of the performance of each module proposed therein has been verified.

Claims

1. A distributed service migration method for vehicle-mounted edge computing, characterized in that: Establish a three-layer heterogeneous vehicle-mounted edge computing environment, and obtain a distributed service migration solution based on an improved multi-agent deep reinforcement learning algorithm in the three-layer heterogeneous vehicle-mounted edge computing environment; The entities of the three-layer heterogeneous vehicle-mounted edge computing environment are divided into a cloud center layer, an edge layer, and a user layer; the user layer is responsible for generating tasks and offloading them to nearby edge nodes first to reduce latency; the edge node layer is responsible for processing tasks locally; the cloud center layer provides global optimization support to ensure the efficiency and reliability of the entire system; The optimization objective is modeled as a decentralized partially observable Markov decision process based on the improved multi-agent deep reinforcement learning algorithm; The improved multi-agent deep reinforcement learning algorithm is based on the improved QMIX algorithm, which constructs a distributed service migration strategy so that each agent completes the service migration decision based on local information and shared role information; the agent behavior state trajectory embedding is constructed for the local Q network and the hybrid network according to the agent historical trajectory information, so as to capture the time-related behavior pattern and solve the partial observability problem; further, the role encoder based on contrastive learning is used to extract the unique behavior patterns of different agents to generate role representation, so as to capture the long-term characteristics of the agent behavior; A multi-head attention mechanism is introduced into the hybrid network to dynamically allocate attention weights.

2. The distributed service migration method for vehicle-mounted edge computing according to claim 1 is characterized in that: The improved multi-agent deep reinforcement learning algorithm is specifically: (1) Constructing decision variables and optimizing target models: A distributed decision-making approach is adopted, with each edge node as the decision-making body. Each edge node makes a migration decision for the service instance it carries in each time slot, that is, the target server of the migration service, thereby minimizing the user service response cost; 1) Decision variables: define variable α u (t)∈{0,…,M} represents the target server to which the service instance of user u migrates at time slot t; the set of target servers for all users at time t is as follows; α(t)={α1(t),…,α U (t)} (1) α u (t) = m∈M, indicating that the target server for the migration of the service instance of user u at time slot t is m, and the service deployment variable p is updated at the same time. u,m (t) = 1; 2) Optimization goal: For the service migration problem in the three-layer heterogeneous vehicle edge environment, the ultimate goal to be achieved is to minimize the weighted sum of latency and energy consumption of all tasks, that is, the cost sum; the optimization objective function is defined as shown in the following formula; Formula (2) represents the minimization of the long-term response cost of all users based on service migration decisions, including the cost of uploading service requests Cost of service request forwarding Cost of access request calculation and service migration costs α(t) is the migration decision made by the edge node for all vehicle users in time slot t; the migration target of each user service can only be the edge server in the environment; each user can only connect to one base station in a certain time slot, and its service instance can only be deployed on one edge server; the computing resources allocated to the user cannot exceed the total resource limit of the server.

3. The distributed service migration method for vehicle-mounted edge computing according to claim 1 is characterized in that: In the improved QMIX algorithm, the agent is an edge node, the observation is all task information received by the agent in each time slot, the current location of the user, and the number of services carried by the server, the state is the joint observation state of all agents, and the action is the migration decision made by the edge node for user services in each time slot; the following improvements are made to the local Q network and hybrid network in the QMIX algorithm to obtain an improved local Q network and an improved hybrid network.

4. The distributed service migration method for vehicle-mounted edge computing according to claim 3 is characterized in that: The improved local Q network is a local Q network of the QMIX algorithm reconstructed by constructing an agent behavior trajectory embedding module and an agent role representation module; At each time slot t, each agent m sends the current observation O m (t) and action A in the previous time slot m (t-1) is input into a fully connected neural network for preliminary feature extraction. The extracted features and the agent behavior trajectory of the previous time slot are embedded in e m (t-1) is input to the agent behavior trajectory embedding module to generate the agent behavior trajectory embedding e for the current time slot m (t); embed the agent behavior trajectory into e m (t) Input to the agent role representation module to generate the current agent role representation z m (t); Finally, the role representation and the previously generated agent behavior trajectory are embedded in e m (t) Input a fully connected layer to obtain the Q value of the agent's local Q network based on the current observation.

5. The distributed service migration method for vehicle-mounted edge computing according to claim 4 is characterized in that: The agent behavior trajectory embedding module is specifically as follows: in the local Q network of each agent m, the agent behavior trajectory {(O1, A0),…, (O |T| ,A |T|-1 )} to encode and obtain the agent behavior trajectory embedding; after the agent m obtains the observation and action, it combines the hidden state e of the previous step generated by the GRU network in the agent behavior trajectory embedding module m (t-1) Generate the current hidden state e m (t); e m (t) is the behavioral trajectory embedding of agent m; e m The calculation formula of (t) is as follows: e m (t)=f φ (O m (t),A m (t-1),e m (t-1)) (3) Where φ represents the shared GRU encoder of the agent, O m (t) represents the current observation of agent m, A m (t-1) represents the last action of agent m, e m (t-1) is the previous hidden state of GRU.

6. The distributed service migration method for vehicle-mounted edge computing according to claim 4 is characterized in that: The agent role representation module is specifically: a) Role representation definition: In a collaborative multi-agent task, each agent m∈M is associated with a role m ∈Role, the role is used to describe the behavior mode of the intelligent agent; role role m The behavior pattern is represented by a role m ∈Z for quantization, which is obtained by training a role encoder f, as shown in the following formula; z m =f(ω m ) (4) Among them, ω m is the behavior trajectory of agent m, that is, the observation and action sequence of the agent in the entire time period; behavior trajectory ω m It includes the decision-making behavior of the agent in different states and its interaction with the environment; the distance between the agent behavior trajectory embeddings obtained by the agent behavior trajectory embedding module is used as an indicator to quantify the agent role representation; after introducing the role representation Z through the agent local trajectory, the individual strategy of the agent m is further improved As shown in the following formula; Among them, O represents the observation of the agent; A represents the action of the agent; Z represents the role representation of the agent; in this context, the individual strategy Not only the current local observation of the agent is considered, but also the role representation z m Information on behavioral patterns provided; b) Role representation discrimination method: The mutual information maximization mechanism is introduced to formalize the learning of role representation, and a contrastive learning objective is derived through InfoNCE; the contrastive learning objective optimizes the role encoder of each agent in a self-supervised manner, and the role encoder embeds the agent behavior trajectory into {e1,…,e M } encoding, and finally obtain the appropriate role representation {z1,…,z M }; The specific approach is to periodically divide all agents into K clusters based on the agent behavior trajectory embedding. For agent m, a role encoder θ q Generate its role representation z m , and set it as the query vector q; through a momentum encoder θ k Generate role representations of other agents as keys The agent role representations in the same cluster as agent m are considered as positive keys {k + }, other agent roles are considered as negative keys The similarity score function is constructed using bilinear product, and the similarity between the query and the key is calculated as q T Wk, where W is a learnable parameter matrix. The final InfoNCE loss function is shown in the following formula. The comparative learning goal is to minimize the loss function. During training, the role encoder is updated through back propagation; the momentum encoder needs to generate stable samples, so it is updated using momentum update. The formula is as follows: θ k ←βθ k +(1-β)θ q (7)where β∈[0,1) is the momentum coefficient.

7. The distributed service migration method for vehicle-mounted edge computing according to claim 6 is characterized in that: The improved hybrid network is a hybrid network that reconstructs the QMIX algorithm by constructing an agent state trajectory embedding module and a multi-head attention module; at each time slot t, the agent global state S(t) is processed by a fully connected layer and input into the agent state trajectory embedding module together with the agent state trajectory embedding τ(t-1) of the previous time slot to obtain the state trajectory embedding τ(t) of this time slot; the state trajectory embedding τ(t) of this time slot and the current role representation z(t) of all agents are input into the multi-head attention module to obtain the hybrid output τ of the multi-head attention module mha (t); Output τ mha (t) is concatenated with the global state S(t) and input into the super-network of the QMIX original hybrid network together with the local Q values ​​of all agents. A set of dynamic weights is calculated to perform weighted summation of the local Q values ​​of each agent and finally obtain the global Q value.

8. The distributed service migration method for vehicle-mounted edge computing according to claim 7, characterized in that: The agent state trajectory embedding module is based on the gated recurrent unit GRU to embed the historical global state {S0,…,S T } is encoded to obtain the state trajectory embedding of the agent; after obtaining the global state at the current moment, the agent state trajectory embedding module combines the hidden state τ(t-1) of the previous moment generated by the GRU network in the module as input to generate the hidden state τ(t) at the current moment; τ(t) is the state trajectory embedding of the agent; the calculation formula of τ(t) is as follows; τ(t)=f φ (S(t),τ(t-1)) (8) Among them, φ represents the GRU encoder of the global state, S(t) represents the current global state of all agents, and τ(t-1) is the previous hidden state of GRU.

9. The distributed service migration method for vehicle-mounted edge computing according to claim 7, characterized in that: The multi-head attention module specifically sets the state trajectory embedding τ as the query of the multi-head attention mechanism, sets the role representation z as the key and value of the multi-head attention mechanism, and the weighted combination of the role representation τ atten As shown in the following formula; Among them, μ m is the shared parameter matrix W V Right m The attention weight is obtained by linear transformation Compute the state trajectory embedding τ and the role representation z of the mth agent m The correlation between them, applying a softmax function to obtain the weight As shown in the following formula; Among them, W Q and W K is the shared parameter matrix for the linear transformation of query-key pairs, is the factor that scales the dot product attention; multi-head attention is used to make the hybrid network simultaneously focus on the position information from different representation subspaces, and the aggregated output is expressed as follows; in, is used and The attention output calculated by the projection of , h∈H={1,…,H} represents the number of heads of the multi-head attention mechanism, W O is the parameter matrix that combines the outputs of all heads; finally, the multi-head attention output is combined with the global state to generate the weights of the hybrid network.

Images