V2E authentication load balancing method based on multi-agent reinforcement learning

Abstract

The invention discloses a V2E authentication load balancing method based on multi-agent reinforcement learning, and the method comprises the steps: firstly constructing a V2E authentication model, constructing a mobility prediction model based on an attention mechanism space-time diagram convolutional network, and predicting an authentication task demand based on the traffic data and road network topology of the V2E authentication model. Secondly, the optimization problem of the V2E authentication model is converted into a Markov decision process, the Markov decision process is solved, and an optimal dynamic adjustment strategy is learned in a collaborative mode; finally, the intelligent agent outputs a decision instruction in real time according to the learned optimal dynamic adjustment strategy, and dynamic balance of the global authentication load is achieved. According to the method, the contradiction between authentication task domain dependency and system load balancing is effectively solved, the limitation that a traditional task unloading scheme cannot be applied to an authentication scene is fundamentally overcome, and the average authentication delay of the system is remarkably reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network management of Internet of Vehicles, and particularly relates to a V2E authentication load balancing method based on multi-agent reinforcement learning. BACKGROUND

[0002] Vehicle Ad Hoc Networks (VANETs) is a key supporting technology for Intelligent Transportation Systems (ITS), in which Vehicle-to-Edge (V2E) authentication as the primary step to ensure network security, plays an important role in verifying the legitimacy of vehicle identity, preventing unauthorized access and malicious attacks. Mobile Edge Computing (MEC) provides low-latency processing capabilities for V2E authentication by sinking computing and storage resources to the network edge. MEC nodes are usually co-deployed with base stations and are responsible for processing vehicle authentication requests forwarded by Roadside Units (RSUs) within their coverage. Due to the high mobility of vehicles and uneven spatial distribution, authentication requests are dynamically and unevenly distributed in time and space dimensions, leading to long-term overload of some MEC nodes and idle resources of other nodes, affecting authentication efficiency.

[0003] Existing technologies mainly try to solve this problem from two directions: one is the load balancing scheme using task offloading mechanism, but due to the strict domain dependency of V2E authentication, it must be processed by the MEC node of the current domain of the vehicle, and cannot be freely migrated between nodes like ordinary computing tasks; the second is to improve authentication efficiency from the micro level by designing lightweight authentication protocols, but this method cannot solve the problem of uneven load distribution at the macro level. In addition, static resource allocation strategies are difficult to adapt to dynamic load changes caused by vehicle movement, while centralized optimization algorithms lack scalability, leading to phenomena such as increased authentication delay and severe load imbalance in existing systems during peak periods or in vehicle-dense areas. These limitations not only reduce the quality of service of the system, but also may cause traffic safety hazards, restricting the overall performance improvement of the Intelligent Transportation System.

[0004] In summary, the existing technologies face fundamental limitations in solving the V2E authentication load balancing problem in VANETs: task scheduling technologies are not applicable due to the non-migratory nature of authentication tasks, while protocol optimization technologies cannot solve the macro load imbalance problem. Therefore, there is an urgent need for an innovative technical solution that can dynamically optimize the RSU-MEC association relationship in a forward-looking manner by integrating authentication demand prediction while meeting the authentication domain dependency, thereby achieving authentication load balancing at the system level and significantly improving the security communication efficiency and service quality of VANETs, providing reliable technical support for Intelligent Transportation Systems. SUMMARY

[0005] In view of the problems of uneven space-time distribution of V2E authentication load caused by vehicle mobility in the Internet of Vehicles, and the contradiction between authentication task domain dependency and load balancing demand, the application proposes an authentication load balancing method based on MADQN. The method first obtains prospective authentication demand information through the ASTGCN mobility prediction model, then takes each MEC node as an agent, adopts a distributed collaborative learning method, dynamically optimizes the logical association relationship between RSU and MEC based on real-time state and prediction information, realizes dynamic balanced allocation of system-level authentication load on the premise of meeting authentication domain dependency, thereby significantly reducing authentication delay and improving system resource utilization efficiency.

[0006] The technical scheme adopted by the application is as follows:

[0007] S1, a V2E authentication model containing vehicles, RSUs and MEC nodes is constructed, authentication delay and load are quantified, load balancing indicators are defined, a joint optimization problem is formalized and constraint conditions are determined.

[0008] S11, a three-layer system model (V2E authentication model) composed of vehicles, RSUs and MEC nodes is constructed. Vehicles communicate with adjacent RSUs through on-board units (OBU); RSUs are responsible for receiving and forwarding V2E authentication requests of vehicles; each MEC node manages an independent edge domain and is responsible for processing authentication requests forwarded by its associated RSU and maintaining authentication state information of the domain.

[0009] S12, the mth MEC node processes the authentication request from the vehicle v i The total delay required by the authentication request forwarded by the rth RSU in its management range is composed of multiple links, including transmission delay including V2R link delay of vehicle to RSU and R2M link delay of RSU to MEC, the value of which depends on the transmission rate of the communication link and the authentication request packet size D mri queuing delay caused by the processing capacity limitation of the current time slot MEC node, the authentication request needs to wait in the task queue, and the delay time is positively correlated with the current task queue length Q m (t) of the MEC. Authentication delay can be divided into intra-domain authentication processing time and cross-domain authentication processing time Since cross-domain authentication requires additional cross-domain verification and key negotiation process, its processing time is significantly greater than the intra-domain authentication processing time

[0010] S13, Quantification of authentication load and load balancing is achieved by mathematically defining the specific implementation. The authentication load L of the m-th MEC at time slot t is defined as m (t) is defined as the ratio of the total amount of computing resources consumed by all authentication tasks that the node needs to process to its total computing capacity. This indicator reflects the real-time resource utilization of the MEC node. In order to measure the load balancing degree among multiple MEC nodes in the V2E authentication model, the statistical method is used to define the load balancing indicator σ of the V2E authentication model L (t) is the standard deviation of the authentication load values of all MEC nodes at time slot t. The smaller the standard deviation value, the more uniform the load distribution among the MEC nodes, and the more ideal the system load balancing effect.

[0011] S14, Then the dynamic RSU-MEC association problem is formalized as a multi-objective joint optimization problem. The optimization goal is to minimize the average authentication delay of the V2E authentication model At the same time, the balanced distribution of authentication load among MEC nodes is achieved. The goal can be expressed as seeking the optimal RSU-MEC association strategy to minimize the weighted objective function: Where ω1 and ω2 are non-negative weight coefficients used to balance the importance of the two optimization goals of delay and load balancing. At the same time, the optimization process needs to follow the following system constraints: each RSU can only be assigned to one MEC in each time slot, and can be reassigned at most once; limit the maximum number of RSUs that can be associated with each MEC; the total authentication delay of each MEC does not exceed its maximum allowed delay, and the authentication load does not exceed its maximum processing capacity.

[0012] S2, Based on the attention mechanism spatio-temporal graph convolutional network (ASTGCN), a mobility prediction model is constructed to predict the authentication task demand of each RSU in the future based on the traffic data and road network topology of the V2E authentication model.

[0013] S21, Data preprocessing and input construction. Collect the vehicle count data of all RSUs in a certain historical period, and obtain the path network topology composed of RSUs and connection relationships. Standardize the vehicle count data to eliminate the dimension effect, and construct the spatio-temporal feature tensor as the input of the prediction model.

[0014] S22, Construct the ASTGCN mobility prediction model. The model is stacked by multiple spatio-temporal blocks, each of which contains spatial attention mechanism, temporal attention mechanism, graph convolutional network and temporal convolutional network, to cooperatively capture the spatio-temporal dynamic characteristics of authentication load.

[0015] S23, spatial dependency modeling. The spatial attention mechanism is used to dynamically calculate the correlation weights of different input RSU node sequences. This mechanism can identify node groups with strong spatial correlations and quantify the degree of mutual influence between nodes. This provides key spatial correlation information for subsequent graph convolution operations.

[0016] S24, temporal dependency modeling. While completing the spatial dependency modeling, the time attention mechanism is used to adaptively focus on the time steps in the historical sequence that are most relevant to future prediction based on the input RSU node sequence. This temporal dimension attention calibration complements the spatial attention, and together they provide support for subsequent spatio-temporal feature extraction. The time attention mechanism adaptively focuses on the time steps in the historical sequence that are most relevant to future prediction, and the output time attention weights are used to enhance the features of key time steps and suppress noise.

[0017] S25, spatial graph convolution. Based on this, the model combines the input RSU node sequence with the dynamic correlation weights calculated by the spatial attention mechanism, then performs spatial graph convolution to aggregate neighbor information. The first-order approximation form of Chebyshev graph convolution is used to normalize the adjacency relationship that has been integrated with dynamic correlation weights, thereby guiding each RSU node to adaptively complete feature fusion according to its correlation strength with neighboring nodes, and output the fused features.

[0018] S26, temporal convolution. Based on the enhanced spatial features by graph convolution and the key time step selection by the time attention mechanism, the model first weights the fused features by the time attention weights, then uses one-dimensional temporal convolution for deep feature extraction, and outputs spatio-temporal features. This step fully utilizes the output of the previous spatial and temporal attention mechanisms, effectively capturing the local temporal patterns and long-term trends contained in the feature sequence.

[0019] S27, after multiple spatio-temporal blocks extract high-order features from spatio-temporal features, the model maps the final features to the prediction output of the future T p time slices of each RSU , i.e., the predicted number of vehicles in each time slice in the future. Based on this, the authentication task volume can be calculated.

[0020] S28, the authentication demand prediction result of each MEC obtained in step S25 is passed to the corresponding agent as an important part of the subsequent decision state information to achieve forward-looking load balancing optimization.

[0021] S3, combining the prediction results of the mobility prediction model, the optimization problem of the V2E authentication model is converted into a Markov decision process, and the state space, action space and reward function of each agent are clearly defined.

[0022] where each MEC node acts as an independent agent, participating in distributed decision-making through a well-defined state space, action space and reward function. The state s m (t) contains its current authentication load L m (t), total delay and next time slot predicted authentication demand action a m (t) is defined as the association adjustment operation on the RSU to which it belongs, i.e. selecting to re-allocate a specific RSU to a certain target MEC node; the reward function r(t) is designed as the weighted negative value of the average authentication delay of the V2E authentication model and the standard deviation of the load balancing, and a foresight reward term is introduced to guide the agent to learn the optimal strategy to reduce the delay and balance the load.

[0023] S4, solving the Markov decision process, each MEC agent learns the optimal RSU-MEC association dynamic adjustment strategy through exploration and interaction with the environment based on the MADQN algorithm.

[0024] S41, for each MEC agent, a separate deep Q network DQN is deployed to represent the action-value function Q(s, a). Each DQN contains a current network and a target network, and its structure is usually a multi-layer perceptron. In the initialization stage, the deep Q network parameters of all agents are randomly initialized, and a shared experience replay pool is established to store the experience tuples generated by system exploration.

[0025] S42, at each decision-making time slot t, each agent selects an action based on the ε-greed strategy. That is, with a probability of ε, the action space is randomly explored, and with a probability of 1-ε, the action with the maximum Q value in the current state is selected. The actions of all agents together constitute a joint action, which adjusts the association between RSU-MEC in the environment. According to the new association relationship, vehicle movement and request arrival, the environment evolves to a new state and generates a common immediate reward. Then, the global experience tuple (s m (t), a m (t), r(t), s m (t+1)) is stored in the shared experience replay pool.

[0026] S43, periodically sample a batch of experience data from the experience replay pool. For each agent, use its target network to calculate the target Q value. Then, update the parameters of the current network by minimizing the mean square error loss between the predicted Q value of the current network and the target Q value. To stabilize the training process, periodically synchronize the parameters of the current network to the target network through soft update.

[0027] S44. Through the iterative training process described above, the Q-networks of each agent gradually converge, enabling accurate evaluation of the long-term expected reward of performing different actions in a specific state. Ultimately, each agent learns an optimal policy that is based on its observed local state s. m (t) maps to an optimal action a. m (t), thereby dynamically optimizing the overall authentication latency and load balancing of the V2E authentication model while satisfying the constraint of authentication domain dependency.

[0028] S5. The agent, based on the learned optimal dynamic adjustment strategy, outputs decision instructions in real time, dynamically adjusting the logical relationship between RSUs and MECs. Each RSU forwards its processed authentication requests to the designated MEC node according to the new relationship, ultimately achieving dynamic balancing of the global authentication load.

[0029] The beneficial effects of this invention are:

[0030] (1) Effectively resolves the contradiction between authentication task domain dependency and system load balancing: By dynamically adjusting the logical relationship between RSU and MEC, rather than directly migrating the authentication task itself, under the premise of strictly adhering to the V2E authentication domain dependency requirements and integrating forward-looking authentication demand prediction, the authentication load is cleverly and accurately redistributed among different MEC nodes, fundamentally overcoming the limitation that traditional task offloading schemes cannot be applied to authentication scenarios.

[0031] (2) Significantly improves system performance indicators: By using collaborative decision-making based on multi-agent reinforcement learning and combining the future load information provided by the ASTGCN mobility prediction model, this invention can more intelligently and proactively direct authentication requests to MEC nodes with lighter loads, thereby significantly reducing the average authentication latency of the system.

[0032] (3) Possesses excellent adaptability and scalability: The MADQN algorithm enables each MEC agent to make distributed decisions based on its local state, which integrates real-time and predictive information, without centralized control. This distributed characteristic naturally provides excellent scalability and can adapt to large-scale vehicle network environments. At the same time, the predictive reward term based on prediction comparison in the reward function enables the system to avoid potential high load risks in advance and has a stronger adaptive capability to dynamic changes brought about by vehicle mobility.

[0033] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the V2E authentication load balancing method provided in an embodiment of the present invention;

[0035] Figure 2 A V2E authentication load balancing scenario diagram provided for an embodiment of the present application;

[0036] Figure 3 An MADQN training process diagram of V2E authentication load balancing provided for an embodiment of the present application;

[0037] Figure 4 is the average reward value curve of the present application in the training process with the number of training rounds;

[0038] Figure 5 is a load balancing degree comparison diagram of the present application method and static association scheme under different RSU deployment scales. DETAILED DESCRIPTION

[0039] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings, which show several embodiments of the present application. However, the present application can be realized in different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present application more thorough and comprehensive.

[0040] Embodiment: a vehicle-to-everything (V2E) authentication load balancing method based on multi-agent reinforcement learning, as shown in Figures 1 to 5 , comprising the following steps:

[0041] S1, constructing a V2E authentication model containing vehicles, RSUs and MEC nodes, quantifying authentication delay and load, defining load balancing indicators, formalizing joint optimization problems and determining constraint conditions.

[0042] Specifically, the process of step S1 includes:

[0043] S11, constructing a three-layer system model (V2E authentication model) composed of vehicles, RSUs and MEC nodes, as shown in Figure 2 . The vehicle communicates with the adjacent RSU through the on-board unit (OBU); the RSU is responsible for receiving and forwarding the V2E authentication request of the vehicle; each MEC node manages an independent edge domain and is responsible for processing the authentication request forwarded by the associated RSU and maintaining the authentication state information of the domain.

[0044] The present application adopts a discrete time model for the V2E authentication model, and divides the entire communication time period into T time slots, denoted as In each time slot t, the quasi-static scenario is considered, in which the connection state between the vehicle and the RSU remains unchanged. The present application considers a V2E authentication model containing M MECs, wherein the set of MECs is denoted as The r RSUs are contained in the management range of each m-th MEC, and the set of RSUs is denoted as Suppose that vehicles are driving along the road segment and are about to enter the coverage of the RSU r .

[0045] When the vehicles are driving along the road segment from other RSUs to the coverage of the r-th RSU, the authentication request is sent to the RSU. Each authentication request is described as where ID mri is the specific identity information of the vehicle V mri , CD mri is the current domain identifier of the vehicle v i , and TS mri is the timestamp of the authentication request. After receiving the authentication request of the vehicle, the RSU does not process the information in any way, but directly forwards the request to the MEC to which it belongs.

[0046] S12, the m-th MEC processes the authentication request forwarded by the RSU from the vehicle v i The total delay required by the authentication request forwarded by the RSU is composed of the delays of multiple links. The transmission delay includes the V2R link delay from the vehicle to the RSU and the R2M link delay from the RSU to the MEC, the values of which depend on the transmission rate of the communication link and the size D mri of the authentication request data packet.

[0047] The V2R link transmission rate R i between the vehicle v V2R and the RSUr can be represented as:

[0048]

[0049] where B ir represents the transmission channel bandwidth between v i and the r-th RSU, P ir represents the transmission power, and N0 represents the Gaussian white noise power. Based on the free space propagation model, the channel gain H ir can be defined as:

[0050]

[0051] where g i is the gain of the vehicle transmitting antenna, g r is the gain of the RSU receiving antenna, A is the signal wavelength, and d ir is the distance between v i and the r-th RSU. Based on this, the transmission rate R V2M. Thus, the authentication request is sent from the v i Transmission delay to MEC for processing can be represented as:

[0052]

[0053] Queueing delay The authentication request needs to wait in the task queue caused by the current time slot MEC node processing capacity limit. MEC processes tasks in a first-come-first-served manner, strictly following their arrival order and their delay time. The process of vehicle arrival at RSU within each time slot t obeys Poisson distribution:

[0054] V r (t) ~ Poisson (λ r Δt)

[0055] Where the vehicle arrival rate is λ r . It is assumed that the MEC has limited authentication task processing capacity per time slot, and is defined as μ m . Let Q m (t) represent the number of authentication tasks waiting to be processed in the queue of the mth MEC at time slot t. Based on these definitions, the evolution of the task queue over time can be modeled using the following update equation:

[0056]

[0057] Therefore, the queueing delay of each authentication task can be approximately represented as:

[0058]

[0059] After the MEC receives the authentication task of V i , it uses the domain identifier CD mri to determine whether to perform intra-domain authentication or cross-domain authentication. The identity verification process includes request parsing, challenge response, and result delivery. Since the cross-distributed edge domain needs to reconstruct the centralized certificate hierarchy, it needs to iterate the remote authentication request, so the cross-domain authentication requires more computing cycles and bandwidth than the intra-domain authentication. After verification, the result is sent back to the vehicle. If the authentication is accepted, the domain identifier CD mri is updated to the domain identifier of the MEC, and its identity information is stored in the local database of the MEC, so that the vehicle can directly perform intra-domain authentication in the future, avoiding repeated cross-domain processing. The processing delay of intra-domain authentication tasks and cross-domain authentication tasks is represented as:

[0060]

[0061] Where C IA and C CA represent the number of CPU cycles required to complete intra-domain and cross-domain identity verification tasks, respectively. Thism represents the computing capacity of the MECs, in CPU cycles per second.

[0062] Optimizing RSU-MEC associations can enhance the flexibility of load balancing. While authentication tasks must be handled by the MECs in the current edge domain, these associations can be dynamically reconfigured. This allows RSUs to forward identity verification requests to different MECs, indirectly redistributing identity verification load across domains. To model this behavior, a binary variable b nm,r (t) e {0, 1} is defined to represent whether the rth RSU originally in the nth MEC’s management scope is reassigned to the mth MEC at time slot t. If a reassignment occurs, b nm,r (t) takes the value 1; otherwise, it is 0. In addition, a binary variable a mri (t) e {0, 1} is used to distinguish authentication types, which are dynamically determined in real-time according to vehicle locations. Here, a mri (t) = 1 represents a cross-domain task, and a mri (t) = 0 represents an intra-domain task. The total authentication delay of an RSU at time slot t can be represented as:

[0063]

[0064] While optimizing RSU-MEC associations is flexible, it can also introduce handover delays. Therefore, the optimization cost at time slot t is defined as:

[0065]

[0066] where δ models the cost of reassigning RSUs, including signaling overhead, configuration delay, and service interruption probability. This enables the system to penalize frequent optimization, thereby balancing delay reduction and system stability during optimization. Finally, the total delay can be expressed as:

[0067]

[0068] S13, quantifies the authentication load and the degree of balancing through mathematical definitions of specific implementations.

[0069] To quantify the authentication load on each MEC, utilization levels are defined based on the cumulative CPU cycles consumed by authentication tasks. First, the total amount of computing resources consumed by each RSU to receive authentication at time slot t needs to be quantified, defined as:

[0070]

[0071] Therefore, the authentication load L m (t) of each MEC at time slot t is defined as:

[0072]

[0073] Afterwards, each MEC will adjust the RSUs in its edge domain. The adjusted network status will become the initial condition of the next time slot t+1. The optimized load of each MEC is defined as:

[0074]

[0075] To measure the degree of load balancing among multiple MEC nodes in the whole system, a statistical method is used to define the system load balancing indicator σ L (t) is the standard deviation of the authentication load values of all MEC nodes at time slot t:

[0076]

[0077] The smaller the standard deviation value is, the more uniform the load distribution among MEC nodes is, and the more ideal the load balancing effect of the V2E authentication model is. Among them represents the average workload at time slot t:

[0078]

[0079] S14, then the dynamic RSU-MEC association problem is formalized as a multi-objective joint optimization problem. The optimization goal is to minimize the average authentication delay of the V2E authentication model while achieving balanced distribution of authentication load among MEC nodes. This goal can be expressed as seeking the optimal RSU-MEC association strategy to minimize the weighted objective function:

[0080]

[0081] where ω1 and ω2 are non-negative weight coefficients used to balance the importance of the two optimization goals of delay and load balancing. At the same time, the optimization process needs to follow the following constraints:

[0082]

[0083]

[0084] Constraint C1 ensures that b nm,r (t), a mri (t) is binary. C3 and C4 specify that each RSU must and can only be reassigned to one target MEC, and at most one association adjustment is made. C5 sets the upper limit of the number of RSUs migrated by any MEC to C6 and C7 set the total authentication delay threshold for each vehicle and the authentication load threshold for MECs.

[0085] S2, a mobility prediction model is constructed based on an attention mechanism spatio-temporal graph convolution network (ASTGCN), to predict the authentication task demand of each RSU in the future based on the historical traffic data of the V2E authentication model and the road network topology.

[0086] Specifically, the process of step S2 includes:

[0087] S21, data preprocessing and input construction. Collect the vehicle quantity data of all RSUs in the historical time period [t0-T h +1,t0] and obtain the path network topology graph composed of RSUs and connection relationships r The vehicle count data is standardized to eliminate the dimensional influence and the spatio-temporal feature tensor is constructed as the input of the prediction model.

[0088] S22, constructing an ASTGCN mobility prediction model. The model is stacked by multiple spatio-temporal blocks, each of which contains a spatial attention mechanism, a temporal attention mechanism, a graph convolution network and a time series convolution network, to cooperatively capture the spatio-temporal dynamic characteristics of the authentication load.

[0089] The attention spatio-temporal graph convolution network (ASTGCN) is a deep learning model specially designed for spatio-temporal data prediction. Its core dynamically captures the topological correlation between road network nodes through a spatial attention mechanism, and focuses on key time steps in historical data through a temporal attention mechanism, effectively solving the defects of traditional methods in modeling dynamic spatio-temporal relationships. The authentication demand prediction model constructed based on ASTGCN can use historical RSU traffic flow data and network topology structure to accurately predict the number of vehicles arriving at each RSU and the amount of authentication requests in the future, which can be used to predict the authentication request amount and type of each MEC in advance, thereby providing key data support for load balancing decision-making, effectively reducing authentication delay and improving overall system efficiency.

[0090] S23, spatial dependency modeling. Through the spatial attention mechanism, the mutual influence degree of different input RSU node sequences in the road network topology is dynamically evaluated and quantified. This mechanism can automatically identify spatially closely related RSU groups based on real-time and historical traffic flow data, such as nodes on the same trunk road or traffic corridor, whose load fluctuations have a high degree of linkage. The calculated spatial attention weight is used as a key input for subsequent graph convolution operations to guide the model to aggregate the feature information of adjacent RSUs, and to assign higher weights s to nodes with stronger correlation, thereby accurately capturing the spatial propagation pattern of authentication load determined by road connectivity. Its calculation formula is:

[0091] ​​

[0092] where Θ1, Θ2 are learnable weight matrices, β s is a bias vector, and σ is a Sigmoid activation function. V s is a learnable weight matrix, which is used to scale the computed raw attention scores to obtain the final spatial attention weight matrix S. The spatial attention weight is used to guide the graph convolution layer to aggregate the feature information of neighboring RSUs.

[0093] S24, Time-dependent modeling. While completing the spatial-dependent modeling, the time attention mechanism is used to adaptively focus on the time steps in the historical sequence that are most relevant to future prediction according to the input RSU node sequence. This attention calibration in the time dimension complements the spatial attention, and together they provide support for subsequent spatio-temporal feature extraction. Through the time attention mechanism, dynamic patterns in the authentication load time series are analyzed in depth, and those historical time steps that have the most impact on future prediction are adaptively focused on. This mechanism can identify historical time periods that have a strong correlation with the prediction target time point (such as the congestion pattern of the morning rush hour, which may have a predictive effect on the load of the same period in the future), while downplaying the influence of irrelevant or noise time points. The generated time attention weight E is used to adjust the focus of the time series convolution layer, thereby enhancing the ability to extract features of periodic, trend, and sudden changes in authentication demand, effectively improving prediction accuracy. The calculation formula is:

[0094]

[0095] where Φ1, Φ2 are learnable parameters, β e is a bias vector. The time attention weight is used to enhance the features of key time steps and suppress noise. e is a learnable weight matrix, which is used to scale the computed raw attention scores to obtain the final time attention weight matrix E. The time attention weight is used to enhance the features of key time steps and suppress noise.

[0096] S25, under the synergistic effect of the time attention mechanism, the model further integrates information from neighboring RSU nodes using spatial graph convolution. By constructing a neighborhood relationship that can adapt to real-time traffic conditions using the dynamic correlation strength reflected by the spatial attention matrix and the input RSU node sequence, a fused feature is output. In this framework, each RSU node adaptively weights and fuses the features of its neighboring nodes based on the attention weights calculated in the previous steps, so that nodes with stronger relevance play a more important role in the feature aggregation process. This design enables the model to break through the limitations of fixed topological structures and accurately capture the complex and dynamic spatial propagation patterns of authentication load in the road network.

[0097] S26, after the spatial features are enhanced via graph convolution, the model further uses one-dimensional time convolution to deeply mine the processed time series of each node. The model first weights and combines the fused features through time attention weights, and then uses one-dimensional time convolution for deep feature extraction to output the spatio-temporal features. With a window sliding along the time axis, the features of adjacent time points are combined to identify the local patterns contained in the time series. The model can effectively extract the periodicity, trend and sudden change characteristics in the time series.

[0098] S27, feature extraction and output. After the spatio-temporal features are extracted by multiple spatio-temporal blocks, the model maps the final features to the prediction output of the future T p time slices of each RSU, that is, the predicted number of vehicles in the future time slices of each RSU, by which the authentication task amount can be calculated.

[0099] S28, prediction information transmission. The authentication demand prediction result obtained in step S25 is transmitted to the corresponding MEC agent as an important part of the subsequent decision state information of the MEC agent to realize forward-looking load balancing optimization.

[0100] S3, combine the prediction results of the mobility prediction model to convert the optimization problem of the V2E authentication model into a Markov (MDP) decision process, and clearly define the state space, action space and reward function of each agent.

[0101] Specifically, the process of step 3 includes:

[0102] The MDP decision process includes agents, environment, action space and reward function. Each MEC node acts as an independent agent and participates in distributed decision-making through a clearly defined state space, action space and reward function.

[0103] The state s m of each agent includes its current authentication load L m (t), total processing delay predicted authentication demand in the next time slot The specific definitions are as follows:

[0104]

[0105] The action a m (t) is defined as the association adjustment operation on the RSU, that is, selecting to reassign a specific RSU to a target MEC node:

[0106]

[0107] ​wherein R r represents the RSU to be migrated, M k represents the target MEC. If M k = M m , it means that the RSU is not migrated.

[0108] The reward function r(t) is designed as the weighted negative value of the system average authentication delay and the load balancing standard deviation, and a foresight reward term ψ(t) is introduced to realize forward-looking optimization. The reward term is generated by comparing the current actual load balancing index σ L (t) with the predicted load benchmark from the ASTGCN model. If a positive constant reward C is given, otherwise it is zero. This mechanism directly encourages the agent to learn a strategy that can actively improve the future load situation. The specific definition is as follows:

[0109]

[0110] S4, a Markov decision process is solved, and each MEC agent learns the optimal RSU-MEC association dynamic adjustment strategy based on the MADQN algorithm by interacting with the environment and collaborative learning.

[0111] Multi-agent deep Q network (MADQN) as an advanced distributed reinforcement learning algorithm provides a new technical path to solve the above problems. The algorithm regards each MEC node as an independent agent, combines deep neural network with Q-learning (Q-Learning), and adopts experience replay and target network mechanism. In the present application, each agent integrates real-time environmental observation and foresight authentication demand information from the ASTGCN mobility prediction model for collaborative training, thereby learning the optimal RSU-MEC association strategy. This method is particularly suitable for handling the authentication load balancing problem with spatial distribution characteristics and dynamic change characteristics in the Internet of Vehicles, which can not only maintain the domain dependence requirement of authentication tasks, but also achieve foresighted system-level load dynamic balancing based on prediction. Through this distributed decision-making framework integrated with prediction information, MADQN can more effectively cope with the uncertainty brought by vehicle mobility, directly optimize the distribution of authentication tasks by adjusting the association relationship between RSU and MEC in real time, thereby significantly improving the overall system performance.

[0112] Specifically, the process of step S4 includes:

[0113] S41, an independent deep Q network DQN is deployed for each MEC agent to approximate its action-value function Q(s, a). Each DQN contains a current network Q m (s, a; θ m ) and a target Q network The structure is usually a multi-layer perceptron. In the initialization stage, the network parameters of all agents are randomly initialized, and a shared experience pool is established to store the experience tuples generated by system exploration.

[0114] S42, at each decision time slot t, each agent selects an action based on the ε-greed strategy. That is, with a probability of ε, the action space is randomly explored, and with a probability of 1-ε, the action with the maximum Q value in the current state is selected. The actions of all agents together constitute a joint action, which adjusts the association relationship of RSU-MEC in the environment. The environment evolves to a new state according to the new association relationship, vehicle movement and request arrival, and generates a common immediate reward. Then, the global experience tuple (s m (t),a m (t),r(t),s m (t+1)) is stored in the shared experience replay pool.

[0115] S43, periodically sample a batch of experience data from the experience replay pool. For each agent, use its target network to calculate the target Q value, and the calculation formula is as follows:

[0116]

[0117] Then, the mean square error between the Q value predicted by the current network and the target Q value is used as the loss function to update the parameters of the current network, and the calculation formula is as follows:

[0118] Loss m =(Q m (s,a;θ m )-y m ) 2

[0119] To stabilize the training process, periodically synchronize the parameters of the current network to the target network through soft update.

[0120]

[0121] Where α is the learning rate and τ is the soft update rate. Through such a training process, each MEC agent learns how to choose the optimal RSU migration adjustment strategy according to the current complex traffic environment.

[0122] S44, through the above iterative training process as shown in Figure 3 , the Q network of each agent gradually converges and can accurately evaluate the long-term expected return of executing different actions in a specific state. Finally, each agent learns an optimal strategy that can map an optimal action a m (t) from the local state s m(t), thereby dynamically optimizing the overall authentication delay and load balancing of the V2E authentication model under the constraint of satisfying the authentication domain dependency.

[0123] S5, the intelligent agent outputs decision instructions in real time according to the learned optimal dynamic adjustment strategy, and dynamically adjusts the logical association relationship between the RSU and the MEC. According to the new association relationship, each RSU forwards the authentication request processed by it to the designated MEC node, and finally realizes the dynamic balancing of the global authentication load.

[0124] To verify the effectiveness of the method of the application, simulation experiments were carried out in the specific implementation process, and the key performance indicators were quantitatively analyzed. The experimental results fully prove the significant advantages of the application in improving the convergence performance and load balancing effect of the system.

[0125] Figure 4 is the average reward value of the application in the training process with the change curve of the number of training rounds. Analysis of the graph shows that the application starts to converge after about 600 training rounds, and finally stabilizes at a relatively high average reward value, showing good convergence and stability. The curve shows that by fusing the prediction information of ASTGCN, the MADQN intelligent agent can more effectively learn high-quality RSU-MEC association strategies, thereby obtaining higher long-term returns.

[0126] Figure 5 is the load balancing degree comparison of the application method and the static association scheme under different RSU deployment scales. The load balancing degree is measured by the improvement percentage of the load standard deviation, and the higher the value represents the better the balancing effect. Analysis of the graph shows that in different scale network environments, the load balancing performance of the application is significantly better than the baseline scheme of static association, and the improvement range of load balancing degree is still maintained at more than 15%.

[0127] The above describes the application in conjunction with the drawings, and it is obvious that the specific implementation of the application is not limited by the above method. As long as this non-essential improvement is made by using the method concept and technical solution of the application, or the concept and technical solution of the application is directly applied to other occasions without improvement, it is within the protection scope of the application.

Claims

1. A V2E authentication load balancing method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Construct a V2E authentication model that includes vehicles, RSUs, and MEC nodes, quantify authentication latency and load, define load balancing metrics, formalize the joint optimization problem, and determine the constraints. S2. A mobility prediction model is constructed based on the attention mechanism spatiotemporal graph convolutional network ASTGCN. Based on the traffic data and road network topology of the V2E authentication model, the future authentication task requirements of each RSU are predicted. S3. Combining the prediction results of the mobility prediction model, the optimization problem of the V2E authentication model is transformed into a Markov decision process, defining the state space, action space and reward function of each agent. S4. Solve the Markov decision process. Each MEC agent, based on the MADQN algorithm, explores and interacts with the environment to collaboratively learn the optimal RSU-MEC association dynamic adjustment strategy. S5. The agent outputs decision instructions in real time based on the learned optimal dynamic adjustment strategy, dynamically adjusts the logical relationship between RSU and MEC, and each RSU forwards the authentication request it processes to the designated MEC node according to the new relationship, so as to achieve dynamic balancing of global authentication load.

2. The V2E authentication load balancing method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step S1 is implemented as follows: S11. Construct a V2E authentication model consisting of vehicles, RSUs, and MEC nodes; vehicles communicate with neighboring RSUs through onboard units (OBUs); RSUs are responsible for receiving and forwarding V2E authentication requests from vehicles; each MEC node manages an independent edge domain, is responsible for processing authentication requests forwarded by its associated RSUs, and maintains the authentication status information of that domain. S12, the m-th MEC node processes data from vehicle v in time slot t. i The total latency required for the authentication request forwarded through the r-th RSU within its management scope. It consists of delays from multiple stages, including: transmission delay. This includes the V2R link delay from the vehicle to the RSU and the R2M link delay from the RSU to the MEC, the values ​​of which depend on the transmission rate of the communication link and the size D of the authentication request packet. mri Queue delay Due to the current processing capacity limitations of the MEC nodes in the current time slot, authentication requests need to wait in the task queue, and the delay time is equal to the length Q of the current pending task queue in the MEC. m (t) are positively correlated; authentication delay is categorized into intra-domain authentication processing time based on authentication type. Cross-domain authentication processing time Processing time Significantly longer than intra-domain authentication processing time S13, the authentication load L of the m-th MEC in time slot t. m (t) is defined as the ratio of the total computing resources consumed by all authentication tasks to be processed by the node to its total computing power. The load balancing index σ of the V2E authentication model is defined using statistical methods. L (t) represents the standard deviation of the certified load values ​​of all MEC nodes in time slot t; S14. The optimization objective is to minimize the average authentication latency of the V2E authentication model. Simultaneously, the goal is to achieve a balanced distribution of authentication load across all MEC nodes; this objective is expressed as finding the optimal RSU-MEC association strategy to minimize the weighted objective function: Where ω1 and ω2 are non-negative weighting coefficients, the optimization process follows the following constraints: each RSU can only be assigned to one MEC in each time slot and can only be reassigned once at most; the maximum number of RSUs associated with each MEC is limited; the total authentication latency of each MEC does not exceed its maximum allowable latency, and the authentication load does not exceed the maximum processing capacity.

3. The V2E authentication load balancing method based on multi-agent reinforcement learning according to claim 2, characterized in that, The specific implementation process of step S2 is as follows: S21. Collect vehicle count data for all RSUs within a certain historical time period and obtain the path network topology composed of RSUs and their connections; standardize the vehicle count data and construct a spatiotemporal feature tensor as the input to the prediction model. S22. Construct the ASTGCN mobility prediction model, which is composed of multiple spatiotemporal blocks stacked together. Each spatiotemporal block contains a spatial attention mechanism, a temporal attention mechanism, a graph convolutional network, and a temporal convolutional network. S23. After extracting spatiotemporal features through multiple spatiotemporal blocks, the final features are mapped to the future T through a fully connected layer. p Predicted output for each time slice That is, the predicted number of vehicles for each RSU in each future time slice; S24. Obtain the certification demand prediction results for each MEC obtained in step S23. This information is transmitted to the corresponding intelligent agent as an important component of its subsequent decision-making state information, in order to achieve forward-looking load balancing optimization.

4. The V2E authentication load balancing method based on multi-agent reinforcement learning according to claim 3, characterized in that, The specific implementation process of the spacetime block is as follows: Spatial dependency modeling: Dynamically calculate the association weights of different input RSU node sequences through a spatial attention mechanism; Time-dependent modeling: Based on the input RSU node sequence, the time attention mechanism adaptively focuses on the time steps in the historical sequence that are most relevant to future predictions, and outputs time attention weights; Spatial graph convolution: Combining the dynamic association weights calculated by the spatial attention mechanism, the input RSU node sequence is weighted and combined with the dynamic association weights, and then spatial graph convolution is performed to aggregate neighbor information. The first-order approximation form of Chebyshev graph convolution is adopted to normalize the adjacency relationship fused with dynamic association weights, guiding each RSU node to adaptively complete feature fusion according to its association strength with neighboring nodes, and output fused features. Temporal convolution: The fused features are weighted and combined using temporal attention weights, and then a one-dimensional temporal convolution is used to extract deep features and output spatiotemporal features.

5. The V2E authentication load balancing method based on multi-agent reinforcement learning according to claim 4, characterized in that, The specific implementation process of step S3 is as follows: Each MEC node acts as an independent agent, participating in distributed decision-making through a defined state space, action space, and reward function; the agent's state s m (t) contains its current authentication load L m (t), total delay and the next time slot prediction certification requirements Action a m (t) is defined as the associated adjustment operation of the RSU, that is, to select to reassign the RSU to a target MEC node; the reward function r(t) is designed as the weighted negative value of the average authentication latency of the V2E authentication model and the standard deviation of load balancing, and a predictive reward term is introduced to guide the agent to learn the optimal strategy of reducing latency and balancing load.

6. The V2E authentication load balancing method based on multi-agent reinforcement learning according to claim 5, characterized in that, The specific implementation process of step S4 is as follows: S41. Deploy an independent Deep Q Network (DQN) for each MEC agent to represent the action-value function Q(s,a). Each DQN contains a current network and a target network. During the initialization phase, the parameters of the Deep Q Networks of all agents are randomly initialized, and a shared experience replay pool is established to store the experience tuples generated by the system exploration. S42. In each decision time slot t, each agent selects an action based on the ε-greed policy. The actions of all agents together constitute a joint action, adjusting the RSU-MEC association in the environment. The environment evolves to a new state based on the new association, vehicle movement, and request arrival status, and generates an immediate reward. Subsequently, the global experience tuple (s) m (t),a m (t),r(t),s m (t+1) is stored in the shared experience replay pool; S43. Periodically sample a batch of experience data from the experience replay pool; for each agent, calculate the target Q value using its target network; Subsequently, the parameters of the current network are updated by minimizing the mean squared error loss between the current network's predicted Q-value and the target Q-value; During training, the parameters of the current network are synchronized to the target network using a soft update method; S44. Through iterative training, the Q-networks of each agent gradually converge, and eventually each agent learns an optimal policy that can be applied based on its observed local state s. m (t) maps to an optimal action a. m (t) dynamically optimizes the overall authentication latency and load balancing of the V2E authentication model under the constraint of authentication domain dependency.

Images