Internet of vehicles optimization method and system based on edge calculation and time division multiple access

Abstract

The invention belongs to the technical field of wireless communication, and discloses an Internet of Vehicles optimization method and system based on edge computing and time division multiple access, and the method comprises the steps: carrying out the joint optimization of service deployment and replacement, RSU deployment, access control, and task unloading rates in VTUs and RSUs, and achieving the optimization of the Internet of Vehicles. And efficient, low-delay and stable task processing service is provided for vehicle-mounted communication in urban traffic, expressways or disaster areas. According to the method, a double-time-scale optimization strategy is adopted, the decision of service deployment and replacement is carried out on a coarse-grained time scale (frame), and other optimization problems are processed in a fine-grained time scale (time slot). The invention provides an innovative hierarchical DRL algorithm, and in the algorithm, a high-level agent adopts a deep Q network (DQN) proxy and is responsible for optimizing a service deployment and replacement strategy in a frame; and the bottom layer intelligent agent adopts a depth deterministic policy gradient (DDPG) for proxy, and specially solves related optimization problems of task unloading rate, access control and the like in the time slot.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, and in particular relates to a vehicle-to-everything (V2X) optimization method and system based on edge computing and time division multiple access. Background Technology

[0002] With the rapid development of Intelligent Transportation Systems (ITS) and the Internet of Vehicles (IoV), intelligent applications such as autonomous driving and real-time traffic monitoring are emerging. These applications place extremely high demands on the real-time performance of data processing, the efficiency of computing power, and the low latency of decision response. However, vehicle terminal units (VTUs) have inherent limitations in terms of computing power and energy efficiency. For example, in autonomous driving scenarios, massive amounts of environmental perception data are generated every second, while the computing power of a typical vehicle CPU is only tens of GFlops and is subject to strict power consumption constraints, making it difficult to meet the processing needs of such demanding applications, resulting in a high task failure rate.

[0003] Multi-access edge computing (MEC) is an effective solution that provides VTUs with near-field computing resources by deploying edge servers with storage and computing capabilities in roadside units (RSUs). This significantly reduces data transmission latency, alleviates in-vehicle computing bottlenecks, and improves task processing success rates. Under a Time Division Multiple Access (TDMA) architecture, the RSU can allocate communication resources to multiple VTUs according to time slots, coordinating task offloading processes and further optimizing resource utilization efficiency. Applying MEC to TDMA-based in-vehicle networks not only promotes the implementation of applications such as real-time data processing and emergency event warning in intelligent transportation scenarios but also provides VTUs with more stable service quality and more efficient resource scheduling.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0005] 1) Time-scale coordination problem

[0006] Optimization of vehicular networks involves decisions at two different time scales: frame-level and time-slot-level. Frame-level service deployment and replacement require long-term resource planning, while time-slot-level task offloading and access control need to respond in real time to dynamically changing network conditions. Traditional optimization methods struggle to effectively coordinate these two time scales, leading to an imbalance between long-term planning and real-time adaptability, which in turn affects the overall success rate of task processing.

[0007] 2) Resource scheduling and coupling problems

[0008] In vehicular networks, service deployment, task offloading, and access control are closely intertwined. Service deployment determines the types of tasks the RSU can handle, the task offloading ratio affects the load distribution between local and edge computing, and access control determines the communication permissions of VTUs within a time slot. Existing technologies often overlook the coupling between these components and optimize them separately, which can easily lead to resource conflicts and increase task failure rates in high-mobility scenarios. Furthermore, under the TDMA architecture, only one VTU is allowed to access the RSU per time slot. How to rationally allocate time slots to match task requirements is also a key challenge in resource scheduling.

[0009] 3) Dynamic environmental adaptation problem

[0010] Vehicular networks are highly dynamic, including channel gain variations caused by the high-speed movement of VTUs, the randomness of task requests, and the time-varying characteristics of wireless channels. Traditional optimization methods struggle to cope with these dynamic factors, while a single DRL algorithm is prone to convergence instability when dealing with mixed action spaces, failing to guarantee a continuous reduction in task processing failure rate.

[0011] 4) Constraint satisfaction problem

[0012] During service deployment, RSUs have limited storage capacity and must meet binary constraints and storage capacity constraints for service deployment and replacement. During task offloading, the ratio of local processing to edge offloading of VTUs must sum to 1. These constraints increase the complexity of the optimization problem, and improper handling can lead to infeasible decisions, further increasing the risk of task failure. Summary of the Invention

[0013] To address the issues of insufficient multi-timescale coordination and neglect of resource scheduling coupling in traditional vehicle network optimization technologies for handling high real-time and high-dynamic tasks in the field of intelligent transportation and vehicle networking, this invention provides a joint optimization method and system for MEC vehicle networks based on multi-timescale hierarchical DRL. It focuses on the core links of vehicle networks, and constructs a joint optimization framework around service deployment and replacement, VTU task offloading rate, and TDMA access control optimization. This breaks through traditional bottlenecks to reduce task failure rate and provides technical support for intelligent applications such as autonomous driving.

[0014] This invention is implemented as follows: a method and system for optimizing vehicle-to-everything (V2X) networks based on edge computing and time-division multiple access. This system, by combining a master-slave dual-layer deep reinforcement learning network, achieves joint optimization of key parameters in the V2X network, such as service deployment and replacement, task offloading rate, and access control. The system includes the following steps:

[0015] S101, System Initialization

[0016] Specifically, the following parameters and configurations need to be set: First, the network parameters need to be initialized, including the parameters ω of the main Q network and the target Q network. - The parameters θ of the main Actor network and the parameters w of the main Critic network are set, and the initial parameters of the target Actor network are set to θ. - =θ, the initial parameters of the target Critic network are set to w - =w; Secondly, the parameters related to iteration and time scale need to be set, including the total number of iterations L, the number of iterations in each iteration (i.e., the number of frames K), and the number of time slots T within each frame; In addition, the key parameters required for the algorithm to run need to be initialized, namely the learning rate α1 of the DQN network, the learning rate α2 of the Critic network and the learning rate β2 of the Actor network, and the discount factor γ used to balance current and future rewards; Finally, the initialization of experience storage and exploration-related components is completed, specifically setting the capacity D of the replay buffer. hi and D l The selection sizes M1 and M2 for small batches of samples, and the OU noise Ξ(t) used to realize the exploration function in the agent action selection process;

[0017] S102 focuses on the interaction process between intelligent agents and the in-vehicle network environment:

[0018] At the beginning of each optimization cycle, the system state s corresponding to the high-level DQN agent must first be determined. hi (k) Complete the initial configuration to ensure that the state parameters accurately reflect the current network conditions; then, select the high-level action a of the DQN agent according to the preset ∈-greedy decision-making strategy. hi (k) Through this action selection process, the service deployment strategy and service replacement strategy under the corresponding frame time scale are finally determined;

[0019] S103, the system determines whether the actions of the high-level units meet the preset constraints;

[0020] First, the agent executes the previously generated high-level action; then, the system determines whether the action meets preset constraints, and this determination is directly used for temporary high-level reward metrics. The generation of the temporary high-level reward indicator; if it is determined that the action satisfies all constraints, then the temporary high-level reward indicator is set to [value]. In this case, the DDPG agent will be invoked in each time slot contained in the current frame to perform optimization work related to the underlying task processing; conversely, if the action does not meet the constraints, the temporary high-level reward metric will be set to a value. And it does not initiate the optimization process for subsequent underlying task processing;

[0021] S104, Generate a VTU task offload rate optimization strategy suitable for the current time slot;

[0022] First, the system state s of the underlying DDPG agent needs to be considered. l (t) Complete the initialization operation to ensure that the state parameter accurately reflects the real-time operational basic information of the vehicular network in the current time slot; subsequently, the DDPG agent executes the corresponding action a according to the preset decision logic. l (t), through this series of actions, the VTUs task offloading rate optimization strategy suitable for the current time slot is finally generated, providing key strategy support for efficient task processing in subsequent time slots;

[0023] S105 is the reward value corresponding to the feedback from the agent;

[0024] First, the underlying agent performs the operation according to the previously generated specific actions. After the action is completed, the system will feed back the corresponding reward value r to the agent based on the actual effect of task processing in the current time slot. l (t); Meanwhile, as actions are executed and the network's operating state dynamically changes, the current state s of the underlying system changes. l (t) will then complete the update transition and transform into the new state s corresponding to the next time slot. l (t+1) provides the latest state basis for subsequent underlying optimization decisions;

[0025] S106, Extract underlying experience data;

[0026] First, the system experience acquired in the current time slot needs to be stored in the preset experience playback buffer D. l If the experience replay buffer D is found during the storage process; l If the capacity limit has been reached, the oldest experience data stored in the buffer is deleted according to the "first-in, first-out" principle to free up storage space for new experience data; then, data is retrieved from the experience set. In this context, a small set of low-level experience data of size M2 is randomly selected, where M2 represents the set of low-level experience data that is selected.

[0027] S107, update the parameters of the target Actor network and the target Critic network;

[0028] First, based on previously stored experience data, the TD target and TD error of the underlying DDPG agent are calculated, providing a key basis for subsequent network parameter adjustments. Next, based on the calculated TD error, the parameters of the main underlying Actor network and the main underlying Critic network are updated and optimized to ensure that the network can better adapt to the current in-vehicle network environment. Finally, after the main network parameters are updated, the parameters of the target Actor network and the target Critic network are updated simultaneously to maintain the synergy and stability of the two-layer network architecture.

[0029] S108, update the parameters of the target Actor network and the target Critic network accordingly;

[0030] First, based on a small batch of experience data extracted from the experience pool, the parameters of the main Actor network and the main Critic network are adjusted and updated respectively to ensure that the network can optimize decision-making capabilities based on the latest experience. Then, based on the updated parameters of the main Actor network and the main Critic network, the parameters of the target Actor network and the target Critic network are updated accordingly to ensure the parameter synergy between the main network and the target network and provide stable support for subsequent optimization decisions.

[0031] S109, Transition Conversion;

[0032] If the current frame reaches its last time slot, it must first wait for the lower-level DDPG agent to complete its action and obtain the corresponding reward; only after this will the higher-level DQN agent receive the higher-level reward r corresponding to that frame. hi (k), and at the same time, the current state s of the higher-level system hi (k) will also complete the transition and update to the new state s corresponding to the next frame. hi (k+1);

[0033] S110, obtain the estimated Q value;

[0034] First, the currently acquired high-level experience needs to be stored in the experience replay buffer D. hi If the buffer D is found during storage; hi If the capacity limit has been reached, the longest-stored experience data in the buffer is removed according to the "first-in, first-out" principle to make room for new experience. Then, a small batch of experience of size M1 is extracted from the experience replay buffer, and the state and action in it are fed into the DQN main network as input data. Through the calculation and processing of the main network, the corresponding estimated Q value is finally obtained.

[0035] S111 focuses on network optimization for high-level DQN agents:

[0036] First, based on previously acquired empirical data, the TD target and TD error of the high-level DQN agent are calculated, providing a core basis for network parameter adjustment. Next, based on the calculated TD error, the parameters of the main DQN network and the target DQN network are updated respectively. Finally, using the updated main Q network parameters as a benchmark, the target Q network parameters are adjusted synchronously to ensure the coordinated adaptation of the two Q network parameters.

[0037] S112, by repeatedly going through the previous optimization iteration process, selects the actions that can achieve the best effect based on the Q value of each action under different system states during training; in this way, the task failure rate of the vehicle network is gradually reduced. After multiple iterations of optimization, the optimal optimization strategy for key aspects such as service deployment and replacement, VTU task offloading rate and access control is finally obtained.

[0038] Furthermore, at the beginning of each cycle, S102 initializes the system state s of the high-level DQN agent. hi (k); where the state is represented as:

[0039] s hi (k)={r(k),N(k-1)}

[0040] in, The task request for the k-th frame representing the VTUs; Indicates whether service h exists on the RSU at the end of frame k-1 or the beginning of frame k; uses a policy to select the high-level action a of the DQN agent. hi (k) yields the service deployment and replacement strategy for the frame, with the action represented as:

[0041]

[0042] in Service placement decision; It is a service replacement decision.

[0043] Furthermore, the formula for calculating the temporary reward in S103 is as follows:

[0044]

[0045] Among them, C1: Service deployment decision δ h (k) and replacement decision All are binary variables (taking only 0 or 1), meaning that for a given service, each frame initially only allows selection of "deploy" or "not deploy" or "delete" or "not delete," avoiding ambiguity in decision-making; C2: only when there is no service n on the RSU. hThe deployment of this service can only be executed when (k-1) = 0, to prevent the duplicate deployment of the same service and the waste of storage resources; C3: Service n can only be deployed on the RSU if it already exists. h The deletion action for this service can only be performed when (k-1) = 1, to avoid invalid operations caused by deleting a non-existent service; C4: For the same service h, deployment and deletion actions cannot be performed simultaneously within the same frame. To avoid logical contradictions; C5: Total storage space for all deployed services on the RSU. Not exceeding its storage capacity limit Π U This ensures compliance with hardware resource limitations. High-level temporary rewards are used to determine whether low-level optimizations are necessary. If... This indicates that the service placement and replacement operations meet the constraints (C1)-(C5), and the strategy is feasible. At this point, the DDPG agent will be invoked to execute the underlying optimization process; if This means that the service placement and replacement strategy does not meet the constraints and is infeasible, making underlying optimization meaningless. Since the high-level solution and the low-level solution together constitute the complete solution to the problem, when the high-level strategy itself is infeasible, even if the low-level optimization yields a result, the overall solution will still fail, so there is no need to perform low-level optimization.

[0046] Furthermore, in S104, the state s of the underlying DDPG agent is initialized. l (t), where the state is represented as:

[0047]

[0048] Where N(k) = {n h (k)},n h (k)∈{0,1}, For service deployment conditions, it indicates whether service h has been deployed on the RSU at the start of the time slot; u(t-1)={u i (t-1)} is the position of VTUi at the end of time slot t-1; η(t-1)={η i (t-1)} is the direction of movement of VTUi; It is the distance between the VTUi and the RSU; It is the channel gain; R(t) = {R i (t)} is the transmission rate; I 1 (t-1,k) and I 2 (t-1,k) are the sets of VTUs with allocated and unallocated time slots, respectively; A(t)={a i (t)} represents the access control policy. We denote the state space of the low-level DDPG agent as S. l It is the set of all lower-level states.

[0049] The DDPG agent executes an action, which is represented as follows:

[0050]

[0051] in For VTUs' task offload rate, This represents the local processing ratio. This represents the proportion of data unloaded to RSU. The action space of the DDPG agent is represented as... It is the set of all feasible solutions to the variables related to task processing.

[0052] Furthermore, in S105, the agent executes the generated action, receives a reward, and the underlying state transitions to the next state; the reward is calculated using the following formula:

[0053]

[0054] C6, This represents the local task offloading rate of the i-th device in the vehicle-to-everything (VTUs) under time slot t. The lower limit is 0, meaning the proportion of tasks processed locally by this device cannot be negative, ensuring the reasonableness of the proportion and preventing negative task allocation ratios. (C7) and in This can be viewed as an unloading rate indicator related to the proportion of local task processing. This refers to the percentage of RSU edge unloading tasks. The maximum local task processing ratio (corresponding to the uninstallation rate metric) is limited to 1, meaning that a maximum of all tasks can be processed locally; while This indicates that for the task of the i-th device in time slot t, it can either be processed locally or offloaded to the RSU edge server for processing. The total task allocation ratio of both is 100%, covering all destinations of task processing and ensuring the integrity of the task allocation logic.

[0055] Furthermore, in step S107, regarding the network computation stage of the underlying DDPG agent, the main Actor network will perform prediction operations based on the current state of the underlying system, and the prediction result can be expressed as:

[0056] μ(s l (t);θ)

[0057] Furthermore, the basis for the target actor network's performance prediction is:

[0058] μ′(s l (t+1); θ - )

[0059] Major commentators' online predictions:

[0060] q(s l (t),a l (t);w)

[0061] Meanwhile, the target evaluator network conducts predictions in the following manner:

[0062] q′(s l (t+1),μ′(s l (t+1); θ - );w - )

[0063] The TD target is calculated according to the following formula:

[0064] y l (t)=r l (t)+γq′(s l (t+1),μ′(s l (t+1); θ - );w - )

[0065] The TD error is obtained by calculation based on the following formula:

[0066] δ l (t)=q(s l (t),a l (t);w)-y l (t)

[0067] The parameters of the critic network must be updated according to the following formula:

[0068]

[0069] The parameters of the actor network must be updated according to the following formula:

[0070]

[0071] Furthermore, the update formula for updating the target Actor and Critic network parameters based on the updated main Actor and Critic network parameters in S108 is as follows:

[0072] θ - ←ζθ+(1-ζ)θ -

[0073] w - ←ζw+(1-ζ)w -

[0074] Here, the symbol θ refers to the parameters of the current policy network. -These correspond to the parameters of the target policy network (i.e., the target actor network); while w represents the parameters of the current value network. - This represents the parameters of the target value network. Additionally, the parameter ζ∈[0,1] is a hyperparameter that needs to be manually configured; its specific value can be adjusted according to the actual training scenario and convergence requirements.

[0075] Furthermore, the calculation formula for the senior management reward in S109 is as follows:

[0076] r hi (k)=r fail (T,k)

[0077] Where r fail (T,k) is the system's task failure rate during the k-th frame, which is our objective function.

[0078] Furthermore, in step S110, a small batch of experience data needs to be extracted from the experience replay buffer. Then, the states and actions from the extracted experience are used as input to the DQN main network for calculation, ultimately obtaining the estimated Q-value used to evaluate the policy value. Its expression is:

[0079] Q(s hi (k),a hi (k);ω)

[0080] Where ω is the weight of the main Q-network;

[0081] s hi Using (k+1) as input to the target network, we obtain:

[0082]

[0083] It should be noted that ω - This symbol represents the weight parameters corresponding to the target Q-network.

[0084] Furthermore, in step S111, the generation of the time difference (TD) target requires calculation according to a specific formula:

[0085]

[0086] And based on the calculated TD error:

[0087] δ hi (k)=Q(s hi (k),a hi (k);ω)-y hi (k)

[0088] Update the main DQN network:

[0089]

[0090] It should be noted that α1 in the formula represents the learning rate; and the parameter updates of the target DQN network must strictly follow the following formula:

[0091] ω - ←ζω+(1-ζ)ω -

[0092] Here, ζ∈[0,1] is a hyperparameter that needs to be manually adjusted.

[0093] Another objective of this invention is to provide a corresponding system adapted to the "MEC vehicular network joint optimization method based on multi-time-scale hierarchical DRL", namely, the MEC vehicular network joint optimization system based on multi-time-scale hierarchical DRL, the composition of which is as follows:

[0094] One of the functions of the system initialization module is to initialize the parameters related to the deep Q-network algorithm, including the parameters ω of the main Q-network and the target Q-network. - ;

[0095] This module is also responsible for initializing the parameters of the deep deterministic policy gradient algorithm, specifically the parameters θ of the main Actor network and w of the main Critic network, while setting the initial parameters of the target Actor network to θ. - =θ, the initial parameters of the target Critic network are set to w - =w;

[0096] This module is also responsible for setting parameters, including setting the total number of iterations L, the number of iterations included in each iteration (i.e. the number of frames K), and the number of time slots T divided within each frame.

[0097] The initialization operations that need to be completed also include: setting the learning rate α1 of the DQN network, and the corresponding learning rates α2 and β2 of the Critic network and Actor network; determining the discount factor γ; and initializing the capacity D of the experience replay buffer. hi and D l The sizes M1 and M2 of the mini-batch are determined, and the noise parameters used for action exploration are initialized.

[0098] The configuration module is used to set application-specific parameters, including various task-related parameters such as the number of vehicle terminals (I).

[0099] The core functions of the intelligent agent module are divided into two parts: First, at the start of each cycle, it generates corresponding actions based on the current state of the high-level network, and this process supports the use of a greedy strategy for the generated actions; Second, also at the start of each cycle, it generates actions based on the current state of the low-level network, and this step has the ability to add exploratory noise to the generated actions.

[0100] The action execution module has two functions: on the one hand, it is responsible for the placement and replacement of services; on the other hand, it is used to implement optimization strategies for VTUs, specifically covering task offload rate control, access control, and other content.

[0101] The reward acquisition module has a dual core function: on the one hand, it is responsible for executing the corresponding actions and calculating the temporary rewards. The design of this module must be based on whether the system constraints are met, and it also undertakes the function of determining whether to start the underlying optimization. On the other hand, this module also needs to calculate the rewards by combining the actions already executed and the current system state. The specific calculation dimensions include two levels: one is the task failure rate of all devices in the system in the current time slot, and the other is the task failure rate of these devices at the end of the frame period.

[0102] State transition module: It undertakes the task of transitioning the state of the underlying system and is responsible for transitioning the current state of the system to the state of the next moment.

[0103] Experience replay module: The core function is to store the experience tuples generated by each interaction. These tuples contain the current state of the underlying system, the action performed, the corresponding reward, and the next state after the transition.

[0104] The data sampling module extracts mini-batch sample data from the experience data stored in the experience playback module to provide data support for subsequent network learning.

[0105] Network update module: Based on the sample information obtained by the data sampling module, this module needs to update the main Q network, main Actor network, and main Critic network. This module has a built-in parameter optimization unit, which adjusts the parameters of each network through gradient descent.

[0106] Parameter update module: It is mainly used to update the parameters of the target Q network, target Actor network and target Critic network with reference to the parameters of the main network. This module adopts a soft update strategy, and its parameter synchronization unit can ensure that the parameters of the target network are always the weighted average of the parameters of the corresponding main network.

[0107] The network update module and the parameter update module work together to update the parameters of the target Actor network and the target Critic network by referring to the main network parameters. This process also adopts a soft update strategy, and the target network parameters need to be synchronized in the form of a weighted average of the main network parameters. This function is completed by the parameter synchronization unit in the module.

[0108] Based on the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solution to be protected by this invention from the following aspects:

[0109] First, this invention has significant advantages in the fusion of MEC in-vehicle networks and hierarchical deep reinforcement learning (DRL), as detailed below:

[0110] Firstly, it achieves a dual improvement in task processing efficiency and failure rate. This invention deeply integrates MEC technology with in-vehicle networks, providing VTUs with near-field computing resources by deploying edge servers on the RSU, effectively compensating for the limited local computing capabilities of VTUs. For latency-sensitive tasks in autonomous driving, such as real-time environmental perception and traffic signal optimization, the collaborative processing mode between the RSU and VTUs significantly shortens the task processing cycle, avoiding task timeouts and failures due to insufficient local computing resources. Simultaneously, the resource allocation mechanism based on the TDMA architecture ensures that only one VTU is allowed to communicate with the RSU per time slot, completely eliminating communication interference between devices and further reducing the risk of task failures due to data transmission conflicts.

[0111] Secondly, it involves the synergy of multi-timescale optimization and dynamic resource management. This invention innovatively divides the timeline into coarse-grained "frames" and fine-grained "time slots," constructing a hierarchical optimization strategy. At the frame level, the higher-level DQN agent is responsible for service deployment and replacement. Based on the historical task request patterns of VTUs and the storage status of RSUs, it pre-caches services with high-frequency demands to RSUs, ensuring the rationality of long-term resource allocation. At the time slot level, the DDPG agent dynamically adjusts the task offloading rate and access control strategy based on real-time status such as vehicle location, channel quality, and the set of VTUs with remaining unallocated time slots, ensuring high service efficiency in the short term. This "long-term planning + real-time adjustment" model avoids blind service deployment and can quickly respond to dynamic changes such as vehicle movement and sudden task bursts, achieving refined resource utilization.

[0112] Finally, the hierarchical DRL algorithm achieves a breakthrough in adaptability to complex scenarios. Addressing the hybrid action space comprised of service deployment (discrete actions) and task offloading rate (continuous actions) in vehicular networks, and the dynamic environmental challenges brought about by the high mobility of vehicles, this invention proposes a hierarchical DRL framework integrating DQN and DDPG. The higher-layer DQN excels at handling discrete decisions, learning optimal service deployment and replacement strategies through experience replay and target network updates, ensuring efficient utilization of RSU storage resources. The lower-layer DDPG is suitable for continuous action optimization, adjusting the task offloading ratio in real time through an Actor-Critic architecture to adapt to rapid changes in vehicle location and channel conditions. Particularly noteworthy is that the lower-layer DDPG, through a constraint-aware reward mechanism, transforms the binary decisions of access control into probabilistic choices, satisfying both the time slot allocation requirements of the TDMA architecture and improving the algorithm's convergence stability.

[0113] Second, the significant technical advancement achieved by this invention lies in the implementation of a hierarchical DRL-based joint optimization method for MEC vehicular networks, which has made significant progress in the following 16 key aspects:

[0114] 1) Efficient service deployment and replacement strategies

[0115] This method leverages a high-level DQN agent to achieve dynamic optimization of service deployment and replacement. The DQN agent comprehensively evaluates the deployment value of different services based on the real-time task requirements of VTUs and the storage status of RSUs: placement is only performed when a service has not yet been deployed; replacement is performed when services are redundant, effectively avoiding storage resource waste caused by duplicate deployments. Through a long-term learning process, the DQN agent can accurately predict high-frequency demand services and complete deployment in advance, thereby reducing task failures caused by service absences.

[0116] 2) Dynamic task unloading optimization scheme

[0117] This method achieves real-time adjustment of the task offloading rate through the underlying DDPG agent. The DDPG agent dynamically allocates the proportion of tasks processed locally and offloaded to the RSU based on the local processing capabilities of the VTUs, the computational load of the RSU, the distance between the vehicle and the RSU, and channel quality, ensuring that the total task latency does not exceed the maximum tolerable threshold. For example, when the vehicle moves further away from the RSU, causing increased communication latency, the algorithm increases the proportion of local processing; when the RSU load is low, the algorithm increases the offloading proportion, making full use of edge computing resources.

[0118] 3) Intelligent access control strategy

[0119] Based on the TDMA architecture, this method designs a distance-priority-based access control mechanism. In each time slot, the RSU prioritizes allocating communication resources to the nearest unallocated time slot VTU, reducing signal attenuation and transmission delay caused by excessive distance. Simultaneously, by maintaining a set of allocated and unallocated time slots for VTUs, it ensures that each VTU is allocated only one time slot per frame, avoiding resource conflicts.

[0120] 4) Multi-timescale collaborative decision-making mechanism

[0121] This method achieves synergy between long-term planning and short-term response through hierarchical decision-making at the frame and time-slot levels. Frame-level service deployment determines the service cache state of RSUs, providing a foundation for time-slot-level task processing; time-slot-level task offloading and access control adjust decisions based on real-time status, feeding back into the optimization of frame-level strategies. This synergistic mechanism ensures both the forward-looking nature of service deployment and the real-time performance of task processing, solving the problem of "at the expense of one aspect" in traditional single-time-scale optimization.

[0122] 5) Innovative Integration of Deep Reinforcement Learning

[0123] By integrating DQN and DDPG algorithms in a hierarchical manner to handle discrete and continuous action spaces respectively, this approach overcomes the limitation of traditional DRL in its difficulty of convergence in mixed action spaces. The high-level DQN learns the long-term value of service deployment through Q-value estimation, while the low-level DDPG learns the real-time policy for task offloading through policy gradients. The two algorithms interact through a reward mechanism, forming a closed-loop optimization.

[0124] 6) Minimize task failure rate optimization

[0125] By directly incorporating task failure rate into the reward function, the algorithm is guided to learn strategies that minimize the failure rate. High-level rewards use end-of-frame failure rate as an indicator to ensure long-term optimization goals; low-level rewards use time-slot failure rate as an indicator to enhance the accuracy of real-time decision-making. This design allows the system to prioritize the completion of critical tasks under resource constraints, significantly reducing the overall failure rate.

[0126] 7) Improved system stability and reliability

[0127] The system's stability is enhanced through a constraint-aware reward mechanism and a soft update strategy. High-level temporary rewards only take effect when service deployment meets constraints such as storage capacity, and low-level rewards only take effect when the offload rate is within a reasonable range, avoiding the learning of ineffective decisions. Target network parameters are smoothly iterated through soft updates, preventing performance fluctuations caused by parameter mutations.

[0128] 8) Adaptability to dynamic environments

[0129] The algorithm can perceive the dynamics of vehicle movement, channel changes, and task arrival in real time and quickly adjust its strategy. Vehicle position is updated in real time through a motion model, channel gain is randomly generated to simulate time-varying characteristics, and task requests are generated according to a specific distribution to simulate randomness. By continuously interacting with the environment, the algorithm constantly optimizes its decisions, ensuring a low failure rate in dynamic scenarios.

[0130] 9) Construction of the joint optimization framework

[0131] This approach overcomes the limitations of existing technologies that optimize service deployment, task offloading, and access control separately, and constructs a framework for joint optimization of these three elements. Service deployment determines the types of tasks that the RSU can handle, task offloading determines the processing location of tasks, and access control determines the timing of task data transmission. These three elements are interconnected and work together to achieve efficient global utilization of network resources.

[0132] 10) Efficient processing of hybrid motion space

[0133] To address the mixed action characteristics of service deployment (discrete) and task offloading (continuous), a layered DRL architecture is adopted to handle them separately: DQN handles discrete service placement / replacement decisions, and DDPG handles continuous offloading rate adjustments. Overall optimization is achieved through information transfer between layers, which solves the problems of slow convergence and poor stability of traditional DRL in mixed spaces.

[0134] 11) Coordinated scheduling of communication and computing resources

[0135] By jointly optimizing transmission rate and computing power, the algorithm achieves coordinated utilization of communication and computing resources. For example, when RSU bandwidth increases, the algorithm increases the task offloading ratio to utilize a higher transmission rate; when RSU computing power is insufficient, the algorithm decreases the offloading ratio to avoid increased processing latency.

[0136] 12) Precise adaptation of vehicle mobility

[0137] By tracking vehicle location and direction in real time, access control and task offloading strategies are dynamically adjusted. For vehicles moving away from the RSU, the local processing ratio is increased in advance; for vehicles moving closer to the RSU, time slots are reserved to prepare for offloading more tasks, ensuring that vehicles can still process tasks efficiently while moving rapidly.

[0138] 13) Efficient allocation of time slot resources

[0139] Based on the TDMA architecture, by maintaining a set of allocated and unallocated time slots for VTUs, it ensures that each VTU is fairly allocated a time slot within a frame, avoiding resource contention. At the same time, time slots are prioritized for VTUs that are closer in distance, improving bandwidth utilization and transmission efficiency, and reducing packet loss and retransmissions caused by excessive distance.

[0140] 14) Innovative design of the reward mechanism

[0141] A multi-tiered reward mechanism was designed: a high-level temporary reward assesses the feasibility of service deployment, a high-level final reward reflects the frame-level failure rate, and a low-level reward reflects the slot-level failure rate. This mechanism ensures the feasibility of decisions while also strengthening the direct optimization of failure rates, making the algorithm's learning objectives clearer.

[0142] 15) Comprehensive advantages of simulation verification

[0143] The superiority of this invention was verified through comparison with various benchmark algorithms. In scenarios with different task sizes, slot lengths, RSU bandwidths, and VTU local processing capabilities, this invention exhibits the lowest task failure rate, with particularly significant advantages in resource-constrained scenarios such as small slots and low bandwidth, demonstrating its broad adaptability.

[0144] 16) Improved convergence speed and stability

[0145] By optimizing the learning rate and adopting a soft update strategy, the algorithm converges faster and exhibits a lower failure rate and less volatility after convergence. Compared to other learning rates, the optimal learning rate enables the algorithm to remain stable while converging quickly, avoiding slow convergence or oscillation problems.

[0146] Third, the core of the vehicle-to-everything (V2X) optimization method and system provided by this invention lies in using mathematical models to guide the system's behavior and learning process. The technical effects brought about by these mathematical models can be explored based on their characteristics:

[0147] 1) The guiding role of the reward mechanism

[0148] The design of high-level temporary rewards, low-level rewards, and high-level final rewards is directly linked to task failure rate and constraint satisfaction. This design allows the algorithm to focus on the core objective of minimizing the failure rate while ensuring that decisions comply with resource constraints. For example, when service deployment violates storage capacity constraints, the temporary reward is 0, preventing the algorithm from learning ineffective strategies; when the task offload rate is not within a reasonable range, the low-level reward is 0, guiding the algorithm to adjust the offload ratio to a reasonable range. In the long run, this reward mechanism can significantly improve the system's task success rate and resource utilization efficiency.

[0149] 2) Optimization efficiency of network updates

[0150] By updating network parameters through small-batch random sampling, parameter oscillations caused by single samples are avoided. Gradient descent is used to update the Critic and Main Q networks, while gradient ascent is used to update the Actor network, allowing the network parameters to iterate towards the optimal direction. For example, the Critic network adjusts its parameters through error correction to improve the accuracy of value estimation; the Actor network adjusts its parameters through policy gradients to enhance the cumulative reward of the policy. This update mechanism enables the network to quickly learn environmental patterns and adapt to dynamically changing in-vehicle scenarios.

[0151] 3) The balancing effect of time difference objectives

[0152] The loss function, calculated based on a time-difference objective, balances the weights of current and future rewards, allowing the algorithm to consider both immediate gains and long-term benefits. The Critic network optimizes value estimation by minimizing the loss function, providing a reliable value assessment for the Actor network and avoiding short-sighted decision-making. This design enhances learning stability, enabling the algorithm to converge to a better strategy even in complex and dynamic environments.

[0153] 4) Smooth transition of parameter updates

[0154] The target network parameters are slowly synchronized through soft updates, ensuring a smooth transition between the target and main network parameters and avoiding learning oscillations caused by sudden changes in target values. For example, when the main network parameters are updated due to new experience, the target network parameters only partially follow, maintaining the relative stability of the target value and making error calculation more reliable. This mechanism significantly improves the convergence stability of the algorithm, especially in highly dynamic environments such as vehicular networks.

[0155] 5) Precise iteration of service status

[0156] The service state is updated through an iterative model, ensuring the uniqueness of service deployments and compliance with storage capacity constraints. This model accurately reflects the service cache state of the RSU, providing clear state feedback to the higher-level DQN agent and enabling more precise service deployment decisions.

[0157] Fourth, as supplementary evidence supporting the inventiveness of the claims of this invention, it can also be reflected in the expected benefits and commercial value generated after the transformation of the technical solution, as follows:

[0158] This invention achieves joint optimization of the MEC (Intelligent Transportation System) vehicular network through hierarchical DRL, significantly reducing the task failure rate and improving the reliability and efficiency of the intelligent transportation system. In terms of technology transfer, its expected benefits and commercial value are mainly reflected in the following aspects:

[0159] From an economic perspective, this invention can significantly reduce the operating costs of intelligent transportation systems. By optimizing service deployment and task offloading, it reduces wasted storage resources and idle computing resources in the Resource Unit (RSU), thereby improving equipment utilization. Through intelligent access control, it improves the efficiency of communication bandwidth utilization and reduces data transmission energy consumption and packet loss rate. It is estimated that compared with traditional methods, this invention can increase RSU resource utilization by 30%-50%, reduce task processing energy consumption by 20%-30%, and significantly reduce long-term operating costs.

[0160] From a social benefit perspective, this invention provides reliable technical support for key applications such as autonomous driving, real-time traffic monitoring, and emergency event early warning. Reducing the task failure rate can decrease traffic accidents caused by data processing delays and improve road traffic efficiency; real-time traffic signal optimization, based on reliable task processing, can reduce traffic congestion and shorten travel time. For example, in an emergency, vehicle warning information can be quickly transmitted to the RSU (Roadside Unit) via the optimized network for processing, promptly notifying surrounding vehicles to avoid the accident and reducing the risk of accidents.

[0161] In terms of commercial applications, this invention can be widely used in scenarios such as urban intelligent transportation networks, highway communication systems, and port / industrial park fleet management. In urban transportation, by deploying the optimized system of this invention, the collaborative capabilities of autonomous vehicles can be improved, enabling efficient vehicle-road cooperative scheduling; on highways, it can ensure real-time communication and data processing for long-distance vehicles, improving road safety; and in closed industrial parks, it can optimize fleet task collaboration and improve operational efficiency.

[0162] Furthermore, the hierarchical DRL framework of this invention has strong versatility and can be extended to other edge computing scenarios, providing optimization solutions for related industries and further expanding their commercial value. In summary, this invention not only achieves a technological breakthrough but also possesses significant economic and social value in practical applications, and has broad market prospects. Attached Figure Description

[0163] Figure 1 This is a flowchart of the vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access provided in an embodiment of the present invention;

[0164] Figure 2 This is an applicable scenario diagram provided by an embodiment of the present invention.

[0165] Figure 3 This is a flowchart of the MEC and TDMA vehicular network optimization method based on hierarchical DRL.

[0166] Figure 4 This is a structural diagram of a vehicle-to-everything (V2X) optimization system based on edge computing and time-division multiple access provided in an embodiment of the present invention.

[0167] Figure 5 This is a comparison chart of the overall convergence performance provided in the embodiments of the present invention. Detailed Implementation

[0168] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0169] like Figure 1 As shown, the vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access provided by an embodiment of the present invention includes:

[0170] S101, System Initialization

[0171] Specifically, the following parameters and configurations need to be set: First, the network parameters need to be initialized, including the parameters ω of the main Q network and the target Q network. - The parameters θ of the main Actor network and the parameters w of the main Critic network are set, and the initial parameters of the target Actor network are set to θ. - =θ, the initial parameters of the target Critic network are set to w - =w; Secondly, the parameters related to iteration and time scale need to be set, including the total number of iterations L, the number of iterations in each iteration (i.e., the number of frames K), and the number of time slots T within each frame; In addition, the key parameters required for the algorithm to run need to be initialized, namely the learning rate α1 of the DQN network, the learning rate α2 of the Critic network and the learning rate β2 of the Actor network, and the discount factor γ used to balance current and future rewards; Finally, the initialization of experience storage and exploration-related components is completed, specifically setting the capacity D of the replay buffer. hi and D l The selection sizes M1 and M2 for small batches of samples, and the OU noise Ξ(t) used to realize the exploration function in the agent action selection process;

[0172] S102 focuses on the interaction process between intelligent agents and the in-vehicle network environment:

[0173] At the beginning of each optimization cycle, the system state s corresponding to the high-level DQN agent must first be determined. hi (k) Complete the initial configuration to ensure that the state parameters accurately reflect the current network conditions; then, select the high-level action a of the DQN agent according to the preset ∈-greedy decision-making strategy. hi (k) Through this action selection process, the service deployment strategy and service replacement strategy under the corresponding frame time scale are finally determined;

[0174] S103, the system determines whether the actions of the high-level units meet the preset constraints;

[0175] First, the agent executes the previously generated high-level action; then, the system determines whether the action meets preset constraints, and this determination is directly used for temporary high-level reward metrics. The generation of the temporary high-level reward indicator; if it is determined that the action satisfies all constraints, then the temporary high-level reward indicator is set to [value]. In this case, the DDPG agent will be invoked in each time slot contained in the current frame to perform optimization work related to the underlying task processing; conversely, if the action does not meet the constraints, the temporary high-level reward metric will be set to a value. And it does not initiate the optimization process for subsequent underlying task processing;

[0176] S104, Generate a VTU task offload rate optimization strategy suitable for the current time slot;

[0177] First, the system state s of the underlying DDPG agent needs to be considered. l (t) Complete the initialization operation to ensure that the state parameter accurately reflects the real-time operational basic information of the vehicular network in the current time slot; subsequently, the DDPG agent executes the corresponding action a according to the preset decision logic. l (t), through this series of actions, the VTUs task offloading rate optimization strategy suitable for the current time slot is finally generated, providing key strategy support for efficient task processing in subsequent time slots;

[0178] S105 is the reward value corresponding to the feedback from the agent;

[0179] First, the underlying agent performs the operation according to the previously generated specific actions. After the action is completed, the system will feed back the corresponding reward value r to the agent based on the actual effect of task processing in the current time slot. l (t); Meanwhile, as actions are executed and the network's operating state dynamically changes, the current state s of the underlying system changes. l (t) will then complete the update transition and transform into the new state s corresponding to the next time slot. l (t+1) provides the latest state basis for subsequent underlying optimization decisions;

[0180] S106, Extract underlying experience data;

[0181] First, the system experience acquired in the current time slot needs to be stored in the preset experience playback buffer D. l If the experience replay buffer D is found during the storage process; lIf the capacity limit has been reached, the oldest experience data stored in the buffer is deleted according to the "first-in, first-out" principle to free up storage space for new experience data; then, data is retrieved from the experience set. In this context, a small set of low-level experience data of size M2 is randomly selected, where M2 represents the set of low-level experience data that is selected.

[0182] S107, update the parameters of the target Actor network and the target Critic network;

[0183] First, based on previously stored experience data, the TD target and TD error of the underlying DDPG agent are calculated, providing a key basis for subsequent network parameter adjustments. Next, based on the calculated TD error, the parameters of the main underlying Actor network and the main underlying Critic network are updated and optimized to ensure that the network can better adapt to the current in-vehicle network environment. Finally, after the main network parameters are updated, the parameters of the target Actor network and the target Critic network are updated simultaneously to maintain the synergy and stability of the two-layer network architecture.

[0184] S108, update the parameters of the target Actor network and the target Critic network accordingly;

[0185] First, based on a small batch of experience data extracted from the experience pool, the parameters of the main Actor network and the main Critic network are adjusted and updated respectively to ensure that the network can optimize decision-making capabilities based on the latest experience. Then, based on the updated parameters of the main Actor network and the main Critic network, the parameters of the target Actor network and the target Critic network are updated accordingly to ensure the parameter synergy between the main network and the target network and provide stable support for subsequent optimization decisions.

[0186] S109, Transition Conversion;

[0187] If the current frame reaches its last time slot, it must first wait for the lower-level DDPG agent to complete its action and obtain the corresponding reward; only after this will the higher-level DQN agent receive the higher-level reward r corresponding to that frame. hi (k), and at the same time, the current state s of the higher-level system hi (k) will also complete the transition and update to the new state s corresponding to the next frame. hi (k+1);

[0188] S110, obtain the estimated Q value;

[0189] First, the currently acquired high-level experience needs to be stored in the experience replay buffer D. hi If the buffer D is found during storage;hi If the capacity limit has been reached, the longest-stored experience data in the buffer is removed according to the "first-in, first-out" principle to make room for new experience. Then, a small batch of experience of size M1 is extracted from the experience replay buffer, and the state and action in it are fed into the DQN main network as input data. Through the calculation and processing of the main network, the corresponding estimated Q value is finally obtained.

[0190] S111 focuses on network optimization for high-level DQN agents:

[0191] First, based on previously acquired empirical data, the TD target and TD error of the high-level DQN agent are calculated, providing a core basis for network parameter adjustment. Next, based on the calculated TD error, the parameters of the main DQN network and the target DQN network are updated respectively. Finally, using the updated main Q network parameters as a benchmark, the target Q network parameters are adjusted synchronously to ensure the coordinated adaptation of the two Q network parameters.

[0192] S112, by repeatedly going through the previous optimization iteration process, selects the actions that can achieve the best effect based on the Q value of each action under different system states during training; in this way, the task failure rate of the vehicle network is gradually reduced. After multiple iterations of optimization, the optimal optimization strategy for key aspects such as service deployment and replacement, VTU task offloading rate and access control is finally obtained.

[0193] like Figure 4 As shown, the vehicle-to-everything (V2X) optimization system based on edge computing and time-division multiple access provided in this embodiment of the invention includes:

[0194] One of the functions of the system initialization module is to initialize the parameters related to the deep Q-network algorithm, including the parameters of the main Q-network and the target Q-network.

[0195] This module is also responsible for initializing the relevant parameters of the deep deterministic policy gradient algorithm, specifically covering the parameters of the main Actor network, the parameters w of the main Critic network, and setting the initial parameters of the target Actor network to and the initial parameters of the target Critic network to .

[0196] This module is also responsible for setting parameters, specifically including setting the total number of iterations L, the number of iterations included in each iteration (i.e. the number of frames K), and the number of time slots T divided within each frame;

[0197] The initialization operations that need to be completed also include: setting the learning rate of the DQN network, as well as the learning rates of the Critic network and the Actor network; determining the discount factor, initializing the capacity of the experience replay buffer, and specifying the mini-batch sizes M1 and M2. In addition, the noise parameters used for action exploration also need to be initialized.

[0198] The configuration module is used to set application-specific parameters, which include various task-related parameters such as the number of vehicle terminals (I).

[0199] The core functions of the intelligent agent module are divided into two parts: First, at the start of each cycle, it generates corresponding actions based on the current state of the high-level network, and this process supports the use of a greedy strategy for the generated actions; Second, also at the start of each cycle, it generates actions based on the current state of the low-level network, and this step has the ability to add exploratory noise to the generated actions.

[0200] The action execution module has two functions: one is to perform the placement and replacement of services; the other is to implement optimization strategies for VTUs, specifically covering task offload rate control, access control, and other content.

[0201] The reward acquisition module has a dual core function: on the one hand, it is responsible for executing the corresponding actions and calculating the temporary rewards. The design of this module must be based on whether the system constraints are met, and it also undertakes the function of determining whether to start the underlying optimization. On the other hand, this module also needs to calculate the rewards by combining the actions already executed and the current system state. The specific calculation dimensions include two levels: one is the task failure rate of all devices in the system in the current time slot, and the other is the task failure rate of these devices at the end of the frame period.

[0202] State transition module: It undertakes the task of transitioning the state of the underlying system and is responsible for transitioning the current state of the system to the state of the next moment.

[0203] Experience replay module: The core function is to store the experience tuples generated by each interaction. These tuples contain the current state of the underlying system, the action performed, the corresponding reward, and the next state after the transition.

[0204] The data sampling module extracts mini-batch sample data from the experience data stored in the experience playback module to provide data support for subsequent network learning.

[0205] Network update module: Based on the sample information obtained by the data sampling module, this module needs to update the main Q network, main Actor network, and main Critic network. This module has a built-in parameter optimization unit, which adjusts the parameters of each network through gradient descent.

[0206] Parameter update module: It is mainly used to update the parameters of the target Q network, target Actor network and target Critic network with reference to the parameters of the main network. This module adopts a soft update strategy, and its parameter synchronization unit can ensure that the parameters of the target network are always the weighted average of the parameters of the corresponding main network.

[0207] The network update module and the parameter update module work together to update the parameters of the target Actor network and the target Critic network by referring to the main network parameters. This process also adopts a soft update strategy, and the target network parameters need to be synchronized in the form of a weighted average of the main network parameters. This function is completed by the parameter synchronization unit in the module.

[0208] The core innovation of this invention lies in proposing a vehicle-to-everything (V2X) optimization method and system based on edge computing and time-division multiple access (TDMA) to optimize the performance of vehicular networks driven by mobile edge computing (MEC). This scheme employs a master-slave two-layer architecture: the upper-layer Deep Q Network (DQN) focuses on frame-level service deployment and replacement strategy formulation, pre-caching high-frequency services to RSUs based on the task requests of vehicle-to-everything (VTUs) and the storage status of roadside units (RSUs), thus ensuring the rationality of resource allocation in the long term; the lower-layer Deep Deterministic Policy Gradient (DDPG) is responsible for slot-level task offloading rate adjustment and access control, optimizing task processing methods and communication resource allocation based on dynamically changing network conditions such as real-time vehicle location and channel quality. This layered architecture not only effectively addresses the dynamic environmental challenges brought about by high vehicle mobility and random task requests, but also achieves collaborative optimization of service deployment, task offloading, and access control, successfully overcoming the limitations of single-dimensional optimization in traditional technologies.

[0209] In practical applications, this invention can be optimized for various scenarios. In urban traffic scenarios, roadside RSUs pre-cached the environmental perception services required for autonomous driving through high-level DQN, while the underlying DDPG adjusted the task offloading ratio of vehicles in real time to ensure that data processing latency met safety requirements. In highway scenarios, the algorithm dynamically allocated communication time slots based on the vehicle's driving direction and distance from the RSU, reducing task failures caused by signal attenuation. In port fleet management, frame-level service deployment coordinated fleet task allocation, and time slot-level optimization ensured efficient synchronization of data transmission between vehicles.

[0210] The advantages of this invention lie in its dynamic adaptability: when vehicles move rapidly, the underlying DDPG can instantly adjust access priorities, allocating time slots preferentially to nearby vehicles; in the face of sudden task requests, the higher-level DQN can quickly update the service cache to ensure the processing of urgent tasks. Simultaneously, the joint optimization mechanism improves resource utilization, with RSU storage resources focused on high-frequency services and communication bandwidth prioritized for critical tasks, reducing equipment energy consumption and operating costs. In summary, this invention achieves efficient optimization of the vehicular network through layered DRL, which can improve traffic safety, increase operational efficiency, and reduce operating costs in practical applications, demonstrating significant technical and application value.

[0211] Example 1: Optimization of Vehicle-Mounted Emergency Response Network

[0212] In urban emergency rescue, the MEC on-board network optimization method based on multi-time-scale hierarchical DRL can efficiently handle task collaboration at the accident site and ensure real-time processing of rescue commands and data.

[0213] S101: The system is initialized as an urban traffic environment, taking into account road distribution, building obstructions, and roadside communication infrastructure. The initial network layout includes multiple vehicle-mounted terminals (VTUs) and roadside units (RSUs) as mobile edge computing nodes within the city. RSUs are deployed at intersections and accident-prone sections to support real-time data processing.

[0214] S102: In the event of a public incident, the high-level DQN agent quickly assesses service deployment and replacement strategies, prioritizing the caching of critical services such as "emergency data fusion" and "rescue command distribution" to RSUs to ensure that VTUs' emergency task requests are processed with priority.

[0215] S103-S111: Refined processing through the underlying DDPG intelligent agent: adjusting the task offloading rate based on the distance between VTUs and the accident site; employing a TDMA mechanism to prioritize the allocation of time slots to vehicles participating in the rescue; dynamically scheduling RSU computing resources to prioritize tasks such as on-site image analysis and path planning. Through iterative optimization, resource utilization is maximized, and the response time is controlled within 1 second.

[0216] S112: The iterative optimization process forms a stable strategy: RSUs pre-cachise high-frequency emergency services, VTUs adjust task processing methods based on real-time location, and time slot resources are allocated to core rescue vehicles. This strategy ensures efficient collaboration between vehicles and RSUs in public events, improves emergency response efficiency, and safeguards public safety in smart city emergency management, demonstrating significant practical value.

[0217] Example 2: Vehicle-Mounted Network Support System for Large Events

[0218] To address the in-vehicle network requirements during large-scale sporting events, music festivals, and other similar activities, the optimized method based on this invention can improve the coordination efficiency between on-site vehicles and roadside units, ensuring stable network services.

[0219] S101: System initialization is set according to the scale of the event venue, deploying RSUs to cover the roads around the event and connecting the vehicle terminal VTUs participating in the event; configure basic parameters to ensure that the processing latency of critical services meets the requirements.

[0220] S102: Before the event begins, the high-level DQN agent determines the service deployment strategy based on the expected number of participating vehicles and the type of task, and caches high-frequency demand services to RSU in advance.

[0221] S103-S111: The underlying intelligent agent adjusts in real time, optimizes the task offloading ratio based on the distance between the vehicle and the RSU, and prioritizes the allocation of communication time slots to vehicles in the core activity area; through iterative optimization, it ensures that resources are tilted towards critical tasks.

[0222] S112: The final optimization strategy can ensure efficient vehicle communication during the event, reduce task delays caused by network congestion, and improve the participant experience.

[0223] This embodiment is applicable to traffic support for large-scale events. By optimizing in-vehicle network services, it provides support for vehicle dispatching and emergency response during the event, and has practical application value.

[0224] Example 3

[0225] Vehicle-mounted emergency response scenarios place extremely stringent demands on the real-time performance and reliability of network task processing. Critical tasks such as issuing rescue orders at accident sites, rapidly analyzing disaster data, and providing risk warnings for surrounding vehicles can be delayed or fail due to improper network resource scheduling, directly impacting rescue efficiency and even threatening personnel safety. However, traditional vehicle-mounted network optimization solutions either lack multi-timescale collaborative decision-making capabilities, failing to balance long-term service deployment with real-time task response; or fail to achieve coordinated optimization of services, offloading, and access, making it difficult to cope with the dynamic challenges posed by high vehicle mobility and sudden task occurrences. The "edge computing + Time Division Multiple Access (TDMA) + master-slave dual-layer DRL" optimization architecture proposed in this invention precisely addresses these pain points: the upper-layer DQN can pre-cacherate high-frequency demand services in emergency scenarios, while the lower-layer DDPG can adjust task offloading and access strategies in real time, providing efficient support for vehicle-mounted emergency response networks. The implementation process is described in detail below.

[0226] To address the problems existing in the prior art, this implementation provides a joint optimization method for MEC vehicular networks based on multi-time-scale hierarchical DRL. Figure 2This is a scenario diagram showing the application of the method of the present invention. First, the number of ground mobile VTUs I is initialized, and the set of VTUs is represented as... An RSU has the following characteristics: number of iterations L, number of iterations per iteration (i.e., number of frames K), and number of time slots T within each frame.

[0227] In the relevant scenario, at the start of each time slot, VTUs need to complete H types of tasks, the set of which consists of... This indicates that a typical task h can be represented using a quintuple. Let D represent this. h λ is the size of the input data for this task (in bits); h It is the processing density of the task, that is, the CPU cycles required to execute each bit of the input data; O h It is the size of the output calculation result, which is larger than the input data D. h Its size is much smaller and will be ignored in the following text; λ h This refers to the storage space required for deploying the service instance of task h; This is the maximum tolerable task processing delay for task h. This paper assumes... That is, the task should be completed within each time slot.

[0228] This invention establishes a multi-time-scale optimization problem in a vehicle-mounted network supporting MEC and TDMA. First, the time axis is divided into coarse-grained time scales called frames. Then, each frame is further divided into fine-grained time units called time slots. Assume there are a total of K frames, and each frame contains T time slots. The sets of frames and time slots are represented as follows: and Each time slot has a length of τ. The system operates using TDMA, meaning different time slots are allocated to different VTUs for data transmission, and each time slot allows only one VTU to communicate with the RSU within a single frame, thus eliminating interference between VTUs. Furthermore, we require that each VTU's tasks be completed within its assigned time slot, encompassing coarse-grained long-term joint service placement and replacement, fine-grained short-term joint task offloading rate of VTUs and RSUs, VTU access control, and RSU deployment optimization, all aimed at maximizing the operator's long-term average economic profit.

[0229] The RSU deploys an MEC server, and the two are considered the same entity. However, in relevant scenarios, such as in remote road sections or areas with signal obstruction, VTUs can only communicate with the RSU. A controller is deployed at the RSU, which collects dynamic information about the VTUs and the tasks. Based on this information, the RSU controller makes decisions regarding network operation and optimization. The collected information can be described as follows: VTU information refers to the information of the onboard equipment itself, such as communication bandwidth, transmit power, local processing capabilities, etc.; RSU information mainly refers to the RSU's feasible communication resources, computing resources, storage resources, deployed services, etc.; application information is used to describe the characteristics of the task, such as application type, required input data volume, computational load, task execution latency, etc.

[0230] Example 4

[0231] During large-scale events such as sporting events and music festivals, surrounding vehicular networks face unique challenges: a large number of vehicles (such as spectator shuttles, emergency support vehicles, and staff vehicles) converge in a short period, leading to a surge in task requests from vehicle-to-everything (VTUs). Furthermore, vehicles are scattered in location and their movement paths are constantly changing. Simultaneously, the storage, computing, and communication resources of roadside units (RSUs) must support multiple tasks, including real-time navigation, vehicle dispatching, and emergency communication. Traditional vehicular network optimization solutions, lacking multi-dimensional collaborative scheduling capabilities, are prone to problems such as service caching lag, time slot allocation conflicts, and task offloading imbalances, leading to data transmission delays and critical task processing failures. This invention, based on edge computing and Time Division Multiple Access (TDMA), employs a multi-time-scale hierarchical DRL optimization scheme. This scheme achieves dynamic resource adaptation and efficient scheduling through a master-slave dual-layer architecture. The implementation process is explained in detail below using a large-scale event vehicular network support scenario.

[0232] S101 is the system initialization step, which requires setting the following parameters and configurations: First, the network parameters are initialized, including the parameters ω of the main Q network and the target Q network. - The parameters θ of the main Actor network and the parameters w of the main Critic network are set, and the initial parameters of the target Actor network are set to θ. - =θ, the initial parameters of the target Critic network are set to w - =w; Secondly, the parameters related to iteration and time scale need to be set, including the total number of iterations L, the number of iterations in each iteration (i.e., the number of frames K), and the number of time slots T within each frame; In addition, the key parameters required for the algorithm to run need to be initialized, namely the learning rate α1 of the DQN network, the learning rate α2 of the Critic network and the learning rate β2 of the Actor network, and the discount factor γ used to balance current and future rewards; Finally, the initialization of experience storage and exploration-related components is completed, specifically setting the capacity D of the replay buffer.hi and D l The selection sizes M1 and M2 for mini-batch samples, and the OU noise Ξ(t) used to achieve the exploration function in the agent action selection process.

[0233] Step S102 focuses on the interaction process between the intelligent agent and the vehicle network environment: at the beginning of each optimization cycle, the system state s corresponding to the high-level DQN intelligent agent needs to be checked first. hi (k) Complete the initial configuration to ensure that the state parameters accurately reflect the current network conditions; then, select the high-level action a of the DQN agent according to the preset ∈-greedy decision-making strategy. hi (k) Through this action selection process, the service deployment strategy and service replacement strategy under the corresponding frame time scale are finally determined.

[0234] S103, First, the agent performs the operation according to the previously generated high-level action; then, the system judges whether the action meets the preset constraints, and the result of this judgment will be directly used for the temporary high-level reward index. The generation of the temporary high-level reward indicator; if it is determined that the action satisfies all constraints, then the temporary high-level reward indicator is set to [value]. In this case, the DDPG agent will be invoked in each time slot contained in the current frame to perform optimization work related to the underlying task processing; conversely, if the action does not meet the constraints, the temporary high-level reward metric will be set to a value. Furthermore, it does not initiate the optimization process for subsequent underlying task processing.

[0235] S104, First, the system state s of the underlying DDPG agent needs to be addressed. l (t) Complete the initialization operation to ensure that the state parameter accurately reflects the real-time operational basic information of the vehicular network in the current time slot; subsequently, the DDPG agent executes the corresponding action a according to the preset decision logic. l (t), through this series of actions, the VTUs task offloading rate optimization strategy suitable for the current time slot is finally generated, providing key strategy support for efficient task processing in subsequent time slots.

[0236] S105, First, the underlying agent performs the operation according to the previously generated specific action. After the action is completed, the system will feed back the corresponding reward value r to the agent based on the actual effect of task processing in the current time slot. l (t); Meanwhile, as actions are executed and the network's operating state dynamically changes, the current state s of the underlying system changes. l (t) will then complete the update transition and transform into the new state s corresponding to the next time slot. l(t+1) provides the latest state basis for subsequent underlying optimization decisions.

[0237] S106, First, the system experience acquired in the current time slot needs to be stored in the preset experience playback buffer D. l If the experience replay buffer D is found during the storage process; l If the capacity limit has been reached, the oldest experience data stored in the buffer is deleted according to the "first-in, first-out" principle to free up storage space for new experience data; then, data is retrieved from the experience set. In this context, a small set of low-level experience data of size M2 is randomly selected, where M2 represents the set of low-level experience data to be selected.

[0238] S107. First, based on previously stored experience data, the TD target and TD error of the underlying DDPG agent are calculated, providing a key basis for subsequent network parameter adjustments. Next, based on the calculated TD error, the parameters of the main underlying Actor network and the main underlying Critic network are updated and optimized respectively to ensure that the network can better adapt to the current vehicle network environment. Finally, after the main network parameters are updated, the parameters of the target Actor network and the target Critic network are updated simultaneously to maintain the coordination and stability of the two-layer network architecture.

[0239] S108. First, based on a small batch of experience data extracted from the experience pool, the parameters of the main Actor network and the main Critic network are adjusted and updated respectively to ensure that the network can optimize decision-making capabilities based on the latest experience. Then, based on the updated parameters of the main Actor network and the main Critic network, the parameters of the target Actor network and the target Critic network are updated accordingly to ensure the parameter coordination between the main network and the target network and provide stable support for subsequent optimization decisions.

[0240] S109, if the current frame reaches its last time slot, it is necessary to wait for the lower-level DDPG agent to complete its action and obtain the corresponding reward; only after this will the higher-level DQN agent receive the higher-level reward r corresponding to that frame. hi (k), and at the same time, the current state s of the higher-level system hi (k) will also complete the transition and update to the new state s corresponding to the next frame. hi (k+1).

[0241] S110, First, the currently acquired high-level experience needs to be stored in the experience replay buffer D. hi If the buffer D is found during storage; hiIf the capacity limit has been reached, the longest-stored experience data in the buffer is removed according to the "first-in, first-out" principle to make room for new experience. Then, a small batch of experience of size M1 is extracted from the experience replay buffer, and the state and action in it are used as input data and fed into the DQN main network. Through the calculation and processing of the main network, the corresponding estimated Q value is finally obtained.

[0242] Step S111 revolves around network optimization of the high-level DQN agent: First, based on previously acquired empirical data, the TD target and TD error of the high-level DQN agent are calculated, providing a core basis for network parameter adjustment; then, based on the calculated TD error, the parameters of the main DQN network and the target DQN network are updated respectively; finally, using the updated main Q network parameters as a benchmark, the target Q network parameters are adjusted synchronously to ensure the coordinated adaptation of the two Q network parameters.

[0243] S112, by repeatedly going through the previous optimization iteration process, selects the actions that can achieve the best effect based on the Q value of each action under different system states during training; in this way, the task failure rate of the vehicle network is gradually reduced. After multiple iterations of optimization, the optimal optimization strategy for key aspects such as service deployment and replacement, VTU task offloading rate and access control is finally obtained.

[0244] S102 initializes the system state s of the high-level DQN agent at the beginning of each cycle. hi (k); where the state is represented as:

[0245] s hi (k)={r(k),N(k-1)}

[0246] in, The task request for the k-th frame representing the VTUs; Indicates whether service h exists on the RSU at the end of frame k-1 or the beginning of frame k; uses a policy to select the high-level action a of the DQN agent. hi (k) yields the service deployment and replacement strategy for the frame, with the action represented as:

[0247]

[0248] in Service placement decision; It is a service replacement decision.

[0249] The formula for calculating the temporary reward in S103 is as follows:

[0250]

[0251] Among them, C1: Service deployment decision δh (k) and replacement decision All are binary variables (taking only 0 or 1), meaning that for a given service, each frame initially only allows selection of "deploy" or "not deploy" or "delete" or "not delete," avoiding ambiguity in decision-making; C2: only when there is no service n on the RSU. h The deployment of this service can only be executed when (k-1) = 0, to prevent the duplicate deployment of the same service and the waste of storage resources; C3: Service n can only be deployed on the RSU if it already exists. h The deletion action for this service can only be performed when (k-1) = 1, to avoid invalid operations caused by deleting a non-existent service; C4: For the same service h, deployment and deletion actions cannot be performed simultaneously within the same frame. To avoid logical contradictions; C5: Total storage space for all deployed services on the RSU. Not exceeding its storage capacity limit Π U This ensures compliance with hardware resource limitations. High-level temporary rewards are used to determine whether low-level optimizations are necessary. If... This indicates that the service placement and replacement operations meet the constraints (C1)-(C5), and the strategy is feasible. At this point, the DDPG agent will be invoked to execute the underlying optimization process; if This means that the service placement and replacement strategy does not meet the constraints and is infeasible, making underlying optimization meaningless. Since the high-level solution and the low-level solution together constitute the complete solution to the problem, when the high-level strategy itself is infeasible, even if the low-level optimization yields a result, the overall solution will still fail, so there is no need to perform low-level optimization.

[0252] In S104, the state s of the underlying DDPG agent is initialized. l (t), where the state is represented as:

[0253]

[0254] Where N(k) = {n h (k)},n h (k)∈{0,1}, For service deployment conditions, it indicates whether service h has been deployed on the RSU at the start of the time slot; u(t-1)={u i (t-1)} is the position of VTUi at the end of time slot t-1; η(t-1)={η i (t-1)} is the direction of movement of VTUi; It is the distance between the VTUi and the RSU; It is the channel gain; R(t) = {R i (t)} is the transmission rate; I 1 (t-1,k) and I 2(t-1,k) are the sets of VTUs with allocated and unallocated time slots, respectively; A(t)={a i (t)} represents the access control policy. We denote the state space of the low-level DDPG agent as S. l It is the set of all lower-level states.

[0255] The DDPG agent executes an action, which is represented as follows:

[0256]

[0257] in For VTUs' task offload rate, This represents the local processing ratio. (The proportion of data unloaded to RSU). The action space of the DDPG agent is represented as... It is the set of all feasible solutions to the variables related to task processing.

[0258] In S105, the agent executes the generated action, receives a reward, and the underlying state transitions to the next state; the reward calculation formula is as follows:

[0259]

[0260] C6, This represents the local task offloading rate of the i-th device in the vehicle-to-everything (VTUs) under time slot t. The lower limit is 0, meaning the proportion of tasks processed locally by this device cannot be negative, ensuring the reasonableness of the proportion and preventing negative task allocation ratios. (C7) and in This can be viewed as an unloading rate indicator related to the proportion of local task processing. This refers to the percentage of RSU edge unloading tasks. The maximum local task processing ratio (corresponding to the uninstallation rate metric) is limited to 1, meaning that a maximum of all tasks can be processed locally; while This indicates that for the task of the i-th device in time slot t, it can either be processed locally or offloaded to the RSU edge server for processing. The total task allocation ratio of both is 100%, covering all destinations of task processing and ensuring the integrity of the task allocation logic.

[0261] In step S107, for the network computation stage of the underlying DDPG agent, the main Actor network will perform prediction operations based on the current state of the underlying system, and the prediction result can be expressed as:

[0262] μ(s l (t);θ)

[0263] Furthermore, the basis for the target actor network's performance prediction is:

[0264] μ′(s l (t+1); θ - )

[0265] Major commentators' online predictions:

[0266] q(s l (t),a l (t);w)

[0267] Meanwhile, the target evaluator network conducts predictions in the following manner:

[0268] q′(s l (t+1),μ′(s l (t+1); θ - );w - )

[0269] The TD target is calculated according to the following formula:

[0270] y l (t)=r l (t)+γq′(s l (t+1),μ′(s l (t+1); θ - );w - )

[0271] The TD error is obtained by calculation based on the following formula:

[0272] δ l (t)=q(s l (t),a l (t);w)-y l (t)

[0273] The parameters of the critic network must be updated according to the following formula:

[0274]

[0275] The parameters of the actor network must be updated according to the following formula:

[0276]

[0277] Furthermore, the update formula for updating the target Actor and Critic network parameters based on the updated main Actor and Critic network parameters in S108 is as follows:

[0278] θ - ←ζθ+(1-ζ)θ -

[0279] w -←ζw+(1-ζ)w -

[0280] Here, the symbol θ refers to the parameters of the current policy network. - These correspond to the parameters of the target policy network (i.e., the target actor network); while w represents the parameters of the current value network. - This represents the parameters of the target value network. Additionally, the parameter ζ∈[0,1] is a hyperparameter that needs to be manually configured; its specific value can be adjusted according to the actual training scenario and convergence requirements.

[0281] The formula for calculating the S109 mid-to-high-level rewards is as follows:

[0282] r hi (k)=r fail (T,k)

[0283] Where r fail (T,k) is the system's task failure rate during the k-th frame, which is our objective function.

[0284] Furthermore, in step S110, a small batch of experience data needs to be extracted from the experience replay buffer. Then, the states and actions from the extracted experience are used as input to the DQN main network for calculation, ultimately obtaining the estimated Q-value used to evaluate the policy value. Its expression is:

[0285] Q(s hi (k),a hi (k);ω)

[0286] Where ω is the weight of the main Q-network;

[0287] s hi Using (k+1) as input to the target network, we obtain:

[0288]

[0289] It should be noted that ω - This symbol represents the weight parameters corresponding to the target Q-network.

[0290] In step S111, the generation of the time difference (TD) target requires calculation according to a specific formula:

[0291]

[0292] And based on the calculated TD error:

[0293] δ hi (k)=Q(s hi (k),ahi (k);ω)-y hi (k)

[0294] Update the main DQN network:

[0295]

[0296] It should be noted that α1 in the formula represents the learning rate; and the parameter updates of the target DQN network must strictly follow the following formula:

[0297] ω - ←ζω+(1-ζ)ω -

[0298] Here, ζ∈[0,1] is a hyperparameter that needs to be manually adjusted.

[0299] To elaborate on the method for minimizing task processing failure rate in vehicular networks based on hierarchical DRL-enabled MEC and TDMA, this invention provides two specific application embodiments, including key details of the implementation scheme.

[0300] Application Example 1: Optimization of Highway Vehicle-to-Vehicle Cooperative Network

[0301] Vehicle-mounted networks (V2V) in highway scenarios face multiple challenges, including high mobility, long-distance coverage, and low-latency requirements. Vehicles (such as freight convoys and passenger vehicles) need to perform tasks such as platooning coordination control, traffic condition warning data synchronization, and emergency fault information transmission in real time. However, high-speed driving can cause channel gain fluctuations and rapid changes in distance from roadside units (RSUs). Traditional V2V network optimization solutions often suffer from data transmission packet loss and excessive latency in critical task processing due to resource scheduling lags and rigid task offloading strategies. For example, if platooning coordination commands for freight convoys cannot be synchronized in time due to network congestion, it may lead to rear-end collisions; if sudden traffic condition warnings are delayed due to insufficient RSU service cache, they will lose their risk-avoidance value. This invention presents a multi-time-scale hierarchical DRL-based MEC and TDMA V2V network optimization method. This method can accurately address the pain points of highway scenarios by using a high-level DQN agent to plan long-term service deployment and a low-level DDPG agent to adapt real-time resource scheduling. The implementation process is detailed below.

[0302] 1) System Initialization

[0303] In the highway vehicle-to-everything (V2X) cooperative system, a master Q network, a master actor, and a critic network are deployed, corresponding to the high-level DQN agent and the low-level DDPG agent, respectively. Network parameters such as learning rate, discount factor, and experience replay buffer size are initialized. Simultaneously, roadside units (RSUs) are deployed at intervals along the highway as mobile edge computing nodes, connecting to the vehicle-mounted terminals (VTUs) of moving vehicles to support low-latency tasks such as vehicle platooning data synchronization and real-time traffic condition warnings.

[0304] 2) Interaction between hierarchical DRL agents and the environment

[0305] The high-level DQN agent is responsible for long-term service deployment and replacement strategies. It generates actions based on the task request patterns of highway VTUs (such as long-haul freight vehicles requesting "platooning coordination control services" more frequently) and RSU storage status. For example, it prioritizes the deployment of "vehicle emergency response services" in RSUs near service areas and removes low-frequency services to free up storage resources.

[0306] The underlying DDPG agent is responsible for adjusting the task offloading rate and access control in the short term. It generates specific actions based on real-time data such as VTU speed, distance from the RSU, and channel gain. For example, when a vehicle is rapidly approaching the RSU, the task offloading ratio is increased; when the channel quality deteriorates, the local processing ratio is increased, and communication resources are allocated to the vehicle with the nearest unallocated time slot to the RSU according to the TDMA mechanism.

[0307] 3) Action execution and reward acquisition

[0308] After executing actions generated by high-level and low-level agents, the system obtains immediate rewards based on changes in the MEC vehicular network performance. The reward signal is calculated based on metrics such as task failure rate (e.g., whether formation control command processing latency exceeds a threshold), RSU computational resource utilization, and data transmission rate. The system state shifts with changes in VTU position (eastward or westward travel), channel gain fluctuations, and new task requests.

[0309] 4) Optimization and Updates

[0310] Based on collected experience data such as states, actions, rewards, and next states, the high-level and low-level agents update their Q-networks, Actor networks, and Critic networks, respectively. Through hierarchical DRL algorithm training, the agents gradually learn optimization strategies adapted to highway scenarios, improving task processing efficiency while reducing energy consumption caused by frequent task unloading by the vehicle.

[0311] 5) Performance evaluation and optimization

[0312] The performance of the hierarchical DRL algorithm was evaluated through simulation experiments. The task failure rate, average processing latency, and RSU resource utilization under different driving speeds and traffic volumes were analyzed and compared with traditional methods such as fixed offloading strategies. Based on the evaluation results, the algorithm parameters were adjusted to further improve the stability and efficiency of the highway vehicle-to-everything (V2X) cooperative network in high-mobility scenarios.

[0313] Application Example 2: Vehicle-mounted Emergency Rescue Communication Support in Disaster Areas

[0314] In disaster relief scenarios, vehicular communication networks are crucial for ensuring efficient rescue operations. Rescue vehicles need to transmit disaster images in real time and receive dispatch instructions from the command center, while vehicles stranded in disaster areas need timely access to evacuation routes and safety warnings. However, disasters often involve infrastructure damage (such as communication base station interruptions), poor channel conditions (such as signal blockage caused by building collapses), and sudden and urgent mission requirements. Traditional vehicular communication solutions, lacking dynamic adaptability, are prone to signal interruptions, data transmission delays, and failures in handling critical rescue tasks, severely hindering rescue efficiency. This invention presents a multi-time-scale hierarchical DRL-based MEC and TDMA vehicular network optimization method. This method can provide stable support for vehicular emergency rescue communication in disaster areas through rapid deployment of emergency RSUs and dynamic adjustment of service deployment and task offloading strategies. The implementation process is described in detail below.

[0315] 1) System Initialization

[0316] In disaster-stricken areas, a primary DQN network and a DDPG network are urgently deployed as high-level and low-level intelligent agents to initialize relevant network parameters. Simultaneously, emergency vehicle-mounted communication equipment is configured, including MEC servers mounted on RSUs and communication terminals on rescue vehicles, covering key rescue routes in the disaster area.

[0317] 2) Interaction between intelligent agents and the environment

[0318] Each RSU and VTU generates actions based on the real-time relief needs and channel conditions in the disaster area, such as adjusting service deployment, task offloading rate, or time slot allocation strategy. Environmental noise and uncertainties are taken into account when generating actions, and emergency communication strategies are optimized through exploration.

[0319] 3) Action execution and reward acquisition

[0320] After an action is executed, immediate rewards are awarded based on key indicators such as the stability of the vehicle communication link, task processing latency, and failure rate. Simultaneously, the system status is updated to reflect vehicle movement trajectories in the disaster area, changes in RSU load, and new rescue mission requests, ensuring dynamic matching of the status with the emergency environment.

[0321] 4) Network updates and optimizations

[0322] By utilizing collected state, action, and reward data, the high-level DQN network and the low-level DDPG network in the hierarchical DRL algorithm are collaboratively optimized. The high-level DQN is responsible for long-term service deployment strategies, while the low-level DDPG is responsible for short-term resource allocation. Through iterative training, the algorithm gradually improves the stability and efficiency of the emergency communication system, ensuring real-time processing of rescue missions.

[0323] 5) Vehicle-mounted emergency rescue communication support

[0324] The hierarchical DRL algorithm of this invention enables intelligent management and optimization of vehicle-mounted emergency communication systems in disaster areas. The algorithm focuses on reducing task failure rates and shortening processing latency, ensuring that rescue vehicles transmit disaster data and receive dispatch instructions in real time. This not only improves the coordination efficiency of rescue operations but also reduces rescue delays caused by communication latency. Through the above steps, this invention provides an efficient and intelligent solution for vehicle-mounted emergency rescue communication support in disaster areas, supporting the smooth implementation of rescue operations.

[0325] In both embodiments, the hierarchical DRL-based MEC vehicular network optimization method provides an efficient and reliable solution, applicable to different scenarios such as highways and disaster emergency response, demonstrating its broad application potential and technical advantages.

[0326] To comprehensively evaluate the performance of this invention, we compared it with several benchmark algorithms. Performance comparisons under identical conditions clearly demonstrate the advantages of this invention in terms of task failure rate and resource utilization, as well as areas requiring further optimization.

[0327] 1. RSU Processing: In this algorithm, the offload rate for all VTUs is a fixed percentage, while the offload rate for RSU is set to a specific value. This means that VTUs will offload some tasks to RSU processing, while the remaining tasks will be processed locally.

[0328] 2. Core Network Processing: In this algorithm, the offload rate of all VTUs is a fixed percentage, while the offload rate of the RSU is set to a specific value. This means that all VTUs offload some or all of their input data to the RSU, and the RSU then offloads the received data to the core network for further processing.

[0329] 3. DDPG: This algorithm relies on random decision-making for service deployment and replacement. DDPG plays a key role in jointly optimizing RSU deployment, task offloading rate and access control within each time slot.

[0330] 4. Random Access Control: Due to the use of TDMA, each time slot can only be used for one VTU. In this algorithm, access control is determined randomly, meaning each time slot is allocated to a random VTU for task offloading.

[0331] 5. Random RSU Deployment: This algorithm involves randomly determining RSU deployment decisions. Therefore, the communication coverage and service caching strategy of the RSU are both randomly set.

[0332] 6. This invention: It represents the proposed multi-timescale hierarchical DRL algorithm.

[0333] exist Figure 3 In the diagram, we demonstrate the convergence performance of the proposed DQN-DDPG algorithm compared to the baseline algorithm under default parameter settings. The convergence performance is evaluated based on three performance metrics: convergence speed, convergence stability, and task failure rate. Figure 5 The results show that the present invention has a fast convergence speed, almost reaching convergence within 10 frames, and can quickly reduce to a low task failure rate. Meanwhile, the proposed algorithm excels in task failure rate and stability: on the one hand, it achieves the lowest task failure rate; on the other hand, after 10 frames, the task failure rate of the present invention remains stable, while other algorithms exhibit higher task failure rates, highlighting the effectiveness of hierarchical optimization.

[0334] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0335] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access, characterized in that, A multi-timescale hierarchical deep reinforcement learning approach is used to optimize the mobile edge computing-driven vehicular network. By adopting a master-slave dual-layer architecture, the upper-layer deep Q network is mainly responsible for frame-level service deployment and replacement strategy decisions, while the lower-layer deep deterministic policy gradient network performs fine-tuning of task offloading rate optimization and access control action execution within each time slot. The specific steps include the following: S101, System Initialization Specifically, the following parameters and configurations need to be set: First, the network parameters need to be initialized, including the parameters ω of the main Q network and the target Q network. - The parameters θ of the main Actor network and the parameters w of the main Critic network are set, and the initial parameters of the target Actor network are set to θ. - =θ, the initial parameters of the target Critic network are set to w - =w; Secondly, the parameters related to iteration and time scale need to be set, including the total number of iterations L, the number of iterations in each iteration (i.e., the number of frames K), and the number of time slots T within each frame; In addition, the key parameters required for the algorithm to run need to be initialized, namely the learning rate α1 of the DQN network, the learning rate α2 of the Critic network and the learning rate β2 of the Actor network, and the discount factor γ used to balance current and future rewards; Finally, the initialization of experience storage and exploration-related components is completed, specifically setting the capacity D of the replay buffer. hi and D l The selection sizes M1 and M2 for small batches of samples, and the OU noise Ξ(t) used to realize the exploration function in the agent action selection process; S102 focuses on the interaction process between intelligent agents and the in-vehicle network environment: At the beginning of each optimization cycle, the system state s corresponding to the high-level DQN agent must first be determined. hi (k) Complete the initial configuration to ensure that the state parameters accurately reflect the current network conditions; then, select the high-level action a of the DQN agent according to the preset ∈-greedy decision-making strategy. hi (k) Through this action selection process, the service deployment strategy and service replacement strategy under the corresponding frame time scale are finally determined; S103, the system determines whether the actions of the high-level units meet the preset constraints; First, the agent executes the operation according to the previously generated high-level action; then, the system determines whether the action meets the preset constraints, and this determination result is directly used for the temporary high-level reward indicator. The generation of the temporary high-level reward indicator; if it is determined that the action satisfies all constraints, then the temporary high-level reward indicator is set to [value]. In this case, the DDPG agent will be invoked in each time slot contained in the current frame to perform optimization work related to the underlying task processing; conversely, if the action does not meet the constraints, the temporary high-level reward metric will be set to a value. And it does not initiate the optimization process for subsequent underlying task processing; S104, Generate a VTU task offload rate optimization strategy suitable for the current time slot; First, the system state s of the underlying DDPG agent needs to be considered. l (t) Complete the initialization operation to ensure that the state parameter accurately reflects the real-time operational basic information of the vehicular network in the current time slot; subsequently, the DDPG agent executes the corresponding action a according to the preset decision logic. l (t), through this series of actions, the VTUs task offloading rate optimization strategy suitable for the current time slot is finally generated, providing key strategy support for efficient task processing in subsequent time slots; S105 is the reward value corresponding to the feedback from the agent; First, the underlying agent performs the operation according to the previously generated specific actions. After the action is completed, the system will provide the agent with a corresponding reward value r based on the actual effect of the task processing in the current time slot. l (t); Meanwhile, as actions are executed and the network's operating state dynamically changes, the current state s of the underlying system changes. l (t) will then complete the update transition and transform into the new state s corresponding to the next time slot. l (t+1) provides the latest state basis for subsequent underlying optimization decisions; S106, Extract underlying experience data; First, the system experience acquired in the current time slot needs to be stored in the preset experience playback buffer D. l If the experience replay buffer D is found during the storage process; l If the capacity limit has been reached, the oldest experience data stored in the buffer is deleted according to the "first-in, first-out" principle to free up storage space for new experience data; then, data is retrieved from the experience set. In this context, a small set of low-level experience data of size M2 is randomly selected, where M2 represents the set of low-level experience data that is selected. S107, update the parameters of the target Actor network and the target Critic network; First, based on previously stored experience data, the TD target and TD error of the underlying DDPG agent are calculated, providing a key basis for subsequent network parameter adjustments. Next, based on the calculated TD error, the parameters of the main underlying Actor network and the main underlying Critic network are updated and optimized to ensure that the network can better adapt to the current in-vehicle network environment. Finally, after the main network parameters are updated, the parameters of the target Actor network and the target Critic network are updated simultaneously to maintain the synergy and stability of the two-layer network architecture. S108, update the parameters of the target Actor network and the target Critic network accordingly; First, based on a small batch of experience data extracted from the experience pool, the parameters of the main Actor network and the main Critic network are adjusted and updated respectively to ensure that the network can optimize decision-making capabilities based on the latest experience. Then, based on the updated parameters of the main Actor network and the main Critic network, the parameters of the target Actor network and the target Critic network are updated accordingly to ensure the parameter synergy between the main network and the target network and provide stable support for subsequent optimization decisions. S109, Transition Conversion; If the current frame reaches its last time slot, it must first wait for the lower-level DDPG agent to complete its action and obtain the corresponding reward; only after this will the higher-level DQN agent receive the higher-level reward r corresponding to that frame. hi (k), and at the same time, the current state s of the higher-level system hi (k) will also complete the transition and update to the new state s corresponding to the next frame. hi (k+1); S110, obtain the estimated Q value; First, the currently acquired high-level experience needs to be stored in the experience replay buffer D. hi If the buffer D is found during storage; hi If the capacity limit has been reached, the longest-stored experience data in the buffer is removed according to the "first-in, first-out" principle to make room for new experience. Then, a small batch of experience of size M1 is extracted from the experience replay buffer, and the state and action in it are fed into the DQN main network as input data. Through the calculation and processing of the main network, the corresponding estimated Q value is finally obtained. S111 focuses on network optimization for high-level DQN agents: First, based on previously acquired empirical data, the TD target and TD error of the high-level DQN agent are calculated, providing a core basis for network parameter adjustment. Next, based on the calculated TD error, the parameters of the main DQN network and the target DQN network are updated respectively. Finally, using the updated main Q network parameters as a benchmark, the target Q network parameters are adjusted synchronously to ensure the coordinated adaptation of the two Q network parameters. S112, by repeatedly going through the previous optimization iteration process, selects the actions that can achieve the best effect based on the Q value of each action under different system states during training; in this way, the task failure rate of the vehicle network is gradually reduced. After multiple iterations of optimization, the optimal optimization strategy for key aspects such as service deployment and replacement, VTU task offloading rate and access control is finally obtained.

2. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, At the beginning of each cycle, S102 initializes the system state s of the high-level DQN agent. hi (k); where the state is represented as: s hi (k)={r(k),N(k-1)} in, The task request for the k-th frame representing the VTUs; Indicates whether service h exists on the RSU at the end of frame k-1 or the beginning of frame k; uses a policy to select high-level actions of the DQN agent. ahi(k) The service deployment and replacement strategy for the obtained frame is represented as follows: in Service placement decision; It is a service replacement decision.

3. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, The formula for calculating the temporary reward in S103 is as follows: Among them, C1: Service deployment decision δ h (k) and replacement decision All are binary variables (taking only 0 or 1), meaning that for a given service, each frame initially only allows selection of "deploy" "not deploy" or "delete" "not delete," avoiding fuzzy decision-making; C2: only when there is no service n on the RSU. h The deployment of this service can only be executed when (k-1) = 0, to prevent the duplicate deployment of the same service and the waste of storage resources; C3: Service n can only be deployed on the RSU if it already exists. h The deletion action for this service can only be performed when (k-1) = 1, to avoid invalid operations caused by deleting a non-existent service; C4: For the same service h, deployment and deletion actions cannot be performed simultaneously within the same frame. To avoid logical contradictions; C5: Total storage space for all deployed services on the RSU. Not exceeding its storage capacity limit Π U Ensure compliance with hardware resource limitations; temporary high-level rewards are used to determine whether underlying optimizations are necessary; if This indicates that the service placement and replacement operations meet the constraints (C1)-(C5), and the strategy is feasible. At this point, the DDPG agent will be invoked to execute the underlying optimization process; if This means that the service placement and replacement strategy does not meet the constraints and is not feasible, so carrying out underlying optimization is meaningless. Since the high-level solution and the low-level solution together constitute the complete solution to the problem, when the high-level strategy itself is not feasible, even if the low-level optimization yields a result, the overall solution will not be valid, so there is no need to carry out low-level optimization.

4. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, In S104, the state s of the underlying DDPG agent is initialized. l (t), where the state is represented as: in For service deployment conditions, it indicates whether service h has been deployed on the RSU at the start of the time slot; u(t-1)={u i (t-1)} is the position of VTUi at the end of time slot t-1; η(t-1)={η i (t-1)} is the direction of movement of VTUi; It is the distance between the VTUi and the RSU; It is the channel gain; R(t) = {R i (t)} is the transmission rate; I 1 (t-1,k) and I 2 (t-1,k) are the sets of VTUs with allocated and unallocated time slots, respectively; A(t)={a i (t)} is the access control policy; the state space of the low-level DDPG agent is denoted as S. l It is the set of all lower-level states; The DDPG agent executes an action, which is represented as follows: in For VTUs' task offload rate, This represents the local processing ratio. (The proportion of data unloaded to RSU); the action space of the DDPG agent is represented as... It is the set of all feasible solutions to the variables related to task processing.

5. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, In step S105, the agent executes the generated action, receives a reward, and the underlying state transitions to the next state; the reward calculation formula is as follows: C6, This represents the local task offloading rate of the i-th device in the vehicle-to-everything (VTUs) under time slot t. The lower limit is 0, meaning the proportion of tasks processed locally by this device cannot be negative, ensuring the reasonableness of the proportion and preventing negative task allocation ratios. (C7) and in This can be viewed as an unloading rate indicator related to the proportion of local task processing. For indicators related to the percentage of RSU edge offloading tasks; The maximum local task processing ratio (corresponding to the uninstallation rate metric) is limited to 1, meaning that a maximum of all tasks can be processed locally; while This indicates that for the task of the i-th device in time slot t, it can either be processed locally or offloaded to the RSU edge server for processing. The total task allocation ratio of both is 100%, covering all destinations of task processing and ensuring the integrity of the task allocation logic.

6. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, The main actor network predictions in S107 are as follows: μ(s l (t);θ) Furthermore, the basis for the target actor network's performance prediction is: μ′(s l (t+1);θ - ) Major commentators' online predictions: q(s l (t),a l (t);w) Meanwhile, the target evaluator network conducts predictions in the following manner: q′(s l (t+1),μ′(s l (t+1);θ - );w - ) The TD target is calculated according to the following formula: y l (t)=r l (t)+γq′(s l (t+1),μ′(s l (t+1);θ - );w - ) The TD error is obtained by calculation based on the following formula: δ l (t)=q(s l (t),a l (t);w)-y l (t) The parameters of the critic network must be updated according to the following formula: The parameters of the actor network must be updated according to the following formula:

7. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, The update formulas for updating the target Actor and Critic network parameters based on the updated main Actor and Critic network parameters in step S108 are as follows: i - ←ζθ+(1-ζ)θ - w - ←ζw+(1-ζ)w - Here, the symbol θ refers to the parameters of the current policy network. - These correspond to the parameters of the target policy network (i.e., the target actor network); while w represents the parameters of the current value network. - Then, it represents the parameters of the target value network; in addition, the parameter ζ∈[0,1] is a hyperparameter that needs to be manually configured, and its specific value can be adjusted according to the actual training scenario and convergence requirements.

8. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, The formula for calculating the senior management reward in S109 is as follows: r hi (k)=r fail (T,k) Where r fail (T,k) is the system's task failure rate during the k-th frame, which is our objective function.

9. The vehicle-to-everything (V2X) optimization method based on edge computing and time-division multiple access as described in claim 1, characterized in that, In step S110, a small batch of experience is sampled from the experience replay buffer, and the state and action are used as inputs to the DQN main network to obtain the estimated Q value. Q(s hi (k),a hi (k);ω) Where ω is the weight of the main Q-network; s hi Using (k+1) as input to the target network, we obtain: Where, ω - These are the weights of the target Q-network; In S111, the Temporal Difference (TD) target is calculated according to the following formula: And based on the calculated TD error: δ hi (k)=Q(s hi (k),a hi (k);ω)-y hi (k) Update the main DQN network: It should be noted that α1 in the formula represents the learning rate; and the parameter updates of the target DQN network must strictly follow the following formula: oh - ←live+(1-g)w - Here, ζ∈[0,1] is a hyperparameter that needs to be manually adjusted.

10. The vehicle-to-everything (V2X) optimization system based on edge computing and time-division multiple access (TDMA) as described in any one of claims 1 to 9, characterized in that, The vehicular network optimization system based on edge computing and time-division multiple access shown includes: One of the functions of the system initialization module is to initialize the parameters related to the deep Q-network algorithm, including the parameters ω of the main Q-network and the target Q-network. - ; This module is also responsible for initializing the parameters of the deep deterministic policy gradient algorithm, specifically the parameters θ of the main Actor network and w of the main Critic network, while setting the initial parameters of the target Actor network to θ. - =θ, the initial parameters of the target Critic network are set to w - =w; This module is also responsible for setting parameters, specifically including setting the total number of iterations L, the number of iterations included in each iteration (i.e. the number of frames K), and the number of time slots T divided within each frame; The initialization operations that need to be completed also include: setting the learning rate α1 of the DQN network, and the corresponding learning rates α2 and β2 of the Critic and Actor networks; determining the discount factor γ; and initializing the capacity D of the experience replay buffer. hi and D l The sizes M1 and M2 of the mini-batch are clearly defined. In addition, the noise parameters used for action exploration need to be initialized. The configuration module is used to set application-specific parameters, which include various task-related parameters such as the number of vehicle terminals (I). The core functions of the intelligent agent module are divided into two parts: First, at the start of each cycle, it generates corresponding actions based on the current state of the high-level network, and this process supports the use of a greedy strategy for the generated actions; Second, also at the start of each cycle, it generates actions based on the current state of the low-level network, and this step has the ability to add exploratory noise to the generated actions. The action execution module has two functions: one is to perform the placement and replacement of services; the other is to implement optimization strategies for VTUs, specifically covering task offload rate control, access control, and other content. The reward acquisition module has a dual core function: on the one hand, it is responsible for executing the corresponding actions and calculating the temporary rewards. The design of this module must be based on whether the system constraints are met, and it also undertakes the function of determining whether to start the underlying optimization. On the other hand, this module also needs to calculate the rewards by combining the actions already executed and the current system state. The specific calculation dimensions include two levels: one is the task failure rate of all devices in the system in the current time slot, and the other is the task failure rate of these devices at the end of the frame period. State transition module: It undertakes the task of transitioning the state of the underlying system and is responsible for transitioning the current state of the system to the state of the next moment. Experience replay module: The core function is to store the experience tuples generated by each interaction. These tuples contain the current state of the underlying system, the action performed, the corresponding reward, and the next state after the transition. The data sampling module extracts mini-batch sample data from the experience data stored in the experience playback module to provide data support for subsequent network learning. Network update module: Based on the sample information obtained by the data sampling module, this module needs to update the main Q network, main Actor network, and main Critic network. This module has a built-in parameter optimization unit, which adjusts the parameters of each network through gradient descent. Parameter update module: It is mainly used to update the parameters of the target Q network, target Actor network and target Critic network with reference to the parameters of the main network. This module adopts a soft update strategy, and its parameter synchronization unit can ensure that the parameters of the target network are always the weighted average of the parameters of the corresponding main network. The network update module and the parameter update module work together to update the parameters of the target Actor network and the target Critic network by referring to the main network parameters. This process also adopts a soft update strategy, and the target network parameters need to be synchronized in the form of a weighted average of the main network parameters. This function is completed by the parameter synchronization unit in the module.

Images