Computing offloading and cache joint optimization method and device based on improved A3C algorithm

Abstract

The application belongs to the technical field of vehicle-mounted edge computing, and particularly relates to a computing offloading and cache joint optimization method and device based on an improved A3C algorithm, the method comprising: being applied to a target network, the target network comprising at least a plurality of roadside units provided with MEC servers, a plurality of terminal vehicles and a cloud server, establishing a vehicle movement model, a time delay energy consumption model of task processing and a cache model; establishing a task priority and task preemption model; according to the above models, a task offloading and cache allocation joint optimization model with constraint conditions is established, with the minimum total average time delay in the system as the target; a deep reinforcement learning framework based on an improved Actor-Critic is constructed to obtain a task offloading decision, a cache allocation decision and a task preemption indication; the application aims to minimize the total average time delay on the basis of meeting the energy consumption time delay constraint.

Description

Technical Field

[0001] This invention belongs to the field of vehicle edge computing technology, specifically relating to a method and apparatus for joint optimization of computation offloading and caching based on an improved A3C algorithm. Background Technology

[0002] With the widespread adoption of 5G and future 6G technologies, various new in-vehicle services are emerging, such as intelligent navigation, voice assistants, and autonomous driving. While these emerging mobile services greatly enrich people's lives, they also consume significant computing and storage resources from mobile devices. In-vehicle edge computing (VEC) has emerged as an efficient solution, allowing vehicles to offload computationally intensive tasks to nearby Remote Units (RSUs) equipped with edge servers, thereby reducing latency, alleviating vehicle load, and improving efficiency. Simultaneously, caching technology is widely used in edge computing to store frequently accessed data, further reducing network transmission latency and improving user experience. However, due to limited resources and unrestricted task execution order, tasks involving large amounts of data transmission can lead to RSU overload, low task completion rates, and significant system latency.

[0003] Most research on vehicle-mounted edge computing only considers task processing for local vehicles, leading to resource waste. For urgent tasks requiring immediate execution, poor queueing rules also cause task failures, significantly reducing system efficiency. Utilizing the computing resources of other moving vehicles on the road and developing reasonable task prioritization and preemption schemes can effectively solve these problems. However, traditional algorithms struggle to adapt to complex, dynamic, and high-dimensional task offloading and caching optimization problems. For example, classic heuristic algorithms, game theory models, and optimization methods can theoretically provide optimal solutions, but they often assume a static network environment or that task requirements are known and unchanging. This makes them ill-suited to sudden changes, resulting in fixed and inflexible decision-making strategies. Deep Reinforcement Learning (DRL), with its adaptive capabilities, long-term reward optimization, lack of need for precise models, and real-time online learning, demonstrates significant advantages in task offloading and caching optimization. DRL, through agent-environment interaction, continuously optimizes strategies through trial and error without a defined model, making it particularly suitable for the dynamic and complex task offloading and caching optimization scenarios in MEC. DRL can optimize decisions in real time under these dynamic changes, avoiding the inefficiency or misjudgment caused by static assumptions in traditional methods. For example, Chinese patent CN119300089A discloses a method and apparatus for mobile edge computing offloading and service caching based on deep reinforcement learning. This application transforms the task offloading and service caching optimization problem into a Markov decision process (MDP) and uses deep reinforcement learning (DRL) technology to explore and optimize the optimal strategy. DRL effectively addresses this challenge by using highly variable communication states as state inputs and outputting computation offloading and service caching strategies through a deep neural network (DNN), thus reducing computational complexity. However, this approach suffers from insufficient exploration, poor strategy stability, and problems adapting to dynamic environments, making it unsuitable for the actual situation of joint optimization of computation offloading and caching. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes a computational offloading and caching joint optimization method and apparatus based on an improved A3C algorithm. It constructs V2V and V2R models using multiple roadside units equipped with MEC servers, multiple terminal vehicles, and cloud servers. These V2V and V2R models, serving as target networks, can reduce server load and minimize resource waste. By introducing the concept of maximum entropy, an improved Soft-A3C algorithm, namely the SA3C algorithm, is obtained. Soft-A3C optimizes the policy by introducing entropy regularization, making the agent focus more on exploration during training, thereby avoiding over-reliance on local optima and improving policy stability and convergence speed. This soft policy method can better cope with complex and dynamic environments, improving learning efficiency and enhancing stability. This approach can be used to solve optimization problems, minimizing the total average system latency.

[0005] One aspect of the present invention provides a joint optimization method for computation offloading and caching based on an improved A3C algorithm, applied to a target network, wherein the target network includes at least multiple roadside units equipped with MEC servers, multiple terminal vehicles, and a cloud server, and the method includes:

[0006] Based on the processing delay of each terminal vehicle in processing the current task locally in the current time slot and the transmission delay of each terminal vehicle in unloading the current task to the corresponding other terminal vehicle in the current time slot, the total delay of all terminal vehicles in the current time slot is obtained.

[0007] Based on the transmission delay of each roadside unit receiving the current task from the terminal vehicle in the current time slot and the processing delay of each roadside unit during the preemption process in the current time slot, the total delay of all roadside units in the current time slot is obtained.

[0008] Based on the processing delay of each roadside unit and / or each terminal vehicle for the previous task, the waiting delay of the current task in the current time slot is obtained;

[0009] Based on the transmission rate at which the cloud server sends the current task to the roadside unit and the task popularity of the current task, the cache latency of the current task in the current time slot is obtained.

[0010] Based on the total latency of all tasks, and with the goal of minimizing the average total latency, a joint optimization problem model for all terminal vehicles and all roadside units is determined.

[0011] The improved A3C algorithm is used to solve the joint optimization problem model, and the preemption indication, unloading strategy and caching strategy of the computing tasks to be processed in each roadside unit and each terminal vehicle are obtained.

[0012] The policy loss function of the improved A3C algorithm is determined by the sum of the product of the advantage and the logarithmic probability gradient of the policy network at each time step, and the product of the control entropy parameter and the policy entropy at each time step.

[0013] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the aforementioned joint optimization of computation offloading and caching based on an improved A3C algorithm.

[0014] Another aspect of the present invention provides a computational offloading and caching joint optimization device based on an improved A3C algorithm, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the computational offloading and caching joint optimization device based on the improved A3C algorithm performs the aforementioned computational offloading and caching joint optimization.

[0015] The present invention has at least the following beneficial effects

[0016] This invention utilizes other moving vehicles on the road and RSUs equipped with MEC servers to construct V2V and V2R models respectively, reducing server load and minimizing resource waste. It prioritizes tasks to ensure orderly execution and designs a task preemption scheme to prioritize urgent tasks, reducing failure rates. Based on Zipf distribution, it utilizes cloud servers to cache RSUs according to task popularity, effectively improving task hit rate. Combining the Asynchronous Advantage Actor Critic (A3C) algorithm based on the Actor-Critic architecture, it introduces the concept of maximum entropy and proposes the Soft-A3C algorithm, or SA3C algorithm, for solving optimization problems, minimizing the system's total average latency. This invention considers user needs in real-world scenarios, achieving intelligence and autonomy while making computation offloading and caching joint optimization more practical. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a VEC system based on task preemption according to an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0019] Figure 3 This is a flowchart of the preemption scheme according to an embodiment of the present invention;

[0020] Figure 4This invention provides an SA3C algorithm framework based on the Actor-Critic architecture. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] The computational offloading and caching joint optimization method and apparatus based on the improved A3C algorithm of this invention are implemented by a target network, which includes at least multiple roadside units equipped with MEC servers, multiple terminal vehicles, and cloud servers.

[0023] Please see Figure 1 For example, the entire target network system is a cloud-edge-device architecture, including a cloud server, three roadside units (RSUs), and multiple vehicle users. Vehicles can communicate with each other and with the RSUs. The cloud server can cache services to the RSUs, and the uplink transmission channel between vehicles and RSUs uses OFDMA technology to eliminate interference between users.

[0024] The cloud server, serving as a remote computing resource center within the target network, provides large-scale data storage and complex computing services. Roadside units (RSUs) are distributed along the roadside, acting as roadside infrastructure responsible for communicating with vehicles, collecting traffic information, sending traffic condition warnings, and acting as relay nodes for communication between vehicles and the cloud server or MEC server. The MEC cloud server is connected to the RSU, with one MEC server adjacent to each RSU. The MEC server deploys computing and storage resources at the network edge, close to vehicles, providing low-latency computing services, reducing latency in vehicle-cloud communication, and enabling rapid processing of vehicle-uploaded data, such as real-time traffic data analysis and vehicle status monitoring. Red vehicles represent task-requesting vehicles, i.e., vehicles with computing needs; blue vehicles are ordinary vehicles that can assist in task processing or communication. In practice, the roles of task-requesting vehicles and ordinary vehicles can be interchanged; that is, vehicle A can act as a task-requesting vehicle at time t1, and as an ordinary vehicle at time t2. Vehicles communicate directly with each other via V2V links to achieve information sharing, such as exchanging vehicle speed and location information. Vehicles and roadside units can communicate directly via V2R links within the coverage area of ​​the roadside unit. Vehicles can send task requests, location information, etc. to the RSU via V2R links, and the RSU can also send traffic rules, road conditions, and other information to the vehicle.

[0025] Please see Figure 2 One aspect of the present invention provides a joint optimization method for computational offloading and caching based on an improved A3C algorithm, the method comprising:

[0026] S1. Based on the processing delay of each terminal vehicle in processing the current task locally in the current time slot and the transmission delay of each terminal vehicle in unloading the current task to the corresponding other terminal vehicle in the current time slot, obtain the total delay of all terminal vehicles in the current time slot.

[0027] In this embodiment of the invention, the latency of a terminal vehicle within the current time slot is determined by its local processing latency and the processing latency on other terminal vehicles. Each terminal vehicle incurs processing latency when processing its current task locally within the current time slot. This latency depends on factors such as the vehicle's own computing power and the complexity of the task. For example, if the vehicle's processor performance is low, or the task involves a large amount of complex computation (such as high-definition image recognition, complex path planning calculations, etc.), then the local processing latency will be relatively long. The local processing latency reflects the time spent by the vehicle processing the task using its own resources. When a terminal vehicle chooses to offload its current task to another corresponding terminal vehicle within the current time slot, a transmission latency occurs. This is mainly related to the communication distance between vehicles, the quality of the communication link (such as signal strength, interference, etc.), and the data transmission rate. For example, if the distance between two vehicles is large, or if there is strong interference in the communication frequency band, causing a decrease in the data transmission rate, then the time required to transmit task data, i.e., the transmission latency, will increase.

[0028] Specifically, in this embodiment of the invention, the calculation process for the total latency of all terminal vehicles within the current time slot may include the following steps:

[0029] S11: Define a vehicle n∈{1,2,…,N}, and the speed of vehicle n is v. n The vehicle's location is (x n ,y n ),

[0030] The speed of the remaining moving vehicle j is v j The position is (x j ,y j The communication range between vehicles is Based on the above data, the communication time between vehicles is calculated as follows:

[0031]

[0032] The connection time between vehicles can be obtained using the above method. This connection time can be used as a processing latency limit for tasks at the vehicle, ensuring that task unloading can be completed at the vehicle.

[0033] With each vehicle as a node and the communication time between vehicles as the edge weight, a topological path is constructed. A depth-first search algorithm is used to find all paths from the target vehicle to which unloading needs to be performed to the current task vehicle, denoted as the hop count H. Then, the single-hop transmission delay of V2V communication is:

[0034]

[0035] Among them, D n V represents the task size. n,j W is the transmission rate from vehicle n to j.n,j P is the transmission bandwidth between vehicles n and j. n,j Let h be the transmission power of vehicle n. n,j For channel gain, σ 2 Let be the noise power. Then the total transmission delay for V2V communication is:

[0036]

[0037] Based on task size D n And the number of CPU cycles required to calculate 1 bit by the user X n To obtain computing resources D n X n Then, based on the vehicle's computing power f n The processing latency for tasks at the vehicle is calculated as follows:

[0038]

[0039] S12: Based on the above data, the total latency for local vehicle task processing is calculated as follows:

[0040]

[0041] in, t represents the total latency of V2V communication. n,l Calculate the latency of the task for the vehicle.

[0042] Using the above formula, the latency of each terminal vehicle in the current time slot can be obtained. Assuming there are N terminal vehicles in total, the latency of these N terminal vehicles can be added together to obtain the total latency of all terminal vehicles in the current time slot.

[0043] S2. Based on the transmission delay of each roadside unit receiving the current task from the terminal vehicle in the current time slot and the processing delay of the task in the current time slot during which preemption occurs for each roadside unit, obtain the total delay of all roadside units in the current time slot.

[0044] In this embodiment of the invention, the latency of each roadside unit within the current time slot is determined by the transmission latency and the processing latency related to task preemption. Each roadside unit experiences transmission latency when receiving the current task from the terminal vehicle in the current time slot. This transmission latency is affected by various factors, such as the distance between the terminal vehicle and the roadside unit; the greater the distance, the longer the signal transmission path, and the greater the transmission latency. The quality of the communication link also plays a role; interference, signal attenuation, and other issues can reduce the data transmission rate, thus increasing the transmission latency. The size of the transmitted data also contributes; the larger the task data volume, the longer the transmission time typically requires. Whether a roadside unit preempts a subsequent task in the current time slot also generates processing latency. If a roadside unit preempts a task in the current time slot, it needs to process the successfully preempted subsequent task, while the current task waits for it to complete before continuing processing. The computational load and complexity of the processed task determine the length of the processing latency. This effectively ensures that urgent tasks can be processed first; if no preemption occurs, the current task continues to be processed.

[0045] Specifically, in this embodiment of the invention, the calculation process of the total delay of all roadside units in the current time slot may include the following steps:

[0046] S21: Define vehicles n∈{1,2,…,N}, RSU roadside units r∈{1,2,…,R}, and the speed of vehicle n is v. n The vehicle's location is (x n ,y n ), RSU position is (x r ,y r Given that the maximum communication distance of the RSU is ψ, and based on the above data, the dwell time of the vehicle within the RSU coverage area is calculated as follows:

[0047]

[0048] Using the above method, the total processing latency of a task between vehicles can be limited to the connectivity time between vehicles. This connectivity time can serve as a latency limit for tasks at the roadside unit, ensuring that task unloading can be completed at the roadside unit.

[0049] S22: Calculate the transmission delay of V2R communication:

[0050]

[0051] Among them, D n V represents the task size. n,r For the transmission rate from vehicle n to RSU r, W n,r P is the transmission bandwidth between vehicle n and RSU r. n,r Let h be the transmission power of vehicle n. n,r For channel gain, σ2 This represents noise power. Based on the task size D... n And the number of CPU cycles required to calculate 1 bit by the user X n To obtain computing resources D n X n Then, based on the roadside unit's computing capacity f r The processing latency of the task on the RSU is calculated as follows:

[0052]

[0053] S23: Based on the above data, the total processing latency on the RSU is calculated as follows:

[0054]

[0055] in, t represents the transmission delay from the vehicle to the RSU. n,r t represents the processing latency of the current task on RSU. n+1,r Indicates the processing latency of subsequent tasks on RSU, m n ∈{0,1} represents a preemption instruction, when m n Preemption occurs when the value is 1; otherwise, preemption is not possible.

[0056] The above formula can be used to obtain the delay of each RSU in the current time slot. Assuming there are a total of R RSUs, the delays of these R RSUs can be added together to obtain the total delay of all roadside units in the current time slot.

[0057] S3. Based on the processing delay of each roadside unit and / or each terminal vehicle for the previous task, obtain the waiting delay of the current task in the current time slot;

[0058] In this embodiment of the invention, the waiting latency of the current task within the current time slot is mainly determined by the processing latency of the roadside unit for the preceding task and the processing latency of the terminal vehicle for the preceding task. When multiple tasks arrive at the roadside unit sequentially, the processing of the preceding task at the roadside unit will incur latency. This latency depends on the computing power of the roadside unit, the complexity of the task, and the task queue situation. For example, if the roadside unit has limited computing resources and multiple complex tasks are queued for processing, the processing latency of the preceding task will be longer, thus affecting the waiting time of the subsequent current task. When the vehicle has multiple tasks executed sequentially, the processing latency of the preceding task will also have an impact. If the vehicle's processor performance is average and the preceding task involves a large amount of data processing or complex algorithm calculations, the time spent processing the preceding task will increase, resulting in a longer waiting time for the current task at the vehicle.

[0059] Specifically, in this embodiment of the invention, the calculation process for the waiting delay of the current task within the current time slot may include the following steps:

[0060] S31: The waiting time for task n is:

[0061]

[0062] in, This indicates the processing delay of the predecessor task j.

[0063] The processing delay of the predecessor task j within the current time slot can be obtained using the above formula. By summing the processing delays of these predecessor tasks, the waiting delay of each current task can be obtained.

[0064] S4. Based on the transmission rate of the cloud server sending the current task to the roadside unit and the task popularity of the current task, obtain the cache latency of the current task in the current time slot;

[0065] In this embodiment of the invention, the caching latency of the current task within the current time slot is determined by the transmission rate at which the cloud server sends the current task to the roadside unit and the task popularity of the current task. If the network bandwidth is sufficient and there is no congestion, the transmission rate may be high, enabling the task data to be sent to the roadside unit quickly, and the caching latency may be correspondingly low. Conversely, if the network is congested or the link quality is poor, the transmission rate decreases, and the time for task data to wait for transmission on the cloud server increases, thus increasing the caching latency. Task popularity reflects how frequently the current task is requested or demanded. If a task has high popularity, it means that many terminal vehicles or other system components may request the task. In order to meet the potential large number of subsequent requests, the cloud server may cache the task data in advance to reduce subsequent transmission pressure and response time.

[0066] Specifically, in this embodiment of the invention, the calculation process of the buffer latency of the current task within the current time slot may include the following steps:

[0067] S41: Define the Zipf slope ∈, and calculate the task popularity using the Zipf distribution as follows:

[0068]

[0069] Where ρ is a normalization factor used to ensure that the sum of probabilities is 1, and N f This represents the total number of task types in the cloud server.

[0070] S42: The cache latency from the compute cloud server to the RSU is:

[0071]

[0072] Where, βn For the decision of unloading, For caching decisions, D n p represents the task size. n R represents the task popularity, and R represents the transmission rate from the cloud server to the RSU.

[0073] The above formula provides the cache latency from the cloud server to the RSU. By summing the cache latency from the cloud server to the RSU, the waiting latency for each current task can be obtained.

[0074] In a preferred embodiment of the present invention, the present invention establishes task priorities to ensure orderly task execution and designs a task preemption scheme to prioritize urgent tasks and reduce task failure rates; the method further includes:

[0075] S4A. Based on the task processing delay normalization constraint, determine the first priority constraint; the first priority constraint is:

[0076]

[0077] Among them, D n Indicates the task size, X n f represents the number of CPU cycles required for a user to calculate 1 bit. r This indicates the computational power of the RSU. This represents the minimum threshold for processing latency. Indicates the maximum processing latency threshold; x n,1 This indicates the processing delay normalization constraint.

[0078] S4B. Based on the normalized constraint of the terminal vehicle's dwell time, the second priority constraint is determined; the second priority constraint is:

[0079]

[0080] in, Indicates the vehicle's dwell time. This indicates the minimum dwell time threshold for vehicles. Indicates the maximum dwell time threshold for vehicles; x n,2 This indicates the normalization constraint for vehicle dwell time.

[0081] S4C, based on the energy consumption normalization constraint, determines the third priority constraint; the third priority constraint is as follows:

[0082]

[0083] Among them, D n Indicates the task size, X n f represents the number of CPU cycles required for a user to calculate 1 bit. rThis indicates the computational power of the RSU. This represents the minimum energy consumption threshold for processing. Indicates the maximum energy consumption threshold for processing; x n,3 This indicates the handling of energy consumption normalization constraints.

[0084] S4D performs a weighted summation of the first, second, and third priority constraints to obtain the task priority; the calculated task priority value is expressed as:

[0085]

[0086] Where λ1, λ2, and λ3 represent the corresponding weights of the three normalization constraints, respectively; This indicates the calculated task priority value.

[0087] S4E: Execute the task according to its priority.

[0088] In a preferred embodiment of the present invention, tasks can be executed according to their priority, so that urgent tasks can be processed first and the failure rate can be reduced.

[0089] like Figure 3 As shown, in this embodiment of the invention, the method for determining whether each roadside unit has preempted in the current time slot includes:

[0090] S4a. If the waiting delay of a subsequent task does not exceed the threshold, no preemption will be performed, and the processing delay without preemption will be obtained based on the processing delay of the current task on the roadside unit.

[0091] S4b. If the waiting delay of a subsequent task exceeds the threshold, determine whether the processing delay of the subsequent task is less than or equal to 0.5 times the maximum waiting delay of the current task. If it is less than or equal to, preemption is performed, and the processing delay after preemption is obtained based on the processing delay of the subsequent task on the roadside unit.

[0092] This invention provides a task preemption scheme, which addresses the issue of a subsequent task waiting for a delay. If the threshold ξ is exceeded, a judgment is initiated, and the processing delay is determined. Less than or equal to 0.5 times the maximum latency of the current task Then it can be seized.

[0093] S5. Based on the total latency of all tasks, and with the goal of minimizing the average total latency, determine the joint optimization problem model for all terminal vehicles and all roadside units; in this embodiment of the invention, the expression for the total average latency of the communication system is calculated according to the system model of the communication system, and the total average latency of the communication system is used as the optimization objective; in this embodiment, the process of determining the optimization objective and constraints based on the above-set system model of the communication system includes:

[0094] S51: Based on the above data, the total system delay is calculated as follows:

[0095]

[0096] Where, β n This represents the unloading decision, when β n When T = 1, unload to RSU for processing; otherwise, execute on the vehicle. n This represents the total latency required to execute task n;

[0097] S52: Define the energy consumption factor δ and calculate the processing energy consumption on the vehicle and RSU respectively:

[0098]

[0099] Among them, D n X represents the task size. n Calculate the number of CPU cycles required for 1 bit for the user, f n For vehicle computing power, f r For RSU computing power;

[0100] S53: Based on the above data, the total energy consumption of the system is calculated as follows:

[0101]

[0102] S54: Considering the total average time delay of the system, the optimization objective and constraints are as follows:

[0103]

[0104] Where N represents the number of tasks; T n T represents the total latency of the nth task; T represents the total system latency. max C1 represents the maximum latency threshold, indicating that the latency cannot exceed the maximum tolerance value; E represents the total system energy consumption. max C1 represents the maximum energy consumption threshold, and C2 indicates that energy consumption cannot exceed the maximum tolerance value. D represents the caching decision. n C represents the task size, C represents the maximum storage capacity, C3 represents the amount of data cached at one RSU cannot exceed the RSU storage capacity; C4, C5, and C6 represent the cache decision, offload decision, and task preemption indication, respectively. C7 indicates the vehicle's dwell time, meaning the total delay of a task being transmitted to the RSU for processing cannot exceed the vehicle's dwell time within the RSU's range. C8 represents the connection time between vehicles, indicating that the total latency of a task being processed between vehicles cannot exceed the connection time between vehicles; C9 represents that the sum of the corresponding weights of the three normalization constraints is 1.

[0105] It should be noted that if we do not consider execution according to task priority, then priority constraint C9 does not need to be considered in the above process.

[0106] S6. The improved A3C algorithm is used to solve the joint optimization problem model to obtain the preemption indication, unloading strategy and caching strategy of the computing tasks to be processed in each roadside unit and each terminal vehicle.

[0107] Please see Figure 4 Determine the state space, action space, and reward function;

[0108] The state-space expression is:

[0109] s(τ)={C V C R}

[0110]

[0111] Among them, C V C represents the vehicle's computing resources. R represents the computational resources of RSU; s(τ) represents the state space at time τ;

[0112] The action space expression is:

[0113] a(τ)={β,m,c r}

[0114] β={β1,β2,…,β N}

[0115] m = {m1, m2, ..., m} N}

[0116]

[0117] Where β represents the unloading decision, m represents the preemption indication (i.e., whether preemption has occurred), and c r This represents the caching decision; a(τ) represents the action space at time τ;

[0118] The expression for the reward function is:

[0119]

[0120] Where N represents the number of tasks, T represents the total system latency, and r(τ) represents the reward function at time τ.

[0121] In this embodiment, the process of solving the optimization problem using the improved A3C algorithm, namely the Soft-A3C algorithm, includes:

[0122] S61: Initialize the policy network π(a(τ)|s(τ); θ) and the value network V(s(τ); θ) v The entropy regularization coefficient β and the learning rate η are set. The policy network is used to output the probability distribution of the current action a(τ) based on the current state s(τ), and the value network is used to evaluate the value of the current state s(τ). The current state s(τ) includes the computing resources of the terminal vehicle and the computing resources of the roadside unit. The current action a(τ) includes the preemption indication, unloading strategy and caching strategy of the computing task to be processed.

[0123] S62: Multiple parallel threads interact with the environment to generate trajectory data (s(τ), a(τ), r(τ), s(τ+1)) and store it in the experience pool; where r(τ) represents the current reward and s(τ+1) represents the predicted state.

[0124] S63A: The cumulative discount reward is calculated using trajectory data:

[0125]

[0126] Where γ represents the discount factor, k represents the k-th step, r(τ+i) represents the reward function for the τ+i-th step, and V(s(τ+k)); θ V () represents the value network value function at steps τ+k;

[0127] S63B: The joint value function calculates the dominance function as follows:

[0128]

[0129] S64A: Based on the concept of maximum entropy, entropy regularization is introduced to calculate the policy loss function L. π (θ) and its gradient are respectively:

[0130] L π (θ)=logπ(a(τ)|s(τ);θ)A(s(τ),a(τ);θ,θ v )+βH(π(s(τ);θ))

[0131]

[0132] Here, β represents the hyperparameter controlling the strength of the entropy regularization term, and H(π(s(τ);θ)) represents the policy entropy. The purpose of introducing the entropy term is to encourage policy exploration, thereby avoiding premature entrapment in local optima.

[0133] The formula for calculating the strategy entropy includes:

[0134]

[0135] Wherein, π(a n (τ)|s(τ)) represents the policy network for the nth action a. n The probability output of (τ) is where N represents the number of tasks to be computed, and each action corresponds to a preemption instruction, unloading strategy, and caching strategy for a task to be computed.

[0136] It should be noted that a higher policy entropy value indicates a more uniform probability distribution of different actions taken by the policy in that state, and thus greater uncertainty; a lower policy entropy value indicates a more concentrated probability distribution and less uncertainty. This embodiment introduces an entropy term into the policy's optimization objective, which is equivalent to encouraging the policy to have higher entropy, i.e., to select different actions more evenly, while maximizing the reward.

[0137] S64B: Based on the TD error, the estimated loss function and its gradient are calculated as follows:

[0138] L v (θ v )=(R(τ)-V(s(τ);θ v )) 2

[0139]

[0140] S65A: Using the RMSProp (Root Mean Square Propagation) optimizer, the exponentially weighted average of the gradients over τ steps is calculated as follows:

[0141] g(τ)=αg(τ-1)+(1-α)Δθ 2

[0142] Where α represents the decay coefficient, and Δθ represents the current gradient of the policy loss function or the estimated loss function at step τ;

[0143] S65B: Update the policy parameter θ and the value network parameter θ using the policy gradient and value gradient respectively. v for:

[0144]

[0145] Where η represents the learning rate, and ∈′ represents a very small positive number to prevent the denominator from being 0;

[0146] S66: Asynchronously upload the gradient of each thread to the global network, update the global parameters, and finally synchronize it to the local network;

[0147] S67: Repeat the above steps until the training reaches the preset termination condition.

[0148] In summary, this invention utilizes other moving vehicles on the road and RSUs equipped with MEC servers to construct V2V and V2R models respectively, reducing server load and minimizing resource waste. This invention prioritizes tasks to ensure orderly execution and designs a task preemption scheme to prioritize urgent tasks, reducing the failure rate. Based on Zipf distribution, this invention utilizes cloud servers to cache RSUs according to task popularity, effectively improving task hit rate. This invention combines the Asynchronous Advantage ActorCritic (A3C) algorithm based on the Actor-Critic architecture, introduces the concept of maximum entropy, and proposes the Soft-A3C algorithm, or SA3C algorithm, for solving optimization problems, minimizing the total average system latency. This invention considers user needs and other factors in real-world scenarios, achieving intelligence and autonomy while making computation offloading and caching joint optimization more aligned with actual conditions.

[0149] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the aforementioned computation offloading and caching joint optimization method based on the improved A3C algorithm.

[0150] Another aspect of the present invention provides an apparatus for a computational offloading and caching joint optimization method based on an improved A3C algorithm, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the apparatus for the computational offloading and caching joint optimization method based on an improved A3C algorithm performs the aforementioned computational offloading and caching joint optimization method based on an improved A3C algorithm.

[0151] Specifically, the memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0152] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for computing offloading and cache joint optimization based on an improved A3C algorithm, characterized in that, The method is applied to a target network, and the target network at least includes a plurality of roadside units provided with MEC servers, a plurality of terminal vehicles, and a cloud server. A total time delay of all terminal vehicles in a current time slot is obtained based on a processing time delay of each terminal vehicle in the current time slot for locally processing a current task and a transmission time delay of each terminal vehicle in the current time slot for offloading the current task to a corresponding other terminal vehicle. A total time delay of all roadside units in the current time slot is obtained based on a transmission time delay of each roadside unit in the current time slot for receiving the current task from the terminal vehicle and a processing time delay of each roadside unit in the current time slot for processing a task in a preemption process. A waiting time delay of the current task in the current time slot is obtained based on a processing time delay of each roadside unit or / and each terminal vehicle for a preceding task. A caching time delay of the current task in the current time slot is obtained based on a transmission rate of the cloud server for sending the current task to the roadside unit and a task popularity of the current task. A joint optimization problem model of all terminal vehicles and all roadside units is determined based on the total time delay of all tasks, with an average total time delay minimum as a target. An improved A3C algorithm is used to solve the joint optimization problem model, so as to obtain a preemption indication, an offloading strategy, and a caching strategy of a to-be-processed computing task in each roadside unit and each terminal vehicle. The policy loss function of the improved A3C algorithm is determined by a product of a logarithmic probability gradient of each time step and a policy network advantage and a sum of a product of a control entropy parameter of each time step and a policy entropy.

2. The method of claim 1, wherein the improved A3C algorithm-based computation offloading and cache joint optimization method is characterized in that, The transmission channel in the target network adopts an orthogonal frequency division multiple access.

3. The method of claim 1, wherein the improved A3C algorithm-based computation offloading and cache joint optimization method is characterized by, The judgment manner of whether each roadside unit in the current time slot occurs preemption includes: If the waiting time delay of a certain subsequent task does not exceed a threshold value, preemption is not performed, and a processing time delay of the current task on the roadside unit is obtained without preemption; If the waiting time delay of a certain subsequent task exceeds the threshold value, it is judged whether the processing time delay of the subsequent task is less than or equal to 0.5 times the maximum waiting time delay of the current task, if yes, preemption is performed, and a processing time delay of the subsequent task on the roadside unit after preemption is obtained.

4. The method of claim 1, wherein the improved A3C algorithm-based computation offloading and cache joint optimization method is characterized by, The caching time delay of the current task in the current time slot is obtained in the following manner: If a certain current task is offloaded on the roadside unit, and there is no current task in the roadside unit, the current task is cached to the roadside unit through the cloud server; The caching time delay of the current task in the current time slot is calculated according to the size of the current task, the task popularity of the current task, and the transmission rate of the cloud server to the roadside unit. The task popularity is obtained by Zipf distribution.

5. The method of claim 1-4, wherein, The method further includes: A first priority constraint is determined according to a processing time delay normalization constraint of the task, and a second priority constraint is determined according to a stay time normalization constraint of the terminal vehicle; A third priority constraint is determined according to an energy consumption normalization constraint; The priority of the task is obtained by weighted summation of the first priority constraint, the second priority constraint, and the third priority constraint. Perform the tasks according to their priorities.

6. The method of claim 5, wherein the improved A3C algorithm-based computation offloading and cache joint optimization method is characterized by, The joint optimization problem model comprises: s.t. C1: 0 < T < T max C2: 0 < E < E max C4: c r ∈ {0,1} C5:β∈{0,1} C6: m e {0, 1} C9: λ1+λ2+λ3=1 where N denotes the number of tasks; T n denotes the total latency of the nth task; T denotes the total system latency, T max denotes the maximum latency threshold, C1 denotes that the latency cannot exceed the maximum tolerance value; E denotes the total system energy consumption, E max denotes the maximum energy consumption threshold, C2 denotes that the energy consumption cannot exceed the maximum tolerance value; denotes the cache decision, D n denotes the task size, C denotes the maximum storage capacity, C3 denotes that the amount of data cached at an RSU cannot exceed the RSU storage capacity; C4, C5, C6 respectively denote the cache decision c r , the offloading decision β, and the task preemption indication m; denotes the vehicle's stay time, C7 denotes that the total latency of a task transmitted to the RSU for processing cannot exceed the vehicle's stay time within the RSU range; denotes the connectivity time between vehicles, C8 denotes that the total latency of a task processed between vehicles cannot exceed the connectivity time between vehicles; C9 denotes that the corresponding weights of the three normalization constraints sum to 1.

7. The method of claim 1, wherein the improved A3C algorithm-based computation offloading and cache joint optimization method is characterized by, The improved A3C algorithm is used to solve the joint optimization problem model, and the preemption indication, the unloading strategy, and the caching strategy of the to-be-processed computing task in each road side unit and each terminal vehicle are obtained. Initialize the policy network and the value network, and set the entropy regularization coefficient β and the learning rate η; the policy network is used to output the probability distribution of the current action a(τ) according to the current state s(τ); the value network is used to evaluate the value of the current state s(τ); the current state s(τ) comprises the computing resources of the terminal vehicle and the computing resources of the road side unit; the current action a(τ) comprises the preemption indication, the unloading strategy, and the caching strategy of the to-be-processed computing task; A plurality of parallel threads interact with the environment to generate trajectory data (s(τ), a(τ), r(τ), s(τ+1)) and store the trajectory data in an experience pool; wherein r(τ) represents the current reward, and s(τ+1) represents the predicted state; The cumulative discounted reward R(τ) is calculated using trajectory data, and the advantage function A(s(τ), a(τ); θ, θ v ) is calculated using the value function. The strategy loss function L is calculated using entropy regularization π (θ) and its gradient, and the estimate loss function and its gradient are calculated according to the TD error. updating the policy parameters θ and the value network parameters θ using policy gradient and value gradient, respectively v ; The gradient of each thread is uploaded to the global network asynchronously, the global parameters are updated, and finally the local network is synchronized again; Repeat the above steps until the training reaches the preset termination condition.

8. The method of claim 7, wherein the improved A3C algorithm-based computation offloading and cache joint optimization method is characterized by, The calculation formula of the policy entropy comprises: wherein H (pi(s (T) ; Q)) represents a policy entropy, pi(s (T) ; Q) represents a probability output in a current state under a policy network Q, and pi(a n (τ) represents a probability output of the policy network for the nth action a n (τ), N represents a number of to-be-computed tasks, and each action corresponds to a preemption indication, an offloading strategy, and a caching strategy of a to-be-computed task.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the computing unloading and caching joint optimization method based on the improved A3C algorithm in any one of claims 1 to 8.

10. An apparatus for computing offloading and cache joint optimization based on an improved A3C algorithm, characterized in that, The device comprises a processor and a memory; the memory is used to store a computer program; the processor is connected with the memory and is used to execute the computer program stored in the memory, so that the computing unloading and caching joint optimization device based on the improved A3C algorithm executes the computing unloading and caching joint optimization method based on the improved A3C algorithm in any one of claims 1 to 8.

Images