Edge computing vehicle networking resource management joint optimization method based on DDPG algorithm

Abstract

This invention discloses a joint optimization method for edge computing-based vehicular network (V2N) resource management based on the DDPG algorithm. The method involves: establishing a network architecture integrating edge services and federated learning in the V2N, initializing network parameters; training a model using a dataset and calculating optimal model parameters; modeling the joint optimization problem as a Markov decision problem and training it using the DDPG algorithm, updating network parameters, recording the average reward, obtaining a decision network, and performing dynamic network offloading scheduling, resource allocation, and service model caching to achieve joint optimization of edge computing-based V2N resource management. This invention improves the real-time performance of V2N services based on edge computing and federated learning, exhibiting good convergence performance and joint optimization effects; it also enhances the security of privacy data on edge servers and the service quality for V2N users, and can be widely applied to practical mobile terminal applications such as path planning and navigation, and remote vehicle diagnostics.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication network technology, and in particular to a joint optimization method for edge computing vehicle network resource management based on the DDPG algorithm. Background Technology

[0002] In recent years, technological advancements in fields such as artificial intelligence, wireless communication, and sensing have driven the rapid development of the Internet of Vehicles (IoV). In traditional IoV systems, due to the limited local computing power of onboard terminal devices, some computationally intensive tasks typically need to be offloaded to remote cloud servers for processing before the results are returned to the terminal devices. The latency in this process often fails to meet the performance requirements of IoV systems, and with the dramatic increase in user devices within IoV systems, this can further congest the cloud computing network. Furthermore, when relevant task datasets are centrally trained on edge servers to obtain service models, the security of privacy data cannot be guaranteed.

[0003] To improve algorithm performance, enable edge servers to make better use of limited space, cache more popular application services, and improve service efficiency, existing research combines reinforcement learning techniques with neural networks. Wang Jingyu et al. proposed a knowledge-driven service offloading decision framework, training a DRL model for offloading decisions on edge nodes with abundant computing resources. The model is then updated based on feedback obtained from asynchronous online learning during vehicle operation in real-world environments, allowing for rapid offloading decisions under different environmental conditions. However, this framework does not consider the complexity of high-dimensional states and actions during task offloading. Chai Haoye et al. proposed a federated learning algorithm based on hierarchical blockchain for intelligent vehicles to share learned environmental knowledge, training the model while considering user privacy and security in the distributed structure of large-scale vehicular networks. The above literature mainly studies how to apply federated learning in vehicle networks, without specifically analyzing the resource consumption of the federated learning process itself in resource-constrained environments. Summary of the Invention

[0004] The purpose of this invention is to provide a joint optimization method for edge computing resource management in vehicle-to-everything (V2X) networks based on the DDPG algorithm. This method applies federated learning to protect user privacy while jointly considering issues such as user task offloading and network resource management, thereby improving the performance of edge computing networks and the quality of user services in V2X networks. It achieves joint optimization of task offloading decisions, resource allocation, and service model caching configuration in V2X networks.

[0005] The technical solution to achieve the purpose of this invention is: a joint optimization method for edge computing vehicle network resource management based on the DDPG algorithm, comprising the following steps:

[0006] Step 1: Establish the network architecture of the vehicle network and initialize network parameters;

[0007] Step 2: Model the joint optimization problem as a Markov decision problem and determine the state space, action space, and reward function in the Markov decision problem model;

[0008] Step 3: Use the DDPG algorithm to train the Markov decision problem of joint optimization, update the network parameters, record the average reward, and stop training until the set network sequence ends.

[0009] Step 4: Based on the obtained decision network, perform dynamic network offloading scheduling, resource allocation, and service model caching to achieve joint optimization of edge computing vehicle network resource management.

[0010] Furthermore, the network architecture of the vehicle-to-everything (V2X) network described in step 1 specifically includes:

[0011] In edge computing vehicle-to-everything (V2X) systems, each edge server provides task offloading services for in-vehicle mobile devices within its respective communication range;

[0012] Edge servers are deployed next to multiple base stations to serve as edge computing nodes for vehicle-mounted terminals in the Internet of Vehicles (IoV).

[0013] When a vehicle terminal generates a request for driving services during driving, the terminal device chooses to offload part of the task to an edge server for processing via wireless communication; the terminal device or edge server can only respond to the user's request task when the relevant driving services are cached.

[0014] Machine learning is used to update the prediction model used in the driving service based on the latest generated dataset, and the edge server learns the cache service model by using the local dataset of the terminal device through federated learning.

[0015] Furthermore, the network parameters mentioned in step 1 specifically include:

[0016] The set of vehicle terminals for the edge server is N = {1,2,...,N}, and the cache space size of the edge server and a single terminal device is C. s C d The computing frequency f of a single CPU core in a server and terminal device s f d The edge server provides a set of caching service types M = {1, 2, ..., M}, and the service cache configuration state matrix Q is between the edge server and the terminal device. s Q d The dataset D used for federated learning in terminal device n n ;

[0017] The virtual task queue state G of terminal device n at decision sequence k. n(k), for task queue G at decision sequence k. n (k) unloading action E n (k), the set of tasks G computed locally at decision sequence k. local (k), where G is the set of unloading tasks with corresponding cache services and the set of unloading tasks without corresponding cache services at decision sequence k. 1 offload (k), G 0 offload (k).

[0018] Further, the initialization of network parameters in step 1 includes:

[0019] Step 1-1: Calculate the task execution time cost T exe (k) is as follows:

[0020]

[0021] in Let G be the set of terminal devices n in sequence k. n User task g in (k) i,n (k) Execution time of this sequence under different unloading decisions; In the current time slice k, after generating the corresponding service cache through federated learning, G last The time required for the task in (k) to complete the computation; G last (k) is the set of user request tasks that were not completed in the previous sequence;

[0022] Step 1-2: Calculate the federated learning time cost T(K) as follows:

[0023]

[0024] Time T of Federated Learning in Time Slice k fed (k) represents the set of service types that require federated learning, Q. learn (k) The longest duration; if there are no service types in a sequence that require federated learning, the duration T(k) of time slice k is a constant. min ;

[0025] Steps 1-3, the miss rate Z(k) of the service cache model when calculating the k-sequence is as follows:

[0026]

[0027] The number of tasks for re-fed learning of the corresponding service model is The total number of tasks in sequence k is N. sum (k).

[0028] Furthermore, the Markov decision problem model in step 2 includes the state S(k), action A(k), and reward function R(k), as detailed below:

[0029] Step 2-1: Calculate the state S(k) as follows:

[0030] S(k)={H G (k),H F (K),Q d (k),Q s (k),G(k),G last (k),X G ,X F ,Y G (k),Y F (k)}

[0031] Where H G (k) represents the number of subcarriers that the edge server has allocated to each terminal device for task offloading communication at the current moment; H F (k) is the current state matrix for the number of subcarriers used for federated learning communication; Q d (k) and Q s (k) represents the current service model cache state matrix of each terminal device and edge server; G(k) is the user task request queue state of all terminals when columns k are displayed; G last (k) represents the set of user request tasks that were not completed in the previous sequence; X G and X F These are the statuses of the number of core processors used for task computation and local update computation in federated learning across all terminal devices; Y G (K) and Y F (K) represents the number of core processors allocated after the resource allocation decision for offloading task computation and the number of core processors for global aggregation computation in federated learning, respectively.

[0032] Step 2-2: Calculate the action space A(k) as follows:

[0033] A(k)={ΔH G (k),ΔH F (k),ΔQ s (k),E(k),ΔY G (k),ΔY F (k)}

[0034] E(k) = {E1(K), E2(K), ..., E N (K)} T It is the unloading decision action for user request task G(k) of all current terminals; ΔH G (k) and ΔHF (k) is used to adjust the allocation strategy of edge server communication subcarriers in this sequence, ΔY G (k) and ΔY F (k) is used to adjust the allocation strategy of edge server computing resources; ΔQ s (k) Adjust the service model cache status in the edge server;

[0035] Steps 2-3: Calculate the reward function R(k) as follows:

[0036] R(k)=-(α1U1(T exe (k))+α2U2(T(k))+α3Z(k))

[0037] α1, α2, α3 ∈ (0, 1) represent the weights of the user request task execution time, the duration of each federated learning sequence, and the miss rate of the service cache model, T exe (k) represents the user request task processing time, T(k) represents the duration of the federated learning sequence, and Z(k) represents the miss rate of the service cache model for sequence k. The Sigmoid function is used to convert T... exe (k) and T(k) are normalized to U1(T) exe (k)) and U2(T(k)).

[0038] Furthermore, the training of the DDPG algorithm described in step 3 includes the following steps:

[0039] Step 3-1: Initialize the algorithm parameters, the system state, and the total reward R and average reward R recorded in each training cycle. avg ;

[0040] Step 3-2: Train the parameters of the decision network. In the previous training step, the μ output parameter μ(s) of the Actor main network was trained. i Add motion noise n i To obtain the actual output action θ μ These are the current network parameters for the Actor; the action noise is set to a Gaussian distribution with a mean of ε0 and a standard deviation of σ0.

[0041] Step 3-3, System resource allocation status s i Standardized to Perform action a i Receive instant reward r i And the next system state s was observed. i+1 System status s i+1 Standardized to The obtained empirical tuple Store in the experience replay pool; determine if experience replay pool B is empty.m Is the capacity full? If so, proceed to step 3-4; otherwise, return to step 3-2.

[0042] Steps 3-4, Usage Experience Randomly replace experiences in the experience pool; randomly sample the smallest batch of experience samples from the experience pool. j = 1,...,N b Update the current network parameters of the Actor and Critic. The target network output of the Actor is μ'(s). j+1 The Critic target network combines mini-batch samples and optimal action μ'(s) j+1 Calculate the current target Q value: y j =r j +γQ'(s j+1 ,μ'(s j+1 |θ μ '),θ Q '), where θ μ 'For the Actor target network parameters and θ Q 'These are the target network parameters for Critic;

[0043] Steps 3-5: Update the parameters of the decision network over multiple training cycles. At the end of each time slice, record the total reward R and update the new system state and time slice count. Stop the training cycle when the set time slice length ends, and calculate the average reward R obtained during each training cycle. avg Determine whether the reward function of the decision network has converged. If it has, end the training; otherwise, return to steps 3-4.

[0044] Furthermore, step 4, which involves performing dynamic network offloading scheduling, resource allocation, and service model caching based on the obtained decision network to achieve joint optimization of edge computing vehicle network resource management, specifically includes:

[0045] Calculate the current state s of each node at each time step. i The state is input into the decision network, and the output is the scheduling method a to be executed. i This refers to the optimal unloading action for each terminal device and the optimal resource allocation action and service model caching for the edge server.

[0046] Compared with the prior art, the present invention has the following significant advantages: (1) It considers the application of federated learning in the Internet of Vehicles and fully connects it with the task offloading provided by MEC. It specifically analyzes the resource allocation and consumption of federated learning based on task offloading, and can reasonably allocate communication and computing resources in the offloading process, so that the task execution time and energy consumption in the system are minimized; (2) Applying the DDPG algorithm to the task offloading problem of the edge network can make the policy optimization more efficient and obtain the optimal solution faster, thereby improving the stability of the policy result; (3) Each terminal device participating in federated learning and the edge server only transmit model parameters through the network, rather than the entire user dataset, thereby avoiding the leakage of privacy information, improving the service quality of Internet of Vehicles users and achieving privacy protection.

[0047] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0048] Figure 1 This is a flowchart of the edge computing vehicle network resource management joint optimization method based on the DDPG algorithm of the present invention.

[0049] Figure 2 This is a diagram of the edge computing vehicle network architecture in an embodiment of the present invention.

[0050] Figure 3 This is a comparison chart of the average task execution time under different execution strategies in the embodiments of the present invention.

[0051] Figure 4 This is a comparison chart of convergence performance under different algorithms in the embodiments of the present invention.

[0052] Figure 5 This is a comparison chart of cache hit rates under different service caching configuration strategies in this embodiment of the invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] Combination Figure 1 This paper presents a joint optimization method for edge computing vehicle network resource management based on the DDPG algorithm, which includes the following steps:

[0055] Step 1: Establish the network architecture of the vehicle network and initialize the necessary network parameters;

[0056] Step 2: Model the joint optimization problem as a Markov decision problem and determine the state space, action space and reward function in the model;

[0057] Step 3: Use the DDPG algorithm to train the Markov decision problem of joint optimization, update the network parameters, record the average reward, and stop training until the set network sequence ends.

[0058] Step 4: Perform joint optimization of edge computing vehicle network resource management based on the obtained decision network.

[0059] In one embodiment, the IoT network architecture described in step 1 specifically includes: in an edge computing vehicle network, each edge server provides task offloading services to in-vehicle mobile devices within its respective communication range. Edge servers are deployed near multiple base stations, serving as edge computing nodes of the vehicle network to provide services to nearby in-vehicle terminals. During driving, in-vehicle terminals generate several tasks requesting driving services. Terminal devices can choose to offload some tasks to edge servers for processing via wireless communication. Only when the terminal device or edge server has cached the relevant driving services can it respond to the user's request tasks. Because the environment in the vehicle network changes rapidly, driving services such as path planning and emergency prediction typically require timeliness and accuracy. Therefore, machine learning can be used to update the prediction models used in driving services based on the latest generated datasets. Edge servers can learn cached service models using the local datasets of terminal devices through federated learning, and can train multiple models simultaneously.

[0060] In one embodiment, the system model parameters described in step 1 specifically include: the set of vehicle terminals of the edge server N = {1,2,...,N}, and the cache space size C of the edge server and a single terminal device. s C d The computing frequency f of a single CPU core in a server and terminal device s f d The edge server provides a set of caching service types M = {1, 2, ..., M}, and the service caching configuration state matrix Q is between the edge server and the terminal device. s Q d The dataset D used for federated learning in terminal device n n The virtual task queue state G of terminal device n at decision sequence k. n (k), for task queue G at decision sequence k. n (k) unloading action E n (k), the set of tasks G computed locally at decision sequence k. local (k), where G is the set of unloading tasks with corresponding cache services and the set of unloading tasks without corresponding cache services at decision sequence k. 1 offload (k), G 0 offload (k).

[0061] Step 1-1: Calculate the task execution time cost T exe (k) is as follows:

[0062] in Let G be the set of terminal devices n in sequence k. n Task g in (k) i,n (k) Execution time of this sequence under different unloading decisions; In the current time slice k, after generating the corresponding service cache through federated learning, G last The time required to complete the computation of task (k) is G. last (k) is the set of user request tasks that were not completed in the previous sequence.

[0063] Step 1-2: Calculate the federated learning time cost T(K) as follows:

[0064]

[0065] Time T of Federated Learning in Time Slice k fed (k) represents the set of service types that require federated learning, Q. learn (k) The longest duration. If there are no service types in a sequence that require federated learning, the duration T(k) of time slice k is a constant. min .

[0066] Steps 1-3, the miss rate Z(k) of the service cache model when calculating the k-sequence is as follows:

[0067]

[0068] The number of tasks for re-fed learning of the service model, and the total number of tasks for the k sequences is N. sum (k).

[0069] In one embodiment, the Markov decision problem model described in step 2 includes a state S(k), an action A(k), and a reward function R(k), specifically:

[0070] Step 2-1: Calculate the state S(k) as follows:

[0071] S(k)={H G (k),H F (K),Q d (k),Q s (k),G(k),G last (k),X G ,X F ,YG (k),Y F (k)}

[0072] Where H G (k) represents the number of subcarriers that the edge server has allocated to each terminal device for task offloading communication at the current moment; H F (k) is the current state matrix for the number of subcarriers used for federated learning communication; Q d (k) and Q s (k) represents the current service model cache state matrix of each terminal device and edge server; G(k) is the user task request queue state of all terminals when columns k are displayed; G last (k) is the set of user request tasks that were not completed in the previous sequence; X G and X F These are the statuses of the number of core processors used for task computation and local update computation in federated learning across all terminal devices; Y G (K) and Y F (K) represents the number of core processors allocated after the resource allocation decision for offloading task computation and the number of core processors for global aggregation computation in federated learning, respectively.

[0073] Step 2-2: Calculate the action space A(k) as follows:

[0074] A(k)={ΔH G (k),ΔH F (k),ΔQ s (k),E(k),ΔY G (k),ΔY F (k)}

[0075] E(k) = {E1(K), E2(K), ..., E N (K)} T This refers to the unloading decision action for user request task G(k) across all current terminals. ΔH G (k) and ΔH F (k) is used to adjust the allocation strategy of edge server communication subcarriers in this sequence, ΔY G (k) and ΔY F (k) is used to adjust the allocation strategy of edge server computing resources. ΔQ s (k) Adjust the service model cache status in the edge server.

[0076] Step 2-3, calculate the reward function R(k) as follows: R(k) = -(α1U1(T) exe (k))+α2U2(T(k))+α3Z(k))

[0077] α1, α2, α3 ∈ (0, 1) represent the weights of the user request task execution time, the duration of each federated learning sequence, and the miss rate of the service cache model, T exe (k) represents the user request task processing time, T(k) represents the duration of the federated learning sequence, and Z(k) represents the miss rate of the service cache model for sequence k. The Sigmoid function is used to convert T... exe (k) and T(k) are normalized to U1(T) exe (k)) and U2(T(k)).

[0078] In one embodiment, step 3, the DDPG algorithm training, specifically includes:

[0079] Step 3-1: First, initialize the algorithm parameters, the system state, and the total reward R and average reward R recorded in each training cycle. avg ;

[0080] Step 3-2: Next, train the parameters of the decision network. In the previous training step, the μ output parameter μ(s) of the Actor main network was used. i To facilitate more effective motion space exploration, motion noise n is added. i The method to obtain the actual output action θ μ These are the current network parameters for the Actor. The action noise is set to a Gaussian distribution with a mean of ε0 and a standard deviation of σ0.

[0081] Step 3-3, System resource allocation status s i Standardized to Perform action a i Receive instant reward r i And the next system state s was observed. i+1 System status s i+1 Standardized to The obtained empirical tuple Store in the experience replay pool. Check experience replay pool B. m Check if the capacity is full; if not, return to step 3-2.

[0082] Steps 3-4, Usage Experience Randomly replace experiences in the experience pool; randomly sample the smallest batch of experience samples from the experience pool. (j=1,...,N b Update the current network parameters of the Actor and Critic. The target network output of the Actor is μ'(s). j+1 The Critic target network combines mini-batch samples and optimal action μ'(s) j+1 Calculate the current target Q value: yj =r j +γQ'(s j+1 ,μ'(s j+1 |θ μ '),θ Q '), where θ μ 'For the Actor target network parameters and θ Q 'These are the target network parameters for Critic;

[0083] Steps 3-5: Finally, the parameters of the decision network are updated over multiple training epochs. At the end of each time slice, the total reward R needs to be recorded, and the new system state and time slice count are updated. The training epoch stops when the set time slice length ends. The average reward R obtained during each training epoch is calculated after each epoch. avg Determine whether the reward function of the decision network has converged; otherwise, return to steps 3-4; otherwise, end the training.

[0084] In one embodiment, the joint optimization of edge computing vehicle network resource management based on the obtained decision network in step 4 is specifically performed by calculating the current state s of each node at each time step. i The state is input into the decision network, and the output is the scheduling method a to be executed. i This refers to the optimal unloading action for each terminal device and the optimal resource allocation action and service model caching for the edge server.

[0085] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] Example

[0087] This embodiment uses a joint optimization method for edge computing vehicle network resource management based on the DDPG algorithm. Firstly, it follows... Figure 2 The architecture establishes an IoT network, setting up a vehicle-to-everything (V2X) simulation environment where services are simultaneously provided to 30 intelligent vehicles within the BS communication range. For the communication resources of the edge servers in the network, the FCA method is used to divide the communication bandwidth between the edge servers and the in-vehicle terminals into 384 subcarriers with a bandwidth of 15kHz. According to the resource allocation strategy, the subcarriers are allocated to the in-vehicle terminals for task offloading and communication of model parameters in federated learning. For the computing resources in the network, the number of CPU cores X in the in-vehicle terminals is set to 4, where the computing power of each core, i.e., the CPU's operating frequency f, is... d The clock speed is 1GHz. The number of CPU cores Y on the edge server is set to 8, and the operating frequency f of each CPU is set to 1GHz. sThe CPU clock speed is 2GHz. The number of CPUs used by the edge server for offloading task processing and global aggregation in federated learning can be adjusted according to the resource allocation strategy. The number of service types M required by the vehicle terminal is set to 50, and the storage space occupied by each service during caching is randomly distributed within a range. In the DDPG neural network, the reward discount factor γ is set to 0.9, and the soft update rate is set to 0.01. The number of neurons in the three hidden layers are set to 400, 300, and 10, respectively. The neural network uses the ReLU function as the activation function. The batch size is set to 32, and the experience cache pool size is set to 10000.

[0088] Figure 3 This is a comparison chart of the average task execution time under different execution strategies in this embodiment of the invention. As can be seen from the chart, the DDPG-based algorithm jointly optimizes task offloading decisions, resource allocation strategies, and service cache configuration updates. It dynamically adjusts decisions based on the real-time status of task queues and resources in the network, thus achieving a lower average long-term execution time for user request tasks.

[0089] Figure 4 This is a comparison graph showing the convergence performance of different algorithms in this embodiment of the invention. The graph compares the convergence results obtained by the two algorithms. It can be seen from the graph that both algorithms can achieve stable convergence of the network, but the reward value of the final convergence obtained by the DQN-based algorithm is much smaller than the reward value obtained by the DDPG-based algorithm.

[0090] Figure 5 This is a comparison chart of cache hit rates under different service caching configuration strategies in this embodiment of the invention. The PSCPP caching strategy only caches a portion of the more popular service models. Since the request probability of each service model corresponding to a user task is pre-fixed, the average service cache hit rate is also basically stable in each cycle. The DDPG-based algorithm needs to first learn the distribution probability of the services required by all vehicle terminals requesting tasks within the service range of the edge server. Therefore, it can be seen that the average service cache hit rate is not high in the initial learning stage, but it gradually increases with the increase of the learning cycle. Around the 500th iteration cycle, the service cache configuration update strategy based on the DDPG algorithm tends to the optimal solution, and the average service cache hit rate of the edge server also basically stabilizes. As can be seen from the curve, when the DDPG-based algorithm basically converges, the obtained service cache configuration update strategy is better than the PSCPP service caching strategy.

[0091] In summary, the method of this invention can improve the real-time performance of vehicle-to-everything (V2X) services based on edge computing and federated learning, and has good convergence performance and joint optimization effect. It can improve the security of privacy data in edge servers and the service quality of V2X users, and can be widely applied to practical application scenarios of mobile terminals such as path planning and navigation, and remote vehicle diagnostics.

Claims

1. An edge computing vehicle networking resource management joint optimization method based on a DDPG algorithm, characterized in that, Includes the following steps: Step 1: Establish the network architecture of the vehicle network and initialize network parameters; Step 2: Model the joint optimization problem as a Markov decision problem and determine the state space, action space, and reward function in the Markov decision problem model; Step 3: Use the DDPG algorithm to train the Markov decision problem of joint optimization, update the network parameters, record the average reward, and stop training until the set network sequence ends. Step 4: Based on the obtained decision network, perform dynamic network offloading scheduling, resource allocation, and service model caching to achieve joint optimization of edge computing vehicle network resource management; The network parameters mentioned in step 1 specifically include: collection of vehicle-mounted terminals for edge servers The size of the cache space for edge servers and individual terminal devices , The computing frequency of a single CPU core in a server and terminal device. , The collection of caching service types provided by edge servers Service cache configuration state matrix in edge servers and terminal devices , Terminal equipment Datasets used for federated learning ; In decision sequence Time terminal equipment Virtual task queue status In the decision sequence For task queues Uninstallation action In the decision sequence Set of tasks for local computation In the decision sequence There are sets of uninstallation tasks with corresponding cache services and sets of uninstallation tasks without corresponding cache services. , ; Step 1, which initializes the network parameters, includes: Step 1-1, Calculate the task execution time cost As follows: wherein is a terminal device at a sequence time set user tasks the sequence execution time under different offloading decisions; is a current time slice , after the corresponding service cache is generated through federated learning, the time required for task completion calculation; is a set of user request tasks that failed to complete in the last sequence; Steps 1-2: Calculate the time cost of federated learning as follows: Time slice Study time in the Middle Federation For the set of service types that require federated learning middle The longest time slice; if there are no service types in a sequence that require federated learning, the time slice... Duration constant value ; Steps 1-3, Calculation Miss rate of the sequential service cache model as follows: The number of tasks for re-fed learning of the corresponding service model is , The total number of tasks in the sequence is ; The Markov decision problem model in step 2 includes states. ,action Reward function The details are as follows: Step 2-1, Calculate the state as follows: in This indicates the number of subcarriers that the edge server has allocated to each terminal device for task offloading communication at the current moment; This is the current state matrix representing the number of subcarriers used for federated learning communications; and These are the current service model cache state matrices for each terminal device and edge server; yes The queue status of user task requests for all terminals is displayed. This is the set of user request tasks that were not completed in the previous sequence. and These are the statuses of the number of core processors used for task computation and local update computation in federated learning across all terminal devices. and These are the number of core processors allocated after resource allocation decisions for offloading task computation and the number of core processors allocated for global aggregation computation in federated learning. Step 2-2: Calculate the motion space as follows: This refers to user request tasks for all current terminals. The decision to unload; and Used to adjust the allocation strategy of edge server communication subcarriers in this sequence. and Used to adjust the allocation strategy of computing resources for edge servers; Adjust the service model cache status in the edge server; Steps 2-3: Calculate the reward function as follows: The weights represent the execution time of the user request task, the duration of each federated learning sequence, and the miss rate of the service cache model. For user request task processing time, For the duration of the federated learning sequence, for The miss rate of the sequential service cache model is calculated using the Sigmoid function. and Normalization and .

2. The edge computing-based vehicular network resource management joint optimization method based on the DDPG algorithm according to claim 1, characterized in that, The network architecture of the vehicle-to-everything (V2X) network described in step 1 specifically includes: In edge computing vehicle-to-everything (V2X) systems, each edge server provides task offloading services for in-vehicle mobile devices within its respective communication range; Edge servers are deployed next to multiple base stations to serve as edge computing nodes for vehicle-mounted terminals in the Internet of Vehicles (IoV). When a vehicle terminal generates a request for driving services during driving, the terminal device chooses to offload part of the task to an edge server for processing via wireless communication; the terminal device or edge server can only respond to the user's request task when the relevant driving services are cached. Machine learning is used to update the prediction model used in the driving service based on the latest generated dataset, and the edge server learns the cache service model by using the local dataset of the terminal device through federated learning.

3. The edge computing-based vehicular network resource management joint optimization method based on the DDPG algorithm according to claim 2, characterized in that, Step 3 describes the training of the DDPG algorithm, which includes the following steps: Step 3-1: Initialize algorithm parameters, system state, and total reward recorded in each training cycle. and average reward ; Step 3-2: Train the parameters of the decision network. In the previous training step, the Actor main network... Output parameters Add motion noise To obtain the actual output action , These are the current network parameters for the Actor; the action noise is set to a Gaussian distribution with a mean of [value missing]. The standard deviation is ; Step 3-3: System Resource Allocation Status Standardized to ; Perform actions Receive instant rewards And the next system state was observed. System status Standardized to The resulting empirical tuple Store in the experience replay pool; check the experience replay pool. Is the capacity full? If so, proceed to step 3-4; otherwise, return to step 3-2. Steps 3-4, Usage Experience Randomly replace experiences in the experience pool; randomly sample the smallest batch of experience samples from the experience pool. Update the current network parameters of the Actor and Critic. The target network output of the Actor is... The Critic target network combines mini-batch samples and optimal actions. Calculate the current target Q value: ,in For Actor target network parameters and Critic target network parameters; Steps 3-5: Update the parameters of the decision network over multiple training cycles, and record the total reward at the end of each time slice. It updates the new system state and time slice count; the training cycle stops when the set time slice length ends, and the average reward obtained during the period is calculated after each training cycle. Determine whether the reward function of the decision network has converged. If it has, end the training; otherwise, return to steps 3-4.

4. The edge computing-based vehicular network resource management joint optimization method based on the DDPG algorithm according to claim 1, characterized in that, Step 4 describes the dynamic offloading and scheduling of the network, resource allocation, and service model caching based on the obtained decision network to achieve joint optimization of edge computing vehicle network resource management. Specifically: Calculate the current state of each node at each time step. The state is input into the decision network, and the output is the scheduling method to be executed. This refers to the optimal unloading action for each terminal device and the optimal resource allocation action and service model caching for the edge server.

Images