A method and system for optimizing network communication resources for autonomous driving
The Deep Deterministic Policy Gradient algorithm, which utilizes multi-objective deep reinforcement learning, combines deep learning and reinforcement learning to solve the optimization problem of control and transmission channels in autonomous driving systems. It achieves the best trade-off between information timeliness and throughput, thereby improving the system's decision-making efficiency and adaptability.
Patent Information
- Application Number
- CN202311647782.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-04
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-12-04
AI Technical Summary
Existing technologies struggle to effectively optimize the trade-off between the timeliness of information in the control channel and the throughput of the transmission channel in autonomous driving systems. Traditional methods assume constant weights and cannot adapt to the importance of dynamic changes in reality, and they are also difficult to handle continuous state and action spaces.
The Deep Deterministic Policy Gradient algorithm, which employs multi-objective deep reinforcement learning, combines the data feature extraction capabilities of deep learning with the decision-making ability of reinforcement learning. Through the General Policy Improvement policy selection module and preference distribution settings, it achieves the optimal trade-off between the timeliness of information in the control channel and the throughput of the transmission channel.
It achieves the optimal trade-off between the AOI of CCH and the throughput of TCH in autonomous driving networks, improves decision-making efficiency and accuracy, enhances the system's adaptability and stability, and demonstrates strong robustness and scalability under different environmental and task requirements.
Smart Images

Figure CN117528581B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of network communication resource optimization technology, and in particular relates to a method for optimizing network communication resources for autonomous driving based on multi-objective deep reinforcement learning. Background Technology
[0002] In the foreseeable future, unmanned vehicle systems (UxVs, including UAVs and UGVs) will dominate in ground, air, surface, and underwater scenarios. This is because advancements in technology have enabled UxVs to achieve greater autonomy, mobility, and carrying capacity. Unmanned vehicle networks play a crucial role in ensuring optimal mission performance and data communication.
[0003] Data transmitted in a UxV network includes collected traffic data (e.g., sensor data) and control data characterizing the network status (e.g., load, channel information). Traffic data is transmitted in the Traffic Channel (TCH), and throughput is a key metric for evaluating it. Control data is transmitted in the Control Channel (CCH), and its timeliness and reliability must be ensured. The "age" of the information (AOI) is a key evaluation metric; if the information is too old, it has lost its timeliness.
[0004] A multi-objective joint optimization method based on deep reinforcement learning (DRL) is proposed, which achieves the optimal trade-off between the AOI of CCH and the throughput of TCH.
[0005] Existing multi-objective optimization algorithms not based on deep reinforcement learning calculate the Pareto front, which indicates that one metric has reached its optimal value without compromising another. However, calculating the Pareto front is unnecessary in practice, as one metric can be sacrificed to optimize a more important one.
[0006] Traditional weighted summation methods transform multi-objective optimization problems into single-objective optimization problems. The performance of single-objective optimization algorithms based on this approach is not ideal. This is mainly because they assume a constant, known weight during the learning and execution phases. However, in the scheme, the weights for each time slot are both variable and unknown. Using these traditional algorithms requires solving several different single-objective optimization problems, each with its own preferences, spanning all time slots. However, considering the non-convexity of the objective function, the high dimensionality of the variables, and their interdependencies, solving them linearly over time is impractical.
[0007] Using deep learning (DL) methods can lead to problems such as a lack of training data, insufficient storage, and computational limitations. Reinforcement learning, on the other hand, does not require training data but still has its limitations.
[0008] Reinforcement learning (RL) methods typically address multi-objective challenges by transforming them into scalar functions through weighted sums. However, this approach is less than ideal because the importance of the different metrics to be optimized varies. Training multiple reinforcement learning models for varying importance is resource-intensive.
[0009] Traditional deep reinforcement learning algorithms, such as Double DQN, can handle high-dimensional state spaces; however, they cannot handle continuous spaces (i.e., where there are an infinite number of possible action or state values). Simply discretizing the continuous space would result in the loss of much useful information. Therefore, Deep Deterministic Policy Gradient (DDPG), a deep reinforcement learning algorithm that handles continuous action spaces, is used.
[0010] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:
[0011] (1) Reasonably optimize multi-objective problems. The Pareto front obtained by an existing multi-objective optimization problem is not necessarily optimal in reality, because in reality, some indicators can be sacrificed to optimize more important indicators. If the multi-objective problem is transformed into a single-objective problem and then optimized, the disadvantage of traditional single-objective optimization problems is that they assume a constant known weight in the learning and execution phases, while in reality, the importance of different optimization objectives varies.
[0012] (2) Reasonably handle the problems of continuous state space and action space. Summary of the Invention
[0013] To address the problems existing in the prior art, this invention provides a method for optimizing network communication resources for autonomous driving based on multi-objective deep reinforcement learning.
[0014] This invention is implemented as follows: a system for optimizing network communication resources in autonomous driving, which employs a multi-objective deep reinforcement learning framework and includes the following modules:
[0015] A Deep Deterministic Policy Gradient algorithm module based on multi-objective deep reinforcement learning combines the data feature extraction capability of deep learning with the decision-making capability of reinforcement learning to achieve the optimal trade-off between the Age of Information (AOI) of the control channel and the throughput of the transmission channel in autonomous vehicle networks.
[0016] A General Policy Improvement strategy selection module is used to select and improve the best strategy from multiple strategies to achieve the best trade-off between different objectives;
[0017] A preference distribution and trade-off path setting module for inputting the preference distribution D ω The path P with balancing weight λ λ , where λ increases from 0 to 1.
[0018] The system further includes:
[0019] A neural network parameter initialization module is provided to initialize the Critic network and the Actor network, as well as the corresponding target networks Q′ and μ′.
[0020] A loop execution module is used to execute a predetermined number of M loops, each loop containing a series of sub-steps from receiving the initial observation state to updating the neural network parameters based on environmental feedback;
[0021] An action exploration and state update module is used to initialize action exploration, receive the state, execute actions, update the environment state, and calculate the reward value r in each loop. t .
[0022] The system further includes:
[0023] An action selection and execution module selects and executes actions based on the Actor network output and added random noise, while considering the current state s. t and preference ω;
[0024] An environmental feedback processing module is used to handle environmental state updates and reward values r after the autonomous vehicle performs an action. t The calculation of the reward value takes into account the AOI of CCH and the throughput of TCH.
[0025] A neural network update module is used to collect samples from the replay buffer R and update the neural network parameters after a certain time step is reached;
[0026] A loop termination judgment module is used to check whether the loop termination condition is met and control whether the loop continues or stops.
[0027] Another objective of this invention is to provide a method for optimizing network communication resources for autonomous driving based on multi-objective deep reinforcement learning, comprising:
[0028] Step 1: Use the Deep Deterministic Policy Gradient (MORL+DDPG) algorithm of multi-objective deep reinforcement learning to leverage the data feature extraction capability of deep learning and the decision-making capability of reinforcement learning to achieve the optimal trade-off between the AOI of CCH and the throughput of TCH in the autonomous vehicle network.
[0029] Step two utilizes the concept of GPI (Geometric Productivity). GPI considers a set of policies interacting with the same environment, with the goal of selecting and improving the policy that outperforms all other policies. GPI trains policies that operate under different preferences, aligning with multi-objective learning problems and determining the optimal trade-offs between different objectives.
[0030] Furthermore, the algorithm is as follows:
[0031] Step 1: Input: D ω P λ ;D ω : Distribution of preferences; P λ The path through which the balancing weight λ increases from 0 to 1;
[0032] Step 2: Initialize the neural network parameters;
[0033] Step 3: Execute M loops, with episodes 1 through M.
[0034] Furthermore, the method for initializing the neural network parameters is as follows:
[0035] Step 2.1: Randomly initialize the Critic network and the Actor network, with network parameters θ respectively. Q and θ μ ;
[0036] Step 2.2: Initialize the target critic network Q′ and the target actor network μ′, with weights θ respectively. Q′ ←θ Q ,θ μ′ ←θ μ .
[0037] Step 2.3: Initialize the replay buffer R and set λ = 0.
[0038] Furthermore, the execution involves M loops, with episodes ranging from 1 to M:
[0039] Step 3.1: Initialize the stochastic process N for action exploration;
[0040] Step 3.2: Receive the initial observation state s0; where s t The state of the environment at time t is composed of the following:
[0041]
[0042] Where Lp[t] represents the position of the p-th UAV controller at time t, including three-dimensional coordinates; l n[t]: Represents the position of the nth unmanned vehicle in the data plane at time t, including three-dimensional coordinates; An[t]: Represents the "age" (AOI) of the information of the nth unmanned vehicle. The higher the age, the lower the timeliness of the information; If the information has not been transmitted after each time step, this value is incremented by 1; P represents the number of UAV controllers, and N represents the number of unmanned vehicles;
[0043] Step 3.3: Sample a preference ω from the sample Dω; ω: preference, representing the different importance of different physical quantities (AOI and throughput). The more important a physical quantity is, the higher this value will be.
[0044] Step 3.4: Execute T sub-loops, where t ranges from 0 to T:
[0045] Step 3.4.1: Select and execute an action based on the output of the deep reinforcement learning neural network, using the following formula:
[0046] a t =μ GPI (s t , w|θ μ )+N t
[0047] Where μ represents the output action of the Actor network, and the network parameters are θ. μ Input state s t And preference ω, N t This represents the "noise" at time t, i.e., a random value; this is because adding a random value to the action output by the Actor network increases uncertainty, thereby enhancing the network's exploratory ability in the early stages of training; the specific composition of the action vector is as follows:
[0048]
[0049] λ n [t]=(λ c npk [t],λ d npk [t]), the two variables in parentheses have been mentioned earlier, p n The value represents power, ΔL represents the displacement of the UAV controller, p represents the serial number of the UAV controller, and n represents the serial number of the unmanned vehicle; P and N represent the total number of UAVs and unmanned vehicles, respectively.
[0050] Step 3.4.2: Instruct the autonomous vehicle to perform actions in the environment, while simultaneously updating the environmental state and obtaining the reward value r. t ;r t This refers to the reward that an autonomous vehicle can receive after performing a certain action in the environment.
[0051] Step 3.4.3: Transfer a sample (S)t a t r t S t+1 , ω) are stored in R;
[0052] Step 3.4.4: If a certain number of time steps are reached, update the neural network. The specific steps are as follows: First, sample N from the replay buffer R. τ There are 10 samples, and then θ is updated once using each sample. Q and θ μ ;
[0053] Step 3.4.5: Check whether the loops in Step 3 and Step 3.4 meet the termination conditions, i.e., whether t has reached T and whether episode has reached M. If they are met, stop; otherwise, continue the loop.
[0054] Furthermore, the autonomous vehicle performs actions in the environment while simultaneously updating the environmental state to obtain a reward value r. t r t This represents the reward that an autonomous vehicle can receive after performing a certain action in the environment, and it consists of the following:
[0055]
[0056] r c [t]: Measure the reward of CCH's AOI.
[0057] r d [t]: Rewards that measure the throughput of TCH.
[0058] A N [t-1]-A N [t] represents the decrease in AOI, A max [t] represents the maximum AOI value of the driverless car.
[0059] θ d N [t]: The average throughput of all driverless vehicles at time t.
[0060] θ d max [t]: The maximum throughput of all autonomous vehicles at time t.
[0061] θ d n [t]: Measures the throughput of TCH. The calculation formula is as follows:
[0062]
[0063]
[0064] Where b is the bandwidth; γmin ou γ: The threshold for signal-to-noise ratio. d nou : The signal-to-noise ratio of the nth unmanned vehicle in the data plane communicating with the oth drone in the control plane through the uth channel.
[0065] Furthermore, the calculation process of the signal-to-noise ratio;
[0066] First, calculate the probability P of using the communication link. LoS mo [t]: The probability that the m-th UGV and the o-th UAV controller use a line-of-sight (LoS) communication link in an air-to-ground (A2G) communication link.
[0067]
[0068] c1 and c2 are two constant coefficients, where, It represents the transmission angle from the UGV to the UAV controller; z represents the height of the UAV, and d represents the distance between the UGV and the UAV.
[0069] Clearly, the probability of a non-line-of-sight (NLoS) communication link is:
[0070]
[0071] Then calculate the distance d between the UAV control plane and the unmanned vehicle. np [t]: The distance between the data plane and the control plane, calculated using the following formula:
[0072]
[0073] Calculate path loss based on distance:
[0074] h LoS mp [t]: Path loss in Loss, calculated using the following formula:
[0075]
[0076] h NLoS mp [t]: Path loss in NLoS, calculated using the following formula:
[0077]
[0078] In these formulas, n represents the path loss exponent, fc represents the carrier frequency, and c represents the speed of light; μ LoS and μ NLoS The variables correspond to the average additional losses of the LoS and NLoS links, respectively; d represents the distance between the m-th unmanned vehicle and the p-th drone controller;
[0079] h mp [t]: The average link loss between the m-th UGV and the o-th UAV controller in the air-to-ground (A2G) communication link, as shown in the following formula:
[0080]
[0081] Then, to calculate the signal-to-noise ratio, we first introduce a channel-correlated binary variable:
[0082] λ c npk : Binary variable, 1 indicates that the nth UAV is associated with the unmanned vehicle controller p on the kth channel and performs CCH transmission, 0 indicates disconnection;
[0083] λ d npk : Binary variable, 1 indicates that the nth UAV is associated with the unmanned vehicle controller p on the kth channel and performs TCH transmission, 0 indicates disconnection;
[0084] λ n [t]: will λ c npk With λ d npk Encapsulate it into a vector, λ n =[λ c npk , λ d npk ].
[0085] p c n [t]: The power of the unmanned vehicle transmitting information on the CCH at time t.
[0086] p d n [t]: The power of the unmanned vehicle transmitting information at time t on the TCH.
[0087] p n [t]: p c n [t] and p d n [t] is encapsulated into a vector, p n [t] = [p] c n [t], p d n [t]].
[0088] γ c npk [t]: The signal-to-noise ratio of the CCH transmitted by the nth unmanned vehicle to the UAV controller p through the kth sub-channel, as shown in the following formula:
[0089]
[0090] γ d npk [t]: The signal-to-noise ratio of the TCH transmitted by the nth unmanned vehicle to the UAV controller p through the kth sub-channel, as shown in the following formula:
[0091]
[0092] The interference is:
[0093]
[0094] The symbols have been explained above, and their specific meanings can be inferred from the superscripts and subscripts. i represents the unmanned vehicle serial number, o represents the drone serial number, and k represents the channel serial number. If any different UxV (denoted as i) sends data packets to its associated drone controller using the same channel that n is using, interference will occur. This communication will interfere with n, regardless of whether it is using CCH or TCH.
[0095] p: Penalty. If the following constraints are violated, p will take a large negative value; otherwise, it will be 0.
[0096] Maximum speed constraint:
[0097]
[0098] Δx, Δy, and Δz represent displacements in three dimensions, and the total displacement cannot exceed the maximum velocity multiplied by time.
[0099] Minimum distance constraints between drones:
[0100]
[0101] This indicates that the coordinate distance between the q-th and i-th drones cannot be less than a threshold.
[0102] Channel constraints:
[0103]
[0104]
[0105]
[0106] K represents the number of channels, P represents the number of drone controllers, and N represents the number of unmanned vehicles.
[0107] Furthermore, the step of updating the neural network if a certain time step is reached specifically involves: firstly, collecting N data from the replay buffer R. τThere are 10 samples, and then θ is updated once using each sample. Q and θ μ The formula is as follows:
[0108]
[0109]
[0110] L is the loss function, and θ is the parameter in the Critic network Q. Q Optimization is achieved by minimizing the mean square error of the multi-objective vectorized time difference error, while updating using vectorized Sarsa; θ μ′ and θ Q′ These are the parameters of the Actor and Critic target networks, respectively; R is the cumulative reward from the current step to the end of the episode; E represents the expectation from the distribution corresponding to the index; Ω represents the preference space; w t Represents the preference vector w t ={w c [t],w d [t]|w c [t]+w d [t] = 1}, where w c and w d These represent the weights of throughput in AOI and TCH, respectively;
[0111] This means that in programming, mean square optimization code can be used directly to update network parameters.
[0112] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:
[0113] First, this invention employs the Deep Deterministic Policy Gradient (MORL+DDPG) algorithm, a multi-objective deep reinforcement learning approach, leveraging the data feature extraction capabilities of deep learning and the decision-making capabilities of reinforcement learning to achieve the optimal trade-off between the AOI (Action of Interest) of CCH (Continuous Traffic Flow) and the throughput of TCH (Traffic Flow) in autonomous vehicle networks. This invention also utilizes the GPI (Geometric Policy Optimization) concept, which considers a set of policies interacting with the same environment, aiming to select and improve the policy that outperforms all others. GPI can train policies operating under different preferences, aligning with multi-objective learning problems and thus helping to determine the optimal trade-off between different objectives.
[0114] Second, considering the technical solution as a whole or from a product perspective, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:
[0115] (1) The performance of the present invention was verified in a variety of environments, and its performance exceeded that of four classic baseline algorithms.
[0116] (2) The algorithm of the present invention has strong robustness and scalability under different environments and different task requirements (i.e., changes in the preference of indicators).
[0117] (3) Regardless of whether the problem involves discrete or continuous actions, the algorithm of this invention can effectively explore the correlation between states, actions, goals, preferences and current policies.
[0118] (4) The algorithm of the present invention can achieve the optimal trade-off of multi-objective optimization, that is, different optimization objectives can be achieved according to different objectives.
[0119] (5) Specifically, in terms of physical quantities, the algorithm of the present invention can reduce the overhead of control information transmission and improve the uplink communication performance.
[0120] This invention employs the Deep Deterministic Policy Gradient (MORL+DDPG) algorithm, a multi-objective deep reinforcement learning approach, leveraging the data feature extraction capabilities of deep learning and the decision-making power of reinforcement learning to achieve the optimal trade-off between the AOI (Action of Interest) of CCH (Conditional Driver Learning) and the throughput of TCH (Traffic Response Learning) in autonomous vehicle networks. This invention utilizes the GPI (Geometric Policy Indicator) concept, which considers a set of policies interacting with the same environment, aiming to select and improve the policy that outperforms all others. GPI can train policies operating under different preferences, aligning with multi-objective learning problems and thus helping to determine the optimal trade-off between different objectives.
[0121] Third, the significant technological advancements achieved by the autonomous driving network communication resource optimization method based on multi-objective deep reinforcement learning provided by this invention are as follows:
[0122] 1) Improve decision-making efficiency and accuracy: By combining the data feature extraction capabilities of deep learning with the decision-making capabilities of reinforcement learning, this method achieves the optimal trade-off between the Age of Information (AOI) of the Control Channel (CCH) and the throughput of the Transmission Channel (TCH) in autonomous vehicle networks, significantly improving decision-making efficiency and accuracy.
[0123] 2) Optimize strategy selection and improvement: This method uses the concept of General Policy Improvement (GPI) to effectively select and improve strategies that are superior to other strategies in multi-objective learning problems, thus providing a more flexible and efficient solution for the trade-off between different objectives.
[0124] 3) Network parameter optimization: By effectively initializing and updating the Critic and Actor networks in a timely manner, this method can better adapt to complex network communication environments and improve the overall system performance and stability by continuously learning and optimizing network parameters.
[0125] 4) Enhanced environmental adaptability: Through continuous cyclic execution and real-time status updates, this method can more accurately simulate and respond to dynamic changes in the real world, enabling unmanned vehicles to have stronger adaptability and response capabilities in complex communication networks.
[0126] 5) Refined reward design: This method refines the design of reward values (r) c [t] and r d [t]) can more accurately reflect the performance of autonomous vehicles in actual operation, thereby enabling the system to learn and optimize performance more effectively.
[0127] 6) Effective signal-to-noise ratio processing: By accurately calculating the signal-to-noise ratio, this method has made significant progress in ensuring the quality and stability of communication between unmanned vehicles and drones, which is crucial for complex autonomous driving network communication environments.
[0128] 7) Efficient network parameter update: By collecting samples from the replay buffer and updating network parameters, this method can ensure continuous learning and adaptation to changes in the environment, thereby improving efficiency while ensuring the stability and reliability of network communication.
[0129] In summary, this method has achieved significant technological progress in optimizing network communication resources for autonomous driving, not only improving the efficiency and accuracy of decision-making, but also enhancing the system's adaptability and stability. Attached Figure Description
[0130] Figure 1 This is a flowchart of the autonomous driving network communication resource optimization method based on multi-objective deep reinforcement learning provided in an embodiment of the present invention.
[0131] Figure 2 This is an application scenario diagram provided by an embodiment of the present invention. Detailed Implementation
[0132] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0133] Example 1: Optimization of Unmanned Vehicle Network for Urban Traffic Monitoring
[0134] 1) System Construction:
[0135] Deploy a network of driverless vehicles in a simulated urban traffic environment for real-time monitoring and data collection.
[0136] The MORL+DDPG algorithm module is used to optimize the communication resources of the autonomous vehicle network.
[0137] 2) Algorithm Implementation:
[0138] First, set the preference distribution D. ω The path P with balancing weight λ λ This reflects the different needs of real-time monitoring and data transmission.
[0139] Initialize the Critic and Actor networks, and prepare the target network and replay buffer.
[0140] 3) Cyclic execution and adjustment:
[0141] The process is repeated in a loop. In each loop, the autonomous vehicle makes action choices based on its current state and preferences ω, such as adjusting the route to avoid congestion and optimizing data transmission.
[0142] Collect environmental feedback, such as traffic flow and communication latency, along with corresponding reward values, to update the neural network.
[0143] 4) Optimization and Evaluation:
[0144] Gradually adjust λ to find the optimal balance between monitoring efficiency (AOI) and data throughput.
[0145] The communication efficiency of the entire network is optimized through continuous iteration and updating of neural network parameters.
[0146] Example 2: Communication Optimization of Unmanned Vehicles in Disaster Relief Scenarios
[0147] 1) System Deployment:
[0148] Deploy unmanned vehicle networks in simulated disaster environments for rescue operations and information gathering.
[0149] The MORL+DDPG algorithm is applied to adjust the allocation of communication resources for autonomous vehicles.
[0150] 2) Algorithm Application:
[0151] Define the preference distribution D ω The path P with balancing weight λ λ Considering the different needs for information timeliness and transmission stability in rescue missions.
[0152] Initialize the relevant neural network and prepare the replay buffer.
[0153] 3) Dynamic adjustment and execution:
[0154] In multiple loops, dynamic actions are adjusted based on real-time environmental conditions and preferences ω, such as optimizing rescue routes and communication strategies.
[0155] The environmental state and reward value are updated and fed back into the neural network for learning and adjustment.
[0156] 4) Performance evaluation and optimization:
[0157] Adjust λ to balance the real-time nature of information and the reliability of communication during rescue missions.
[0158] By continuously optimizing network parameters, the communication efficiency and response speed of unmanned vehicles in complex disaster environments can be improved.
[0159] In these two embodiments, through specific algorithm applications and network parameter optimization, the autonomous vehicle network can achieve efficient and reliable communication resource optimization for different application scenarios. This not only improves the performance of the autonomous vehicle network but also enhances its adaptability and effectiveness in different environments.
[0160] Application scenario diagram as follows Figure 2 As shown, the control plane consists of drones, responsible for issuing action commands, including which communication channels to use, communication power, and drone displacement. Note that the data collection plane, which is composed of drones, consists of drones and unmanned vehicles (UAVs). For simplicity, the equipment in the control plane is collectively referred to as UAVs. Figure 2 As shown, the unmanned vehicle collects data and uploads the environmental status to the drone. The drone then transmits the status to the deep reinforcement learning model, which outputs actions, including power, channel selection, and drone displacement. The drone then executes the corresponding commands.
[0161] The optimization objectives of this invention for AOI and throughput are:
[0162]
[0163] Where A N Let θ be the average AOI and λ be the average throughput. λ represents the communication channel used between the UAV and the unmanned vehicle, p represents the power, and L represents the UAV's position.
[0164] To address this optimization objective, we employ a deep reinforcement learning algorithm. We use a deep learning neural network for feature extraction, inputting the environmental state into the network and outputting the most appropriate action in the current state and its value. Then, we optimize the network parameters using the Bellman equation to maximize the reward function. The design of the reward function is aligned with the optimization objective; maximizing the reward function means maximizing the optimization objective.
[0165] like Figure 1 As shown, this invention provides a method for optimizing network communication resources for autonomous driving based on multi-objective deep reinforcement learning, comprising the following steps:
[0166] S101 uses the Deep Deterministic Policy Gradient (MORL+DDPG) algorithm based on multi-objective deep reinforcement learning, leveraging the data feature extraction capabilities of deep learning and the decision-making capabilities of reinforcement learning to achieve the optimal trade-off between the AOI of CCH and the throughput of TCH in autonomous vehicle networks.
[0167] S102 uses the GPI concept, which considers a set of policies that interact with the same environment. The goal is to select and improve the policy that is superior to all other policies. GPI will train policies that operate under different preferences, which is consistent with multi-objective learning problems and determines the optimal trade-off between different objectives.
[0168] The algorithm provided by this invention is as follows:
[0169] Step 1: Input: D ω P λ ;D ω : Distribution of preferences; P λ The path through which the balancing weight λ increases from 0 to 1;
[0170] Step 2: Initialize the neural network parameters;
[0171] Step 3: Execute M loops, with episodes 1 through M.
[0172] The method for initializing neural network parameters provided by this invention is as follows:
[0173] Step 2.1: Randomly initialize the Critic network and the Actor network, with network parameters θ respectively. Q and θ μ ;
[0174] Step 2.2: Initialize the target critic network Q′ and the target actor network μ′, with weights θ respectively. Q′ ←θ Q θ μ′ ←θ μ ;
[0175] Step 2.3: Initialize the replay buffer R and set λ = 0.
[0176] This invention provides the execution of M loops, with episodes from 1 to M:
[0177] Step 3.1: Initialize the stochastic process N for action exploration;
[0178] Step 3.2: Receive the initial observation state s0; where s t The state of the environment at time t is composed of the following:
[0179]
[0180] Where Lp[t] represents the position of the p-th UAV controller at time t, including three-dimensional coordinates; l n [t]: Represents the position of the nth unmanned vehicle in the data plane at time t, including three-dimensional coordinates; An[t]: Represents the "age" (AOI) of the information of the nth unmanned vehicle. The higher the age, the lower the timeliness of the information; If the information has not been transmitted after each time step, this value is incremented by 1; P represents the number of UAV controllers, and N represents the number of unmanned vehicles;
[0181] Step 3.3: Sample a preference ω from the sample Dω; ω: preference, representing the different importance of different physical quantities (AOI and throughput). The more important a physical quantity is, the higher this value will be.
[0182] Step 3.4: Execute T sub-loops, where t ranges from 0 to T:
[0183] Step 3.4.1: Select and execute an action based on the output of the deep reinforcement learning neural network, using the following formula:
[0184] a t =μ GPI (s t , w|θ μ )+N t
[0185] Where μ represents the output action of the Actor network, and the network parameters are θ. μ Input state S t And preference ω, N t This represents the "noise" at time t, i.e., a random value; this is because adding a random value to the action output by the Actor network increases uncertainty, thereby enhancing the network's exploratory ability in the early stages of training; the specific composition of the action vector is as follows:
[0186]
[0187] λn[t]=(λ c npk [t],λ d npk [t]), the two variables in parentheses have been mentioned earlier, p n The value represents power, ΔL represents the displacement of the UAV controller, p represents the serial number of the UAV controller, and n represents the serial number of the unmanned vehicle; P and N represent the total number of UAVs and unmanned vehicles, respectively.
[0188] Step 3.4.2: Instruct the autonomous vehicle to perform actions in the environment, and update the environment state to obtain the reward value rt; rt: represents the reward that the autonomous vehicle can obtain after performing a certain action in the environment;
[0189] Step 3.4.3: Transfer a sample (s) t a t r t s t+1 , ω) are stored in R;
[0190] Step 3.4.4: If a certain number of time steps are reached, update the neural network. The specific steps are as follows: First, sample N from the replay buffer R. τ There are 10 samples, and then θ is updated once using each sample. Q and θ μ ;
[0191] Step 3.4.5: Check whether the loops in Step 3 and Step 3.4 meet the termination conditions, i.e., whether t has reached T and whether episode has reached M. If they are met, stop; otherwise, continue the loop.
[0192] The present invention enables unmanned vehicles to perform actions in an environment while updating the environmental state and obtaining a reward value r. t ;r t This represents the reward that an autonomous vehicle can receive after performing a certain action in the environment, and it consists of the following:
[0193]
[0194] r c [t]: Measure the reward of CCH's AOI.
[0195] r d [t]: Rewards that measure the throughput of TCH.
[0196] A N [t-1]-A N [t] represents the decrease in AOI, A max [t] represents the maximum AOI value of the driverless car.
[0197] θ d N [t]: The average throughput of all driverless vehicles at time t.
[0198] θ d max [t]: The maximum throughput of all autonomous vehicles at time t.
[0199] θ d n[t]: Measures the throughput of TCH; the calculation formula is as follows:
[0200]
[0201]
[0202] Where b is the bandwidth; γ min ou Signal-to-noise ratio threshold: γ d nou : The signal-to-noise ratio of the nth unmanned vehicle in the data plane communicating with the oth drone in the control plane through the uth channel.
[0203] The signal-to-noise ratio calculation process provided by this invention;
[0204] First, calculate the probability P of using the communication link. LoS mo [t]: The probability that the m-th UGV and the o-th UAV controller use a line-of-sight (LoS) communication link in an air-to-ground (A2G) communication link.
[0205]
[0206] c1 and c2 are two constant coefficients, where, It represents the transmission angle from the UGV to the UAV controller; z represents the height of the UAV, and d represents the distance between the UGV and the UAV.
[0207] Clearly, the probability of a non-line-of-sight (NLoS) communication link is:
[0208]
[0209] Then calculate the distance d between the UAV control plane and the unmanned vehicle. np [t]: The distance between the data plane and the control plane, calculated using the following formula:
[0210]
[0211] Calculate path loss based on distance:
[0212] h LoS mp [t]: Path loss in Loss, calculated using the following formula:
[0213]
[0214] h NLoS mp [t]: Path loss in NLoS, calculated using the following formula:
[0215]
[0216] In these formulas, n represents the path loss exponent, fc represents the carrier frequency, and c represents the speed of light; μ LoS and μ NLoS The variables correspond to the average additional losses of the LoS and NLoS links, respectively; d represents the distance between the m-th unmanned vehicle and the p-th drone controller;
[0217] h mp [t]: The average link loss between the m-th UGV and the o-th UAV controller in the air-to-ground (A2G) communication link, as shown in the following formula:
[0218]
[0219] Then, to calculate the signal-to-noise ratio, we first introduce a channel-correlated binary variable:
[0220] λ c npk : Binary variable, 1 indicates that the nth UAV is associated with the unmanned vehicle controller p on the kth channel and performs CCH transmission, 0 indicates disconnection;
[0221] λ d npk : Binary variable, 1 indicates that the nth UAV is associated with the unmanned vehicle controller p on the kth channel and performs TCH transmission, 0 indicates disconnection;
[0222] λ n [t]: will λ c npk With λ d npk Encapsulate it into a vector, λ n =[λ c npk , λ d npk ];
[0223] p c n [t]: The power of the unmanned vehicle transmitting information on the CCH at time t.
[0224] p d n [t]: The power of the unmanned vehicle transmitting information at time t on the TCH.
[0225] p n [t]: p c n [t] and p d n [t] is encapsulated into a vector, p n [t] = [p] c n[t], p d n [t]].
[0226] γ c npk [t]: The signal-to-noise ratio of the CCH transmitted by the nth unmanned vehicle to the UAV controller p through the kth sub-channel, as shown in the following formula:
[0227]
[0228] γ d npk [t]: The signal-to-noise ratio of the TCH transmitted by the nth unmanned vehicle to the UAV controller p through the kth sub-channel, as shown in the following formula:
[0229]
[0230] The interference is:
[0231]
[0232] The symbols have been explained above, and their specific meanings can be inferred from the superscripts and subscripts. i represents the unmanned vehicle serial number, o represents the drone serial number, and k represents the channel serial number. If any different UxV (denoted as i) sends data packets to its associated drone controller using the same channel that n is using, interference will occur. This communication will interfere with n, regardless of whether it is using CCH or TCH.
[0233] p: Penalty. If the following constraints are violated, p will take a large negative value; otherwise, it will be 0.
[0234] Maximum speed constraint:
[0235]
[0236] Δx, Δy, and Δz represent displacements in three dimensions, and the total displacement cannot exceed the maximum velocity multiplied by time.
[0237] Minimum distance constraints between drones:
[0238]
[0239] This indicates that the coordinate distance between the q-th and i-th drones cannot be less than a threshold.
[0240] Channel constraints:
[0241]
[0242]
[0243]
[0244] K represents the number of channels, P represents the number of drone controllers, and N represents the number of unmanned vehicles.
[0245] The present invention provides a method for updating the neural network if a certain time step is reached. The specific steps are as follows: First, Nτ samples are collected from the replay buffer R. Then, θQ and θμ are updated once using each sample, as shown in the following formula:
[0246]
[0247]
[0248] L is the loss function, and θ is the parameter in the Critic network Q. Q Optimization is achieved by minimizing the mean square error of the multi-objective vectorized time difference error, while updating using vectorized Sarsa. θ μ′ and θ Q′ These are the parameters of the Actor and Critic target networks, respectively; R is the cumulative reward from the current step to the end of the episode; E represents the expectation from the distribution corresponding to the index; Ω represents the preference space; w t Represents the preference vector w t ={w c [t],w d [t]|w c [t]+w d [t] = 1}, where w c and w d These represent the weights of throughput in AOI and TCH, respectively;
[0249] This means that in programming, mean square optimization code can be used directly to update network parameters.
[0250] This invention employs the Deep Deterministic Policy Gradient (MORL+DDPG) algorithm, a multi-objective deep reinforcement learning approach, leveraging the data feature extraction capabilities of deep learning and the decision-making power of reinforcement learning to achieve the optimal trade-off between the AOI (Action of Interest) of CCH (Conditional Driver Learning) and the throughput of TCH (Traffic Response Learning) in autonomous vehicle networks. This invention utilizes the GPI (Geometric Policy Indicator) concept, which considers a set of policies interacting with the same environment, aiming to select and improve the policy that outperforms all others. GPI can train policies operating under different preferences, aligning with multi-objective learning problems and thus helping to determine the optimal trade-off between different objectives.
[0251] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.
[0252] This invention employs the Deep Deterministic Policy Gradient (MORL+DDPG) algorithm, a multi-objective deep reinforcement learning approach, leveraging the data feature extraction capabilities of deep learning and the decision-making power of reinforcement learning to achieve the optimal trade-off between the AOI (Action of Interest) of CCH (Conditional Driver Learning) and the throughput of TCH (Traffic Response Learning) in autonomous vehicle networks. This invention utilizes the GPI (Geometric Policy Indicator) concept, which considers a set of policies interacting with the same environment, aiming to select and improve the policy that outperforms all others. GPI can train policies operating under different preferences, aligning with multi-objective learning problems and thus helping to determine the optimal trade-off between different objectives.
[0253] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A system for optimizing network communication resources in autonomous driving, the system employing a multi-objective deep reinforcement learning framework, characterized in that, Includes the following modules: A Deep Deterministic Policy Gradient algorithm module based on multi-objective deep reinforcement learning combines the data feature extraction capability of deep learning with the decision-making capability of reinforcement learning to achieve the optimal trade-off between the Age of Information (AOI) of the control channel and the throughput of the transmission channel in autonomous vehicle networks. A General Policy Improvement strategy selection module is used to select and improve the best strategy from multiple strategies to achieve the best trade-off between different objectives; A preference distribution and trade-off path setting module for inputting the preference distribution D ω The path P with balancing weight λ λ , where λ increases from 0 to 1; A neural network parameter initialization module is provided to initialize the Critic network and the Actor network, as well as the corresponding target networks Q′ and µ′. A loop execution module is used to execute a predetermined number of M loops, each loop containing a series of sub-steps from receiving the initial observation state to updating the neural network parameters based on environmental feedback; An action exploration and state update module is used to initialize action exploration, receive the state, execute actions, update the environment state, and calculate the reward value r in each loop. t ; An action selection and execution module selects and executes actions based on the Actor network output and added random noise, while considering the current state s. t and preference ω; An environmental feedback processing module is used to handle environmental state updates and reward values r after the autonomous vehicle performs an action. t The calculation of the reward value takes into account the AOI of CCH and the throughput of TCH. A neural network update module is used to collect samples from the replay buffer R and update the neural network parameters after a certain time step is reached; A loop termination judgment module is used to check whether the loop termination condition is met and control whether the loop continues or stops.
2. A method for optimizing network communication resources for autonomous driving based on multi-objective deep reinforcement learning of the system described in claim 1, characterized in that, Includes the following steps: Step 1: Use the Deep Deterministic Policy Gradient algorithm of multi-objective deep reinforcement learning, which combines the data feature extraction capability of deep learning with the decision-making capability of reinforcement learning, to achieve the best trade-off between the information timeliness of the control channel and the throughput of the transmission channel in autonomous vehicle networks. Step 2: Using the concept of General Policy Improvement, select and improve strategies under different preferences in order to determine the best trade-off between different objectives.
3. The optimization method as described in claim 2, characterized in that, The algorithm includes the following steps: Distribution of input preferences D ω The path P with balancing weight λ λ , where λ increases from 0 to 1; Initialize the neural network parameters; Execute M loops, with episodes 1 to M.
4. The optimization method as described in claim 3, characterized in that, The method for initializing neural network parameters is as follows: Randomly initialize the Critic network and the Actor network, with network parameters θ respectively. Q and θ µ ; Initialize the target Critic network Q′ and the target Actor network µ′ with weights θ respectively. Q′ ←θ Q ,θ µ′ ←θ µ ; Initialize the replay buffer R and set λ = 0.
5. The optimization method as described in claim 3, characterized in that, The steps for executing M loops are as follows: Initialize the random process N for the action exploration; Receive initial observation state s0; From sample D ω A preference ω is sampled in the middle; Execute T sub-loops, where t ranges from 0 to T, including selecting and executing actions, updating the environment state, and calculating the reward value r. t The neural network is updated when a certain number of time steps are reached.
6. The optimization method as described in claim 5, characterized in that, The reward value r obtained during the process of performing actions and updating the environmental state t Its composition is as follows: r c [t]: Measure the reward for CCH's AOI; r d [t]: Rewards that measure the throughput of TCH; This includes calculating the AOI reduction value, the average throughput of the autonomous vehicle, and the maximum throughput.
7. The optimization method as described in claim 5, characterized in that, The calculation process for signal-to-noise ratio includes: Calculate the probability P of the line-of-sight communication link between the unmanned vehicle and the UAV controller in the air-to-ground communication link. LoS mo [t] is the probability of non-line-of-sight communication links; Calculate the distance d between the autonomous vehicle and the drone controller. np [t] and the corresponding path loss; To calculate the signal-to-noise ratio, we first introduce a channel-correlated binary variable: λ c npk : Binary variable, 1 indicates that the nth UAV is associated with the unmanned vehicle controller p on the kth channel and performs CCH transmission, 0 indicates disconnection; λ d npk : Binary variable, 1 indicates that the nth UAV is associated with the unmanned vehicle controller p on the kth channel and performs TCH transmission, 0 indicates disconnection; λ n [t]: will λ c npk With λ d npk Encapsulate it into a vector, λ n =[λ c npk , λ d npk ]; γ c npk [t]: The signal-to-noise ratio of the CCH transmitted by the nth unmanned vehicle to the UAV controller p through the kth sub-channel; γ d npk [t]: The signal-to-noise ratio of the TCH transmitted by the nth unmanned vehicle to the unmanned vehicle controller p through the kth sub-channel.
8. The optimization method as described in claim 5, characterized in that, The step of updating the neural network when a certain time step is reached includes: N is sampled from the replay buffer R. τ One sample; Update the parameters θ in the Critic network Q once for each sample. Q and the parameters θ in the Actor network µ µ ; The loss function L is minimized using the mean square error optimization method, thereby optimizing the loss function by minimizing the mean square error of the multi-objective vectorized time difference error, while updating using vectorized Sarsa.
