Age minimization method for data distribution information of Internet of Vehicles

Abstract

The invention relates to an Internet of Vehicles data distribution information age minimization method, which obtains a near-end optimization model through training in advance, and executes information age updating and unmanned aerial vehicle track selection through a base station and the model in the Internet of Vehicles, so that the calculation and decision burden of an unmanned aerial vehicle can be reduced, and the unmanned aerial vehicle is enabled to mainly focus on physical movement and communication tasks; therefore, the energy efficiency and the task execution effect of the unmanned aerial vehicle are improved, and the base station, as a central control node, can collect and process the state information of the whole network so as to make a global optimal decision.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation systems and vehicle networking communication technology, and in particular to a method for minimizing the age of vehicle networking data distribution information. Background Art

[0002] In the intelligent transportation system, vehicles transmit data through wireless communication networks, and base stations collect and process vehicle information. However, when the base station is located in a remote location in the city, the long distance transmission between the base station and the vehicle in the city will affect the quality of the data packet, which in turn affects the base station's ability to decode and recover data, and ultimately affects the decision-making of the urban traffic management system.

[0003] The flexibility and reliable line-of-sight characteristics of unmanned aerial vehicles (UAVs) enable them to act as mobile relay nodes to assist base stations in collecting vehicle information. UAVs can dynamically adjust their positions and communication strategies while moving to optimize the efficiency of data transmission. However, in this scenario, the freshness of packet information (i.e., age of information, AoI) is a key indicator because it is directly related to the timeliness and accuracy of urban traffic decision-making.

[0004] So, when drones are used as mobile relay nodes in the Internet of Vehicles environment, how should the data packets of moving vehicles be scheduled to ensure that their information age is the minimum? When the information age is the minimum, how should the drone trajectory be planned? There is no direct answer to this question in the existing technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to achieve the optimal solution for drone-assisted vehicle data packet scheduling and drone trajectory planning in a vehicle networking environment. In order to overcome the defects of the above-mentioned prior art, the present invention provides a method for minimizing the age of vehicle networking data distribution information.

[0006] The present invention provides a method for minimizing the age of Internet of Vehicles data distribution information, comprising the following steps:

[0007] S1: Based on the Internet of Vehicles with drones as mobile relay nodes, an optimization model is established with the goal of minimizing the average information age of all vehicles in the Internet of Vehicles and the energy consumption of the drones;

[0008] S2: obtaining a Markov process of interaction between the drone and the constrained environment where the Internet of Vehicles is located based on the optimization model, and specifying the number of cycles of the Markov process;

[0009] S3: Based on the Markov process, the UAV is trained by a proximal strategy optimization algorithm and reaches the number of cycles to obtain a proximal optimization model;

[0010] S4: In the base station, the UAV is steered to operate using the proximal optimization model to minimize the average information age of all vehicles and the energy consumption of the UAV.

[0011] The method for minimizing the information age of vehicle network data distribution disclosed in the present invention obtains a proximal optimization model through advance training, and uses the model to perform information age update and drone trajectory selection, which can reduce the calculation and decision-making burden of the drone, so that it can focus mainly on physical movement and communication tasks, thereby improving the energy efficiency and task execution effect of the drone. Moreover, the base station, as a central control node, can collect and process the status information of the entire network to make a global optimal decision. In addition, by optimizing the information age update and drone movement path through the proximal optimization model, network resources and energy can be effectively utilized, unnecessary communication and mobile overhead can be reduced, and the energy efficiency of the system can be improved. It is not only suitable for vehicle network data distribution in intelligent transportation systems, but also can be applied to multiple fields such as drone group collaborative tasks, emergency rescue and disaster management, and has broad application prospects and commercial value.

[0012] In a possible implementation, step S1 includes the following steps:

[0013] S11: establishing a communication network structure in which a base station is associated with a drone, and one drone is associated with multiple vehicles, thereby obtaining the vehicle network;

[0014] S12: Divide the designated time into multiple time slots, and obtain the connection constraints between the UAV and each vehicle in each time slot based on the communication fact that the change of the position of the UAV will cause the change of its coverage range;

[0015] S13: Obtaining a channel gain between the UAV and each vehicle in each time slot based on the communication channel characteristics of the Internet of Vehicles;

[0016] S14: obtaining the data transmission rate between the UAV and each vehicle in each time slot based on the channel gain between the UAV and each vehicle in each time slot obtained in step S13;

[0017] S15: obtaining the propulsion energy consumption of the drone in each time slot according to the energy consumption algorithm of the drone propulsion process, and obtaining the total energy consumption of the drone according to the fact that the drone hovers in a scheduling operation;

[0018] S16: setting the information sampling behavior of all the vehicles to follow the sampling update of all the time slots, and using the minimum data amount of the state update to represent the minimum data amount required for a successful state update of each vehicle, and then obtaining the data amount of the state update of each vehicle in each time slot based on the data transmission rate obtained in step S14, and obtaining the information age update algorithm of each vehicle according to the comparison between the data amount of the state update of each vehicle in each time slot and the minimum data amount of the state update;

[0019] S17: based on the total energy consumption of the drone in each time slot obtained in step S15, obtain the average energy consumption of the drone at the specified time, based on the information age update algorithm obtained in step S16, obtain the average information age of all the vehicles at the specified time, and take the minimum value of the sum of the average energy consumption and the average information age as the target;

[0020] S18: specifying a working area of ​​the drone, using the working area and the connection constraints obtained in step S12 as constraints corresponding to the target, and obtaining the optimization model accordingly;

[0021] Compared with the existing technology, the above technical solution can provide constraints and goals for optimizing information age and reducing energy consumption based on the display constraints of the Internet of Vehicles, thereby providing a guarantee for the establishment of the Markov process and the training of the proximal optimization model based on it.

[0022] In a possible implementation manner, the formula for the connection constraint between the drone and each vehicle obtained in step S12 is:

[0023]

[0024] In the formula,

[0025] i=0,1,2,…I-1 represents the communication between the drone and the vehicle, and I is the total number of vehicles;

[0026] The specified fixed time is divided into N time slots, n=1, 2, ..., n represents the number of time slots;

[0027] is a binary variable, represents that the i-th vehicle is dispatched by the drone, otherwise

[0028] Compared with the prior art, the above technical solution can take into account the mobility of the UAV and provide a constraint for UAV vehicle scheduling based on the fact that changes in the position of the UAV will lead to changes in its coverage range.

[0029] In a possible implementation manner, the formula for the channel gain obtained in step S13 is:

[0030]

[0031] In the formula,

[0032] represents the channel gain between the UAV and vehicle i in time slot n;

[0033] represents the position of vehicle i at time n;

[0034] A two-dimensional coordinate representing the drone;

[0035] q BS =(x BS ,y BS ) represents the location of the base station;

[0036] h0 represents the channel gain at the reference distance d0 = 1m;

[0037] H represents the flight altitude of the drone;

[0038] d i,u (n) represents the distance between vehicle i and the UAV at time slot n;

[0039] This scheme provides a fast and accurate way to calculate the channel gain between the UAV and vehicle i in each time slot, thereby achieving accurate acquisition of information gain.

[0040] In a possible implementation manner, the data transmission rate obtained in step S14 is calculated as follows:

[0041]

[0042] In the formula,

[0043] represents the data transmission rate between the UAV and vehicle i in time slot n;

[0044] B represents the communication bandwidth;

[0045] σ 2 stands for additive white Gaussian noise;

[0046] P represents the transmission power of the vehicle;

[0047] This scheme can achieve accurate acquisition of the data transmission rate between the UAV and vehicle i in each time slot, thereby providing a guarantee for the calculation of the data volume of the status update of each vehicle in each time slot.

[0048] In a possible implementation, in step S15, the energy consumption algorithm of the UAV propulsion process is calculated as follows:

[0049]

[0050] In the formula,

[0051] P(v n ) represents the propulsion energy consumption of the UAV in time slot n;

[0052] P0 represents the blade power of the UAV when it is hovering;

[0053] P1 represents the induced power of the UAV when it is hovering;

[0054] v n represents the speed of the drone at time slot n;

[0055] U tip represents the tip speed of the rotor blade of the UAV;

[0056] v0 represents the average rotor induced speed of the UAV when it is hovering;

[0057] d represents the fuselage drag ratio of the UAV;

[0058] ρ represents the air density;

[0059] s represents the rotor solidity of the UAV;

[0060] A represents the rotor disc area of ​​the UAV;

[0061] The total energy consumption of the UAV is calculated as:

[0062]

[0063] P h =P0+P1,

[0064] In the formula,

[0065] T s represents the flight time of the drone;

[0066] T c Represents the hovering time of the drone;

[0067] P h Represents the power consumption of the drone in a hovering state;

[0068] This solution is based on the fact that the main energy consumption of UAVs is composed of propulsion, hovering and communication. Combined with the fact that the energy consumed by communication is the smallest, the communication energy consumption is ignored, thereby reducing the complexity of energy consumption calculation while ensuring accuracy.

[0069] In a possible implementation manner, in step S16, the data amount of the status update of each vehicle in each time slot is calculated as follows:

[0070]

[0071] In the formula,

[0072] The amount of data for the status update of vehicle i in time slot n;

[0073] δ n represents the duration of time slot n;

[0074] The formula of the information age update algorithm of each of the vehicles is:

[0075]

[0076] In the formula,

[0077] represents the information age of vehicle i in time slot n;

[0078] The minimum amount of data representing the state update of vehicle i;

[0079] Represents a collection of vehicles;

[0080] This solution introduces the minimum data amount for the status update of vehicle i and provides an update strategy based on it, thereby ensuring the orderly update of the information age.

[0081] In a possible implementation, the Markov process obtained in step S2 is expressed as follows:

[0082] The drone is used as an intelligent agent, and the intelligent agent interacts cyclically with the constrained environment where the Internet of Vehicles is located. The number of cyclic interactions is the number of cycles of the Markov process specified in step S2, and the number of steps performed in each cycle is the same;

[0083] For each step, the agent observes the constraint environment at the beginning of the step, obtains the observation state of the current step, and based on the observation state and the agent's own strategy, the agent selects a mixed action from the mixed action space in the step;

[0084] The hybrid action space consists of a discrete action space and a continuous action space;

[0085] The agent selects a mixed action from the mixed action space in this step, which means that the agent selects a discrete action from the discrete action space for the vehicle whose information age needs to be updated, and selects two continuous actions for the moving speed and direction of the drone in the continuous action space;

[0086] After the agent performs the mixed action in this step, the state of the constraint environment changes, and the agent obtains a corresponding reward value, so this step ends and proceeds to the next step;

[0087] This solution realizes the conversion of the optimization model into a Markov process and provides a prototype of the model for training drones, which can train the intelligent agent through the proximal optimization algorithm, thus providing the prerequisite for obtaining the proximal optimization model.

[0088] In a possible implementation manner, step S3 includes the following steps:

[0089] S31: According to the requirements of the proximal strategy optimization algorithm, a neural network including a performer network and a critic network is used, all parameters of the neural network are initialized, and the experience replay pool of the agent is initialized, and the number of cycles in the training phase is set to the number of cycles of the Markov process, and the number of steps in each cycle of the training phase is stipulated to be the same as the number of steps set by the Markov process;

[0090] The performer network is composed of a discrete action network module for outputting discrete actions and a continuous action network module for outputting continuous actions;

[0091] S32: In the current step, the agent starts observing the constraint environment at the current step and obtains the current observation state of each time slot, and the current observation state is expressed as:

[0092]

[0093] In the formula,

[0094] State represents the current observation state;

[0095] represents a set of time slots;

[0096] S33: inputting the current observation state into the performer network and the critic network simultaneously;

[0097] The discrete action network module of the performer network outputs the dispatch vehicle index of the agent in each time slot, and sends a state update message to the index representative vehicle in turn according to the dispatch vehicle index of each time slot, wherein the state update message contains the information age of the vehicle involved, and resets the information age of the vehicle involved at the agent to δ n , while the information age of vehicles that do not send status update messages increases by δ n , the discrete action of this step is updated;

[0098] The continuous action network module of the performer network calculates the current output value of the continuous action in each time slot, and the current output value includes the moving direction and the moving speed of the agent. The agent executes in sequence according to the current output value, and the continuous action of this step is updated.

[0099] After the discrete actions and continuous actions of this step are updated, the agent obtains a reward value based on the constraint environment update of this step and obtains the next observation state of each time slot based on the reward and the output of the critic network;

[0100] S34: storing the current step state, action, reward, and next step observation state of each time slot in the experience replay pool;

[0101] S35: Determine whether a cycle has been completed. If not, take the next observation state of each time slot as the current observation state of each time slot and return to execute step S32;

[0102] If yes, proceed to the next step;

[0103] S36: Determine whether the current number of cycles has reached the number of cycles or the storage limit of the experience replay pool. If at least one of the above is satisfied, obtain the proximal optimization model by performing several time-based iterations based on all data collected by the experience replay pool and embed the proximal optimization model into the base station. Otherwise, take the next observation state of each time slot as the current observation state of each time slot and return to execute step S32.

[0104] This solution can optimize in both discrete action space and continuous action space, making information update and UAV movement path planning more efficient and accurate. This dual-space optimization method can significantly reduce information age and energy consumption and improve overall system performance.

[0105] In a possible implementation, in step S33, the reward value is calculated as follows:

[0106]

[0107] In the formula,

[0108] The first item is the average age of all vehicles at the end of time slot n;

[0109] The second item is the energy consumption of the UAV in time slot n;

[0110] ξ and ψ are the non-negative weighted information age and energy consumption components, respectively;

[0111] This solution provides a clear reward mechanism by rewarding the agent based on the target update, which guarantees the efficiency of model building and ensures the accuracy and reliability of the target of optimizing the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 A flow chart of a method for minimizing the age of Internet of Vehicles data distribution information disclosed in an embodiment of the present invention;

[0113] Figure 2 is a flow chart of step S1 disclosed in an embodiment of the present invention;

[0114] Figure 3 is a flow chart of step S3 disclosed in an embodiment of the present invention;

[0115] Figure 4 Schematic diagram of the neural network structure involved in the embodiment of the present invention. DETAILED DESCRIPTION

[0116] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0117] In the embodiments of the present application, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0118] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0119] See also Figure 1 to Figure 3 As shown, the embodiment of the present application discloses a method for minimizing the age of vehicle network data distribution information, see Figure 1 As shown, the method comprises the following steps:

[0120] S1: Based on the Internet of Vehicles with drones as mobile relay nodes, an optimization model is established with the goal of minimizing the average information age of all vehicles in the Internet of Vehicles and the energy consumption of drones.

[0121] See also Figure 2 As shown, in this embodiment, step S1 includes the following steps:

[0122] S11: Establish a communication network structure in which a base station is associated with a drone, and a drone is associated with multiple vehicles, thereby obtaining a vehicle network;

[0123] Of course, the specific Internet of Vehicles can be composed of a communication network structure in parallel or in parallel. As a specific example, assume that the Internet of Vehicles has a main base station, one vehicle, and one drone. The communication between the drone and the base station or mobile device is achieved through a dedicated frequency band to ensure the reliability and stability of the communication.

[0124] S12: Divide the designated time into multiple time slots, and obtain the connection constraints between the UAV and each vehicle in each time slot based on the communication fact that the change of the UAV position will cause the change of its coverage range;

[0125] In this embodiment, the formula for the connection constraint between the UAV and each vehicle obtained in step S12 is:

[0126]

[0127] In the formula,

[0128] i=0,1,2,…I-1 represents the communication between drones and vehicles, and I is the total number of vehicles;

[0129] The specified time is divided into N time slots, n = 1, 2, ..., N represents the number of time slots;

[0130] is a binary variable, represents that the i-th vehicle is dispatched by the drone, otherwise

[0131] S13: Obtain the channel gain between the UAV and each vehicle in each time slot based on the communication channel characteristics of the Internet of Vehicles;

[0132] In this embodiment, the formula for the channel gain obtained in step S13 is:

[0133]

[0134] In the formula,

[0135] represents the channel gain between UAV and vehicle i in time slot n;

[0136] represents the position of vehicle i at time n;

[0137] Represents the two-dimensional coordinates of the drone;

[0138] q BS =(x BS ,y BS ) represents the location of the base station;

[0139] h0 represents the channel gain at the reference distance d0 = 1m;

[0140] H represents the flight altitude of the drone;

[0141] d i,u (n) represents the distance between vehicle i and the UAV at time slot n.

[0142] S14: obtaining the data transmission rate between the UAV and each vehicle in each time slot based on the channel gain between the UAV and each vehicle in each time slot obtained in step S13;

[0143] In this embodiment, the data transmission rate obtained in step S14 is calculated as:

[0144]

[0145] In the formula,

[0146] represents the data transmission rate between UAV and vehicle i in time slot n;

[0147] B represents the communication bandwidth;

[0148] σ 2 stands for additive white Gaussian noise;

[0149] P represents the transmission power of the vehicle.

[0150] S15: obtaining the propulsion energy consumption of the UAV in each time slot according to the energy consumption algorithm of the UAV propulsion process, and obtaining the total energy consumption of the UAV according to the fact that the UAV is hovering by the scheduling operation;

[0151] In step S15 of this embodiment, the energy consumption algorithm of the drone propulsion process is calculated as follows:

[0152]

[0153] In the formula,

[0154] P(v n ) represents the propulsion energy consumption of the UAV in time slot n;

[0155] P0 represents the blade power when the UAV is hovering;

[0156] P1 represents the induced power of the drone when it is hovering;

[0157] v n represents the speed of the drone at time slot n;

[0158] U tip represents the tip speed of the UAV's rotor blades;

[0159] v0 represents the average rotor induced speed of the UAV when it is hovering;

[0160] d represents the drag ratio of the drone’s fuselage;

[0161] ρ represents the air density;

[0162] s represents the rotor solidity of the UAV;

[0163] A represents the rotor disc area of ​​the drone.

[0164] At the same time, in step S15, the total energy consumption of the drone is calculated as:

[0165]

[0166] P h =P0+P1,

[0167] In the formula,

[0168] T s Represents the flight time of the drone;

[0169] T c Represents the drone's hovering time;

[0170] P h Represents the power consumption of the drone in hovering state.

[0171] S16: The information sampling behavior of all vehicles is set to follow the sampling update of all time slots, and the minimum data amount of the state update represents the minimum data amount required for a successful state update of each vehicle, and then the data amount of the state update of each vehicle in each time slot is obtained based on the data transmission rate obtained in step S14, and the information age update algorithm of each vehicle is obtained according to the comparison between the data amount of the state update of each vehicle in each time slot and the minimum data amount of the state update;

[0172] Specifically, in step S16 of this embodiment, the formula for the data amount of each vehicle status update in each time slot is:

[0173]

[0174] In the formula,

[0175] The amount of data for the status update of vehicle i in time slot n;

[0176] δ n Represents the duration of time slot n.

[0177] At the same time, in step S16, the formula of the information age update algorithm of each vehicle is:

[0178]

[0179] In the formula,

[0180] represents the information age of vehicle i in time slot n;

[0181] Represents the minimum data amount for the status update of vehicle i;

[0182] Represents a collection of vehicles.

[0183] S17: based on the total energy consumption of the drones in each time slot obtained in step S15, the average energy consumption of the drones at the specified time is obtained, based on the information age update algorithm obtained in step S16, the average information age of all vehicles at the specified time is obtained, and the minimum value of the sum of the average energy consumption and the average information age is taken as the target;

[0184] Specifically, in step S17 of this embodiment, the average information age of all vehicles in the Internet of Vehicles and the average energy consumption of drones at a specified time can be expressed as:

[0185]

[0186] In the formula,

[0187] ξ and η are positive values, indicating the weight of the corresponding item in the optimization objective.

[0188] S18: specifying the working area of ​​the UAV, taking the working area and the connection constraints obtained in step S12 as constraints corresponding to the target, and obtaining the optimization model accordingly;

[0189] Based on this, the specific expression of the obtained optimization model is as follows:

[0190]

[0191] In the formula,

[0192] X max and Y max Represents the maximum plane area of ​​drone activity.

[0193] S2: Based on the optimization model, a Markov process of interaction between the UAV and the constrained environment of the Internet of Vehicles is obtained, and the number of cycles of the Markov process is specified.

[0194] In this embodiment, step S2 is actually to convert the solution process of the above optimization model into a Markov process. The obtained Markov process is expressed as follows:

[0195] The drone is used as an intelligent agent, and the intelligent agent is composed of a corresponding experience replay pool. The intelligent agent interacts cyclically with the constraint environment where the Internet of Vehicles is located. The number of cyclic interactions is the number of cycles of the Markov process specified in step S2, and the number of steps performed in each cycle is the same;

[0196] For each step, the agent observes the constraint environment at the beginning of this step and obtains the observation state of the current step. Based on this observation state and the agent's own strategy, the agent selects a mixed action from the mixed action space in this step;

[0197] The hybrid action space consists of discrete action space and continuous action space;

[0198] The agent selects a mixed action from the mixed action space in this step, which means that the agent selects a discrete action from the discrete action space for the vehicle whose information age needs to be updated, and selects two continuous actions for the speed and direction of the drone in the continuous action space;

[0199] After the agent performs the mixed action in this step, the state of the constraint environment changes, and the agent obtains the corresponding reward value. This step ends and proceeds to the next step.

[0200] S3: Based on the Markov process, the UAV is trained by the proximal strategy optimization algorithm and reaches the number of cycles to obtain the proximal optimization model;

[0201] The Proximal Policy Optimization (PPO) algorithm is a policy-based method that uses two deep networks. The corresponding network structure diagram is shown in Figure 4 As shown, the first network is the actor network, which is the object of training and maintains the current strategy π θ The Performer network takes the state as input and outputs a list of probabilities corresponding to each action in the action space. These probabilities form a distribution based on which actions can be sampled. The second network is the Critic network, which is used to select the state value V according to its estimate. θ(a|s) to “criticize” the current strategy of the performer network. After performing an action, the critic network evaluates whether the action taken increases or decreases the expected feedback based on the current state.

[0202] See also Figure 3 As shown, in this embodiment, step S3 includes the following steps:

[0203] S31: According to the requirements of the proximal strategy optimization algorithm, a neural network containing a performer network and a critic network is used, all parameters of the neural network (i.e., network weights) are initialized, and the experience replay pool of the agent is initialized, and the number of cycles in the training phase is set to the number of cycles of the Markov process, and the number of steps in each cycle of the training phase is stipulated to be the same as the number of steps set by the Markov process;

[0204] The performer network consists of a discrete action network module for outputting discrete actions and a continuous action network module for outputting continuous actions;

[0205] As a specific example, the number of loops is set to MaxIter, each loop has N steps, and each step is an Episode.

[0206] S32: In the current step (Episode), the agent observes the constraint environment at the beginning of the current step and obtains the current observation state of each time slot. The current observation state is expressed as:

[0207]

[0208] In the formula,

[0209] State represents the current observation state;

[0210] Represents a collection of time slots.

[0211] S33: Input the current observation state into the performer network and the critic network at the same time;

[0212] The discrete action network module of the performer network outputs the agent dispatch vehicle index of each time slot, and sends a state update message to the index representative vehicle in turn according to the dispatch vehicle index of each time slot. The state update message contains the information age of the vehicle involved, and resets the information age of the vehicle involved at the agent to δ n , while the information age of vehicles that do not send status update messages increases by δ n , the discrete action of this step is updated;

[0213] The continuous action network module of the performer network calculates the current output value of the continuous action in each time slot. The current output value includes the moving direction and moving speed of the agent. The agent executes in sequence according to the current output value. The continuous action of this step is updated at this point.

[0214] After the discrete actions and continuous actions of this step are updated, the agent obtains the reward value based on the constraint environment of this step and obtains the next observation state of each time slot based on the reward and the output of the critic network;

[0215] In step S33 of this embodiment, the calculation formula of the reward value is:

[0216]

[0217] In the formula,

[0218] The first item is the average age of all vehicles at the end of time slot n;

[0219] The second item is the energy consumption of the UAV in time slot n;

[0220] ξ and ψ are the non-negative weighted information age and energy consumption components, respectively.

[0221] See also Figure 4 As shown, based on the proximal strategy optimization algorithm, by observing the probability π of the current strategy θ (a(n0)|s(n0)), and get the mixed action a(n) selected by the agent in the current step. Then the agent performs the mixed action a(n) in the current step. After the mixed action is completed, that is, the drone moves to a new position, the information transmission of vehicle i is completed, the constraint environment is updated, and the agent obtains the reward value r(n) in the current step. Then the agent observes the environment at the end of the current step (that is, before the next Episode starts) and obtains the new observation state s(n+1).

[0222] The Actor-Critic structure consists of a Critic Network and an Actor Network. Figure 4 As shown, the hidden layers of the performer network and the critic network consist of two fully connected layers, and the input layer and the two fully connected layers have 256, 128, and 64 neurons respectively. In this embodiment, the strategy and value function are represented by a deep neural network. Therefore, the discrete action strategy, continuous action strategy, and action value function are represented as and V θ (s(n)).

[0223] S34: store the current step state, action, reward, and next observation state of each time slot in the experience replay pool;

[0224] S35: Determine whether a cycle has been completed. If not, take the next observation state of each time slot as the current observation state of each time slot and return to execute step S32;

[0225] If yes, proceed to the next step;

[0226] S36: Determine whether the current number of cycles has reached the number of cycles or the storage limit of the experience replay pool. If at least one of them is met, the proximal optimization model is obtained by performing several time-based iterations on all data collected from the experience replay pool and the proximal optimization model is embedded in the base station. Otherwise, the next observation state of each time slot is used as the current observation state of each time slot and the step S32 is executed again.

[0227] As a specific example, the storage limit of the experience replay pool can be set to the condition that the number of steps n at this time satisfies the condition that it is divisible by the storage volume L of the experience replay pool, that is, the operation of step S33 is performed once every L time steps. Then, the collected data is iterated several times. In each iteration, the algorithm randomly collects small batch samples of data in the experience replay pool. Each batch of samples contains B experiences, which are determined by the latest version of the strategy. Generate. Using the collected rewards and current state value The generalized advantage estimation (GAE) algorithm can be used to generate advantage estimates For each batch, the collected states are passed into the performer and critic networks to establish new strategies and corresponding criticism target values ​​respectively.

[0228] In the environmental interaction phase of step S33, the state obtained by the agent is sent to the performer network and the critic network. Then, the discrete action network determines the discrete decision of dispatching the vehicle, and the continuous action network determines the direction and speed of the drone's flight. The two are combined to form a hybrid action and feedback to the environment.

[0229] The experience storage pool (denoted by D) uses S, A and R to represent the state, action and reward collected in the experience storage pool D. After each reward feedback, D = (s(n), a(n), r(n), s(n+1)) will be stored in the experience pool D. Once enough samples are collected, the algorithm enters the learning phase.

[0230] First, calculate the odds estimate, which is as follows:

[0231]

[0232] In the formula,

[0233] N' represents the stop time of the current learning iteration;

[0234] γ∈[0,1] represents the discount factor, in this example, γ=0.99;

[0235] λ p Represents the variance in the model.

[0236] Then a small batch of experiences is extracted from the experience pool memory. For each batch of experiences, the states in the small batch of experiences are passed to the performer network and the critic network to generate new policies π respectively. θ And the target advantage value, from which the loss function is derived. Among them, the loss function of discrete actions is:

[0237]

[0238] In the formula,

[0239] represents the ratio of the discrete action strategy to the old discrete action strategy, and

[0240]

[0241] represents the advantage estimate of the corresponding empirical sample in batch i;

[0242] The Clip function is defined as:

[0243]

[0244] The Clip function is added to ensure that the update of each importance sampling is not too large. Similarly, the loss function of the continuous action strategy can be obtained as:

[0245]

[0246] In the formula,

[0247] Represents the ratio of the old continuous action strategy to the new continuous action strategy.

[0248] The loss function of the critic network is:

[0249]

[0250] This example uses the Adam optimizer. In addition, the discount factor γ = 0.99, the learning rate of the performer network α = 0.01, the learning rate of the critic network α = 0.01, and the ∈ clip =0.1.

[0251] S4: In the base station, the UAV is steered to operate with a proximal optimization model to minimize the average information age of all vehicles and the energy consumption of the UAV.

[0252] Compared with the prior art, the method for minimizing the information age of Internet of Vehicles data distribution disclosed in the embodiment of the present application, compared with updating the information age at the drone, updates the information age at the base station, which can reduce the calculation and decision-making burden of the drone, so that it can focus mainly on physical movement and communication tasks, thereby improving the energy efficiency and task execution effect of the drone. As a central control node, the base station can collect and process the status information of the entire network to make the global optimal decision. By optimizing the information age update and the drone movement path, network resources and energy can be effectively utilized, unnecessary communication and movement overhead can be reduced, and the energy efficiency of the system can be improved. In addition, simultaneous optimization in discrete action space and continuous action space makes information update and drone movement path planning more efficient and accurate. This dual-space optimization method can significantly reduce the age of information (AoI) and energy consumption, and improve the overall system performance.

[0253] The technical solution of the embodiment of the present invention is not only applicable to vehicle network data distribution in intelligent transportation systems, but can also be applied to multiple fields such as drone swarm collaboration tasks, emergency rescue and disaster management, and has broad application prospects and commercial value.

[0254] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0255] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0256] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for minimizing the age of vehicle network data distribution information, characterized in that: The steps include: S1: Based on the Internet of Vehicles with drones as mobile relay nodes, an optimization model is established with the goal of minimizing the average information age of all vehicles in the Internet of Vehicles and the energy consumption of the drones; S2: obtaining a Markov process of interaction between the drone and the constrained environment where the Internet of Vehicles is located based on the optimization model, and specifying the number of cycles of the Markov process; S3: Based on the Markov process, the UAV is trained by a proximal strategy optimization algorithm and reaches the number of cycles to obtain a proximal optimization model; S4: In the base station, the UAV is steered using the proximal optimization model to minimize the average information age of all vehicles and the energy consumption of the UAV.

2. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 1, characterized in that: The step S1 comprises the following steps: S11: establishing a communication network structure in which a base station is associated with a drone, and one drone is associated with multiple vehicles, thereby obtaining the vehicle network; S12: Divide the designated time into multiple time slots, and obtain the connection constraints between the drone and each vehicle in each time slot based on the communication fact that the drone's position changes and its coverage changes; S13: Obtaining a channel gain between the UAV and each vehicle in each time slot based on the communication channel characteristics of the Internet of Vehicles; S14: obtaining the data transmission rate between the UAV and each vehicle in each time slot based on the channel gain between the UAV and each vehicle in each time slot obtained in step S13; S15: obtaining the propulsion energy consumption of the drone in each time slot according to the energy consumption algorithm of the drone propulsion process, and obtaining the total energy consumption of the drone according to the fact that the drone is hovering in a scheduling operation; S16: setting the information sampling behavior of all the vehicles to follow the sampling update of all the time slots, and using the minimum data amount of the state update to represent the minimum data amount required for a successful state update of each vehicle, and then obtaining the data amount of the state update of each vehicle in each time slot based on the data transmission rate obtained in step S14, and obtaining the information age update algorithm of each vehicle according to the comparison between the data amount of the state update of each vehicle in each time slot and the minimum data amount of the state update; S17: based on the total energy consumption of the drone in each time slot obtained in step S15, obtain the average energy consumption of the drone at the specified time, based on the information age update algorithm obtained in step S16, obtain the average information age of all the vehicles at the specified time, and take the minimum value of the sum of the average energy consumption and the average information age as the target; S18: Specify the working area of ​​the UAV, use the working area and the connection constraints obtained in step S12 as constraints corresponding to the target, and obtain the optimization model accordingly.

3. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 3, characterized in that: The formula for the connection constraint between the drone and each vehicle obtained in step S12 is: In the formula, i=0,1,2,…I-1 represents the communication between the drone and the vehicle, and I is the total number of vehicles; The specified fixed time is divided into N time slots, n=1, 2, ..., N represents the number of time slots; is a binary variable, represents that the i-th vehicle is dispatched by the drone, otherwise 4. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 3, characterized in that: The formula of the channel gain obtained in step S13 is: In the formula, represents the channel gain between the UAV and vehicle i in time slot n; represents the position of vehicle i at time n; A two-dimensional coordinate representing the drone; q BS =(x BS ,y BS ) represents the location of the base station; h0 represents the channel gain at the reference distance d0 = 1m; H represents the flight altitude of the drone; d i,u (n) represents the distance between vehicle i and the UAV at time slot n.

5. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 4, characterized in that: The formula for the data transmission rate obtained in step S14 is: In the formula, represents the data transmission rate between the UAV and vehicle i in time slot n; B represents the communication bandwidth; σ 2 stands for additive white Gaussian noise; P represents the transmission power of the vehicle.

6. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 5, characterized in that: In step S15, the energy consumption algorithm of the UAV propulsion process is calculated as follows: In the formula, P(v n ) represents the propulsion energy consumption of the UAV in time slot n; P0 represents the blade power of the UAV when it is hovering; P1 represents the induced power of the UAV when it is hovering; v n represents the speed of the UAV at time slot n; U tip represents the tip speed of the rotor blade of the UAV; v0 represents the average rotor induced speed of the UAV when it is hovering; d represents the fuselage drag ratio of the UAV; ρ represents the air density; s represents the rotor solidity of the UAV; A represents the rotor disc area of ​​the UAV; The total energy consumption of the UAV is calculated as: P h =P0+P1, In the formula, T s represents the flight time of the drone; T c Represents the hovering time of the drone; P h Represents the power consumption of the drone in hovering state.

7. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 6, characterized in that: In step S16, the data amount of each vehicle status update in each time slot is calculated as follows: In the formula, The amount of data for the status update of vehicle i in time slot n; δ n represents the duration of time slot n; The formula of the information age update algorithm of each of the vehicles is: In the formula, represents the information age of vehicle i in time slot n; The minimum amount of data representing the state update of vehicle i; Represents a collection of vehicles.

8. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 7, characterized in that: The obtained Markov process in step S2 is expressed as follows: The drone is used as an intelligent agent, and the intelligent agent interacts cyclically with the constrained environment where the Internet of Vehicles is located. The number of cyclic interactions is the number of cycles of the Markov process specified in step S2, and the number of steps performed in each cycle is the same; For each step, the agent observes the constraint environment at the beginning of the step, obtains the observation state of the current step, and based on the observation state and the agent's own strategy, the agent selects a mixed action from the mixed action space in the step; The hybrid action space consists of a discrete action space and a continuous action space; The agent selects a mixed action from the mixed action space in this step, which means that the agent selects a discrete action from the discrete action space for the vehicle whose information age needs to be updated, and selects two continuous actions for the moving speed and direction of the drone in the continuous action space; After the agent performs the mixed action in this step, the state of the constraint environment changes, and the agent obtains a corresponding reward value, so this step ends and proceeds to the next step.

9. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 8, characterized in that: The step S3 comprises the following steps: S31: According to the requirements of the proximal strategy optimization algorithm, a neural network including a performer network and a critic network is used to initialize all parameters of the neural network, and the experience replay pool corresponding to the agent is initialized, and the number of cycles in the training phase is set to the number of cycles of the Markov process, and the number of steps in each cycle of the training phase is stipulated to be the same as the number of steps set by the Markov process; The performer network is composed of a discrete action network module for outputting discrete actions and a continuous action network module for outputting continuous actions; S32: In the current step, the agent starts observing the constraint environment at the current step and obtains the current observation state of each time slot, and the current observation state is expressed as: In the formula, State represents the current observation state; represents a set of time slots; S33: inputting the current observation state into the performer network and the critic network simultaneously; The discrete action network module of the performer network outputs the dispatch vehicle index of the agent in each time slot, and sends a state update message to the index representative vehicle in turn according to the dispatch vehicle index of each time slot, wherein the state update message contains the information age of the vehicle involved, and resets the information age of the vehicle involved at the agent to δ n , while the information age of vehicles that do not send status update messages increases by δ n , the discrete action of this step is updated; The continuous action network module of the performer network calculates the current output value of the continuous action in each time slot, and the current output value includes the moving direction and the moving speed of the agent. The agent executes in sequence according to the current output value, and the continuous action of this step is updated. After the discrete actions and continuous actions of this step are updated, the agent obtains a reward value based on the constraint environment update of this step and obtains the next observation state of each time slot based on the reward and the output of the critic network; S34: storing the current step state, action, reward, and next step observation state of each time slot in the experience replay pool; S35: Determine whether a cycle has been completed. If not, take the next observation state of each time slot as the current observation state of each time slot and return to execute step S32; If yes, proceed to the next step; S36: Determine whether the current number of cycles has reached the number of cycles or the storage limit of the experience replay pool. If at least one of them is satisfied, obtain the proximal optimization model by performing several time-based iterations based on all the data collected by the experience replay pool and embed the proximal optimization model into the base station. Otherwise, use the next observation state of each time slot as the current observation state of each time slot and return to execute step S32.

10. The method for minimizing the age of Internet of Vehicles data distribution information according to claim 9, characterized in that: In step S33, the calculation formula of the reward value is: In the formula, The first item is the average age of all vehicles at the end of time slot n; The second item is the energy consumption of the UAV in time slot n; ξ and ψ are the non-negative weighted information age and energy consumption components, respectively.

Images