Internet of vehicles distributed resource allocation method based on multi-agent consensus construction

Abstract

An Internet of Vehicles distributed resource allocation method based on multi-agent consensus construction comprises the following operation steps: (1) deploying a consensus constructor and an action device on vehicles forming the Internet of Vehicles, and deploying an evaluator on a central base station; (2) the vehicles forming the Internet of Vehicles input the observed local network state values of the Internet of Vehicles into the consensus constructor, and the consensus constructor outputs the category of the global network state of the Internet of Vehicles; and (3) under the guidance of an evaluator deployed by the central base station, the action devices of the vehicles forming the Internet of Vehicles generate an Internet of Vehicles distribution resource allocation strategy according to the types of the local network state of the Internet of Vehicles and the global network state of the Internet of Vehicles. According to the method, the cooperative capability between the vehicles is further improved, and the resource allocation effect is improved.

Description

Technical Field

[0001] The present invention relates to a vehicle networking distributed resource allocation method based on multi-agent consensus, and belongs to the field of information technology, in particular to the field of vehicle networking technology. Background Art

[0002] With the development of autonomous driving technology, vehicle-to-vehicle communication (V2X) technology has become a research focus, aiming to achieve advanced autonomous driving and intelligent transportation systems through vehicle-to-everything connections. These technologies require the network to have timely message delivery, low latency and high reliability. However, the existing C-V2X (cellular vehicle-to-everything) technology faces the problem of inefficient resource allocation, especially in dynamically changing communication environments, such as high mobility and the Doppler effect. Traditional resource allocation methods are difficult to meet the needs of efficient network management.

[0003] Although the existing multi-agent reinforcement learning (MARL) method has shown potential in vehicle network resource allocation, problems such as high communication overhead and poor vehicle cooperation in dynamic environments have not been effectively solved. How to use the technical advantages of multi-agent reinforcement learning to optimize the coordination between vehicles and improve resource utilization efficiency to meet the growing demand for V2X networks is a technical problem that urgently needs to be solved in the field of vehicle networking technology. Summary of the invention

[0004] In view of this, the purpose of the present invention is to solve the problem of vehicle agent cooperation in V2X resource allocation by introducing a multi-agent consensus building mechanism, so as to achieve the technical goal of improving the efficiency of vehicle network resource utilization.

[0005] In order to achieve the above object, the present invention proposes a method for allocating distributed resources in an Internet of Vehicles based on multi-agent consensus, and the method comprises the following steps:

[0006] (1) Deploy consensus builders and actors on the vehicles that make up the Internet of Vehicles, and deploy evaluators at the central base station;

[0007] (2) The vehicles constituting the Internet of Vehicles input the local network state values ​​of the vehicle network observed by themselves into the consensus builder, and the consensus builder outputs the category of the global network state of the Internet of Vehicles;

[0008] (3) Under the guidance of the evaluator deployed by the central base station, the actors of the vehicles constituting the IoV generate an IoV distributed resource allocation strategy based on the categories of the local network status of the vehicle network and the global network status of the IoV.

[0009] The consensus builder is a classification neural network trained by contrastive learning method, which can generate classification of global network status through limited local observations and guide local observation agents to perform resource allocation with global vision;

[0010] The actor and evaluator are a group of actor-critic networks trained using a multi-agent deep deterministic policy gradient method, the actor corresponds to the actor network part of the actor-critic network, and the evaluator corresponds to the critic network part of the actor-critic network.

[0011] The consensus builder is pre-trained first, and then the pre-trained consensus builder is finally trained together with the actor and the evaluator; the specific process is as follows:

[0012] The pre-training method of the consensus builder is as follows:

[0013] The consensus builder is constructed by P in the unlabeled knowledge distillation method during the pre-training stage. T Teacher Network and P S Student network composition, the two networks have the same network structure and the same initial network parameters;

[0014] Initialize the consensus builder, that is, T Teacher Network and P S The student networks are initialized;

[0015] Initialize the simulated road and vehicle networking environment, including a central base station B and K vehicles, where the vehicle set is represented as vehicle Through the V2V broadcast link (based on PC5 interface) to communicate with surrounding vehicles j∈V i communication, and at the same time communicate with the central base station through the V2I link (based on the Uu interface); i represents the set of other vehicles within the communication range of vehicle i;

[0016] Initialize a random resource allocation strategy for the vehicle and initialize the observation storage area for storing local observations

[0017] Pre-training starts, for each vehicle Collect local observations before each time step t Then all the local observation records of all vehicles are unified. t Store to observation storage area Local observation Contains the channel state information of each sub-band of the link from vehicle i to each communication target before time step t, the interference power of each sub-band measured by each communication target at the receiver, and the communication index;

[0018] The length of each time step is the same as the update interval of the small-scale fading of the channel; each round contains several time steps, and the length of each round is consistent with the update interval of the large-scale fading of the channel; centralized training is performed after each round; each training starts from the observation storage area Small batch sampling of S local observations Where oτ represents the local observation vector of the τth sample selected

[0019] By minimizing the following cross entropy loss function S The parameters of the student network are updated:

[0020]

[0021] In the above formula, P T The teacher network is used to generate consensus pseudo labels, P S Student network learning P T The teacher network’s consensus representation of other observations; Indicates P S The student network consists of local observations The generated consensus classification probability distribution has a dimension of M; express The mth dimension of Indicates P T The teacher network is composed of local observations of other vehicles The generated consensus probability distribution pseudo label has a dimension of M; express The mth dimension; the function H(a, b) = -a log b is used to measure the gap between two probabilities a and b; M is a hyperparameter, representing the total number of categories of the current vehicle network environment state;

[0022] P S After the student network is updated, P T The teacher network is derived from P by exponential moving average method. S The student network updates parameters to achieve P T Teacher Network and P S joint updating of student networks;

[0023] After multiple rounds of the above pre-training process, the pre-training process of the consensus builder is completed; S The student network then becomes a preliminary consensus builder;

[0024] The pre-trained consensus builder is trained together with the actor and evaluator in the final training method as follows:

[0025] The actor for each vehicle consists of a policy actor network Network with target actor Policy actor network Network with target actor Have the same neural network structure and the same initial network parameters;

[0026] The central base station evaluator includes the prediction critic network Q φ With the target critic network Q φ′ , predict the critic network Q φ With the target critic network Q φ′ Have the same neural network structure and the same initial network parameters;

[0027] The consensus builder P obtained by pre-training the consensus builder S Student Network and P T The teacher network is deployed to each vehicle, and the pre-trained consensus builder P S The student network can obtain basically consistent global network state classification from local observations of vehicles;

[0028] Initialize the actuators of each vehicle, initialize the central base station evaluator, and initialize the experience playback area for storing state transition records The simulated road and Internet of Vehicles environment settings are consistent with the consensus builder pre-training method described above;

[0029] Training starts, for each vehicle Collect local observations of the initial state before each time step t Using the pre-trained consensus builder P S Student network generates consensus classification Leveraging a strategic actor network Compute resource allocation actions in, is Gaussian noise, which is used to improve the exploratory nature of the vehicle resource allocation strategy; each vehicle jointly performs resource allocation actions and receives its own rewards And collect local observations of the final state before t+1 Then all the vehicle records (o t , a t , r t , o t+1 )Save to the experience playback area

[0030] Local observation The definition of is consistent with the pre-training process of the consensus builder mentioned above; vehicle resource allocation action It consists of two parts: power selection and sub-band selection. Under the dual-antenna design of the vehicle, the vehicle allocates power and sub-bands for both the V2I link and the V2V link. They represent the V2I link transmission power of vehicle i in time step t and the subband occupied by the V2I link respectively; They represent the V2V broadcast transmission power of vehicle i in time step t and the subband occupied by V2V broadcast respectively;

[0031] Each round contains several time steps, and concentrated training is performed after each round; each training starts from Small batch sampling S records (o τ , a τ , r τ , o′ τ ), τ = {1, ..., S}, where o τ , a τ , r τ , o′ τ Represent the local observation vectors of the initial state Motion Vector Reward Vector and the local observation vector of the final state

[0032] Among them, the reward of vehicle i at time step t is defined as:

[0033]

[0034] In the above formula, x and y are proportional coefficients; interruption interval It is defined as the number of time steps from the last broadcast link interruption of vehicle i to time step t; is the distance weight from vehicle i to nearby vehicle j in time step t, which is used to encourage vehicles to pay more attention to the link interruption in the case of dense vehicles and close distance to avoid traffic accidents; d msb Indicates the safe braking distance of the vehicle; represents the distance between vehicle i and nearby vehicle j; |V i | represents the set of vehicles V near vehicle i i Number of vehicles included in; Age of information represents the number of time steps from the last successful V2I transmission to time step t, in order to capture the timeliness of V2I messages;

[0035] The policy actor network is updated by gradient descent:

[0036]

[0037] in, Representing a policy actor network Input local observation and consensus After the output action, Represents the predicted critic network Q φ Enter o τ , The predicted Q value output later;

[0038] The prediction critic network is updated according to the Bellman equation to minimize the predicted Q value Q φ (o τ , a τ ) and target Q value The mean square error between:

[0039]

[0040] in, Defined as:

[0041]

[0042] Where γ is the discount factor, is the set of actions obtained based on the local observation and consensus of the final state, used to estimate the Q value of the final state, is the consensus inferred by vehicle i based on the local observation of the final state, With Q φ′ are the network parameters of the target actor network and the target critic network, respectively. The target actor network and the target critic network periodically soft-update their network parameters from the corresponding policy actor network and prediction critic network;

[0043] After multiple rounds of training, the training of the actor and the evaluator is completed;

[0044] After the actor and evaluator are trained, the consensus builder is trained according to the consensus builder pre-training method. S The student network is trained to adapt to the local observation relationships under the current actor strategy; similar to the pre-training process of the consensus builder mentioned above, S Student Network Update P T The teacher network is derived from P by exponential moving average. S Update parameters at the student network.

[0045] After multiple rounds of training, P S The student network becomes a consensus builder that adapts to resource allocation strategies.

[0046] The interruption interval Calculate according to the following formula:

[0047]

[0048] In the above formula, represents the minimum channel capacity required to maintain the V2V link connection state; C i→j represents the channel capacity of the V2V link from vehicle i to vehicle j within its communication range; the condition Indicates that the link with the minimum channel capacity in the V2V broadcast link of vehicle i is interrupted, V i Represents the set of other vehicles within the communication range of vehicle i.

[0049] The beneficial effect of the present invention is that the method proposed by the present invention further improves the cooperation ability between vehicles and improves the resource allocation effect on the basis of the existing resource allocation method based on reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the vehicle network distributed resource allocation method based on multi-agent consensus proposed by the present invention;

[0051] Figure 2 It is a schematic diagram of the cellular vehicle networking environment constructed in the experiment in the embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0053] See also Figure 1 , introduces a vehicle network distributed resource allocation method based on multi-agent consensus proposed by the present invention, the method comprising the following steps:

[0054] (1) Deploy consensus builders and actors on the vehicles that make up the Internet of Vehicles, and deploy evaluators at the central base station;

[0055] (2) The vehicles constituting the Internet of Vehicles input the local network state values ​​of the vehicle network observed by themselves into the consensus builder, and the consensus builder outputs the category of the global network state of the Internet of Vehicles;

[0056] (3) Under the guidance of the evaluator deployed by the central base station, the actors of the vehicles constituting the IoV generate an IoV distributed resource allocation strategy based on the categories of the local network status of the vehicle network and the global network status of the IoV.

[0057] The consensus builder is a classification neural network trained by contrastive learning method, which can generate classification of global network status through limited local observations and guide local observation agents to perform resource allocation with global vision;

[0058] The actor and evaluator are a group of actor-critic networks trained using a multi-agent deep deterministic policy gradient method, the actor corresponds to the actor network part of the actor-critic network, and the evaluator corresponds to the critic network part of the actor-critic network.

[0059] The consensus builder is pre-trained first, and then the pre-trained consensus builder is finally trained together with the actor and the evaluator; the specific process is as follows:

[0060] The pre-training method of the consensus builder is as follows:

[0061] The consensus builder is constructed by P in the unlabeled knowledge distillation method during the pre-training stage. T Teacher Network and P S Student network composition, the two networks have the same network structure and the same initial network parameters;

[0062] Initialize the consensus builder, that is, T Teacher Network and P S The student networks are initialized;

[0063] Initialize the simulated road and vehicle networking environment, including a central base station B and K vehicles, where the vehicle set is represented as vehicle Through V2V (vehicle to vehicle) broadcast link (based on PC5 interface) to communicate with surrounding vehicles j∈V i communication, while communicating with the central base station via a V2I (vehicle to infrastructure) link (based on the Uu interface); V i represents the set of other vehicles within the communication range of vehicle i;

[0064] Initialize a random resource allocation strategy for the vehicle and initialize the observation storage area for storing local observations

[0065] Pre-training starts, for each vehicle Collect local observations before each time step t Then all the local observation records of all vehicles are unified. t Store to observation storage area Local observation Contains the channel state information of each sub-band from vehicle i to each communication target before time step t (including channel gain, interference channel gain), the interference power of each sub-band measured by each communication target at the receiver, and communication indicators (including information age and interruption interval);

[0066] The length of each time step is the same as the update interval of the small-scale fading of the channel, which is generally set to 1ms in the embodiment; each round contains several time steps, and the time length of each round is consistent with the update interval of the large-scale fading of the channel, which is generally set to 100ms in the embodiment; centralized training is performed after each round; each training is performed from the observation storage area Small batch sampling of S local observations Where oτ represents the local observation vector of the τth sample selected

[0067] By minimizing the following cross entropy loss function S The parameters of the student network are updated:

[0068]

[0069] In the above formula, P T The teacher network is used to generate consensus pseudo labels, P S Student network learning P T The teacher network’s consensus representation of other observations; Indicates P S The student network consists of local observations The generated consensus classification probability distribution has a dimension of M; express The mth dimension of Indicates P T The teacher network is composed of local observations of other vehicles The generated consensus probability distribution pseudo label has a dimension of M; express The mth dimension; the function H(a, b) = -a log b is used to measure the gap between two probabilities a and b; M is a hyperparameter, representing the total number of categories of the current vehicle network environment state;

[0070] P S After the student network is updated, P T The teacher network is derived from P by exponential moving average. S The student network updates parameters to achieve P T Teacher Network and P S joint updating of student networks;

[0071] After multiple rounds of the above pre-training process, the pre-training process of the consensus builder is completed; SThe student network then becomes a preliminary consensus builder.

[0072] The pre-trained consensus builder is trained together with the actor and evaluator in the final training method as follows:

[0073] The actor for each vehicle consists of a policy actor network Network with target actor Policy actor network Network with target actor Have the same neural network structure and the same initial network parameters;

[0074] The central base station evaluator includes the prediction critic network Q φ With the target critic network Q φ′ , predict the critic network Q φ With the target critic network Q φ′ Have the same neural network structure and the same initial network parameters.

[0075] The consensus builder P obtained by pre-training the consensus builder S Student Network and P T The teacher network is deployed to each vehicle, and the pre-trained consensus builder P S The student network can obtain basically consistent global network state classification from local observations of vehicles;

[0076] Initialize the actuators of each vehicle, initialize the central base station evaluator, and initialize the experience playback area for storing state transition records The simulated road and Internet of Vehicles environment settings are consistent with the consensus builder pre-training method described above;

[0077] Training starts, for each vehicle Collect local observations of the initial state before each time step t Using the pre-trained consensus builder P S Student network generates consensus classification Leveraging a strategic actor network Compute resource allocation actions in, is Gaussian noise, which is used to improve the exploratory nature of the vehicle resource allocation strategy; each vehicle jointly performs resource allocation actions and receives its own rewards And collect local observations of the final state before t+1 Then all the vehicle records (o t , a t , r t , o t+1 )Save to the experience playback area

[0078] Local observation The definition of is consistent with the pre-training process of the consensus builder mentioned above; vehicle resource allocation action It consists of two parts: power selection and sub-band selection. Under the dual-antenna design of the vehicle, the vehicle allocates power and sub-bands for both the V2I link and the V2V link. They represent the V2I link transmission power of vehicle i in time step t and the subband occupied by the V2I link respectively; They represent the V2V broadcast transmission power of vehicle i in time step t and the subband occupied by V2V broadcast respectively;

[0079] Each round contains several time steps, and concentrated training is performed after each round; each training starts from Small batch sampling S records (o τ , a τ , r τ , o′ τ ), τ = {1, ..., S}, where o τ , a τ , r τ , o′ τ Represent the local observation vectors of the initial state Motion Vector Reward Vector and the local observation vector of the final state

[0080] Among them, the reward of vehicle i at time step t is defined as:

[0081]

[0082] In the above formula, x and y are proportional coefficients, and the values ​​in the embodiment are 0.05 and 0.05 respectively; the interruption interval It is defined as the number of time steps from the last broadcast link interruption of vehicle i to time step t; is the distance weight from vehicle i to nearby vehicle j in time step t, which is used to encourage vehicles to pay more attention to the link interruption in the case of dense vehicles and close distance to avoid traffic accidents; d msb Indicates the safe braking distance of the vehicle; represents the distance between vehicle i and nearby vehicle j; information age represents the number of time steps from the last successful V2I transmission to time step t, in order to capture the timeliness of V2I messages;

[0083] The policy actor network is updated by gradient descent:

[0084]

[0085] in, Representing a policy actor network Input local observation and consensus After the output action, Represents the predicted critic network Q φ enter The predicted Q value output later;

[0086] The prediction critic network is updated according to the Bellman equation to minimize the predicted Q value Q φ (o τ , a τ ) and target Q value The mean square error between:

[0087]

[0088] in, Defined as:

[0089]

[0090] Wherein γ is a discount factor, which is 0.99 in the embodiment; is the set of actions obtained based on the local observation and consensus of the final state, used to estimate the Q value of the final state, is the consensus inferred by vehicle i based on the local observation of the final state, With Q φ′ are the network parameters of the target actor network and the target critic network, respectively. The target actor network and the target critic network periodically soft-update their network parameters from the corresponding policy actor network and prediction critic network;

[0091] After multiple rounds of training, the training of the actor and the evaluator is completed;

[0092] After the actor and evaluator are trained, the consensus builder is trained according to the consensus builder pre-training method. S The student network is trained to adapt to the local observation relationships under the current actor strategy; similar to the pre-training process of the consensus builder mentioned above, S Student Network Update P T The teacher network is derived from P by exponential moving average. S Update parameters at the student network.

[0093] After multiple rounds of training, P SThe student network becomes a consensus builder that adapts to resource allocation strategies.

[0094] The interruption interval Calculate according to the following formula:

[0095]

[0096] In the above formula, represents the minimum channel capacity required to maintain the V2V link connection state; C i→j represents the channel capacity of the V2V link from vehicle i to vehicle j within its communication range; the condition Indicates that the link with the minimum channel capacity in the V2V broadcast link of vehicle i is interrupted, V i Represents the set of other vehicles within the communication range of vehicle i.

[0097] Table 1

[0098]

[0099] The inventors have conducted a large number of experiments on the method proposed in the present invention, and the experimental results are as follows:

[0100] See also Figure 2 Based on the urban road structure of the Manhattan grid, the inventors used Python to build a typical cellular vehicle network environment. 16 vehicle nodes were randomly generated in the environment, and a single base station was set in the center of the road. The main environmental simulation parameters are shown in Table 1. In the simulation environment, time is discretized into time slots of 1 ms in length, corresponding to the minimum time unit and time step length of resource allocation. Under the training parameter setting of 500 rounds and 100 time steps per round, the method proposed in the present invention has better performance in terms of reward convergence speed than the pure DDPG algorithm. On the other hand, the method proposed in the present invention also achieved higher convergence rewards, indicating that the consensus building mechanism helps vehicles learn to obtain a better resource allocation mode. For details of the DDPG algorithm, please refer to the literature Lowe R, Wu YI, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J]. Advances inneural information processing systems, 2017, 30.

[0101] The inventors also verified the comparison of communication indicators such as the information age of the V2I link, the V2V link interruption rate, and the channel utilization rate of the proposed method under different environmental parameters (including the number of vehicle nodes and the number of sub-bands). The experimental results show that the method of the present invention shows superior performance under various experimental settings.

[0102] Experimental results prove that the method proposed in the present invention is effective.

Claims

1. A distributed resource allocation method for Internet of Vehicles based on multi-agent consensus, characterized by: The method comprises the following steps: (1) Deploy consensus builders and actors on the vehicles that make up the Internet of Vehicles, and deploy evaluators at the central base station; (2) The vehicles constituting the Internet of Vehicles input the local network state values ​​of the vehicle network observed by themselves into the consensus builder, and the consensus builder outputs the category of the global network state of the Internet of Vehicles; (3) Under the guidance of the evaluator deployed by the central base station, the actors of the vehicles constituting the IoV generate an IoV distributed resource allocation strategy based on the categories of the local network status of the vehicle network and the global network status of the IoV.

2. The method for allocating distributed resources in the Internet of Vehicles based on multi-agent consensus construction according to claim 1 is characterized in that: The consensus builder is a classification neural network trained by contrastive learning method, which can generate classification of global network status through limited local observations and guide local observation agents to perform resource allocation with global vision; The actor and evaluator are a group of actor-critic networks trained using a multi-agent deep deterministic policy gradient method, the actor corresponds to the actor network part of the actor-critic network, and the evaluator corresponds to the critic network part of the actor-critic network.

3. The method for allocating distributed resources in the Internet of Vehicles based on multi-agent consensus construction according to claim 2 is characterized in that: The consensus builder is pre-trained first, and then the pre-trained consensus builder is finally trained together with the actor and evaluator; the specific process is as follows: The pre-training method of the consensus builder is as follows: The consensus builder is constructed by P in the unlabeled knowledge distillation method during the pre-training stage. T Teacher Network and P S Student network composition, the two networks have the same network structure and the same initial network parameters; Initialize the consensus builder, that is, T Teacher Network and P S The student networks are initialized; Initialize the simulated road and vehicle networking environment, including a central base station B and K vehicles, where the vehicle set is represented as vehicle Through the V2V broadcast link (based on PC5 interface) to communicate with surrounding vehicles j∈V i communication, and at the same time communicate with the central base station through the V2I link (based on the Uu interface); i represents the set of other vehicles within the communication range of vehicle i; Initialize a random resource allocation strategy for the vehicle and initialize the observation storage area for storing local observations Pre-training starts, for each vehicle Collect local observations before each time step t Then all the local observation records of all vehicles are unified. t Store to observation storage area Local observation Contains the channel state information of each sub-band of the link from vehicle i to each communication target before time step t, the interference power of each sub-band measured by each communication target at the receiver, and the communication index; The length of each time step is the same as the update interval of the small-scale fading of the channel; each round contains several time steps, and the length of each round is consistent with the update interval of the large-scale fading of the channel; centralized training is performed after each round; each training starts from the observation storage area Small batch sampling of S local observations τ={1,…,S}, where o τ Represents the local observation vector of the τth sample selected By minimizing the following cross entropy loss function S The parameters of the student network are updated: In the above formula, P T The teacher network is used to generate consensus pseudo labels, P S Student network learning P T The teacher network’s consensus representation of other observations; Indicates P S The student network consists of local observations The generated consensus classification probability distribution has a dimension of M; express The mth dimension of Indicates P T The teacher network is composed of local observations of other vehicles The generated consensus probability distribution pseudo label has a dimension of M; express The mth dimension of the network; the function H(a,b) = -a log b is used to measure the gap between two probabilities a and b; M is a hyperparameter, representing the total number of categories of the current vehicle network environment state; P S After the student network is updated, P T The teacher network is derived from P by exponential moving average. S The student network updates parameters to achieve P T Teacher Network and P S joint updating of student networks; After multiple rounds of the above pre-training process, the pre-training process of the consensus builder is completed; S The student network then becomes a preliminary consensus builder; The pre-trained consensus builder is trained together with the actor and evaluator in the final training method as follows: The actor for each vehicle consists of a policy actor network Network with target actor Policy actor network Network with target actor Have the same neural network structure and the same initial network parameters; The central base station evaluator includes the prediction critic network Q φ With the target critic network Q φ′ , predict the critic network Q φ With the target critic network Q φ′ Have the same neural network structure and the same initial network parameters; The consensus builder P obtained by pre-training the consensus builder S Student Network and P T The teacher network is deployed to each vehicle, and the pre-trained consensus builder P S The student network can obtain basically consistent global network state classification from local observations of vehicles; Initialize the actuators of each vehicle, initialize the central base station evaluator, and initialize the experience playback area for storing state transition records The simulated road and Internet of Vehicles environment settings are consistent with the consensus builder pre-training method described above; Training starts, for each vehicle Collect local observations of the initial state before each time step t Using the pre-trained consensus builder P S Student network generates consensus classification Leveraging a strategic actor network Compute resource allocation actions in, is Gaussian noise, which is used to improve the exploratory nature of the vehicle resource allocation strategy; each vehicle jointly performs resource allocation actions and receives its own rewards And collect local observations of the final state before t+1 Then all the vehicle records (o t ,a t ,r t ,o t+1 )Save to the experience playback area Local observation The definition of is consistent with the pre-training process of the consensus builder mentioned above; vehicle resource allocation action It consists of two parts: power selection and sub-band selection. Under the dual-antenna design of the vehicle, the vehicle allocates power and sub-bands for both the V2I link and the V2V link. They represent the V2I link transmission power of vehicle i in time step t and the subband occupied by the V2I link respectively; They represent the V2V broadcast transmission power of vehicle i in time step t and the subband occupied by V2V broadcast respectively; Each round contains several time steps, and concentrated training is performed after each round; each training starts from Small batch sampling S records (o τ ,a τ ,r τ ,o′ τ ),τ={1,…,S}, where o τ ,a τ ,r τ ,o′ τ Represent the local observation vectors of the initial state Motion Vector Reward Vector and the local observation vector of the final state Among them, the reward of vehicle i at time step t is defined as: In the above formula, x and y are proportional coefficients; interrupt interval It is defined as the number of time steps from the last broadcast link interruption of vehicle i to time step t; is the distance weight from vehicle i to nearby vehicle j in time step t, which is used to encourage vehicles to pay more attention to the link interruption in the case of dense vehicles and close distance to avoid traffic accidents; d msb Indicates the safe braking distance of the vehicle; represents the distance between vehicle i and nearby vehicle j; |V i | represents the set of vehicles V near vehicle i i Number of vehicles included in; Age of information represents the number of time steps from the last successful V2I transmission to time step t, in order to capture the timeliness of V2I messages; The policy actor network is updated by gradient descent: in, Representing a policy actor network Input local observation and consensus After the output action, Represents the predicted critic network Q φ Enter o τ , The predicted Q value output later; The prediction critic network is updated according to the Bellman equation to minimize the predicted Q value Q φ (o τ ,a τ ) and target Q value The mean square error between: in, Defined as: Where γ is the discount factor, is the set of actions obtained based on the local observation and consensus of the final state, used to estimate the Q value of the final state, is the consensus inferred by vehicle i based on the local observation of the final state, With Q φ′ are the network parameters of the target actor network and the target critic network, respectively. The target actor network and the target critic network periodically soft-update their network parameters from the corresponding policy actor network and prediction critic network; After multiple rounds of training, the training of the actor and the evaluator is completed; After the actor and evaluator are trained, the consensus builder is trained according to the consensus builder pre-training method. S The student network is trained to adapt to the local observation relationships under the current actor strategy; similar to the pre-training process of the consensus builder mentioned above, S Student Network Update P T The teacher network is derived from P by exponential moving average. S Update parameters at the student network; After multiple rounds of training, P S The student network becomes a consensus builder that adapts to resource allocation strategies.

4. The method for allocating distributed resources in the Internet of Vehicles based on multi-agent consensus construction according to claim 3 is characterized by: The interruption interval Calculate according to the following formula: In the above formula, represents the minimum channel capacity required to maintain the V2V link connection state; C i→j represents the channel capacity of the V2V link from vehicle i to vehicle j within its communication range; the condition Indicates that the link with the minimum channel capacity in the V2V broadcast link of vehicle i is interrupted, V i Represents the set of other vehicles within the communication range of vehicle i.

Images