Resource management method and system for densely interfering Internet of Vehicles

Abstract

The present invention relates to the technical field of Internet of Things, and specifically discloses a resource management method and system for densely interfering vehicle networks, which establishes a communication model for densely interfering vehicle networks; models the conflict relationship of densely interfering vehicle networks as a conflict model; establishes a service quality model for the communication model; analyzes the interference relationship between multiple users of the conflict model through a hyperchannel, establishes a hypergraph interference model, and uses the hypergraph interference model to calculate the interference degree of the entire network; establishes a resource allocation model for the communication model based on the hypergraph interference model and the service quality model to maximize the network throughput while considering the interference degree and service quality constraints; converts the resource allocation model into a Markov decision model; and solves the Markov decision model. Simulation results show that the system and method proposed by the present invention significantly improve the data transmission throughput of the vehicle network affected by dense interference.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a resource management method and system for densely interfering Internet of Vehicles. Background Art

[0002] With the emergence of Internet of Vehicles (IoV) applications such as autonomous vehicles and smart connected cars, the traffic volume generated by vehicles is increasing rapidly, bringing higher communication needs. Vehicle-to-vehicle (V2V) communication has evolved as a key technology to meet these needs by enabling cooperation between nearby vehicles. In a densely populated IoV environment, multiple vehicle user equipment (VUE) using the same spectrum resources will cause co-channel interference. Given the limited spectrum resources, it is not feasible to simply allocate more spectrum to meet the growing demand of VUE. This increasingly prominent discrepancy between the service needs of vehicles and the limited spectrum resources has prompted extensive research on resource optimization in IoV.

[0003] Researchers have proposed various solutions to the resource allocation problem in IoVs. Although optimization methods for resource allocation have been widely adopted, these methods are no longer sufficient in densely deployed IoVs environments due to the rapid increase in the number of devices, the diversity of user services, and the inherent complexity of optimization schemes.

[0004] Resource allocation methods based on machine learning can effectively improve the efficiency of resource allocation. However, considering the high reliability and low latency of data transmission and the scale of network devices, the computational overhead of the network increases dramatically. In order to reduce the computational pressure, improve the throughput and resource reuse rate, IoV resource allocation based on federated learning is a good solution. As a distributed artificial intelligence (DAI) method, federated learning (FL) can solve complex learning, planning and decision-making problems, and has become the main development trend of DAI-based resource management in IoV. However, in densely distributed vehicular networks, a large amount of overlap in communication radius can cause conflicts between V2V communication links (VLs), causing serious interference to neighboring users. Existing research on federated learning in vehicular networks has paid limited attention to this interference. It remains a severe challenge to effectively use DAI in densely deployed IoVs to alleviate the resource allocation pressure caused by large-scale network conflicts, achieve conflict-free resource management, and improve VUE experience. Summary of the invention

[0005] The present invention provides a resource management method and system for a densely interfering vehicle network, and solves the technical problem of how to allocate resources efficiently and without conflict in a densely interfering vehicle network.

[0006] In order to solve the above technical problems, the present invention provides a resource management method for densely interfering Internet of Vehicles, including:

[0007] Construct a communication model for densely interfering Internet of Vehicles;

[0008] Modeling the conflict relationship in the communication model as a conflict model;

[0009] Using service quality indicators to measure the demand of the communication model and establishing a service quality model of the communication model;

[0010] Establishing a hypergraph interference model based on the communication model and the conflict model;

[0011] Establishing a resource allocation model of the communication model based on the hypergraph interference model and the quality of service model;

[0012] Converting the resource allocation model into a Markov decision model;

[0013] The Markov decision model is solved to obtain a resource allocation strategy for the densely interfering Internet of Vehicles at the current moment.

[0014] Furthermore, the communication model of the densely interfering Internet of Vehicles is constructed including L vehicle user equipments, namely VUEs, N V2V communication links, namely VLs, and a roadside base station, namely BS; the link between the VUE and the base station has been allocated communication channels and transmission powers in advance, the lth channel is used by the lth VUE, so there are L communication channels in total, and the bandwidth of each channel is W, l = 1, 2, …, L; the channel of each VUE is allowed to be reused by multiple VLs, but a VL can only access one channel in a time slot.

[0015] Furthermore, the conflict model includes two conflict relationships, direct conflict and hidden conflict; direct conflict means that in the same time slot t, two VLs use the same resource block, i.e., RB, and have the same sending or receiving vertex; hidden conflict means that in the same time slot t, two VLs are simultaneously allocated the same RB, and the sending or receiving vertex of one VL is within the communication range of the sending or receiving vertex of the other VL.

[0016] Furthermore, the service quality model of the communication model is constructed as follows:

[0017]

[0018]

[0019] in, is the transmission rate that the lth VUE can achieve, The minimum data rate for each VUE, is the instantaneous signal interference plus noise ratio (SINR) of the nth VL when it is assigned to the lth channel, and γ0 is the SINR threshold; is the delay of the nth VL in time slot t Greater than or equal to the maximum acceptable delay D for VL data transmission max The probability of is the queue length of the nth VL at the beginning of time slot t Greater than or equal to the delay exceeding D max Data packet Q max The probability, p o is the probability threshold.

[0020] Furthermore, in the hypergraph interference model, each VUE is a vertex, and VL connects the vertices as hyperedges, so that the vertices can communicate with each other; the hypergraph interference model of the t time slot is represented by the incident matrix I t To represent, the matrix I t The rows represent vertices, the columns represent hyperedges, and the matrix I t The element i at row n and column e n,e = 1, indicating that the nth transitive vertex is included in the eth hyperedge, otherwise i n,e =0;

[0021] Using the network conflict degree matrix Φ t =log(max(I′ t R t ,1)) represents the situation of dense interference in the Internet of Vehicles at time slot t, I′ t is the incident matrix I t The transpose of R t is the resource matrix at time slot t, resource matrix R t The element at row n and column j Indicates that the dth resource block, i.e., RB, is allocated to the nth VL, otherwise

[0022] The overall network conflict degree of densely interfering vehicle networks Calculated as: Φ t 1 represents Φ t The number of 1s in Indicates that no conflict occurs, otherwise

[0023] Furthermore, the resource allocation model is constructed as follows:

[0024]

[0025]

[0026]

[0027]

[0028] Among them, max is maximization, st is to satisfy, C1 to C9 are different constraints; T is the number of time slots, is the signal interference plus noise ratio of the lth VUE at time slot t, For VUE collection, is the VL set, is the transmission power of the nth VL, is the maximum transmit power for each VL, is the allocation index of VL, Indicates that the lth channel is assigned to the nth VL, otherwise

[0029] Furthermore, in the Markov decision model: each VL is represented as an agent, and each agent is trained to develop a strategy to determine the optimal action for its channel selection so as to maximize its reward function; the Markov decision model of N agents is defined as a tuple in is the state space, It is the action space. is the transition probability, is the reward function, γ is the discount factor;

[0030] The state space includes the channel coefficients of the V2V signal link VUE channel coefficient Channel allocation of the VL adjacent to the nth VL in the previous time slot t-1 and the current length of the queue at the nth VL transmitter It refers to the channel power gain associated with the nth V2V link on the lth channel in the current time slot t. Refers to the channel interference parameter of the nth VL to the lth VUE when the nth channel is reused for communication in the current time slot t;

[0031] The action space consists of the actions that each agent performs, including selecting a channel and transmit power

[0032] The reward function at time slot t is designed as:

[0033]

[0034] The penalty function U(z) imposes a penalty when any part of the reward is not met, and z refers to the value inside the penalty function brackets. U(z) is as follows:

[0035]

[0036] Among them, the constant A1<0, λ1, λ2 and λ3 are the weight coefficients of the three components of the defined reward.

[0037] Furthermore, when solving the Markov decision model, a DQN algorithm is used to approximate the Q value function of the Markov decision model; in the DQN algorithm, each agent is composed of two neural networks, namely a Q network and a target Q network, and an experience replay pool; the input of the DQN algorithm is the current state of the densely interfering Internet of Vehicles, and the output is a set of actions, namely a resource allocation plan;

[0038] In a Q-network, the agent receives state information s at each time step t t As input, and output the current state s t Execute each action a in the action space t The action Q value Q(s t ,a t ; w), where w is the weight of the Q network; after obtaining the estimated Q value, the greedy algorithm strategy is used to select the best action a t , that is, select action a with the highest Q value t ;

[0039] In the agent designated action a t Afterwards, the system will receive a return r t , and enter the next state s t+1 ; A set of data for each time step (s t ,a t ,r t ,s t+1 ) is stored in an experience replay buffer of size D using the experience replay method;

[0040] In the target Q network, the randomly sampled tuples (s t ,a t ,r t ,s t+1 ) is used to calculate the expected value of the target Q network, namely:

[0041] y t =r t +maxQ(s t+1 ,a t+1 ;w - ).

[0042] Q(s t+1 ,a t+1 ;w - ) is the target Q network in the next state s t+1 Execute each next action a in the action space t+1Q value, w - are the parameters of the target Q network obtained in the previous training process; the target Q network can be trained by minimizing the loss function, that is:

[0043] L(w)=(y t -Q(s t ,a t ;w)) 2 .

[0044] The loss function is calculated using stochastic gradient descent, and the weights of the target Q network are used to update the weights of the target Q network.

[0045] Furthermore, the federated learning method is used to improve the convergence speed of the DQN algorithm, specifically:

[0046] All agents use the global model to build local DQN networks. During the global model training process, the kth agent reads from its own replay buffer D k A small batch of samples B is randomly selected from k , train to minimize the loss function and update the local parameters Then, at the end of a round of learning, the global parameters of the global model are obtained by weighted averaging of the parameters;

[0047] At time slot t, the loss function L(w t ) is minimized to:

[0048]

[0049] Among them, w t are the network parameters of the global model at time slot t, is the network parameter of the kth agent at time slot t, K is the total number of agents;

[0050] At time slot t, the network parameters of the global model are updated as follows:

[0051]

[0052] Where η is the learning rate;

[0053] During a round of training, at the current time slot t, each agent first obtains the latest parameters w of the global model t-1 As its latest parameter Calculate its loss function Then, each agent calculates the gradient of the loss function Get new local parameters Update the local model; after local training is completed, each agent will update the new local parameters Sent to the server of the global model; the server of the global model aggregates the parameters of all agents to obtain a new global parameter w t And send the parameter to each agent in batches for the next round of training.

[0054] The present invention also provides a resource management system for densely interfering vehicle networks, the key of which is that the system adopts the resource management method for densely interfering vehicle networks to allocate resources to the densely interfering vehicle networks.

[0055] The resource management method and system of densely interfering vehicle networks provided by the present invention establish a communication model of densely interfering vehicle networks; model the conflict relationship of densely interfering vehicle networks as a conflict model; use service quality indicators to measure the needs of the communication model and establish a service quality model of the communication model; analyze the interference relationship between multiple users of the conflict model through super channels, establish a hypergraph interference model, and use the hypergraph interference model to calculate the interference degree of the entire network; based on the hypergraph interference model and the service quality model, establish a resource allocation model (resource allocation optimization problem) of the communication model to maximize the network throughput while considering the interference degree and service quality (QoS) constraints; convert the resource allocation model into a Markov decision (MDP) model; for vehicle networks with dense interference, finally solve the Markov decision model, which can improve the transmission rate and the overall network throughput. Simulation results show that the system and method proposed by the present invention significantly improve the data transmission throughput of vehicle networks affected by dense interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic diagram of a network model of a densely interfering Internet of Vehicles provided by an embodiment of the present invention;

[0057] Figure 2 is an example diagram of a conflict model provided by an embodiment of the present invention;

[0058] Figure 3 It is an architecture diagram of a conflict-free resource allocation algorithm based on FedAvg-DQN provided in an embodiment of the present invention;

[0059] Figure 4 It is a comparison diagram of the convergence process of different learning rates of the conflict-free resource allocation algorithm based on FedAvg-DQN provided by an embodiment of the present invention;

[0060] Figure 5 is a comparison chart of average network throughputs of four algorithms provided by an embodiment of the present invention;

[0061] Figure 6 is an energy efficiency comparison diagram of four algorithms provided by an embodiment of the present invention;

[0062] Figure 7This is a comparison chart of average SINRs of four algorithms provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0064] The resource management method for densely interfering vehicle networks provided by an embodiment of the present invention includes the following steps:

[0065] Construct a communication model for densely interfering Internet of Vehicles;

[0066] Modeling conflicting relationships in the communication model as a conflict model;

[0067] Use service quality indicators to measure the needs of the communication model and establish a service quality model for the communication model;

[0068] A hypergraph interference model is established based on the communication model and conflict model;

[0069] Based on the hypergraph interference model and service quality model, a resource allocation model of the communication model is established;

[0070] Convert the resource allocation model into a Markov decision model;

[0071] The Markov decision model is solved to obtain the resource allocation strategy for the densely interfering Internet of Vehicles at the current moment.

[0072] The present invention establishes a communication model for a densely interfered Internet of Vehicles; models the conflict relationship of the densely interfered Internet of Vehicles as a conflict model; adopts a service quality indicator to measure the demand of the communication model and establishes a service quality model of the communication model; establishes a hypergraph interference model by analyzing the interference relationship between multiple users of the conflict model through a hyperchannel, and uses the hypergraph interference model to calculate the interference degree of the entire network; establishes a resource allocation model (resource allocation optimization problem) of the communication model based on the hypergraph interference model and the service quality model to maximize the network throughput while considering the interference degree and service quality (QoS) constraints; converts the resource allocation model into a Markov decision (MDP) model; and finally solves the Markov decision model for the Internet of Vehicles with dense interference, so as to improve the transmission rate and the overall network throughput.

[0073] Each step is described in detail below.

[0074] (1) Constructing a communication model for densely interfering Internet of Vehicles

[0075] The communication model of densely deployed Internet of Vehicles is as follows: Figure 1 As shown, the communication model includes L vehicle user equipment (VUE) (with a set ) and N V2V communication links (denoted by the set denoted by ). Both the VUE and the V2V communication link (VL) are equipped with a single antenna to ensure that they can maintain communication with the RSU (roadside) base station (BS). The link between the VUE and the base station is called the V2I link.

[0076] Assume that this is a time slot communication system, define t = {1, 2, ..., T} as the time slot parameter, that is, there are T time slots in total, and the current time slot is denoted as t. Since the VUE link has been allocated communication channels and transmission power in advance, the lth channel is used by the lth VUE, so there are L communication channels in total, and the bandwidth of each channel is W.

[0077] use As the allocation index of VL, where Indicates that the lth channel is assigned to the nth VL, otherwise Assume that the channel of each VUE can be reused by multiple VLs, but a VL can only access one channel in a time slot, which can be described by the formula:

[0078]

[0079] In the current time slot t, the channel power gain associated with the nth V2V link on the lth channel is named And there are:

[0080]

[0081] in, is the small-scale attenuation power component, which is assumed to be distributed in an exponential form with unit mean. Represents the large-scale attenuation parameter that is not affected by frequency. The network will be subject to three different types of interference, as follows:

[0082] 1) In the current time slot t, when the nth channel is reused for communication, the channel interference parameters of the nth VL to the lth VUE.

[0083] 2) In the current time slot t, when the lth channel is reused for communication, the channel interference parameters of the lth VUE to the nth VL.

[0084] 3) In the current time slot t, when the lth channel is reused for communication, the n'th

[0085] The channel interference parameter of the nth VL to the nth VL (n′≠n).

[0086] When the nth VL and the lth VUE reuse the lth communication channel at the same time in time slot t, the path loss and small-scale fading from the nth VL to the lth VUE are defined as and Similarly, when the nth VL and the lth VUE reuse the lth communication channel at the same time in time slot t, we define and are the path loss and small-scale fading from the lth VUE to the nth VL, respectively. When the n′th VL and the nth VL reuse the lth communication channel at the same time in time slot t, we define and are the path loss and small-scale fading from the n′th VL to the nth VL, respectively.

[0087] Therefore, the signal to interference plus noise ratio (SINR) of the lth VUE at time slot t is expressed as:

[0088]

[0089] in, is the transmission power of the lth VUE in time slot t, is the transmission power of the nth VL. In addition, σ 2 is the power spectral density of the additive white Gaussian noise (AWGN) present in the channel. is the channel interference parameter of the lth VUE when the nth channel is reused for communication in the current time slot t.

[0090] Similarly, when assigned to the lth channel, the SINR (signal to interference plus noise ratio) of the nth pair of V2V receivers in time slot t can be expressed as:

[0091]

[0092] in, They are the interference from the lth VUE in time slot t and the interference generated when the nth VL reuses the lth communication channel:

[0093]

[0094] (2) Modeling the conflict relationship in the communication model as a conflict model

[0095] In densely deployed IoV systems, avoiding data transmission conflicts is a critical and challenging task. Figure 2 As shown in the figure, network conflicts in the Internet of Vehicles can be divided into two types: direct conflicts and hidden conflicts.

[0096] 1) Direct conflict: In the same time slot t, two VLs use the same resource block (RB) and have the same transmit or receive vertex, e.g. Figure 2 If VUE1 in the example sends RB as the sender, VL1 and VL2 will conflict directly.

[0097] 2) Hidden conflict: In the same time slot t, two VLs are allocated the same RB at the same time, and the sending or receiving vertex of one VL is within the communication range of the sending or receiving vertex of the other VL. Specifically, Figure 2 In the example, VUE1 as the sender sends data RB, while VUE2 and VUE3 as the receivers receive the information at the same time, and direct conflicts will inevitably occur between them. At the same time, VL4 and VL3 connected to VUE2 and VUE3 respectively will also be indirectly affected by the information transmission.

[0098] (3) Establishing a Quality of Service (QoS) Model for the Communication Model

[0099] In the IoV scenario, due to the different requirements of VUE and VL information transmission, QoS indicators are used to measure the requirements.

[0100] 1) VUE data rate: Use the minimum data rate The QoS performance is quantitatively expressed as a coefficient to ensure that users have a smooth service experience. The quantitative definition of this requirement is:

[0101]

[0102] in, is the transmission rate that the lth VUE can achieve, for example:

[0103]

[0104] For simplicity, it is assumed that each VUE requires a uniform minimum data transfer rate.

[0105] 2) Reliability of VL: The transmission reliability of VL can be judged by the interruption probability of VL. The probability of being below the threshold SINR (denoted by γ0) is called the outage probability, which is expressed as:

[0106]

[0107] Among them, p o is a critical value and is the probability threshold.

[0108] The reliability constraint of the nth VL can be rewritten as:

[0109]

[0110] 3) Delay of VL: Assume that the transmitter in VL is equipped with a buffer with limited capacity and generates packets at a steady rate of λ bits per second. The transmission rate of the nth VL can be express:

[0111]

[0112] in, is the instantaneous SINR of the nth VL.

[0113] because It may change in different time slots, which will cause the generated data packets to not match the instantaneous throughput, which in turn causes the data packets to be queued and delayed at the transmitter. At the beginning of time slot t, the queue length of the nth VL is Its expression is as follows:

[0114]

[0115] in, is the queue length of the transmitter in time slot t-1, τλ is the total number of bits arriving in the queue in each time slot, λ is the number of bits of data packets generated per second at a stable rate, τ is the time length of each time slot, is the number of bits sent to the receiver in the t-1 time slot, and max{} represents the maximum value in {}.

[0116] The delay in VL information transmission is the main cause of queuing. Definition D max is the maximum acceptable delay for VL data transmission, Q max =λD max For a delay exceeding D max According to the above definition, the delay constraint of the nth VL can be expressed as follows:

[0117]

[0118] in, is the delay of the nth VL in time slot t.

[0119] (4) Establishing a hypergraph interference model based on the communication model and conflict model

[0120] In the IoV scenario, due to the massive number of devices and scarce spectrum resources, large-scale intensive interference will be generated. In order to effectively deal with the interference problem in the IoV scenario, the present invention proposes a user-centric hypergraph interference model.

[0121] exist Figure 2In this paper, each VUE is taken as a vertex and its peripheral links are colored. The color of each vertex should be different from the color of the link between every two hops. This is a more complex graph star coloring that is different from the traditional coloring method. Therefore, this example proposes a novel hypergraph-based dense IoV conflict model to analyze network conflicts, which transforms graph coloring into hypergraph coloring. Next, some basic definitions of hypergraphs are introduced.

[0122] Definition 1 (Hypergraph): A hypergraph is a generalization of a graph in which any subset of a given set can be an edge, rather than a two-element subset. Let N x ={n1,n2,...,n x} is a finite set, let N x A family of subsets of is represented by E = {e1,e2,...,e y}. It can be expressed as:

[0123]

[0124] Specifically, N x Define t as the set of vertices and E as the set of hyperedges. Therefore, the hypergraph at time slot t It can be defined as VL connects the vertices of the hyperedge, enabling them to communicate. The set of VL is equal to the hypergraph In other words, Defining a Hypergraph Each hyperedge of contains at most N t Vertices.

[0125] Definition 2 (Incident Matrix): In this example, the incident matrix I(N x ,E) to characterize MatrixI(N x ,E) rows represent vertices, columns represent hyperedges, and the matrix I(N x ,E) The element in row i and column j is defined as:

[0126]

[0127] For hypergraphs, this example will The incident matrix I(N x ,E) is expressed as where i n,e = 1 if the nth transitive vertex is included in the eth hyperedge, otherwise i n,e =0, is the number of hyperedges.

[0128] Definition 3 (vertex degree): Define E(n), is the set of all hyperedges that contain vertex n. The insertion degree of E(n) |E(n)| is the degree of vertex n.

[0129] This example uses is the resource matrix, which represents the relationship between VL and resources, so the link conflict degree is expressed as Indicates that D1 represents the total number of resource blocks. Indicates that the dth RB is allocated to the nth VL, otherwise

[0130] According to the above introduction, this example introduces a network conflict degree matrix Φ t , which is a matrix based on the resource matrix R t and the matrix of the incident matrix, which is:

[0131] Φ t =log(max(I′ t R t ,1))

[0132] Among them, I′ t is the incident matrix I t The transpose of . log(·) is the operation of taking the logarithm of each element in the matrix, for example,

[0133]

[0134] By comparing each initial value in the matrix with 1, the maximum value can be expressed as max(·,1), for example,

[0135]

[0136] In the network conflict matrix In e,d > 0 means that the VL belonging to the e-th hyperedge has a conflict because it is assigned to the d-th resource, otherwise ψ e,d ≤0. Therefore, the overall network conflict degree of the entire system can be calculated as follows:

[0137]

[0138] in, Indicates that no conflict occurs in the IoV scenario, otherwise Φ t 1 is Φ t The number of 1s in the game.

[0139] (5) Establishing a resource allocation model

[0140] In this paper, our goal is to implement DAI-based resource allocation in densely deployed IOV systems with overlapping interference scenarios, thereby improving resource reuse, increasing network throughput, and improving network efficiency. The resource allocation model (optimization problem) in densely deployed IOV systems can be mathematically formulated as follows:

[0141]

[0142]

[0143]

[0144]

[0145] C1 to C9 are different constraints. is the maximum transmit power for each VL. Constraints C1 and C2 manage bandwidth allocation for different services to ensure comprehensive end-to-end service quality; constraint C3 ensures that the resources allocated to each IoV link meet the minimum data transmission rate requirements, thereby enhancing the user's smooth service experience; constraint C4 ensures that the transmit power of the nth VL is less than the maximum power of the channel; then, constraint C5 ensures that the queue length of the nth VL does not exceed the total capacity of the link; constraints C6 and C7 ensure that the VL using specific resources is limited by the total number of VLs. Constraint C8 ensures that the resources allocated to each VL are limited by whether the RB has been allocated; constraint C9 is that the degree of network conflict for the entire system is limited by link resources. Services such as voice, video, and critical data can be prioritized by implementing QoS mechanisms and transmitted through network devices.

[0146] (6) Convert the resource allocation model into a Markov decision model

[0147] Resource allocation model is a computationally intensive challenge in the field of time-averaged combinatorial optimization. Due to the rapid growth of computational requirements, traditional centralized solutions cannot effectively solve it. In order to solve this problem, this example transforms the resource allocation problem in the dense Internet of Vehicles scenario into a Markov decision model, namely the MDP model.

[0148] For IoV with dense interference, with VL as the agent, each agent is trained to develop a strategy to determine the optimal action for its channel selection, thereby maximizing its utility function: 1) state; 2) action; 3) reward. The MDP of N agents can be defined as a tuple in is the state space, It is the action space. is the transition probability, is the reward function, and γ is the discount factor. In other words, at a certain time slot t, the agent determines the current state of the environment by t , and make the next action according to the strategy π(a|s). Then, the agent receives the reward r t And add it to the next state s t+1 .

[0149] 1. State space: The state includes the channel state information and resource allocation in IoV, while the action is the resource allocation strategy of the base station in IoV. The channel state information and resource allocation in IoV include:

[0150] 1) Channel coefficient of V2V signal link

[0151] 2) VUE channel coefficient

[0152] 3) Channel allocation of the VL adjacent to the nth VL in the previous time slot t-1,

[0153] in, is the number of neighboring VLs that reused channel l in the previous time slot t-1;

[0154] 4) The current length of the queue on the nth VL transmitter

[0155] The size of the state space depends on the number of available channels in the IoV. When each agent makes a decision, the calculator on the VL estimates the channel coefficients And send it to the sender. VUE channel coefficient It is estimated by the BS and propagated by the BS to each VUE within the communication coverage of the system.

[0156] 2. Action space: Each VL is represented as an agent. The operations performed by agent n include selecting a channel and transmit power The transmission power is divided into N p +1 level, for example,

[0157]

[0158] 3. Reward function: The reward value for the agent to take corresponding actions according to the state. In order to optimize the energy efficiency of IoV and meet the QoS requirements of UE and V2V communication links, the reward function is designed as:

[0159]

[0160] The penalty function U(z) (z refers to the value inside the penalty function brackets) imposes a penalty when any part of the reward is not met, as follows:

[0161]

[0162] Among them, the constant A1<0, λ1, λ2 and λ3 are the weight coefficients of the three components of the defined reward. All agents work normally and can obtain the global reward The agent completes an evaluation of the global reward at the end of each time slot.

[0163] (7) Solve the Markov decision model and allocate resources

[0164] In the MDP problem model constructed in this example, the Q-value function (i.e., the cumulative reward value of the action taken by the agent in the state space S) is the key function for solving the MDP problem, and its definition is as follows:

[0165]

[0166] Among them, Q π (s t ,a t ) is in the current state s t Take action a based on policy π t The cumulative value of the reward obtained, E[] is the expected value, R t |s t ,a t For the current state s t Take action a t Rewards received; t+k |s t ,a t For the current state s t Take action a t The reward value at t+k, r t+k is the reward at time t+k, r t is the reward at the current moment, Q π (s t+1 ,a t+1 )|s t ,a t Under policy π, from state s t+1 Start taking action t+1 Q value.

[0167] The maximum Q value function is obtained by continuously seeking the optimal strategy π * The obtained expression is:

[0168]

[0169] in, a t For state s t+1 Next, select the one that makes the Q value Maximizing action a t+1 The corresponding Q value is a t For the optimal strategy π * Next, from state s t Execute action a t Q value.

[0170] However, the maximum Q value function is a nonlinear equation and has no analytical solution. In order to solve this problem, a neural network is used to approximate the Q value function, namely the DQN algorithm.

[0171] The DQN algorithm structure constructed in this example is as follows Figure 3 As shown in Figure 2, each agent consists of two neural networks (i.e., Q network and target Q network) and an experience replay pool. The network input is the current state of the densely interfered Internet of Vehicles, and the output is a set of actions, i.e., the resource allocation plan.

[0172] In a Q-network, the agent receives state information s at each time step t t As input, and output the current state s t Execute each action a in the action space t The action Q value Q(s t ,a t ; w), where w is the weight of the Q network. After obtaining the estimated Q value, in order to improve the agent's ability to explore actions, a greedy algorithm strategy is used to select the best action a t , that is, select action a with the highest Q value t :

[0173] a t = argmaxQ(s t ,a t ; w)

[0174] In the agent designated action a t Afterwards, the system will receive a return r t , and enter the next state s t+1 . A set of data (s) for each time step t ,a t ,r t ,s t+1 ) is stored in an experience replay buffer of size D using the experience replay method. When the buffer is full, the previous data set is deleted to provide storage space for the latest data.

[0175] The data in the experience replay pool is generated by exploring the system sequentially, but during the neural network training process, these data should be independent and identically distributed. Therefore, instead of updating w in each time slot based on the current state of the graph, M tuples (s) are randomly sampled from the experience replay pool. t ,a t ,r t ,s t+1 ) to eliminate strong correlations between elements and thus reduce the variance of the update.

[0176] In the target Q network, the randomly sampled tuples (s t ,a t ,r t ,s t+1 ) is used to calculate the expected value of the target Q network, namely:

[0177] y t =r t +maxQ(s t+1 ,a t+1 ;w - )

[0178] Q(s t+1 ,a t+1 ;w - ) is the target Q network in the next state s t+1 Execute each next action a in the action space t+1 Q value, w - are the parameters of the target Q network obtained in the previous training process and remain unchanged in each learning process. The target Q network improves the robustness of learning because it updates Q(s t ,a t ; w) and update influence y t A delay is added between the two to avoid oscillation or dispersion of the policy. The Q network can be trained by minimizing the loss function, namely:

[0179] L(w)=(y t -Q(s t ,a t ;w)) 2

[0180] The loss function is calculated using stochastic gradient descent, and the weights of the target Q network are used to update the weights of the target Q network.

[0181] However, the convergence speed of the DQN algorithm is very slow. In order to solve this problem, the federated learning method (FedAvg) is used to improve the convergence speed of the DQN algorithm. The overall algorithm is called FedAvg-DQN algorithm, that is, the dueling dual deep Q network algorithm based on federated learning. In the considered IoV network, all agents use the global model from the FedAvg-DQN server to establish a local DQN network. During the global model training process, each agent randomly extracts a small amount of data set B from the local experience replay pool D to update its own local DQN model. The kth RL agent extracts a small amount of data set B from its own replay buffer D k A small batch of samples B is randomly selected from k , train to minimize the loss function and update the local parameters Then, at the end of a round of learning, the global parameters of the FedAvg-DQN global network are obtained by weighted averaging of the parameters. At time slot t, the loss function L(w t ) can be expressed as follows:

[0182]

[0183] Among them, w t are the network parameters of the global model at time slot t, is the network parameter of the kth agent at time slot t, and K is the total number of agents.

[0184] At time slot t, the server network parameters of the global model are updated as follows:

[0185]

[0186] Where η is the learning rate. At the current time slot t, each agent first obtains the latest parameter w of the global model from the server t-1 As its latest parameter Calculate its loss function Then, each agent calculates the gradient of the loss function Get new local parameters Update the local model. After local training is completed, the agent will update the new local parameters Sent to the global server. The global server aggregates the parameters of all agents and obtains the new global parameter w t The parameters are sent to each agent for the next round of training. Figure 3 As shown in Figure 2, the larger the number of agents, the faster the FedAvg-DQN algorithm can be trained.

[0187] Based on the above method, an embodiment of the present invention further provides a resource management system for densely interfering vehicle networks, which uses the above resource management method for densely interfering vehicle networks to allocate resources for densely interfering vehicle networks.

[0188] (8) Experimental simulation

[0189] Using the urban scenario in 3GPP TR 36.885, the effectiveness of the proposed algorithm is verified through simulation experiments. Table 1 lists the parameters required for the simulation experiment.

[0190] Table 1

[0191]

[0192] In order to verify the effectiveness of the method proposed in this paper (FedAvg-DQN algorithm, abbreviated as FL-DQN algorithm), its simulation performance is compared with the following algorithms, including random network resource allocation (RM), network resource allocation based on greedy algorithm (GA) and network resource allocation based on maximum node degree algorithm (MND).

[0193] A. Convergence of Training

[0194] Figure 4 The convergence behavior of the network resource management algorithm based on FedAvg-DQN under different learning rates is described. During the entire training phase, when the number of V2V links is fixed at 20, the reward value tends to stabilize as the training progresses, and when the learning rate is set to 0.001, the convergence performance is significantly improved. Too high a learning rate can lead to unstable parameter updates and hinder stable convergence. On the contrary, too low a learning rate may cause the algorithm to fall into a local optimum. Therefore, in subsequent experiments, a learning rate of 0.001 was used. Figure 4 The effectiveness of the FedAvg-DQN network resource management algorithm is emphasized.

[0195] The performance evaluation indicators of the Internet of Vehicles include:

[0196] 1) Network throughput: This performance indicator evaluates the network throughput Γ of the IoV after the resource allocation algorithm allocates all communication link resources, which can be expressed as follows:

[0197]

[0198] in, is the average signal-to-dryness ratio, P b is the maximum tolerable bit error rate.

[0199] 2) Energy efficiency: This performance indicator can describe the number of transmission bits that can be obtained when the IoV system consumes one unit of energy, reflecting the efficiency of the system in utilizing energy resources η EE , which can be expressed as:

[0200]

[0201] Among them, P1 is energy resources.

[0202] 3) Signal to Interference and Noise Ratio: The ratio of the signal to the sum of interference and noise in the system. The specific calculation formula is shown above.

[0203] B. Performance comparison of different solutions

[0204] Figure 5 The network throughput achieved using various algorithms in the context of IoV is shown. It is clear that as the number of V2V connections increases, the network throughput also increases. This can be attributed to the enhanced transmission links, which can transmit more data simultaneously, thereby improving the overall network throughput. In addition, the FL-DQN algorithm provided by the present invention is significantly better than the other algorithms. This superiority can be attributed to the algorithm's strong adaptive ability. Therefore, Figure 5 It is proved that the FL-DQN algorithm provided by the invention significantly improves the network throughput of the system.

[0205] Figure 6 The relationship between different transmission powers and energy efficiency is described. In a communication system, higher energy efficiency indicates higher utilization of energy resources. When the number of V2V links is fixed at 20, since energy efficiency is inversely proportional to power, energy efficiency will obviously decrease as the transmission power increases. In addition, Figure 6 The energy efficiency of the FL-DQN algorithm provided by the invention and other three comparison algorithms at different transmission powers is compared. Figure 6 It can be seen that the resource reuse rate of the FL-DQN algorithm exceeds that of the comparison algorithm. This result can be attributed to the fact that the FL-DQN algorithm provided by the invention can adaptively select the best resource allocation scheme according to the current network status. Therefore, Figure 6 It is verified that the FL-DQN algorithm provided by the invention can significantly improve energy efficiency.

[0206] Figure 7 The relationship between the number of V2V links and SINR is shown. The higher the SINR value, the less interference the network has. Figure 7 The SINR values ​​of the FL-DQN algorithm provided by the invention and the other three comparison algorithms under different transmission links are compared. Compared with the other three algorithms, the FL-DQN algorithm provided by the invention has a higher signal-to-noise ratio. This is because the FL-DQN algorithm provided by the invention can effectively avoid interference, improve the signal-to-noise ratio, and ensure the transmission rate by adaptively selecting the best resource allocation scheme. Figure 7 It is verified that the FL-DQN algorithm provided by the invention can effectively avoid interference and improve the signal-to-noise ratio.

[0207] Simulation experiments have proved that the resource management method and system for densely interfering IoV provided by the present invention can meet the delay requirements of V2V communication while reducing interference in IoV.

[0208] In summary, the embodiments of the present invention provide a resource management method and system for a densely interfering Internet of Vehicles, which are used to efficiently and conflict-freely allocate resources in a densely interfering Internet of Vehicles. The contributions of the method and system can be summarized as follows:

[0209] 1) To deal with the overlapping interference problem in dense IoV, an interference hypergraph model is proposed, which analyzes the interference relationship between multiple users through hyperchannels. Using this model, the interference degree of the entire network can be calculated. Then, a resource allocation optimization problem is proposed, which aims to maximize the network throughput while taking into account the interference degree and QoS constraints.

[0210] 2) In order to dynamically make multiple decisions for resource allocation goals in a complex and ever-changing dense IoV environment, this problem is transformed into an MDP model. Each VL is regarded as an agent that runs and interacts in the environment and dynamically selects its actions to maximize the total reward, that is, to allocate resources as efficiently as possible.

[0211] 3) In order to improve the convergence speed of the learning algorithm and network throughput in densely deployed Internet of Vehicles based on distributed computing, a FedAvg-DQN algorithm is proposed to solve the wireless resource allocation problem. The convergence speed of the learning algorithm is improved by using a distributed architecture. The results of simulation experiments show that the FedAvgDQN algorithm significantly improves the network throughput and energy efficiency in the Internet of Vehicles.

[0212] It should also be noted that other Internet of Things networks having the same network structure as the Internet of Vehicles provided by the present invention are also applicable to the present invention.

[0213] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A resource management method for densely interfering Internet of Vehicles, characterized in that: include: Construct a communication model of densely interfering vehicle networks; the constructed communication model of densely interfering vehicle networks includes L vehicle user equipments, namely VUEs, N V2V communication links, namely VLs, and a roadside base station, namely BS; the link between the VUE and the base station has been allocated communication channels and transmission power in advance, the lth channel is used by the lth VUE, so there are L communication channels in total, and the bandwidth of each channel is W, l = 1, 2, ..., L; each VUE channel is allowed to be reused by multiple VLs, but a VL can only access one channel in a time slot; The conflict relationship in the communication model is modeled as a conflict model; the conflict model includes two conflict relationships, direct conflict and hidden conflict; direct conflict means that in the same time slot t, two VLs use the same resource block, i.e., RB, and have the same sending or receiving vertex; hidden conflict means that in the same time slot t, two VLs are simultaneously allocated the same RB, and the sending or receiving vertex of one VL is within the communication range of the sending or receiving vertex of the other VL; The service quality index is used to measure the demand of the communication model, and a service quality model of the communication model is established; the service quality model of the communication model is constructed as follows: in, is the transmission rate that the lth VUE can achieve, The minimum data rate for each VUE, is the instantaneous signal interference plus noise ratio (SINR) of the nth VL when it is assigned to the lth channel, and γ0 is the SINR threshold; is the delay of the nth VL in time slot t Greater than or equal to the maximum acceptable delay D for VL data transmission max The probability of is the queue length of the nth VL at the beginning of time slot t Greater than or equal to the delay exceeding D max Data packet Q max The probability of o is the probability threshold; A hypergraph interference model is established based on the communication model and the conflict model; in the hypergraph interference model, each VUE is used as a vertex, and VL connects the vertices as hyperedges, so that the vertices can communicate with each other; the hypergraph interference model of the t time slot is used as the incident matrix I t To represent, the matrix I t The rows represent vertices, the columns represent hyperedges, and the matrix I t The element i at row n and column e n,e = 1, the nth transitive vertex is included in the eth hyperedge, otherwise i n,e =0; Using the network conflict degree matrix Φ t =log(max(I′ t R t ,1)) represents the situation of dense interference in the Internet of Vehicles at time slot t, I′ t is the incident matrix I t The transpose of R t is the resource matrix at time slot t, resource matrix R t The element at row n and column j Indicates that the dth resource block, i.e., RB, is allocated to the nth VL, otherwise The overall network conflict degree of densely interfering vehicle networks Calculated as: Represents Φ t The number of 1s in Indicates that no conflict occurs, otherwise Based on the hypergraph interference model and the service quality model, a resource allocation model of the communication model is established; the resource allocation model is constructed as follows: Among them, max is maximization, st is to satisfy, C1 to C9 are different constraints; T is the number of time slots, γ l t is the signal interference plus noise ratio of the lth VUE at time slot t, For VUE collection, is the VL set, is the transmission power of the nth VL, is the maximum transmit power for each VL, is the allocation index of VL, The lth channel is assigned to the nth VL, otherwise Converting the resource allocation model into a Markov decision model; In the Markov decision model: each VL is represented as an agent, and each agent is trained to develop a strategy to determine the optimal action for its channel selection to maximize its reward function; the Markov decision model of N agents is defined as a tuple in is the state space, It is the action space. is the transition probability, is the reward function, γ is the discount factor; The state space includes the channel coefficients of the V2V signal link VUE channel coefficient Channel allocation of the VL adjacent to the nth VL in the previous time slot t-1 and the current length of the queue at the nth VL transmitter It refers to the channel power gain associated with the nth V2V link on the lth channel in the current time slot t. Refers to the channel interference parameter of the nth VL to the lth VUE when the nth channel is reused for communication in the current time slot t; The action space consists of the actions that each agent performs, including selecting a channel and transmit power The reward function at time slot t is designed as: The penalty function U(z) imposes a penalty when any part of the reward is not met, and z refers to the value inside the penalty function brackets. U(z) is as follows: Among them, the constant A1<0, λ1, λ2 and λ3 are the weight coefficients of the three components of the defined reward; The Markov decision model is solved to obtain a resource allocation strategy for the densely interfering Internet of Vehicles at the current moment.

2. The resource management method for densely interfering vehicle networks according to claim 1 is characterized in that: When solving the Markov decision model, the DQN algorithm is used to approximate the Q value function of the Markov decision model; in the DQN algorithm, each agent consists of two neural networks, namely, a Q network and a target Q network, and an experience replay pool; the input of the DQN algorithm is the current state of the densely interfered Internet of Vehicles, and the output is a set of actions, namely, a resource allocation plan; In a Q-network, the agent receives state information s at each time step t t As input, and output the current state s t Execute each action a in the action space t The action Q value Q(s t ,a t ; w), where w is the weight of the Q network; after obtaining the estimated Q value, the greedy algorithm strategy is used to select the best action a t , that is, select action a with the highest Q value t ; In the agent designated action a t Afterwards, the system will receive a return r t , and enter the next state s t+1 ; A set of data for each time step (s t ,a t ,r t ,s t+1 ) is stored in an experience replay buffer of size D using the experience replay method; In the target Q network, the randomly sampled tuples (s t ,a t ,r t ,s t+1 ) is used to calculate the expected value of the target Q network, namely: and t =r t +maxQ(s t+1 ,to t+1 ;w - ). Q(s t+1 ,a t+1 ;w - ) is the target Q network in the next state s t+1 Execute each next action a in the action space t+1 Q value, w - are the parameters of the target Q network obtained in the previous training process; the target Q network can be trained by minimizing the loss function, that is: L(w)=(y t -Q(s t ,a t ;w)) 2 The loss function is calculated using stochastic gradient descent, and the weights of the target Q network are used to update the weights of the target Q network.

3. The resource management method for densely interfering vehicle networks according to claim 2 is characterized in that: The federated learning method is used to improve the convergence speed of the DQN algorithm. Specifically: All agents use the global model to build local DQN networks. During the global model training process, the kth agent reads from its own replay buffer D k A small batch of samples B is randomly selected from k , train to minimize the loss function and update the local parameters Then, at the end of a round of learning, the global parameters of the global model are obtained by weighted averaging of the parameters; At time slot t, the loss function L(w t ) is minimized to: Among them, w t are the network parameters of the global model at time slot t, is the network parameter of the kth agent at time slot t, K is the total number of agents; At time slot t, the network parameters of the global model are updated as follows: Where η is the learning rate; in one round of training, at the current time slot t, each agent first obtains the latest parameter w of the global model t-1 As its latest parameter Calculate its loss function Then, each agent calculates the gradient of the loss function Get new local parameters Update the local model; after local training is completed, each agent will update the new local parameters Sent to the server of the global model; the server of the global model aggregates the parameters of all agents to obtain a new global parameter w t And send the parameter to each agent in batches for the next round of training.

Images