Multi-agent system networking and resource optimization method
Through molecular dynamics and interference control methods, combined with the MIADC-DDPG model, resource optimization of multi-agent systems is achieved, resource allocation and interference problems in cellular vehicle networking are solved, network performance and spectrum utilization are improved, and communication quality is ensured and stability.
Patent Information
- Application Number
- CN202410939587.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
In multi-agent systems, especially in cellular vehicle networking scenarios, traditional resource management methods are difficult to meet the resource allocation needs of high-speed changing mobile agent topology networks, resulting in shortage of spectrum resources and interference problems, affecting communication quality and stability.
Using molecular dynamics and interference control methods, the MIADC-DDPG multi-agent system resource optimization collaborative model is used to realize the steady-state cluster management of the agent and the optimal allocation of spectrum resources, reducing interference and improving spectrum utilization.
It effectively improves the network performance of multi-agent systems, improves spectrum utilization, reduces packet loss rate and end-to-end delay, and ensures communication quality and stability.
Smart Images

Figure CN118921712A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of multi-agent system and mobile communication, and specifically relates to a multi-agent system networking and resource optimization method. Background Art
[0002] A multi-agent system (MAS) is a computing system composed of multiple agents interacting in an environment. Cellular-Vehicle to Everything (C-V2X) is one of the typical MAS. In 5G communication networks and future 6G communication networks, large-scale vehicle agents are connected to cellular networks, which brings about an explosive growth in the amount of communication data, causing high loads on network bandwidth and spectrum resources, resulting in high costs and other issues. Therefore, it is extremely necessary to use resource allocation technology to allow each node to collaborate and share limited spectrum resources to improve the resource utilization of cellular networks.
[0003] In the process of networking and resource optimization of multi-agent systems composed of vehicle agents, the following are involved:
[0004] (1) Multi-agent system networking resource optimization model:
[0005] Resource allocation, as a research hotspot in wireless communications, plays a vital role in cellular vehicle network communications. Wireless communication resources include time-frequency resources, power resources, and space resources. How to efficiently optimize wireless resources while ensuring users' needs for diversified and customized services and improving network resource utilization is an urgent problem to be solved. In the MAS scenario, mobile intelligent vehicles have the characteristics of high-speed mobility and frequent topology changes. There are differences in communication and computing capabilities between different ranges and different processing units (intelligent body carrier units, roadside infrastructure). In large-scale MAS, traditional cellular network resource management methods can no longer meet the needs of dynamically changing mobile intelligent bodies. Research on how to perform efficient resource allocation management in a high-speed changing mobile intelligent body topology network is of practical significance for ensuring MAS communication and service quality, and laying a solid foundation for the actual deployment and management of MAS in the later stage.
[0006] (2) Interference coordination among mobile agents:
[0007] Although resource reuse can efficiently utilize spectrum resources and alleviate the shortage of spectrum resources for mobile agents, it also further aggravates the interference in mobile agent communications. In a cell, the V2V link can provide communication services for cellular users and other mobile agents by reusing the wireless spectrum resources of the V2I link, but it also leads to high data loads on the base station and interference between cellular users and mobile agents. In addition, V2X communication may interfere with cellular users or other V2X communications of shared resources, reduce the quality of MAS communication, or even interrupt communication, affecting the stability and safety of mobile agents.
[0008] A suitable interference control model is crucial to ensure the stability and performance of C-V2X communication systems, and has become a research focus in the field of MAS communications. However, the complexity and changes in the channel environment have brought great challenges to the research on C-V2X communication interference control. In the field of wireless communication interference modeling and control, interference modeling and control of cellular network and self-organizing network communications play a key reference role in the research of MAS communications.
[0009] (3) Dynamic clustering among mobile agents:
[0010] Wireless networks are subject to a variety of impairments, including shadowing, path loss, time and / or frequency selective wireless channels, interference and / or multi-user interference. To address these issues, radio resources (such as time slots, frequency bands, transmit power, etc.) should be allocated in a more optimized manner to adapt to different channels and network conditions. Among them, the mobile intelligent agent dynamic cluster (MIADC) solution is the preferred solution, because clustering can classify vehicles with similar behavior patterns or low mutual interference into one category, which can realize centralized management and scheduling of mobile agent communication needs, reduce MAS networking interference, avoid conflicts, ensure multi-agent steady-state networking, and reduce communication congestion and resource waste, which can improve resource utilization efficiency and improve the overall performance of the system. Summary of the invention
[0011] The technical problem to be solved by the present invention is that the present invention provides a multi-agent system networking and resource optimization method, which aims to use molecular dynamics and interference control to perform steady-state clustering of mobile agents, realize centralized management and scheduling of the communication needs of each agent, and based on this, use the resource reuse collaborative model MIADC-DDPG to optimize the allocation of MAS spectrum resources to improve the network performance of mobile agents and improve spectrum utilization.
[0012] To achieve the above object, the technical solution adopted by the present invention is:
[0013] A multi-agent system networking and resource optimization method comprises the following steps:
[0014] S1) Define the cellular vehicle network MAS scenario:
[0015] S2) establishing a MAS communication model and a delay model;
[0016] S3) Establish a molecular dynamics-enabled agent steady-state strategy:
[0017] S4) establishing a clustering algorithm for the steady state of the intelligent agent cluster and controlling interference through intelligent agent networking;
[0018] S5) through MAS resource optimization model;
[0019] S6) solving the multi-agent resource optimization problem through the MIADC-DDPG multi-agent system resource optimization collaborative model;
[0020] S7) Optimize the results through spectrum resources.
[0021] As a further improvement of the present invention, the step S1) defines the cellular vehicle network MAS scenario as follows;
[0022] Applied to highway scenarios, the base station acts as a central control node and data forwarding node to coordinate the communication between mobile intelligent vehicle systems, and provides communication services within the coverage area through centralized scheduling, congestion control and interference coordination. In a single-base station single-cell, the base station is set in the center of the cell, and the mobile intelligent agents are scattered on the roads within the cell according to the spatial Poisson distribution. The relevant information of the V2I link and the V2V link will be generated at different periodic frequencies according to the mobility of the intelligent agent and the topology of the vehicle network, and applied to advanced driving services;
[0023] In the V2X networking model of the Internet of Vehicles, base stations are deployed in the cellular center, which are responsible for collecting information about intelligent vehicle terminal devices and V2X interactions and sensing the communication status from a global perspective. There are M V2I links and K V2V links in the Internet of Vehicles communication system. The set of V2I links is denoted as M = {1, ..., M}, and the set of V2V links is denoted as K = {1, ..., K}. The V2I link provides downlink data services for high-speed mobile multi-agents, and the V2V link supports multi-agent vehicle ad hoc networking communications. The transceiver ends of all terminal devices are equipped with an omnidirectional antenna, and the cellular-based V2X vehicle network realizes the integration of 5G cellular network and vehicle ad hoc network.
[0024] As a further improvement of the present invention, the MAS communication model in step S2) is specifically as follows:
[0025] The path loss model of V2I communication in the cellular vehicle network highway scenario is defined as:
[0026]
[0027] Among them, d 2D Indicates the distance between the transmitter and the receiver, unit: m, d BP is the cutoff distance, and d is defined BP =4h BS h UT f c / c;
[0028] When 10m≤d 2D ≤d BP When , the path loss of the V2I link in the highway scenario is defined as:
[0029] PL1=32.4+21log 10 (d 3D )+20log 10 (f c )
[0030] Among them, f c Indicates the carrier frequency, d 3D Represents the 3D distance between the source intelligent vehicle and the target base station. The specific calculation method is defined as
[0031] When BP ≤d 2D When the distance is ≤5km, the path loss model of the V2I link in the highway scenario is defined as:
[0032] PL2=32.4+40log 10 (d 3D )+20log 10 (f c )-9.5log 10 ((d BP ) 2 +(h BS -h UT ) 2 )
[0033] Among them, f c Indicates the carrier frequency, h BS and h UT Respectively represent the deployment heights of roadside base stations and intelligent connected vehicles, d 3D Represents the 3D distance between the source intelligent vehicle and the target base station. The specific calculation method is defined as
[0034] For V2V link communication in the cellular vehicle network highway scenario, the path loss model is defined as:
[0035] PLV2V =36.85+30log 10 (d 2D )+18.9log 10 (f c )
[0036] In order to improve the spectrum utilization rate of the Internet of Vehicles networking resources, the downlink spectrum resources orthogonally allocated by the V2I link are reused by the V2V link. The channel power gain between the mth V2I link and the base station is expressed as:
[0037] g m,B =PL V2I α m,B h m,B
[0038] Among them, h m,B is the frequency-dependent small-scale rapid fading, which obeys the exponential distribution with unit mean; α m,B Large-scale fading, including path loss and shadow fading;
[0039] The signal-to-interference-to-noise ratio of the receiving user in the V2I link is expressed as:
[0040]
[0041] The signal-to-interference-to-noise ratio of the receiving user in the V2V link is expressed as:
[0042]
[0043] Among them, P m , P k Denote the transmission power of V2I and V2V links respectively, g k represents the channel gain between the kth V2V channel and the interference channel, σ 2 represents the noise power, I m is the interference power received by the V2I link, I k is the interference power received by the V2V link;
[0044] The channel capacity of the V2I link is expressed as:
[0045] C m =Wlog(1+SINR m )
[0046] The channel capacity of the V2V link is expressed as:
[0047] C k =Wlog(1+SINR k )
[0048] Where W is the spectrum bandwidth;
[0049] V2V link interference exists at the transmitting and receiving ends of the V2V link, and its interference power is expressed as:
[0050] I i,j =P k g i,j .
[0051] As a further improvement of the present invention, the MAS communication delay model in step S2) is specifically as follows:
[0052] The average delay of the Internet of Vehicles includes the sum of propagation, transmission, queuing and processing delays. Processing delay refers to the delay caused by the host or router processing after receiving the packet. Transmission delay refers to the time required for the data packet to be sent to the link. Queuing delay refers to the time it takes for the data packet to queue in the input queue and the output queue after entering the router and wait for processing. Transmission delay depends on the size of the data packet and the transmission rate.
[0053] According to Shannon's formula, the wireless link rate is expressed as:
[0054]
[0055] Among them, k i represents the channel fading coefficient, P is the vehicle v i The transmission power, PL i represents the path loss between the vehicle and the target node, where the target node refers to the neighboring vehicle or the roadside base station, σ 2 is the Gaussian white noise power, B represents the channel bandwidth;
[0056] The transmission delay is expressed as:
[0057]
[0058] The propagation delay is expressed as:
[0059]
[0060] Where c is the speed of light;
[0061] Combining formulas (19) and (20), the total delay is expressed as:
[0062] t i =t r +t p
[0063] As a further improvement of the present invention, the molecular dynamics-enabled agent steady-state strategy in step S3) is specifically as follows:
[0064] When a vehicle is driving, it always tries to maintain an appropriate balance distance with surrounding vehicles, that is, the vehicle tends to maintain a dynamic required safety distance so that the vehicle neither lags behind nor follows closely, thereby driving safely and efficiently. The vehicle required safety distance is expressed as:
[0065]
[0066] Where: t0 is the driver's reaction time, a0 is the maximum deceleration of the vehicle, and 1 / 2a0 = 0.07 m / s 2 , v is the instantaneous speed of the vehicle;
[0067] On the basis of the required safety distance and combined with the actual inter-vehicle distance of intelligent agents, the saturation coefficient of the required distance is further obtained:
[0068]
[0069] Where: Y is the saturation coefficient of the required distance, L is the actual vehicle distance;
[0070] When Y < 1, there is basically no risk of collision between the rear vehicle and the front vehicle. However, from the perspective of lane utilization and traffic operation efficiency, the rear vehicle should increase its speed to make Y = 1, thereby achieving efficient driving. When Y > 1, there is a certain risk of collision. Therefore, the rear vehicle should reduce its speed to make Y = 1.
[0071] The interaction between vehicles is caused by the combined effects of repulsion and attraction, and is affected by the required safety distance. The potential function is a mathematical expression that describes the interaction between molecules, and its form is:
[0072]
[0073] Where: τ is equal to the depth of the potential energy well, σ is the distance between the two bodies when the potential energy of the interaction is exactly zero. In the vehicle modeling based on molecular dynamics, σ = X. r is the distance between the two bodies at the bottom of the potential energy well, at this time r = L. Therefore, in vehicle molecular dynamics, the required safe distance X is the critical distance when the interaction potential between vehicles is 0, and the actual distance between vehicles L is the distance between the two bodies at the bottom of the potential energy well;
[0074] Therefore, in vehicle molecular dynamics, the potential function of the interaction between vehicles is expressed as:
[0075]
[0076] Among them: the first term corresponds to the effect of mutual repulsion between vehicle molecules at close distances, and the second term corresponds to the effect of mutual attraction between vehicle molecules at long distances;
[0077] From the potential function, we can deduce the vehicle interaction acceleration:
[0078]
[0079] Where: m is the vehicle mass
[0080] The vehicle interaction acceleration is simplified to:
[0081]
[0082] The urban road is a two-way multi-lane model. Considering the communication radius and networking steady-state requirements of the Internet of Vehicles, the division of vehicle clusters should be based on the vehicle transmission range. The vehicle transmission range changes with the change of node density on the road. The calculation formula for the vehicle transmission range radius is:
[0083]
[0084] Where: R is the transmission range, λ is the vehicle node density, N is the number of vehicles on the road section, and P is the probability of road section connectivity;
[0085] The communication distance of the vehicle node is defined as R. When the distance between two nodes is less than R, a two-way communication link is formed between the nodes. Once the distance between the nodes exceeds R, the communication link is broken. While maintaining a certain network connectivity probability, the increase in node density will lead to a decrease in the transmission radius, and vice versa.
[0086] As a further improvement of the present invention, the clustering algorithm of the agent cluster in the steady state and the interference control through the agent networking in step S4) are specifically as follows;
[0087] A two-stage resource allocation algorithm is proposed. In the first stage, the interference is minimized based on the bipartite graph matching theory. In the second stage, the DDPG algorithm is used to optimize the resource management strategy of cellular vehicle networks.
[0088] Introducing channel interference factors and establishing optimization goals, V2V link communications are divided into mutually exclusive spectrum sharing clusters to minimize vehicle networking system interference and maintain 5G cellular vehicle networking steady state while ensuring the stability and efficiency of the molecular dynamics system of the connected vehicle.
[0089]
[0090] The first optimization objective is the interference between vehicles, and the second is the saturation coefficient of the required safety distance of the intelligent networked vehicle, minimizing |Y i,j -1|, so that Y i,j As close to 1 as possible, it can reduce the risk of collision between intelligent network vehicles and ensure road safety while improving road utilization and traffic efficiency. i,j≤R is the constraint condition of the optimization problem, which constrains the distance between vehicles in the cluster to not exceed the vehicle communication distance R to ensure that the vehicles in the cluster can ensure normal communication and information exchange;
[0091] A clustering mechanism is introduced to cluster all intelligent connected vehicles in the cellular vehicle network system. In the cellular vehicle network system, vehicles regularly report their positions and speeds to the base station, so the base station understands the vehicle network topology. Each intelligent connected vehicle belongs to a cluster, each cluster corresponds to a sub-channel, and vehicles in the same cluster will be assigned to the same sub-channel. In the clustering problem, the clustering results will tend to place vehicles with strong interference and high collision risks into different clusters, which is equivalent to separating them into different sub-channels, which helps to reduce coarse-scale inter-cell interference;
[0092] Graph theory is introduced to describe the optimization problem. Interference management and molecular dynamics management between vehicles are captured by weighted undirected graphs. The weight of the undirected graph is w i,j =I i,j , where I i,j Interference between vehicles, including path loss, shadowing, and small-scale fading;
[0093] Each vehicle on the road is modeled as a vertex in an undirected graph. When two vertices interfere with each other, they are connected by an edge to form an undirected graph G, whose vertex set is V(G) and edge set is E(G). The clustering goal is to divide the K vertices of the weighted undirected graph into N non-intersecting clusters L1, L2, ..., L N , so that L1∪L2∪...∪L N =V(G), which minimizes the intra-cluster interference of all vehicle clusters and ensures safe and efficient driving of vehicles based on molecular dynamics;
[0094] The maximum cut problem is solved by the improved Kernighan-Lin algorithm.
[0095] As a further improvement of the present invention, the heuristic algorithm for solving the MAXN-CUT problem based on the Kernighan-Lin algorithm in step S4) is specifically as follows:
[0096] Initialize the cluster;
[0097] Definition: The weight of each cluster When the cluster has no nodes assigned to it, W i =0;
[0098] Step 1: Select weight w i,j The largest edge connects nodes i and j, assigning nodes i and j to the two clusters C that are not assigned nodes. i =[i],C j= [j], and in the process of initializing the cluster, w i,j Set to 0, and repeat this operation until there is a node in each of the n clusters;
[0099] Step 2: For node k that is not assigned a cluster, calculate the time required to assign the node k to cluster C. i When , the total weight added to the cluster Assign the node k to the weight gain ΔW i The smallest corresponding cluster, C i =[C i ,k];
[0100] Step 3: Repeat Step 2 until each node is assigned to a cluster and n original clusters are obtained.
[0101] As a further improvement of the present invention, the step S6) MIADC-DDPG multi-agent system resource optimization collaborative model is specifically as follows:
[0102] The design objective function is as follows:
[0103]
[0104] C m ≥C0 (b)
[0105]
[0106] t i ≤t0 (e)
[0107] In the above optimization problem, the objective function is to maximize the system downlink throughput, which is expressed as ∑C m , constraint (a) means that the transmission power of M V2I links cannot be greater than the maximum transmission power of the V2I link Constraint (b) indicates that the capacity of each V2I channel needs to be greater than the minimum required throughput C0 of the link, and constraint (c) indicates that the transmit power of the K V2V links needs to be less than or equal to the maximum transmit power of the V2V link. p0 represents the tolerable interruption probability of the link, constraint (d) indicates that the transmission rate of K V2V links must be less than or equal to the tolerable interruption probability, that is, to ensure the connectivity of the V2V link, constraint (e) indicates that the delay generated by the V2I link and the V2V link cannot be greater than the maximum delay constraint of the Internet of Vehicles, and t0 represents the maximum delay that can be tolerated by the Internet of Vehicles communication;
[0108] The above problems are solved through the MIADC-DDPG multi-agent collaborative model;
[0109] Assume that the strategy executed at time t is A t , working status is St , the reward is R t , the above strategies, states and rewards are defined as follows:
[0110] State space: At time slot t, the state of the intelligent connected vehicle is described as:
[0111] s k (t) = {G k (t),I k ,P k (t-1),SINR k}
[0112] Among them, G k I (t) is the channel gain associated with agent k at time t; k is the interference power received by agent k; P k (t-1) represents the transmission power of agent k in the previous time slot; SINR k is the signal-to-noise ratio of agent k;
[0113] According to the state and strategy, each agent will make decisions on transmission power allocation and channel selection one by one. The action set of the agent in time slot t is defined as:
[0114] a k (t) = {P k (t),ρ k (t)}
[0115] Among them, P k (t) represents the transmission power of agent k in time slot t; ρ k (t) is the channel selection of agent k;
[0116] The reward is used to evaluate the agent in state s t ∈S, take action a t ∈A, transition to the next state s t+1 ∈S, the system incentive is used to evaluate the effect of the agent taking a specific action. The goal of resource optimization is to maximize the uplink throughput of the Internet of Vehicles by interacting with the environment while satisfying constraints such as QoS. The reward function is defined as the throughput of the cellular Internet of Vehicles system;
[0117]
[0118] The model updates the parameters ω of the Critic current network and θ of the Actor current network through the gradient back propagation of the neural network, thereby updating the Critic target network and the Actor target network;
[0119] Rewards provide evaluable feedback for the decision-making of the intelligent agent performing deep reinforcement learning, and the ultimate goal of the strategy π is to obtain the maximum cumulative reward. The intelligent agent uses the rewards of environmental feedback as the basis to optimize the value function, and then uses the value function to evaluate the strategy. The strategy maps the state to the action, and the strategy optimization is to find the optimal mapping. The goal of the DDPG framework is to find the optimal strategy π* in a distributed fully observable Markov decision process and maximize the reward function within a limited time frame;
[0120] In DDPG, the agent uses a deterministic policy network μ(s|θ μ ) to select actions, where θ μ is the parameter of the policy network. During training, the agent adjusts the state of the policy network according to the current state s t Select action a through the policy network t , then, action a t Applied to environments t , the agent observes the next state s t+1 and reward r t The entire transition process is expressed as:
[0121] a t =μ(s|θ μ )
[0122] s t+1 =F(s t ,a t )+σ t
[0123] r t =R(s t ,a t )
[0124] DDPG trains the policy network and the value function network by minimizing the loss of the Q value, where the Q value is the action value function. The calculation of the Q value is expressed as:
[0125] Q(s t ,a t )=r t +γ·Q(s t+1 ,μ(s t+1 |θ μ )|θ Q )
[0126] Where γ is the discount factor, θ Q It is the parameter of the value function network, and the Q value is used to calculate the loss of the policy network to optimize the policy.
[0127] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0128] The present invention provides a multi-agent system networking and resource optimization method, which designs a networking model from two dimensions: cluster motion control and intra-network interference coordination, and combines the intelligent scheduling of the agents and network resources within the cluster to ensure the maximum probability of nodes successfully accessing the channel, thereby effectively improving the performance of the wireless self-organizing network of the agent system. Based on the molecular dynamics theory, the multi-agent cluster steady-state management is carried out to ensure the overall motion consistency of the multi-agent when moving; in order to solve the fast channel deviation caused by high mobility, a cluster interference coordination model is proposed to improve the cluster networking service quality between multi-agents and maintain the networking steady state; a dynamic cluster-deep determination policy gradient algorithm framework based on mobile agents is built to optimize the scheduling and allocation of network resources of the multi-agent system to improve the system throughput, reduce the packet loss rate and end-to-end delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0129] Figure 1 It is a schematic diagram of the overall implementation framework of the present invention;
[0130] Figure 2 It is the C-V2X networking communication model of the present invention;
[0131] Figure 3 An undirected graph representation of the optimization problem of the present invention;
[0132] Figure 4 FIG. 4 is a diagram of the MIADC-DDPG algorithm architecture of the present invention. DETAILED DESCRIPTION
[0133] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:
[0134] As a specific embodiment of the present invention, the present invention is a MAS networking resource optimization model and an intelligent agent steady-state clustering method under the 5G-V2X architecture, which aims to use molecular dynamics and interference control to perform steady-state clustering of mobile intelligent agents, realize centralized management and scheduling of the communication needs of each intelligent agent, and based on this, use the resource reuse collaborative model MIADC-DDPG to optimize the allocation of MAS spectrum resources to improve the network performance of mobile intelligent agents and improve spectrum utilization. The overall implementation framework is as follows: Figure 1 shown.
[0135] Cellular Vehicle Network MAS scenario definition:
[0136] The invention can be applied to highway scenarios. The base station acts as a central control node and data forwarding node to coordinate the communication between mobile intelligent vehicle systems and provide communication services within the coverage area through centralized scheduling, congestion control and interference coordination. In a single-base station single-cell, the base station is set in the center of the cell, and the mobile intelligent agents are scattered on the roads within the cell according to the spatial Poisson distribution. The relevant information of the V2I link and the V2V link will be generated at different periodic frequencies according to the mobility of the intelligent agent and the topology of the vehicle network, and applied to services such as advanced driving.
[0137] In the V2X networking model of the Internet of Vehicles, a base station (BS) is deployed in the cellular center, which is responsible for collecting information about intelligent vehicle terminal devices and V2X interactions and perceiving the communication status from a global perspective. In the Internet of Vehicles communication system, there are M V2I links and K V2V links. The set of V2I links is denoted as M = {1,...,M}, and the set of V2V links is denoted as K = {1,...,K}. The V2I link provides downlink data services for high-speed moving multi-agents, and the V2V link supports multi-agent vehicle ad hoc networking communications. The transceivers of all terminal devices are equipped with an omnidirectional antenna. The cellular-based V2X vehicle network can realize the integration of 5G cellular networks and vehicle ad hoc networks. The C-V2X networking communication model is as follows: Figure 2 shown.
[0138] Molecular dynamics-based agent steady-state strategy:
[0139] Based on the theory of molecular dynamics, there is a "vehicle molecular equilibrium distance" during the driving process of intelligent vehicles. At this time, the distance between vehicles is the required safety distance of the vehicles. The driver does not need to accelerate or decelerate significantly. The intelligent body maintains a constant speed and can maintain a high driving efficiency while ensuring safe driving.
[0140] The required safety distance refers to the minimum safe braking distance required by the intelligent vehicle at a specific speed, from the time the driver starts to react to the time the vehicle completely stops. This distance is similar to the equilibrium distance in molecular dynamics, which means that when the distance between vehicles is maintained, the interaction between vehicles is almost zero, achieving a relatively static safety state. When a vehicle is driving, it always tries to maintain an appropriate equilibrium distance with the surrounding vehicles. That is, the vehicle tends to maintain a dynamic required safety distance, so that the vehicle neither lags behind nor follows closely, thereby driving safely and efficiently. The vehicle's required safety distance can be expressed as:
[0141]
[0142] Where: t0 is the driver's reaction time, a0 is the maximum deceleration of the vehicle, and 1 / 2a0 = 0.07 m / s 2 , v is the instantaneous speed of the vehicle.
[0143] On the basis of the required safety distance and combined with the actual inter-vehicle distance of intelligent agents, the saturation coefficient of the required distance is further obtained:
[0144]
[0145] Where: Y is the saturation coefficient of the required distance, and L is the actual vehicle distance.
[0146] The saturation coefficient of the required distance is an important parameter of the vehicle's operating status, which can keenly reveal the shortcomings of the vehicle during driving. When Y < 1, there is basically no risk of collision between the rear vehicle and the front vehicle. However, from the perspective of lane utilization and traffic operation efficiency, the speed of the rear vehicle can be increased to make Y = 1, thereby achieving efficient driving. When Y > 1, there is a certain risk of collision. Therefore, the rear vehicle should reduce its speed to make Y = 1.
[0147] The interaction between vehicles is caused by the combined effects of repulsion and attraction, and is affected by the required safety distance. The potential function is a mathematical expression that describes the interaction between molecules. The Lennard-Jones potential function is currently the most widely used potential energy function, especially for describing the interaction between inert gas molecules. Its general form is:
[0148]
[0149] Among them: τ is equal to the depth of the potential energy well, σ is the distance between the two bodies when the potential energy of the interaction is exactly zero. In the vehicle modeling based on molecular dynamics, σ = X. r is the distance between the two bodies at the bottom of the potential energy well, at this time r = L. Therefore, in vehicle molecular dynamics, the required safe distance X is the critical distance when the interaction potential between vehicles is 0, and the actual distance between vehicles L is the distance between the two bodies at the bottom of the potential energy well.
[0150] Therefore, in vehicle molecular dynamics, the potential function of the interaction between vehicles can be expressed as:
[0151]
[0152] Among them: the first term corresponds to the effect of mutual repulsion between vehicle molecules at close range, and the second term corresponds to the effect of mutual attraction (for example, through van der Waals force) between vehicle molecules at long distance.
[0153] The potential function can be used to deduce the vehicle interaction acceleration:
[0154]
[0155] Where: m is the vehicle mass
[0156] The vehicle interaction acceleration can be simplified to:
[0157]
[0158] Urban roads are bidirectional multi-lane models. Considering the communication radius and networking steady-state requirements of the Internet of Vehicles, the division of vehicle clusters should be based on the vehicle transmission range, and the vehicle transmission range changes with the change of node density on the road. The calculation formula for the vehicle transmission range radius is:
[0159]
[0160] Where: R is the transmission range, λ is the vehicle node density, N is the number of vehicles on the road section, and P is the probability of road section connectivity.
[0161] The communication distance of the vehicle node is defined as R. When the distance between two nodes is less than R, a two-way communication link is formed between the nodes; once the distance between the nodes exceeds R, the communication link is broken. And while maintaining a certain network connectivity probability, the increase in node density will lead to a decrease in the transmission radius, and vice versa.
[0162] Cellular vehicle networking communication model:
[0163] The path loss model of V2I communication in the cellular vehicle network highway scenario is defined as:
[0164]
[0165] Among them, d 2D Indicates the distance between the transmitter and the receiver (unit: m), d BP is the cutoff distance, and d is defined BP =4h BS h UT f c / c.
[0166] When 10m≤d 2D ≤d BP When , the path loss of the V2I link in the highway scenario is defined as:
[0167] PL1=32.4+21log 10 (d 3D )+20log 10 (f c ) (9)
[0168] Among them, f c Indicates the carrier frequency, d 3D Represents the 3D distance between the source intelligent vehicle and the target base station. The specific calculation method is defined as
[0169] When BP ≤d 2DWhen the distance is ≤5km, the path loss model of the V2I link in the highway scenario is defined as:
[0170] PL2=32.4+40log 10 (d 3D )+20log 10 (f c )-9.5log 10 ((d BP ) 2 +(h BS -h UT ) 2 ) (10)
[0171] Among them, f c Indicates the carrier frequency, h BS and h UT Respectively represent the deployment heights of roadside base stations and intelligent connected vehicles, d 3D Represents the 3D distance between the source intelligent vehicle and the target base station. The specific calculation method is defined as
[0172] For V2V link communication in the cellular vehicle network highway scenario, the path loss model is defined as:
[0173] PL V2V =36.85+30log 10 (d 2D )+18.9log 10 (f c ) (11)
[0174] In order to improve the spectrum utilization of Internet of Vehicles networking resources, the downlink spectrum resources orthogonally allocated by the V2I link are reused by the V2V link. The channel power gain between the mth V2I link and the base station is expressed as:
[0175] g m,B =PL V2I α m,B h m,B (12)
[0176] Among them, h m,B is the frequency-dependent small-scale rapid fading, which obeys the exponential distribution with unit mean; α m,B It is large-scale fading, including path loss and shadow fading.
[0177] The signal-to-interference-to-noise ratio of the receiving user in the V2I link can be expressed as:
[0178]
[0179] The signal-to-interference-to-noise ratio of the receiving user in the V2V link can be expressed as:
[0180]
[0181] Among them, P m , P k Denote the transmission power of V2I and V2V links respectively, g k represents the channel gain between the kth V2V channel and the interference channel, σ 2 represents the noise power, I m is the interference power received by the V2I link, I k is the interference power received by the V2V link.
[0182] The channel capacity of the V2I link can be expressed as:
[0183] C m =Wlog(1+SINR m ) (15)
[0184] The channel capacity of the V2V link can be expressed as:
[0185] C k =Wlog(1+SINR k ) (16)
[0186] Where W is the spectrum bandwidth.
[0187] V2V link interference exists at the transmitting and receiving ends of the V2V link, and its interference power can be expressed as:
[0188] I i,j =P k g i,j (17)
[0189] Cellular vehicle network MAS communication delay model:
[0190] In cellular Internet of Vehicles communications, Internet of Vehicles information needs to be transmitted with high reliability and extremely low transmission delay. However, the traditional mobile communication network architecture cannot meet the ultra-low latency and high reliability requirements of the Internet of Vehicles. Therefore, based on the 5G architecture, establishing a universal Internet of Vehicles optimization model is of great significance to the network structure of the 5G Internet of Vehicles.
[0191] The average delay of the Internet of Vehicles includes the sum of propagation, transmission, queuing and processing delays. Processing delay refers to the delay caused by the host or router processing after receiving the packet (analyzing the header, extracting data, error checking, routing selection, etc.). Due to the powerful computing power of the server, this delay is generally at the microsecond level. Transmission delay refers to the time required for a data message to be sent to the link. The propagation delay is usually very small. Queuing delay refers to the time it takes for a data message to queue in the input queue and the output queue after entering the router and wait for processing. Transmission delay depends on the size of the data packet and the transmission rate.
[0192] According to Shannon's formula, the wireless link rate is expressed as:
[0193]
[0194] Among them, k i represents the channel fading coefficient, P is the vehicle v i The transmission power, PL i represents the path loss between the vehicle and the target node (neighboring vehicle or roadside base station), σ 2 is the Gaussian white noise power, and B is the channel bandwidth.
[0195] The transmission delay is expressed as:
[0196]
[0197] The propagation delay is expressed as:
[0198]
[0199] Where c is the speed of light.
[0200] Combining formulas (19) and (20), the total delay is expressed as:
[0201] t i =t r +t p (twenty one)
[0202] Clustering algorithm considering the steady state of traffic clusters:
[0203] In the case where V2V link users and V2I link users share spectrum resources, minimizing interference while maintaining system throughput is a key research issue in the development of the Internet of Vehicles. To address this problem, a two-stage resource allocation algorithm is proposed. In the first stage, the interference is minimized based on the bipartite graph matching theory, and in the second stage, the DDPG algorithm is used to optimize the resource management strategy of the cellular Internet of Vehicles.
[0204] The channel interference of 5G cellular vehicle network will seriously affect the quality of V2X communication link. In this paper, the channel interference factor is introduced, and the optimization goal is established. By optimizing the V2V link communication into non-intersecting spectrum sharing clusters, the vehicle network system interference is minimized while ensuring the stability and efficiency of the molecular dynamics system of the networked vehicle, so as to improve the communication quality, ensure the high efficiency and stability of the intelligent networked vehicle system, and the ultra-low interference and ultra-high quality of communication in the dynamic and time-varying C-V2X channel environment, and maintain the steady state of 5G cellular vehicle network.
[0205]
[0206] d i,j ≤R (23)
[0207] The first optimization goal is the interference between vehicles. In the 5G cellular vehicle network system with high-speed topology changes, effective interference control is crucial to improving the communication quality and network stability of the vehicle network. The second optimization goal is the saturation coefficient of the required safety distance of the intelligent networked vehicle, which minimizes |Y i,j -1|, so that Y i,j As close to 1 as possible, it can improve road utilization and traffic efficiency while reducing the risk of intelligent network vehicle collision and ensuring road driving safety. Formula (23) is the constraint condition of the optimization problem, which constrains the distance between vehicles in the cluster to not exceed the vehicle communication distance R to ensure that the vehicles in the cluster can ensure normal communication and information exchange.
[0208] Reasonable clustering schemes can reduce control overhead and improve network scalability. In order to effectively utilize heterogeneous technologies by grouping vehicles into clusters and effectively manage and control vehicle network interference, a clustering mechanism is introduced. All intelligent connected vehicles in the cellular vehicle network system are clustered. In the cellular vehicle network system, vehicles regularly report their location and speed to the base station, so the base station understands the vehicle network topology. Each intelligent connected vehicle belongs to a cluster, each cluster corresponds to a sub-channel, and vehicles in the same cluster will be assigned to the same sub-channel. In the clustering problem, the clustering results will tend to place vehicles with strong interference and high collision risks into different clusters, which is equivalent to separating them into different sub-channels, which helps to reduce coarse-scale intercell interference (ICI).
[0209] In order to solve the optimization problem, graph theory is introduced to describe the optimization problem. The interference management between vehicles and molecular dynamics management are captured by weighted undirected graphs. The undirected graph of the optimization problem is as follows: Figure 3 As shown, the weight w of the undirected graph i,j =I i,j , where I i,jis the interference between vehicles, including path loss, shadowing and small-scale fading.
[0210] Each vehicle on the road is modeled as a vertex in an undirected graph. When two vertices interfere with each other, they are connected by an edge to form an undirected graph G, whose vertex set is V(G) and edge set is E(G). The clustering goal is to divide the K vertices of the weighted undirected graph into N non-intersecting clusters L1, L2, ..., L N , so that L1∪L2∪...∪L N =V(G), which minimizes the intra-cluster interference of all vehicle clusters and ensures safe and efficient driving of vehicles based on molecular dynamics. It is also called the MAXN-CUT problem, also called the maximum cut problem. The improved Kernighan-Lin algorithm is used to solve the maximum cut problem.
[0211] The Kernighan-Lin algorithm is the most well-known heuristic algorithm in the field of graph partitioning. It is usually used to divide a large graph into two subgraphs of approximately equal size while keeping the total weight of the two groups minimum. Its basic idea is based on a greedy strategy, which continuously optimizes the partitioning scheme to reduce the total weight after partitioning, and iteratively exchanges nodes to improve the quality of partitioning until the optimal partitioning scheme is reached. The Kernighan-Lin algorithm first randomly divides the nodes in the graph into two subgraphs, and then calculates the gain of each node in the first group and each node in the second group. The exchange gain is equal to the difference in the total weight before and after the exchange of the two nodes. The pairing with the highest gain is exchanged and fixed. The fixed nodes will no longer be exchanged. Repeat the exchange operation until all nodes are fixed, or the exchange gain is not a positive value, and then the cycle ends. Algorithm 1 shown in Table 1 is a heuristic algorithm for solving the MAX N-CUT problem based on the Kernighan-Lin algorithm proposed in this application.
[0212] Table 1
[0213]
[0214] Resource optimization and coordination model of multi-agent system based on MIADC-DDPG;
[0215] The goal of vehicle network resource optimization is to maximize the system downlink throughput while ensuring the reliability of each V2V link to the maximum extent by jointly optimizing the system downlink throughput and V2V link connectivity in a dynamic and time-varying C-V2X networking environment to address the problems of high communication power consumption, spectrum tension and low energy efficiency. In view of this, the design objective function is as follows:
[0216]
[0217] Cm ≥C0(24b)
[0218]
[0219] t i ≤t0(24e)
[0220] In the above optimization problem, the objective function is to maximize the system downlink throughput, which is expressed as ΣC m Constraint (24a) indicates that the transmission power of the M V2I links cannot be greater than the maximum transmission power of the V2I link Constraint (24b) indicates that the capacity of each V2I channel needs to be greater than the minimum required throughput C0 of the link. Constraint (24c) indicates that the transmit power of the K V2V links needs to be less than or equal to the maximum transmit power of the V2V link. p0 represents the tolerable interruption probability of the link, and constraint (24d) indicates that the transmission rate of K V2V links must be less than or equal to the tolerable interruption probability, that is, to ensure the connectivity of the V2V link. Constraint (24e) indicates that the delay generated for the V2I link and the V2V link cannot be greater than the maximum delay constraint of the Internet of Vehicles, and t0 represents the maximum delay that can be tolerated by the Internet of Vehicles communication.
[0221] In order to meet the diverse needs of different links in cellular vehicle networks, namely, large downlink throughput requirements and high V2V connection reliability requirements, the MIADC-DDPG multi-agent collaborative model is used to solve the above problems.
[0222] Assume that the strategy executed at time t is A t , working status is S t , the reward is R t The above strategies, states and rewards are defined as follows:
[0223] State space: At time slot t, the state of the intelligent connected vehicle can be described as:
[0224] s k (t) = {G k (t),I k ,P k (t-1),SINR k} (25)
[0225] Among them, G k I (t) is the channel gain associated with agent k at time t; k is the interference power received by agent k; P k (t-1) represents the transmission power of agent k in the previous time slot; SINR k is the signal-to-noise ratio of agent k.
[0226] According to the state and strategy, each agent will make decisions on transmission power allocation and channel selection one by one. The action set of the agent in time slot t is defined as:
[0227] a k (t) = {P k (t),ρ k (t)} (26)
[0228] Among them, P k (t) represents the transmission power of agent k in time slot t; ρ k (t) is the channel selection of agent k.
[0229] The reward is used to evaluate the agent in state s t ∈S, take action a t ∈A, transition to the next state s t+1 The system incentive obtained by ∈S is used to evaluate the effect of the agent taking a specific action. The goal of resource optimization in this paper is to maximize the uplink throughput of the Internet of Vehicles by interacting with the environment while satisfying constraints such as QoS. Therefore, in this section, the reward function is defined as the throughput of the cellular Internet of Vehicles system.
[0230]
[0231] The DDPG algorithm is a deep reinforcement learning algorithm based on the Actor-Critic architecture. The Actor network and the Critic network estimate the value function and final strategy of the system. The MIADC-DDPG algorithm architecture is shown in the figure below. Figure 4 As shown in the figure, the model updates the parameters ω of the Critic current network and the parameters θ of the Actor current network through the gradient back propagation of the neural network, thereby updating the Critic target network and the Actor target network.
[0232] Rewards provide evaluable feedback for deep reinforcement learning agents to make decisions. The ultimate goal of the strategy π is to obtain the maximum cumulative reward. The agent uses the rewards of environmental feedback as a basis to optimize the value function, and then uses the value function to evaluate the strategy. The strategy maps the state to the action, and strategy optimization is to find the optimal mapping. The goal of the DDPG framework proposed in this paper is to find the optimal strategy π* in a distributed fully observable Markov decision process (DFMDP) to maximize the reward function within a limited time frame.
[0233] In DDPG, the agent uses a deterministic policy network μ(s|θ μ ) to select actions, where θ μis the parameter of the policy network. During training, the agent t Select action a through the policy network t Then, action a t Applied to environments t , the agent observes the next state s t+1 and reward r t The entire transition process can be expressed as:
[0234]
[0235] DDPG trains the policy network and value function network by minimizing the loss of Q value (action value function). The calculation of Q value can be expressed as:
[0236] Q(s t ,a t )=r t +γ·Q(s t+1 ,μ(s t+1 |θ μ )|θ Q ) (29)
[0237] Where γ is the discount factor, θ Q is a parameter of the value function network. The Q value is used to calculate the loss of the policy network, thereby optimizing the policy. The entire transition function and Q value calculation process is a key component of the DDPG algorithm. Algorithm 2 shown in Table 2 is the MIADC-DDPG multi-agent system resource optimization collaborative model proposed in this application.
[0238] Table 2
[0239]
[0240] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.
Claims
1. A multi-agent system networking and resource optimization method, characterized in that: The following steps are involved: S1) Define the cellular vehicle network MAS scenario: S2) establishing a MAS communication model and a delay model; S3) Establish a molecular dynamics-enabled agent steady-state strategy: S4) establishing a clustering algorithm for the steady state of the intelligent agent cluster and controlling interference through intelligent agent networking; S5) through MAS resource optimization model; S6) solving the multi-agent resource optimization problem through the MIADC-DDPG multi-agent system resource optimization collaborative model; S7) Optimize the results through spectrum resources.
2. A multi-agent system networking and resource optimization method according to claim 1, characterized in that: The step S1) defines the cellular vehicle network MAS scenario as follows; Applied to highway scenarios, the base station acts as a central control node and data forwarding node to coordinate the communication between mobile intelligent vehicle systems, and provides communication services within the coverage area through centralized scheduling, congestion control and interference coordination. In a single-base station single-cell, the base station is set in the center of the cell, and the mobile intelligent agents are scattered on the roads within the cell according to the spatial Poisson distribution. The relevant information of the V2I link and the V2V link will be generated at different periodic frequencies according to the mobility of the intelligent agent and the topology of the vehicle network, and applied to advanced driving services; In the V2X networking model of the Internet of Vehicles, base stations are deployed in the cellular center, which are responsible for collecting information about intelligent vehicle terminal devices and V2X interactions and sensing the communication status from a global perspective. There are M V2I links and K V2V links in the Internet of Vehicles communication system. The set of V2I links is denoted as M = {1, ..., M}, and the set of V2V links is denoted as K = {1, ..., K}. The V2I link provides downlink data services for high-speed mobile multi-agents, and the V2V link supports multi-agent vehicle ad hoc networking communications. The transceiver ends of all terminal devices are equipped with an omnidirectional antenna, and the cellular-based V2X vehicle network realizes the integration of 5G cellular network and vehicle ad hoc network.
3. A multi-agent system networking and resource optimization method according to claim 1, characterized in that: The MAS communication model in step S2) is specifically as follows; The path loss model of V2I communication in the cellular vehicle network highway scenario is defined as: Among them, d 2D Indicates the distance between the transmitter and the receiver, unit: m, d BP is the cutoff distance, and d is defined BP =4h BS h UT f c / c; When 10m≤d 2D ≤d BP When , the path loss of the V2I link in the highway scenario is defined as: PL1=32.4+21log 10 (d 3D )+20log 10 (f c ) Among them, f c Indicates the carrier frequency, d 3D Represents the 3D distance between the source intelligent vehicle and the target base station. The specific calculation method is defined as When BP ≤d 2D When the distance is ≤5km, the path loss model of the V2I link in the highway scenario is defined as: PL2=32.4+40log 10 (d 3D )+20log 10 (f c )-9.5log 10 ((d BP ) 2 +(h BS -h UT ) 2 ) Among them, f c Indicates the carrier frequency, h BS and h UT Respectively represent the deployment heights of roadside base stations and intelligent connected vehicles, d 3D Represents the 3D distance between the source intelligent vehicle and the target base station. The specific calculation method is defined as For V2V link communication in the cellular vehicle network highway scenario, the path loss model is defined as: PL V2V =36.85+30log 10 (d 2D )+18.9log 10 (f c ) In order to improve the spectrum utilization rate of the Internet of Vehicles networking resources, the downlink spectrum resources orthogonally allocated by the V2I link are reused by the V2V link. The channel power gain between the mth V2I link and the base station is expressed as: g m,B =PL V2I α m,B h m,B Among them, h m,B is the frequency-dependent small-scale rapid fading, which obeys the exponential distribution with unit mean; α m,B Large-scale fading, including path loss and shadow fading; The signal-to-interference-to-noise ratio of the receiving user in the V2I link is expressed as: The signal-to-interference-to-noise ratio of the receiving user in the V2V link is expressed as: Among them, P m , P k Denote the transmission power of V2I and V2V links respectively, g k represents the channel gain between the kth V2V channel and the interference channel, σ 2 represents the noise power, I m is the interference power received by the V2I link, I k is the interference power received by the V2V link; The channel capacity of the V2I link is expressed as: C m =Wlog(1+SINR m ) The channel capacity of the V2V link is expressed as: C k =Wlog(1+SINR k ) Where W is the spectrum bandwidth; V2V link interference exists at the transmitting and receiving ends of the V2V link, and its interference power is expressed as: I i,j =P k g i,j 。 4. A multi-agent system networking and resource optimization method according to claim 1, characterized in that: The MAS communication delay model in step S2) is specifically as follows; The average delay of the Internet of Vehicles includes the sum of propagation, transmission, queuing and processing delays. Processing delay refers to the delay caused by the host or router processing after receiving the packet. Transmission delay refers to the time required for the data packet to be sent to the link. Queuing delay refers to the time it takes for the data packet to queue in the input queue and the output queue after entering the router and wait for processing. Transmission delay depends on the size of the data packet and the transmission rate. According to Shannon's formula, the wireless link rate is expressed as: Among them, k i represents the channel fading coefficient, P is the vehicle v i The transmission power, PL i represents the path loss between the vehicle and the target node, where the target node refers to the neighboring vehicle or the roadside base station, σ 2 is the Gaussian white noise power, B represents the channel bandwidth; The transmission delay is expressed as: The propagation delay is expressed as: Where c is the speed of light; Combining formulas (19) and (20), the total delay is expressed as: t i =t r +t p 。 5. A multi-agent system networking and resource optimization method according to claim 1, characterized in that: The molecular dynamics-enabled agent steady-state strategy in step S3) is specifically as follows: When a vehicle is driving, it always tries to maintain an appropriate balance distance with surrounding vehicles, that is, the vehicle tends to maintain a dynamic required safety distance so that the vehicle neither lags behind nor follows closely, thereby driving safely and efficiently. The vehicle required safety distance is expressed as: Where: t0 is the driver's reaction time, a0 is the maximum deceleration of the vehicle, and 1 / 2a0 = 0.07 m / s 2 , v is the instantaneous speed of the vehicle; On the basis of the required safety distance and combined with the actual inter-vehicle distance of intelligent agents, the saturation coefficient of the required distance is further obtained: Where: Y is the saturation coefficient of the required distance, L is the actual vehicle distance; When Y < 1, there is basically no risk of collision between the rear vehicle and the front vehicle. However, from the perspective of lane utilization and traffic operation efficiency, the rear vehicle should increase its speed to make Y = 1, thereby achieving efficient driving. When Y > 1, there is a certain risk of collision. Therefore, the rear vehicle should reduce its speed to make Y = 1. The interaction between vehicles is caused by the combined effects of repulsion and attraction, and is affected by the required safety distance. The potential function is a mathematical expression that describes the interaction between molecules, and its form is: Where: τ is equal to the depth of the potential energy well, σ is the distance between the two bodies when the potential energy of the interaction is exactly zero. In the vehicle modeling based on molecular dynamics, σ = X. r is the distance between the two bodies at the bottom of the potential energy well, at this time r = L. Therefore, in vehicle molecular dynamics, the required safe distance X is the critical distance when the interaction potential between vehicles is 0, and the actual distance between vehicles L is the distance between the two bodies at the bottom of the potential energy well; Therefore, in vehicle molecular dynamics, the potential function of the interaction between vehicles is expressed as: Among them: the first term corresponds to the effect of mutual repulsion between vehicle molecules at close distances, and the second term corresponds to the effect of mutual attraction between vehicle molecules at long distances; From the potential function, we can deduce the vehicle interaction acceleration: Where: m is the vehicle mass The vehicle interaction acceleration is simplified to: The urban road is a two-way multi-lane model. Considering the communication radius and networking steady-state requirements of the Internet of Vehicles, the division of vehicle clusters should be based on the vehicle transmission range. The vehicle transmission range changes with the change of node density on the road. The calculation formula for the vehicle transmission range radius is: Where: R is the transmission range, λ is the vehicle node density, N is the number of vehicles on the road section, and P is the probability of road section connectivity; The communication distance of the vehicle node is defined as R. When the distance between two nodes is less than R, a two-way communication link is formed between the nodes. Once the distance between the nodes exceeds R, the communication link is broken. While maintaining a certain network connectivity probability, the increase in node density will lead to a decrease in the transmission radius, and vice versa.
6. A multi-agent system networking and resource optimization method according to claim 1, characterized in that: The clustering algorithm of the agent cluster steady state in step S4) and the interference control through agent networking are as follows; A two-stage resource allocation algorithm is proposed. In the first stage, the interference is minimized based on the bipartite graph matching theory. In the second stage, the DDPG algorithm is used to optimize the resource management strategy of cellular vehicle networks. Introducing channel interference factors and establishing optimization goals, V2V link communications are divided into mutually exclusive spectrum sharing clusters to minimize vehicle networking system interference and maintain 5G cellular vehicle networking steady state while ensuring the stability and efficiency of the molecular dynamics system of the connected vehicle. d i,j ≤R The first optimization objective is the interference between vehicles, and the second is the saturation coefficient of the required safety distance of the intelligent networked vehicle, minimizing |Y i,j -1|, so that Y i,j As close to 1 as possible, it can reduce the risk of collision between intelligent network vehicles and ensure road safety while improving road utilization and traffic efficiency. i,j ≤R is the constraint condition of the optimization problem, which constrains the distance between vehicles in the cluster to not exceed the vehicle communication distance R to ensure that the vehicles in the cluster can ensure normal communication and information exchange; A clustering mechanism is introduced to cluster all intelligent connected vehicles in the cellular vehicle network system. In the cellular vehicle network system, vehicles regularly report their positions and speeds to the base station, so the base station understands the vehicle network topology. Each intelligent connected vehicle belongs to a cluster, each cluster corresponds to a sub-channel, and vehicles in the same cluster will be assigned to the same sub-channel. In the clustering problem, the clustering results will tend to place vehicles with strong interference and high collision risks into different clusters, which is equivalent to separating them into different sub-channels, which helps to reduce coarse-scale inter-cell interference; Graph theory is introduced to describe the optimization problem. Interference management and molecular dynamics management between vehicles are captured by weighted undirected graphs. The weight of the undirected graph is w i,j =I i,j , where I i,j Interference between vehicles, including path loss, shadowing, and small-scale fading; Each vehicle on the road is modeled as a vertex in an undirected graph. When two vertices interfere with each other, they are connected by an edge to form an undirected graph G, whose vertex set is V(G) and edge set is E(G). The clustering goal is to divide the K vertices of the weighted undirected graph into N non-intersecting clusters L1, L2, ..., L N , so that L1∪L2∪...∪L N =V(G), which minimizes the intra-cluster interference of all vehicle clusters and ensures safe and efficient driving of vehicles based on molecular dynamics; The maximum cut problem is solved by the improved Kernighan-Lin algorithm.
7. A multi-agent system networking and resource optimization method according to claim 6, characterized in that: The heuristic algorithm for solving the MAXN-CUT problem based on the Kernighan-Lin algorithm in step S4) is as follows: Initialize the cluster; Definition: The weight of each cluster When the cluster has no nodes assigned to it, W i =0; Step 1: Select weight w i,j The largest edge connects nodes i and j, assigning nodes i and j to the two clusters C that are not assigned nodes. i =[i],C j = [j], and in the process of initializing the cluster, w i,j Set to 0, and repeat this operation until there is a node in each of the n clusters; Step 2: For node k that is not assigned a cluster, calculate the time required to assign the node k to cluster C. i When , the total weight added to the cluster Assign the node k to the weight gain ΔW i The smallest corresponding cluster, C i =[C i ,k]; Step 3: Repeat Step 2 until each node is assigned to a cluster and n original clusters are obtained.
8. A multi-agent system networking and resource optimization method according to claim 1, characterized in that: The step S6) MIADC-DDPG multi-agent system resource optimization collaborative model is as follows: The design objective function is as follows: C m ≥C0 (b) t i ≤t0 (e) In the above optimization problem, the objective function is to maximize the system downlink throughput, which is expressed as ΣC m , constraint (a) means that the transmission power of M V2I links cannot be greater than the maximum transmission power of the V2I link Constraint (b) indicates that the capacity of each V2I channel needs to be greater than the minimum required throughput C0 of the link, and constraint (c) indicates that the transmit power of the K V2V links needs to be less than or equal to the maximum transmit power of the V2V link. p0 represents the tolerable interruption probability of the link, constraint (d) represents that the transmission rate of K V2V links must be less than or equal to the tolerable interruption probability, that is, to ensure the connectivity of the V2V link, constraint (e) represents that the delay generated by the V2I link and the V2V link cannot be greater than the maximum delay constraint of the Internet of Vehicles, and t0 represents the maximum delay that can be tolerated by the Internet of Vehicles communication; The above problems are solved through the MIADC-DDPG multi-agent collaborative model; Assume that the strategy executed at time t is A t , working status is S t , the reward is R t , the above strategies, states and rewards are defined as follows: State space: At time slot t, the state of the intelligent connected vehicle is described as: s k (t)={G k (t),I k ,P k (t-1),SINR k } Among them, G k I (t) is the channel gain associated with agent k at time t; k is the interference power received by agent k; P k (t-1) represents the transmission power of agent k in the previous time slot; SINR k is the signal-to-noise ratio of agent k; According to the state and strategy, each agent will make decisions on transmission power allocation and channel selection one by one. The action set of the agent in time slot t is defined as: a k (t)={P k (t),ρ k (t)} Among them, P k (t) represents the transmission power of agent k in time slot t; ρ k (t) is the channel selection of agent k; The reward is used to evaluate the agent in state s t ∈S, take action a t ∈A, transition to the next state s t+1 ∈S, the system incentive is used to evaluate the effect of the agent taking a specific action. The goal of resource optimization is to maximize the uplink throughput of the Internet of Vehicles by interacting with the environment while satisfying constraints such as QoS. The reward function is defined as the throughput of the cellular Internet of Vehicles system; The model updates the parameters ω of the Critic current network and θ of the Actor current network through the gradient back propagation of the neural network, thereby updating the Critic target network and the Actor target network; Rewards provide evaluable feedback for the decision-making of the intelligent agent performing deep reinforcement learning, and the ultimate goal of the strategy π is to obtain the maximum cumulative reward. The intelligent agent uses the rewards of environmental feedback as the basis to optimize the value function, and then uses the value function to evaluate the strategy. The strategy maps the state to the action, and the strategy optimization is to find the optimal mapping. The goal of the DDPG framework is to find the optimal strategy π* in a distributed fully observable Markov decision process and maximize the reward function within a limited time frame; In DDPG, the agent uses a deterministic policy network μ(s|θ μ ) to select actions, where θ μ is the parameter of the policy network. During training, the agent adjusts the state of the policy network according to the current state s t Select action a through the policy network t , then, action a t Applied to environments t , the agent observes the next state s t+1 and reward r t The entire transition process is expressed as: a t =μ(s|θ μ ) s t+1 =F(s t ,a t )+s t r t =R(s t ,a t ) DDPG trains the policy network and the value function network by minimizing the loss of the Q value, where the Q value is the action value function. The calculation of the Q value is expressed as: Q(s t ,a t )=r t +γ·Q(s t+1 ,μ(s t+1 |θ μ )|θ Q ) Where γ is the discount factor, θ Q It is the parameter of the value function network, and the Q value is used to calculate the loss of the policy network to optimize the policy.
Citation Information
Cited By
Channel resource allocation method and system based on hierarchical reinforcement learning
CN119967616A
Intelligent adaptive control method and system for 5G wireless communication antenna
CN120074591A
