Vehicle-mounted ad hoc network routing method based on reinforcement learning
By adopting the Q-learning algorithm and RSU relay hierarchical routing framework in the on-board ad hoc network, the problems of packet transmission delay and network congestion in urban market scenarios are solved, and low-latency and high-reliability packet transmission is achieved. Simplifies the routing decision process.
Patent Information
- Application Number
- CN202411067677.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
Smart Images

Figure CN119071869A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer network applications, and in particular to a vehicle-mounted ad hoc network routing method based on reinforcement learning, which ensures the reliable transmission of data packets. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.
[0003] (1) Location-based routing protocol for VANET
[0004] With the rapid development of the Internet of Vehicles (IoV), researchers have improved traditional mobile ad hoc networks (MANETs) routing protocols to adapt to the highly dynamic IoV and applied various emerging technologies to the design of VANET routing protocols. Topology-based routing protocols use routing tables to store information about routing paths. Before communication, routes to all destinations must be established, and each node regularly broadcasts control packets to share information with other nodes in the network and updates the routing table. The node forwards the packet to the next hop on the selected route in the routing table. Since a large number of routing entries are stored, the complete route from the source node to the destination node is available. Therefore, topology-based routing protocols can effectively reduce the packet loss rate, but at the same time they can also increase bandwidth overhead. Through the application of global positioning system (GPS) technology, vehicle nodes can obtain their location coordinates on a real-time map, and then forward packets based on the location information shared between nodes. Location-based routing can reduce the impact of high mobility of nodes and avoid the need to maintain routing tables for the entire network. Due to its high delivery ratio, short transmission delay, and low communication overhead, location-based routing protocols have become the most suitable, scalable, and promising strategy for VANETs.
[0005] To adapt to the dynamic nature and changing environment of VANETs, segmentation metrics and traffic awareness are introduced in location-based geographic routing protocols. Because the topology of vehicular networks is limited by roads and buildings in urban scenarios, the best path is selected by evaluating the adjacent segments at each intersection. A set of intersection sequences between source and destination nodes is selected as the packet transmission trajectory. A widely used method to obtain segmentation and traffic information in the field of Internet of Vehicles (IoV) is real-time link evaluation by transmitting collector packets (CPs) at intersections. CPs forward from the current intersection to the next intersection, collect the network and traffic status of each segment between intersections, and then assign weights to the segments according to the link quality. When transmitting packets, the path with the largest weight is selected to meet the requirements of high-rate or low-latency transmission of packets. Although these protocols are suitable for various dynamic VANET scenarios, the continuous availability of routing metrics requires frequent interactions between nodes and continuous updates of calculation results during packet communication. The continuous measurement process requires additional communication bandwidth and time, which increases overhead and latency.
[0006] (2) Routing protocols based on machine learning (ML) and reinforcement learning (RL)
[0007] Since vehicles in daily life usually have some obvious patterns and characteristics from a macro perspective, such as regular visits to specific places and relatively stable vehicle density in different areas, the latest routing technologies have introduced machine learning methods to train adaptive network models to improve network performance. In these protocols, network characteristics can be learned through machine learning (ML) or reinforcement learning (RL) methods based on historical traffic data without sending collector packets (CPs), thereby saving bandwidth resources and shortening communication delays. Existing routing protocols implemented through ML and RL can predict the movement and trajectory of vehicles to ensure more reliable and efficient packet transmission. However, the large number of states of vehicle behavior has high uncertainty, making it difficult to train a network model to make accurate predictions. As more and more vehicles join VANETs, the algorithm will become more and more complex and difficult to converge. Summary of the invention
[0008] The purpose of the present invention is to provide a vehicle-mounted ad hoc network routing method based on reinforcement learning in view of the limitations of the above research, which is based on the V2X (vehicle-to-everyting, V2X) routing protocol at the intersection, and the segmented road sections are monitored in real time by roadside units (RSUs) to prevent possible network congestion; by extracting the historical traffic flow characteristics of the road, training the Q learning algorithm to obtain the Q table, and then selecting the road section with the largest Q value, the optimal path can be found; before the data packet is transmitted, the method does not obtain real-time link information by broadcasting traffic perception data packets, so the routing of the method has lower overhead and delay; the method also takes all intersections as the state (state space) of Q learning, and all road sections as the action (action space) of Q learning, that is, data is sent from a certain intersection and reaches a certain road section through routing selection. This greatly reduces the number of states and makes the algorithm easy to implement. In the previous RL routing protocol with geographical environment as the state, the geographical area is divided into grids, the optimal grid is first selected, and then the data packet is forwarded to the next hop in the grid. However, intersections and buildings in the grid affect signal transmission, especially in urban scenarios. In the present invention, RSU (roadside units) are introduced as relays at intersections to alleviate the problems caused by sudden changes in vehicle direction and signal attenuation; at the same time, the Q table in the previous method is calculated offline and remains static during the routing process, so the routing decision cannot be adjusted by observing the network load. The load of the vehicle depends on the length of the buffer queue; in the present invention, a monitoring module is installed on the RSU (roadside unit) at the intersection to obtain the total number of data packets and estimate the average load of the local network on the road section; if the network on one path is overloaded, the data packet will be forwarded to another section.
[0009] The technical solution of the present invention is as follows:
[0010] A vehicle-mounted ad hoc network routing method based on reinforcement learning, comprising:
[0011] Design a hierarchical routing framework that combines Q-learning-based routing path planning with location-based repeater selection to forward packets to the road with the best communication conditions and then select the most appropriate repeater on the road;
[0012] A multidimensional Q table is calculated by the Q learning algorithm to realize the routing path selection between any two intersections, and the agent makes routing decisions based on the multidimensional Q table;
[0013] A congestion control mechanism is introduced to monitor the connection status in real time. Once the network is congested, an alternative section will be selected.
[0014] Furthermore, the hierarchical routing framework includes:
[0015] In the scenario, a two-way road section is adjacent to an intersection; a drive test unit RSU connected to a server is deployed at the intersection to communicate with vehicles for reliable packet forwarding in the intersection area;
[0016] The vehicle is equipped with a GPS system and digital maps, which can obtain the real-time geographic location of itself and the road test unit RSU.
[0017] Further, forwarding the data packet to the road with the best communication condition includes:
[0018] The server trains the Q learning algorithm based on the historical traffic flow information of the road section and sends the multi-dimensional Q table to the road test unit RSU at the intersection;
[0019] When a data packet arrives at an intersection, the road test unit RSU determines the direction of the intersection to which the data packet is sent by selecting the path with the largest Q value in the multidimensional Q table;
[0020] In the process of sending a data packet from a road test unit RSU at a certain intersection to a road test unit RSU at another adjacent intersection, a position-based greedy forwarding strategy is adopted to select a repeater, that is, a vehicle, on the path between two adjacent intersections.
[0021] Furthermore, the congestion control mechanism includes:
[0022] The server estimates the resource utilization status, that is, whether the transmission network of the section is congested, based on the statistical information of the total amount of data packets entering and leaving the section from the intersections at both ends; when the resource utilization rate exceeds the threshold, the segment with the second largest Q value is selected as the backup routing path to avoid entering the congested network segment.
[0023] Furthermore, the Q learning algorithm includes:
[0024] The state space S is the set of all intersections in the area, that is, the set of intersections; S t ∈S is described as It means that the data packet arrives at the intersection l at time t i , i is the identifier of the intersection;
[0025] The action space A is a number of road segments adjacent to an intersection, including roads in the east, west, south, and north directions; a t ∈A is the action taken by the agent at time t, which determines the road segment to which the information is sent;
[0026] Select the action of forwarding the data packet to a segment so that the location of the data packet changes from Transition to
[0027] After executing the action of forwarding a data packet to one of the road segments, the agent is immediately given a reward R; R t is the agent's current state S after executing the operation t Transition to the next state S t+1 The reward obtained when the packet is forwarded to the intersection D where the destination is located k , then the reward is φ, φ>0; otherwise, the reward is 0;
[0028] A multidimensional Q table is obtained by training the Q learning algorithm with historical road traffic flow in urban scenes. The Q values in the Q table are used for path selection and packet forwarding.
[0029] Furthermore, the multidimensional Q table is a three-dimensional matrix, including the current intersection, the target intersection and the action, which are represented by I, D and a respectively; the Q value is represented by Q( D k , ), indicating that at the current intersection Action Arrive at the destination intersection D k The Q value of Represents the action of forwarding a data packet to a road segment, r∈{east,west,south,north}.
[0030] Further, the agent distributes the Q-table to the road test units RSU at the intersection;
[0031] The vehicle obtains its own position and the position of the destination vehicle through GPS, and obtains the road topology through digital maps;
[0032] Then, determine the intersection where the source point is located and the intersection where the target position is located;
[0033] At the beginning of routing, the data packet is forwarded to the intersection where the source point is located; first, the road test unit RSU determines the current state (state, s) by checking the current intersection and the target intersection; then, the road test unit RSU searches the Q table and selects the action (action, a) with the largest "state-action value" in the current state; the road test unit RSU takes action to forward the data packet to the selected path, and the data packet reaches the adjacent intersection of the road through the V2V forwarding strategy and V2I forwarding strategy on the path, that is, the road test unit RSU sends the data packet to one or more relays (cars) on the road in turn, and then the relay sends it to the road test unit RSU at the intersection.
[0034] Furthermore, the multidimensional Q table is updated and iterated by the following formula:
[0035]
[0036] Where: α represents the learning rate; γ represents the discount factor as a dynamic parameter related to vehicle density and distance to the destination;
[0037] The definition of γ is as follows:
[0038] γ t =β×SP t +(1-β)×DP t
[0039] Where: β is the weight factor; SP t is the influencing factor of vehicle density, DP t The degree to which the distance from the starting point to the end point is shortened;
[0040] SP t and DP t They are defined according to the following formulas:
[0041]
[0042]
[0043] Where: s max is the maximum vehicle density; s min is the minimum vehicle density on the road segment; s best is the optimal road section vehicle density; s ij for Intersection with The vehicle density of the road section between intersections; dist t Current intersection Intersection point D k distance t+1 For the next intersection Intersection point D k distance.
[0044] Furthermore, at both ends of the road where the source vehicle and the destination vehicle are located, there are two candidate intersections;
[0045] For the two candidate intersections of the source point, they are the intersections on both sides of the road section where the source point is located, and their distances to the destination vehicle are compared respectively, and then the intersection with the shorter distance is selected as the intersection where the source point is located;
[0046] For the destination intersection, the target intersection is selected based on the distance to the target vehicle and the cosine value of the vehicle's motion angle. The formula is as follows:
[0047]
[0048] in:
[0049] ω is a weight factor between 0 and 1; dist i Is the candidate destination intersection D i The distance from the destination vehicle vd; c is a constant for normalizing the distance; is the displacement vector of the destination vehicle at the beacon interval Δt; is the position vector from the destination vehicle to the candidate destination intersection;
[0050] Choose the weight ω i The larger candidate intersection is selected as the target intersection.
[0051] Furthermore, the congestion control mechanism is introduced. By monitoring the connection status in real time, once the network is congested, an alternative route will be selected, including:
[0052] When the RSU is about to forward a data packet along one of the roads, the server extracts data from the RSUs at both ends of the road and calculates the average load of the link according to the following formula:
[0053]
[0054] Where: r_in i From the crossroads I i The number of packets entering road r, r outi From the crossroads I i The number of packets leaving road r; Num r is the total number of vehicles on road r;
[0055] If the average load exceeds the preset threshold θ, the node's buffer queue is close to full, the network on this road is congested, and the road section with the second largest Q value is selected as an alternative path.
[0056] Compared with the prior art, the present invention has the following beneficial effects:
[0057] 1. The present invention designs a hierarchical routing framework to realize data packet routing in urban scenarios, combines routing path planning based on the Q learning algorithm and repeater selection based on the location algorithm, forwards data packets to the road with the best communication conditions, and then selects the most suitable repeater on the road, thereby improving the reliability of transmission.
[0058] 2. The present invention calculates a multidimensional Q table through a Q learning algorithm to realize the routing path selection between any two intersections, and the agent makes routing decisions based on the multidimensional Q table; at the same time, in the Q learning algorithm, traffic flow characteristics such as vehicle density can also be used as parameters, so that the overhead and delay of traffic perception are greatly reduced.
[0059] 3. The present invention introduces a congestion control mechanism, which monitors the connection status in real time. Once the network is congested, an alternative section will be selected, thereby improving the success rate of packet transmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 Schematic diagram of using a road test unit (RSU) as a repeater at an intersection;
[0061] Figure 2 Schematic diagram of the routing scheme;
[0062] Figure 3 is a schematic diagram of a candidate source intersection and a destination intersection;
[0063] Figure 4 This is a schematic diagram of the AC forwarding policy routing solution;
[0064] Figure 5 Schematic diagram of V2X routing solution;
[0065] Figure 6 This is a schematic diagram of the V2V forwarding strategy;
[0066] Figure 7 To determine the routing scheme diagram;
[0067] Figure 8 Schematic diagram for selecting a relay routing solution;
[0068] Fig. 9 Schematic diagram of RSU continuous relay forwarding routing solution. DETAILED DESCRIPTION
[0069] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0070] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.
[0071] Embodiment 1
[0072] There are two common routing protocols for VANETs: location-based routing protocols (for VANETs) and routing protocols based on machine learning (ML) and reinforcement learning (RL). The main disadvantages of these two types of routing protocols are:
[0073] Location-based routing protocols: During packet communication, the widely used traffic-aware routing obtains traffic status by periodically broadcasting collector packets, which requires frequent interactions between nodes and continuous updates of calculation results. This process requires additional communication bandwidth and time, thereby increasing overhead and latency.
[0074] Routing protocols based on machine learning (ML) and reinforcement learning (RL):
[0075] The behavior of a vehicle has a large number of states and high uncertainty, so it is difficult to train a network model to make accurate predictions. As more and more vehicles join VANETs, the algorithm will become more and more complex and difficult to converge.
[0076] This embodiment is improved based on the reinforcement learning (RL) method to solve the problem that the model is difficult to converge.
[0077] This embodiment is based on a V2X (vehicle-to-everyting, V2X) routing protocol based on an intersection based on a Q learning algorithm, and real-time monitoring of segmented sections is performed through roadside units (RSU) to prevent possible network congestion. By extracting the historical traffic flow characteristics of the road, training the Q learning algorithm to obtain the Q table, and then selecting the section with the largest Q value, the optimal path can be found. The method of this embodiment does not obtain real-time link information by broadcasting traffic perception packets before data packets are transmitted. Therefore, the routing of this method has lower overhead and delay. It takes all intersections as the state (state space) of Q learning and all sections as the action (action space) of Q learning, that is, data is sent from a certain intersection and reaches a certain section through routing selection. This greatly reduces the number of states and makes the algorithm easy to implement. In the previous RL routing protocol with geographical environment as the state, the geographical area is divided into grids, the optimal grid is first selected, and then the data packet is forwarded to the next hop in the grid. However, intersections and buildings in the grid affect signal transmission, especially in urban scenarios. In this embodiment, RSU (roadside units, RSUs) are introduced as relays at intersections to alleviate the problems caused by sudden changes in vehicle direction and signal attenuation. The Q table in the previous method is calculated offline and remains static during the routing process, so the routing decision cannot be adjusted by observing the network load. The load of the vehicle depends on the length of the buffer queue. In this embodiment, a monitoring module is installed on the RSU (roadside unit) at the intersection to obtain the total number of data packets and estimate the average load of the local network on the road section. If the network on one path is overloaded, the data packet will be forwarded to another section.
[0078] For details, please refer to Figure 1 , a vehicle-mounted ad hoc network routing method based on reinforcement learning, comprising:
[0079] A hierarchical routing framework is designed to implement packet routing in urban scenarios. The routing path planning based on the Q-learning algorithm and the repeater selection based on the location algorithm are combined to forward the data packets to the road with the best communication conditions, and then the most suitable repeater is selected on the road, thereby improving the reliability of transmission.
[0080] The Q-learning algorithm is used to calculate a multidimensional Q-table to realize the routing path selection between any two intersections. The agent makes routing decisions based on the multidimensional Q-table. At the same time, in the Q-learning algorithm, traffic flow characteristics such as vehicle density can also be used as parameters, which greatly reduces the overhead and delay of traffic perception.
[0081] A congestion control mechanism is introduced. By monitoring the connection status in real time, an alternative section will be selected once the network is congested, thereby improving the success rate of packet transmission.
[0082] In this embodiment, specifically, the hierarchical routing framework includes:
[0083] In the scenario, a two-way road section is adjacent to an intersection; a drive test unit RSU connected to a server is deployed at the intersection to communicate with vehicles for reliable packet forwarding in the intersection area;
[0084] The vehicle is equipped with a GPS system and digital maps, which can obtain the real-time geographic location of itself and the road test unit RSU.
[0085] In this embodiment, specifically, forwarding the data packet to a road with the best communication condition includes:
[0086] The server trains the Q learning algorithm based on the historical traffic flow information of the road section and sends the multi-dimensional Q table to the road test unit RSU at the intersection;
[0087] When a data packet arrives at an intersection, the road test unit RSU determines the direction (one of the east, south, west, and north) of the intersection to which the data packet is sent by selecting the path with the largest Q value in the multidimensional Q table;
[0088] In the process of sending data packets from a road test unit RSU at one intersection to a road test unit RSU at another adjacent intersection, a location-based greedy forwarding strategy is adopted to select a repeater, that is, a vehicle, on the path between two adjacent intersections; the server estimates the resource utilization status, that is, whether the transmission network of the section is congested, based on the statistical information of the total amount of data packets entering and leaving the section from the intersections at both ends; when the resource utilization rate exceeds the threshold, the segment with the second largest Q value is selected as the backup routing path to avoid entering the congested network segment.
[0089] The routing solution of this embodiment is as follows: Figure 2 As shown in the figure, in this routing scheme, there are two elements: intersections and road sections (road sections have relays). There is an RSU (roadside unit) at the intersection, which is a unit that can transmit signals. A relay is a vehicle on the road section. There are zero to multiple vehicles on the road section between two intersections. These vehicles can be used as relays to transmit information. The vehicle at the source (red vehicle) sends the generated data packet to the vehicle at the destination (green vehicle). The specific steps are as follows:
[0090] The red vehicle between intersections 1 and 2 is the source vehicle, generating a data packet.
[0091] Step 1: Send the data packet to one of the RSUs at the two intersections near the road section (one of 1 and 2 in the figure above), which is the source intersection. And take one of the RSUs at the two intersections of the road section where the end point is located (one of 8 and 9 in the figure above) as the destination intersection. In this example, select intersection 2 as the source intersection and intersection 9 as the destination intersection.
[0092] Step 2: The source intersection and the destination intersection confirm the current state (state, s), which is (2, 9) in this example. The RSU at intersection 2 consults the Q table and finds that the action with the largest Q value corresponding to the state (2, 9) is south, so it sends the data packet to the south intersection 5. When sending from the RSU at intersection 2 to the RSU at intersection 5, it must pass through one or more relay nodes, that is, a car on the road section.
[0093] Step 3: Repeat step 2 until the data packet reaches the destination vehicle.
[0094] Specifically, there are the following steps:
[0095] The red vehicle between intersections 1 and 2 is the source vehicle, generating a data packet. The green vehicle between intersections 8 and 9 is the target vehicle. Intersection 2 is closer to the target vehicle than intersection 1, so it is selected as the source intersection S. The target vehicle is heading towards intersection 9 and is shorter from intersection 9, so intersection 9 is selected as the destination intersection D. The source vehicle first forwards the data packet to the RSU at the source intersection S. The RSU determines (2,9) as the current state and finds the segment 2-5 with the largest Q value in the Q table. The data packet is forwarded to intersection 5 by the relay vehicle on the road, such as the yellow vehicle in the figure. The RSU at intersection 5 determines (5,9) as the current state and finds the segment 5-6 with the largest Q value in the Q table. However, the RSU finds that the segment 5-6 with the largest value in the current state is congested, so 5-8 is selected as an alternative path. When the packets arrive at intersection 8, the RSU forwards them to the destination intersection D. If the destination vehicle is in the neighbor table (a table that stores information about other vehicles or RSUs), the packet is forwarded directly to the destination vehicle and the routing is successful.
[0096] Therefore, in order to implement the above method, this embodiment needs to implement the following points:
[0097] 1) Nodes in the network (RSUs and vehicles) need to exchange status information regularly, in order to allow them to select appropriate relays when forwarding data packets. Therefore, vehicles and RSUs broadcast beacon packets at regular intervals, including position, velocity, and acceleration information.
[0098] 2) Establish a multi-dimensional Q table to transfer information from the source point (intersection) to the destination (intersection). The traditional two-dimensional Q table only provides the path from a certain point to several destinations. In order to meet the needs of the actual situation, communication between any two nodes should be allowed, that is, (source point, destination) as the current state and the direction of the next hop (east, south, west, north) as the action, so a multi-dimensional Q table is needed.
[0099] 3) Data packets need to be forwarded to the path with the best routing performance, which depends on the vehicle density and network load. Therefore, we chose an approach that combines RL and real-time vehicle density monitoring.
[0100] The method is further explained below based on the following three models: 1) multi-dimensional Q learning model, 2) routing path selection model, and 3) final communication model.
[0101] 1. Multidimensional Q-learning model (i.e. Q-learning algorithm)
[0102] The agent of Q learning is a server with historical traffic information, which explores the VANET network and implements routing path planning for packet forwarding. The agent is the learner and decision maker in the Q learning process, and learns control strategies through continuous interaction with the environment.
[0103] The state space (denoted as S) is the set of all intersections (intersections) in the area. t ∈S can be described as It means that the data packet arrives at the intersection l at time t i , i is the identifier of the intersection.
[0104] The action space (denoted as A) is the multiple road segments adjacent to an intersection, including roads in the east, west, south, and north directions. t ∈A is the action taken by the agent at time t, which determines the road segment to which the information is sent.
[0105] Select the action of forwarding the data packet to a segment so that the location of the data packet changes from Transition to
[0106] After executing the action of forwarding a data packet to one of the road segments, the agent is immediately given a reward (denoted as R). t is the agent's current state S after executing the operation t Transition to the next state S t+1 If the data packet is forwarded to the intersection D where the destination is located k , then the reward is φ, φ>0; otherwise, the reward is 0. The reward will be distributed according to the following formula:
[0107]
[0108] In this embodiment, a multidimensional Q table is obtained by training a Q learning algorithm with historical road traffic flow in an urban scene. The Q values in the Q table are used for path selection and packet forwarding. The packet travels along the road to one intersection after another until it reaches the destination. At each intersection, it decides which road to take. The current intersection is Consider it as the state at time t. Select the action to forward the data packet to a segment Change the status from Transition to r is an element of a set of roads adjacent to the intersection, usually in four adjacent directions, i.e., r∈{east,west,south,north}. For example, if RSU takes action Then the data packet will be forwarded to the north road of the intersection. The original two-dimensional Q matrix is improved into a three-dimensional matrix, including the current intersection, the target intersection and the action, represented by I, D and a respectively. The state dimension in the original matrix is replaced by the two dimensions of the current intersection and the target intersection. Q( D k , ) indicates that at the current intersection Action Arrive at the destination intersection D k The Q value of the agent. By adding a dimension, the state of the agent becomes easier to observe. All intersections can be reached through the optimal routing path based on the Q table. The formula for iteratively updating the Q table is defined as follows:
[0109]
[0110] The learning rate α is set to 0.1 based on experience. In order to select a path with reliability and continuity, the discount factor γ is used as a dynamic parameter related to the vehicle density and the distance to the destination. γ is defined as follows:
[0111] γ t =β×SP t +(1-β)×DP t
[0112] In the above formula, β is the weight factor, 0≤β≤1; SP t is the influencing factor of vehicle density, DP t The degree to which the distance from the starting point to the end point is shortened. t and DP t They are defined according to the following formulas:
[0113]
[0114]
[0115]
[0116] In the above formula, s max is the maximum vehicle density; s min is the minimum vehicle density on the road segment; s best is the optimal road section vehicle density. According to experimental simulation, the network performance is best when there are 28 adjacent vehicles within the communication range. ij for Intersection with The vehicle density of the road section between intersections, dist t Current intersection Intersection point D k distance t+1 For the next intersection Intersection point D k If the distance from the next intersection to the target intersection exceeds the distance from the current intersection to the target intersection, then DP t will be 0. When the optimal vehicle density is approached and the next intersection is closer to the destination intersection, the discount factor γ is larger. To the intersection The Q value is updated based on the historical traffic data of the road. Through the dynamic parameter γ, we ensure that the data packet is transmitted on the path where the vehicle density is as close to the optimal value as possible. Although adding a dimension to the Q table increases the number of states, γ t The introduction of SP (a quantity related to vehicle density) and DP (a quantity related to distance) in the definition formula improves the convergence speed of Q learning.
[0117] The source vehicle sends a data packet to an RSU at a nearby intersection (source intersection). The RSU uses an intersection near the destination vehicle as the destination intersection and sends the data packet to the RSU at that intersection. There are two candidate intersections at each end of the road where the source (vehicle) and the destination (vehicle) are located. For the two candidate intersections of the source point, they are the intersections on both sides of the road section where the source point is located. The distances from them to the destination (vehicle) are compared respectively, and then the intersection with the shorter distance is selected as the intersection where the source point is located. The schematic diagram of the candidate source intersection and the destination intersection is shown in the figure below. Figure 3 shown.
[0118] For the destination intersection, the target intersection is selected based on the distance to the target vehicle and the cosine value of the vehicle's motion angle. The formula is as follows. iThe larger candidate intersection is selected as the target intersection.
[0119]
[0120] in:
[0121]
[0122] Among them, dist i Is the candidate destination intersection D i The distance from the destination vehicle vd; c is a constant for normalizing the distance; is the displacement vector of the destination vehicle at the beacon interval (each vehicle will periodically send a beacon packet to the RSU, and this time interval is called the beacon interval) Δt; is the position vector from the destination vehicle to the candidate destination intersection.
[0123] ω i In the expression of , it represents the relationship between the vehicle's direction of movement and the position of the candidate destination intersection. If the vehicle is moving toward the intersection, the value is positive, otherwise it is negative. i The ω in the expression is a weight factor between 0 and 1.
[0124] The multidimensional Q-learning algorithm is as follows (Algorithm 1):
[0125] Symbols: ε, α, β: hyperparameters; S: state space, all intersections; A: action space, all road segments; D k : The intersection where the target address is located; R: The reward value in reinforcement learning
[0126]
[0127]
[0128] 2. Routing Path Selection Model
[0129] The agent gets the Q table and distributes it to the RSUs at the intersection. The source point (vehicle) obtains its own position and the position of the destination vehicle through GPS, and obtains the road topology through the digital map. Then, the intersection where the source point is located and the intersection where the destination position is located can be determined.
[0130] like Figure 4As shown in the figure, at the beginning of routing, the data packet is forwarded to the intersection where the source point is located. The RSU at the intersection forwards the data packet to the appropriate road section according to the Q value. First, the RSU determines the current state (state, s) by checking the current intersection and the intersection of the target. Then, the RSU looks up the Q table and selects the action (action, a) with the largest "state-action value" in the current state. The RSU takes action to forward the data packet to the selected path, and the data packet reaches the adjacent intersection of the road through V2V communication (vehicle to vehicle communication) on the path, that is, the RSU sends the data packet to one or more relays (cars) on the road in turn, and then the relay sends it to the RSU at the intersection.
[0131] The above process is repeated until the data packet reaches the destination intersection.
[0132] Each vehicle will periodically send a beacon packet to the RSU, and this time interval is called the beacon interval.
[0133] Through the Q table, routing makes full use of historical traffic information, but it is impossible to understand the link conditions in real time, increasing the possibility of network congestion. Using the above routing process, all data packets passing through the intersection will be forwarded by the RSU. Assuming that vehicles on each road generate data packets at the same speed, the resource utilization on the road can be estimated by calculating the number of input and output data packets. Each RSU contains a table that records the number of input and output data packets from roads adjacent to the intersection. The record table is shown in Table 1. During the beacon interval, when the RSU receives a packet from road r∈{north, south, east, west}, r_out increases; when the RSU forwards a packet to road r, r_in increases. The values in the table are updated periodically.
[0134] The record table of the number of incoming and outgoing packets is as follows:
[0135] Table 1: Number of incoming and outgoing packets
[0136] North South East West r_in North_in South_in East_in West_in r_out North_out South_out East_out West_out
[0137] When an RSU is about to forward a data packet along one of the roads, the server extracts data from the RSUs at both ends of the road and calculates the average load of the link (road segment, the section between two intersections) according to the following formula:
[0138]
[0139] Among them, r_in i is the number of packets entering road r from intersection Ii, r_out iis the number of packets leaving road r from intersection Ii. The intersections Ii and Ij are located at the two ends of the road r to be evaluated. Numr is the total number of vehicles on road r. The average load of each vehicle node can be obtained by the above formula. If this value exceeds the preset threshold θ, the buffer queue of the node is close to full and the network on this road is congested. Set the default value of θ to 0.5. Select the section with the second largest Q value as an alternative path (if the θ value of the section with the second largest Q value does not reach the threshold). At each intersection, the RSU forwards the data packet to a path with a greater distance reduction and good communication conditions, thereby ensuring routing performance.
[0140] The process of selecting a routing path is shown in the following algorithm (Algorithm 2):
[0141]
[0142]
[0143] 3. V2X Communication Model
[0144] Vehicles maintain a neighbor table containing state information of neighbor nodes. The neighbor table is updated regularly by exchanging beacon packets (a packet sent by vehicles to periodically update information) to adapt to highly dynamic networks and segment routing paths according to the location of intersections.
[0145] The neighbor table is where each vehicle records its neighbors. When a vehicle sends a beacon packet, its neighboring vehicles receive the beacon packet and add or update the vehicle's information in their own neighbor tables.
[0146] A greedy forwarding strategy is implemented on the path segment between two adjacent intersections. In order to achieve the goal of sending data packets from the source vehicle to the target vehicle, two algorithms are introduced: the V2V (vehicle to vehicle) forwarding strategy in the road area and the V2I (vehicle to intersection) forwarding strategy in the intersection area.
[0147] Figure 5 Schematic diagram of V2I and V2V;
[0148] 1) V2V forwarding strategy
[0149] Vehicles on the road use a V2V forwarding strategy to select relays and transmit data packets. The source vehicle generates a data packet and determines the location of the target vehicle. If the source vehicle and the destination vehicle are on the same road, the target address TD (target destination) is the destination vehicle VD (destination vehicle), and the vehicle forwards the packet directly to the destination vehicle through a greedy strategy. Otherwise, the data packet will first be forwarded to a path close to the destination address. When the data packet is on the path between the source intersection and the destination intersection, the target address TD (target destination) is the next intersection determined at each intersection by the algorithm for selecting the routing path (i.e., Algorithm 2). A greedy forwarding strategy is executed on the road section between intersections to forward the data packet to the target destination. The vehicle node selects the neighbor closest to the target intersection or target vehicle as the next hop for relaying the data packet. When there are no available neighbor nodes within the communication range, the problem of the data packet being temporarily unable to be sent out occurs. The vehicle node stores the data packet in a buffer queue and continues to move until it encounters an appropriate relay and forwards the data packet to the next hop, such as Figure 6 shown.
[0150] The following algorithm shows the V2V forwarding strategy (Algorithm 3).
[0151] Symbols: VD (destination vehicle), Relay, VS (source vehicle), TD (target destination). The difference between VD and TD is that VD is the final destination, while TD is the destination sent by the vehicle to the next hop.
[0152]
[0153]
[0154] 2) V2I forwarding strategy
[0155] Once the data packet reaches the intersection area through V2V communication on the path, the RSU at the intersection forwards it to the optimal section selected by Algorithm 2 and selects a reliable relay from the road. The RSU takes the action selected in Algorithm 2. Determine the routing path r and the next intersection adjacent to the road segment like Figure 7 shown.
[0156] Then, RSU selects a neighbor vehicle as a relay on path r, such as Figure 8 shown.
[0157] If the current intersection is the destination intersection D, and the RSU forwards the data packet to the node closest to the destination vehicle. When no available neighbor vehicles are observed in the intersection area, the RSU carries the data packet until an appropriate relay appears, such as Fig. 9 shown.
[0158] The V2I forwarding strategy is as follows (Algorithm 4):
[0159] Symbols: VD (destination vehicle), D (destination intersection), (intersection i at the current time t), (Action calculated based on multi-dimensional Q table), Relay (relay), TD (target destination)
[0160]
[0161]
[0162] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.
[0163] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.
Claims
1. A vehicle-mounted ad hoc network routing method based on reinforcement learning, characterized in that: include: Design a hierarchical routing framework that combines Q-learning-based routing path planning with location-based repeater selection to forward packets to the road with the best communication conditions and then select the most appropriate repeater on the road; A multidimensional Q table is calculated by the Q learning algorithm to realize the routing path selection between any two intersections, and the agent makes routing decisions based on the multidimensional Q table; A congestion control mechanism is introduced to monitor the connection status in real time. Once the network is congested, an alternative section will be selected.
2. According to claim 1, a vehicle ad hoc network routing method based on reinforcement learning is characterized in that: The hierarchical routing framework comprises: In the scenario, a two-way road section is adjacent to an intersection; a drive test unit RSU connected to a server is deployed at the intersection to communicate with vehicles for reliable packet forwarding in the intersection area; The vehicle is equipped with a GPS system and digital maps, which can obtain the real-time geographic location of itself and the road test unit RSU.
3. A vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 2, characterized in that: The forwarding of the data packet to the road with the best communication condition comprises: The server trains the Q learning algorithm based on the historical traffic flow information of the road section and sends the multi-dimensional Q table to the road test unit RSU at the intersection; When a data packet arrives at an intersection, the road test unit RSU determines the direction of the intersection to which the data packet is sent by selecting the path with the largest Q value in the multidimensional Q table; In the process of sending a data packet from a road test unit RSU at a certain intersection to a road test unit RSU at another adjacent intersection, a position-based greedy forwarding strategy is adopted to select a repeater, that is, a vehicle, on the path between two adjacent intersections.
4. The vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 1, characterized in that: The congestion control mechanism comprises: The server estimates the resource utilization status, that is, whether the transmission network of the section is congested, based on the statistical information of the total amount of data packets entering and leaving the section from the intersections at both ends; when the resource utilization rate exceeds the threshold, the segment with the second largest Q value is selected as the backup routing path to avoid entering the congested network segment.
5. The vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 1, characterized in that: The Q learning algorithm comprises: The state space S is the set of all intersections in the area, that is, the set of intersections; S t ∈S is described as It means that the data packet arrives at the intersection l at time t i , i is the identifier of the intersection; The action space A is a number of road segments adjacent to an intersection, including roads in the east, west, south, and north directions; a t ∈A is the action taken by the agent at time t, which determines the road segment to which the information is sent; Select the action of forwarding the data packet to a segment so that the location of the data packet changes from Transition to After executing the action of forwarding a data packet to one of the road segments, the agent is immediately given a reward R; R t is the agent's current state S after executing the operation t Transition to the next state S t+1 The reward obtained when the packet is forwarded to the intersection D where the destination is located k , then the reward is φ, φ>0; otherwise, the reward is 0; A multidimensional Q table is obtained by training the Q learning algorithm with historical road traffic flow in urban scenes. The Q values in the Q table are used for path selection and packet forwarding.
6. A vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 5, characterized in that: The multidimensional Q table is a three-dimensional matrix, including the current intersection, the target intersection and the action, which are represented by I, D and a respectively; the Q value is represented by Indicates that at the current intersection Action Arrive at the destination intersection D k The Q value of Represents the action of forwarding a data packet to a road segment, r∈{east,west,south,north}.
7. The vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 3, characterized in that: The agent distributes the Q table to the road test units RSU at the intersection; The vehicle obtains its own position and the position of the destination vehicle through GPS, and obtains the road topology through digital maps; Then, determine the intersection where the source point is located and the intersection where the target position is located; At the beginning of routing, the data packet is forwarded to the intersection where the source point is located; first, the RSU determines the current state (state, s) by checking the current intersection and the target intersection; then, the RSU searches the Q table and selects the action (action, a) with the largest "state-action value" in the current state; the RSU takes action to forward the data packet to the selected path, and the data packet reaches the adjacent intersection of the road through the V2V forwarding strategy and V2I forwarding strategy on the path, that is, the RSU sends the data packet to one or more relays (cars) on the road in sequence, and then the relay sends it to the RSU at the intersection.
8. The vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 6, characterized in that: The multidimensional Q table is updated and iterated by the following formula: Where: α represents the learning rate; γ represents the discount factor as a dynamic parameter related to vehicle density and distance to the destination; The definition of γ is as follows: c t =β×SP t +(1-β)×DP t Where: β is the weight factor; SP t is the influencing factor of vehicle density, DP t The degree to which the distance from the starting point to the end point is shortened; SP t and DP t They are defined according to the following formulas: Where: s max is the maximum vehicle density; s min is the minimum vehicle density on the road segment; s best is the optimal road section vehicle density; s ij for Intersection with The vehicle density of the road section between intersections; dist t is the current intersection point l i t Intersection point D k distance t+1 For the next intersection Intersection point D k distance.
9. The vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 1, characterized in that: There are two candidate intersections at both ends of the road where the source vehicle and the destination vehicle are located; For the two candidate intersections of the source point, they are the intersections on both sides of the road section where the source point is located, and their distances to the destination vehicle are compared respectively, and then the intersection with the shorter distance is selected as the intersection where the source point is located; For the destination intersection, the target intersection is selected based on the distance to the target vehicle and the cosine value of the vehicle's motion angle. The formula is as follows: in: ω is a weight factor between 0 and 1; dist i Is the candidate destination intersection D i The distance from the destination vehicle vd; c is a constant for normalizing the distance; is the displacement vector of the destination vehicle at the beacon interval Δt; is the position vector from the destination vehicle to the candidate destination intersection; Choose the weight ω i The larger candidate intersection is selected as the target intersection.
10. The vehicle-mounted ad hoc network routing method based on reinforcement learning according to claim 4, characterized in that: The congestion control mechanism is introduced. By monitoring the connection status in real time, once the network is congested, an alternative route will be selected, including: When the RSU is about to forward a data packet along one of the roads, the server extracts data from the RSUs at both ends of the road and calculates the average load of the link according to the following formula: Where: r_in i From the crossroads I i The number of packets entering road r, r outi From the crossroads I i The number of packets leaving road r; Num r is the total number of vehicles on road r; If the average load exceeds the preset threshold θ, the node's buffer queue is close to full, the network on this road is congested, and the road section with the second largest Q value is selected as an alternative path.
Citation Information
Patent Citations
Routing method based on Q learning and electronic map in vehicular ad hoc network
CN107454650A
Routing method based on Q learning and trust model in vehicle-mounted ad hoc network
CN110049530A
Internet of vehicles hybrid routing method based on deep reinforcement learning
CN117676764A
Intersection-based vehicle-mounted ad hoc network routing method based on Q learning
CN117692983A
Cited By
Vehicle ad hoc network system and method based on deep reinforcement learning
CN119854904A
Layered deep reinforcement learning routing protocol method based on multi-link ad hoc network
CN121037284A
