Hierarchical reinforcement learning-based routing method and system for supporting grouped RSU in urban vanet

Abstract

Proposed are a hierarchical reinforcement learning-based routing method and system for supporting a grouped RSU in an urban VANET. The hierarchical reinforcement learning-based routing method for supporting a grouped RSU in an urban VANET proposed by the present invention comprises the steps of: generating a data packet to be transmitted, determining a source roadside unit (RSU), and transmitting the packet to adjacent vehicles on the basis of a link cost table, by a user vehicle; identifying an RSU of the nearest destination vehicle among the adjacent vehicles, and determining a group to which the RSU of the destination vehicle belongs, by the source RSU; selecting a next RSU in a group Q-table on the basis of information of the destination vehicle; retransmitting an ACK message including information of the next RSU having a maximum Q-value in the group Q-table to the source RSU; and updating a local Q-table of the source RSU.

Description

Hierarchical reinforcement learning-based routing method and system supporting grouped RSUs in urban VANETs

[0001] The present invention relates to a hierarchical reinforcement learning-based routing method and system with grouped RSU support in an urban VANET.

[0002] Rapid urban growth and the increasing number of vehicles on the road present significant challenges for transportation systems. Intelligent Transportation Systems (ITS) and the Internet of Vehicles (IoV) are promising solutions to address these challenges. ITS systems typically rely on a variety of data sources, including sensors, cameras, and GPS devices, to collect real-time traffic and transportation data, then process and analyze it to generate insights and optimize transportation systems.

[0003] In recent years, the Internet of Things (IoT) has garnered significant attention as a means of connecting and integrating diverse devices and systems. One important application of IoT is the Internet of Vehicles (IoV), a system that enables communication and information exchange between vehicles, infrastructure, and other entities such as pedestrians. IoV has the potential to revolutionize transportation by enabling a variety of safety, efficiency, and convenience applications, including collision avoidance, traffic management, and entertainment.

[0004] Vehicular Ad Hoc Networks (VANETs) are a critical component of the Internet of Vehicles (IoV) and Intelligent Transport Systems (ITS). VANETs are a type of wireless network that enables vehicles to communicate directly with each other (V2V) and with roadside units (RSUs) and other infrastructure elements (Vehicle-to-Infrastructure (V2I). VANETs facilitate data exchange and improve traffic management and safety.

[0005] The technical problem to be achieved by the present invention is to provide a method and system for a hierarchical Q-learning-based routing algorithm with Grouped RSU (HQGR) supporting grouped RSUs in an urban VANET. According to embodiments of the present invention, RSUs are grouped, and group Q-vectors including all RSU group information are exchanged through V2X (Vehicle-to-Everything) communication, and Q-Vector-based Road-Segment (QVRS) control packets defined in the network are periodically broadcast in each segment to update V2X evaluation metrics obtained through the vehicle's position, speed, direction, and communication status, and this integrated V2X metric is presented as a reward function in the Q-learning algorithm. The present invention provides a method for hierarchically training in response to a destination vehicle based on a packet forwarding process from a global perspective through a group Q-table and a local Q-table.

[0006] In one aspect, the method for routing based on hierarchical reinforcement learning supporting grouped RSUs in an urban VANET proposed in the present invention comprises the steps of generating a data packet to be transmitted by a user vehicle, determining a source RSU (roadside unit) and transmitting the packet to neighboring vehicles based on a link cost table, identifying the RSU of the nearest destination vehicle among the neighboring vehicles by the source RSU and confirming a group to which the RSU of the destination vehicle belongs, selecting a next RSU from a group Q-table based on information of the destination vehicle, transmitting an ACK message including information of a next RSU with a maximum Q-value in the group Q-table back to the source RSU, and updating a local Q-table of the source RSU.

[0007] The step of generating a data packet to be transmitted by the user vehicle, determining a source RSU, and transmitting the packet to adjacent vehicles based on a link cost table comprises setting a plurality of groups based on the positions of the RSUs at each intersection on the road, and transmitting the packet through data transmission between a plurality of RSUs included in the plurality of groups and a plurality of vehicles including the user vehicle.

[0008] The step of identifying the RSU of the nearest destination vehicle among the adjacent vehicles by the source RSU and checking the group to which the RSU of the destination vehicle belongs is as follows: the source RSU broadcasts a QVRS (Q-Vector-based Road-Segment) control message to the RSU through the destination vehicle, the QVRS message includes a Q-vector including a time stamp, a sequence number of the source RSU, an address, and group information of the RSU of the destination vehicle; when the destination vehicle receives the QVRS message, the location, speed, and direction of the destination vehicle are recorded, an integrated path cost related to a propagation delay, a hop count, a minimum link quality, and a minimum expected expiration life are calculated, and the integrated path cost is stored in a local link cost table, and a relationship with a neighboring RSU is established and maintained.

[0009] The step of selecting a next RSU from the group Q-table based on the information of the destination vehicle includes encoding the state of the RSU into a Q-vector using a group Q-table storing a global Q-value and a local Q-table storing the local Q-value of an RSU within the group to update group information through a Q-learning algorithm, measuring an expected total reward received by performing a specific action in a specific state representing a group of destination vehicles or an RSU within a group of destination vehicles according to a current position of the source RSU, evaluating the source RSU's Q-vector and V2X metrics including a propagation delay, a number of hops, a link quality, and a minimum expected expiration lifetime, and selecting a next RSU from the group Q-table using the V2X metrics as a reward function.

[0010] In another aspect, the hierarchical reinforcement learning-based routing system supporting grouped RSUs in an urban VANET proposed in the present invention includes an RSU group identification unit that generates a data packet to be transmitted by a user vehicle, determines a source RSU (roadside unit), and transmits the packet to neighboring vehicles based on a link cost table, and identifies the RSU of the nearest destination vehicle among neighboring vehicles and confirms the group to which the RSU of the destination vehicle belongs, an RSU selection unit that selects the next RSU from a group Q-table based on information of the destination vehicle, and a Q-table management unit that receives an ACK message including information of the next RSU with the maximum Q-value from the group Q-table and updates the local Q-table of the source RSU.

[0011] According to embodiments of the present invention, a method and system for grouped RSUs in an urban VANET, a Hierarchical Q-learning-based routing algorithm with Grouped RSU (HQGR), is provided, in which RSUs are grouped, group Q-vectors including all RSU group information are exchanged through V2X (Vehicle-to-Everything) communication, and Q-Vector-based Road-Segment (QVRS) control packets defined in the network are periodically broadcast in each segment to update V2X evaluation metrics obtained through vehicle position, speed, direction, and communication status. This integrated V2X metric is presented as a reward function in the Q-learning algorithm. Through the group Q-table and the local Q-table, it can be hierarchically trained in response to a destination vehicle based on a packet forwarding process from a global perspective.

[0012] Embodiments of the present invention can reduce broadcast overhead and extend path lifetime while maintaining a high packet delivery ratio and low average end-to-end latency. Furthermore, group design can significantly improve the convergence speed of the learning process.

[0013] FIG. 1 is a diagram of a routing structure grouped based on RSU in a vehicle ad hoc network according to urban roads and vehicle mobility according to one embodiment of the present invention.

[0014] FIG. 2 is a flowchart illustrating a hierarchical reinforcement learning-based routing method supporting grouped RSUs in an urban VANET according to one embodiment of the present invention.

[0015] FIG. 3 is a diagram illustrating the structure of a hierarchical reinforcement learning-based routing system supporting grouped RSUs in an urban VANET according to one embodiment of the present invention.

[0016] FIG. 4 is a block diagram of a hierarchical reinforcement learning-based routing algorithm supporting grouped RSUs in an urban VANET according to one embodiment of the present invention.

[0017] FIG. 5 is a diagram for explaining a group Q-table, a local Q-table, and a Q-vector according to one embodiment of the present invention.

[0018] FIG. 6 is a diagram for explaining a V2X broadcasting mechanism according to one embodiment of the present invention.

[0019] FIG. 7 is a diagram for explaining a data routing process according to one embodiment of the present invention.

[0020] FIG. 8 is a diagram showing a data and ACK packet structure according to one embodiment of the present invention.

[0021] FIG. 9 is a drawing for explaining an example of a scenario according to one embodiment of the present invention.

[0022] In the present invention, a hierarchical reinforcement learning-based routing method and system supporting grouped RSUs in an urban VANET are proposed.

[0023] In an embodiment of the present invention, a large-scale urban road area is depicted as having RSUs deployed at each intersection, and two RSUs connected to a road segment are considered neighboring RSUs. Due to geographical constraints, each RSU may have at least one neighboring RSU. Vehicles on each road segment may move at different speeds and in different ways. Based on the intersection locations, the area is divided into multiple groups, each group containing multiple RSUs. Therefore, there are multiple distinct groups overall, and multiple RSUs are deployed within a group locally. A routing path means that data packets are delivered through multi-hop transmissions, sequentially between vehicles and RSUs.

[0024] The Q-value (or Q-function) of the Q-learning algorithm according to an embodiment of the present invention is used to measure the expected total reward that an agent can receive by performing a specific action in a specific state. Each agent begins exploring the environment with an initial Q-table. The agent selects an RSU, and the environment is the entire network topology. The state represents a destination vehicle group or a destination RSU within the destination vehicle group, depending on the current location of the RSU. The Q-vector collects key information from the group Q-table and the local Q-table, and this information is shared with neighboring RSUs via broadcast messages.

[0025] Broadcasting within RSUs and vehicles is necessary to share information about the network environment and link status changes and to ensure that packets are reliably delivered to the correct destination at the desired time. To achieve efficient and effective communication, a Q-vector-based broadcast mechanism is proposed. Each RSU broadcasts a proposed Q-Vector-based Road-Segment (QVRS) control message to each neighboring RSU via the vehicle. Broadcasting occurs independently for each road segment to reduce network overhead. The QVRS message includes a Q-vector, which includes a timestamp, the sequence number of the sending RSU, its address, and the group information of the corresponding RSU. When a vehicle receives a broadcast message on this road segment, it first records its position, speed, and direction, calculates an integrated path cost associated with the propagation delay, hop count, minimum link quality, and minimum expected expiration time, and stores this in a local link cost table to establish and maintain a relationship with its neighboring RSU. Finally, the vehicle retransmits the QVRS message until it reaches its neighboring RSU. The integrated road segment path cost is calculated by evaluating the propagation delay, number of hops, link quality, and expected expiration lifespan, and is used as a reward function in the Q-learning algorithm. Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0026]

[0027] FIG. 1 is a diagram of a routing structure grouped based on RSU in a vehicle ad hoc network according to urban roads and vehicle mobility according to one embodiment of the present invention.

[0028] Referring to FIG. 1, a routing path (140) is illustrated grouped (130) based on multiple RSUs (110) in a vehicle ad hoc network according to urban roads and vehicle (120) mobility.

[0029] Two RSUs connected by a road segment are considered neighboring RSUs. Due to geographical conditions, each RSU has at least one neighboring RSU. In each road segment, vehicles move at different variable speeds and directions. Based on the location of intersections, the area can be divided into multiple groups. The multiple groups is indicated as , where And NG is the total number of groups. represents a group containing multiple RSUs, It includes multiple RSUs, such as , where the group The total number of my RSUs is am. Silver index with Indicates the RSU ID within the vehicle. Therefore, there are multiple different groups overall, and multiple RSUs are placed in a group locally. In Fig. 1, the arrow line (140) indicates a routing path along which data packets are sequentially transmitted through vehicles and RSUs via multi-hop transmission.

[0030] A Q-learning algorithm is implemented for the scenario of the present invention. The Q-value (or Q-function) is used to measure the expected total reward that an agent can receive by performing a specific action in a specific state. Each agent starts with an initial Q-table and begins exploring the environment. The agent is in state s at time t. t Action in a t When performing and receiving reward R, the Q-value of the Q-table is updated as follows [Mathematical Formula 1] according to the Bellman Optimality Equation:

[0031] [Mathematical Formula 1]

[0032]

[0033] Here, α is the learning rate, is the discount rate. During learning, the Q-value is updated based on the rewards received and the actions performed by the agent. If the Q-value for a particular state-action pair is high, performing that action in that state and following the optimal policy is expected to yield a high total reward. In other words, a high Q-value indicates that the agent has learned that that action is likely to yield high rewards in the long run, and therefore should select that action in that state.

[0034]

[0035] FIG. 2 is a flowchart illustrating a hierarchical reinforcement learning-based routing method supporting grouped RSUs in an urban VANET according to one embodiment of the present invention.

[0036] A routing method based on hierarchical reinforcement learning supporting grouped RSUs in an urban VANET according to an embodiment of the present invention includes the steps of: generating a data packet to be transmitted by a user vehicle, determining a source RSU (roadside unit) and transmitting the packet to adjacent vehicles based on a link cost table (210); identifying the RSU of the nearest destination vehicle among the adjacent vehicles by the source RSU and confirming a group to which the RSU of the destination vehicle belongs (220); selecting the next RSU from a group Q-table based on information of the destination vehicle (230); transmitting an ACK message including information of the next RSU with the maximum Q-value in the group Q-table back to the source RSU (240); and updating a local Q-table of the source RSU (250).

[0037] In step (210), a data packet to be transmitted by a user vehicle is generated, a source roadside unit (RSU) is determined, and the packet is transmitted to adjacent vehicles based on a link cost table. The step of generating a data packet to be transmitted by the user vehicle, determining a source RSU, and transmitting the packet to adjacent vehicles based on a link cost table sets a plurality of groups based on the positions of RSUs at each intersection on the road, and transmits the packet through data transmission between a plurality of RSUs included in the plurality of groups and a plurality of vehicles including the user vehicle.

[0038] In step (220), the source RSU identifies the RSU of the closest destination vehicle among the adjacent vehicles and checks the group to which the RSU of the destination vehicle belongs.

[0039] The step of identifying the RSU of the nearest destination vehicle among the adjacent vehicles by the source RSU and confirming the group to which the RSU of the destination vehicle belongs is such that the source RSU broadcasts a QVRS (Q-Vector-based Road-Segment) control message to the RSU through the destination vehicle. The QVRS message includes a Q-vector including a time stamp, a sequence number of the source RSU, an address, and group information of the RSU of the destination vehicle, and when the destination vehicle receives the QVRS message, the location, speed, and direction of the destination vehicle are recorded, an integrated path cost related to a propagation delay, a hop count, a minimum link quality, and a minimum expected expiration life are calculated, and the integrated path cost is stored in a local link cost table, and a relationship with a neighboring RSU is established and maintained.

[0040] In step (230), the next RSU is selected from the group Q-table based on the information of the destination vehicle. To update group information through a Q-learning algorithm, the state of the RSU is encoded into a Q-vector using the group Q-table storing the global Q-value and the local Q-table storing the local Q-value of the RSU within the group.

[0041] Measures the expected total reward received by performing a specific action in a specific state representing a group of destination vehicles or an RSU within a group of destination vehicles, based on the current location of the source RSU.

[0042] The above source RSU evaluates the V2X metrics including the Q-vector and forwarding delay, number of hops, link quality and minimum expected expiration lifetime, and uses the V2X metrics as a reward function to select the next RSU from the group Q-table.

[0043] Thereafter, in step (240), an ACK message containing information of the next RSU with the maximum Q-value in the group Q-table is retransmitted to the source RSU, and in step (250), the local Q-table of the source RSU is updated.

[0044]

[0045] FIG. 3 is a diagram illustrating the structure of a hierarchical reinforcement learning-based routing system supporting grouped RSUs in an urban VANET according to one embodiment of the present invention.

[0046] The hierarchical reinforcement learning-based routing system (300) according to the present embodiment may include a processor (310), a bus (320), a network interface (330), a memory (340), and a database (350). The memory (340) may include an operating system (341) and a hierarchical reinforcement learning-based routing routine (342) that supports grouped RSUs in an urban VANET. The processor (310) may include an RSU group verification unit (311), an RSU selection unit (312), and a Q-table management unit (313). In other embodiments, the hierarchical reinforcement learning-based routing system (300) may include more components than those in FIG. 3. However, it is not necessary to explicitly illustrate most of the conventional components. For example, the hierarchical reinforcement learning-based routing system (300) may include other components such as a display or a transceiver.

[0047] The memory (340) is a computer-readable storage medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. In addition, the memory (340) may store an operating system (341) and a program code for a hierarchical reinforcement learning-based routing routine (342) that supports grouped RSUs in an urban VANET. These software components may be loaded from a computer-readable storage medium separate from the memory (340) using a drive mechanism (not shown). This separate computer-readable storage medium may include a computer-readable storage medium (not shown) such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (340) through a network interface (330) rather than a computer-readable storage medium.

[0048] The bus (320) may enable communication and data transmission between components of the hierarchical reinforcement learning-based routing system (300). The bus (320) may be configured using a high-speed serial bus, a parallel bus, a storage area network (SAN), and / or other suitable communication technology.

[0049] The network interface (330) may be a computer hardware component for connecting the hierarchical reinforcement learning-based routing system (300) to a computer network. The network interface (330) may connect the hierarchical reinforcement learning-based routing system (300) to the computer network via a wireless or wired connection.

[0050] The database (350) can serve to store and maintain all information necessary for hierarchical reinforcement learning-based routing supporting grouped RSUs in an urban VANET. While FIG. 3 illustrates the database (350) as being built and included within the hierarchical reinforcement learning-based routing system (300), the present invention is not limited thereto and may be omitted depending on the system implementation method or environment, or the entire database or part of the database may exist as an external database built on a separate system.

[0051] The processor (310) may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations of a hierarchical reinforcement learning-based routing system (300). The instructions may be provided to the processor (310) by a memory (340) or a network interface (330), and via a bus (320). The processor (310) may be configured to execute program codes for an RSU group verification unit (311), an RSU selection unit (312), and a Q-table management unit (313). These program codes may be stored in a storage device such as the memory (340).

[0052] The RSU group verification unit (311), the RSU selection unit (312), and the Q-table management unit (313) can be configured to perform steps (210 to 250) of FIG. 2.

[0053] A hierarchical reinforcement learning-based routing system (300) may include an RSU group verification unit (311), an RSU selection unit (312), and a Q-table management unit (313).

[0054] The RSU group verification unit (311) generates a data packet to be transmitted by the user vehicle, determines a source RSU (roadside unit), and transmits the packet to adjacent vehicles based on a link cost table. Then, the RSU of the closest destination vehicle among the adjacent vehicles is identified, and the group to which the RSU of the destination vehicle belongs is verified.

[0055] According to an embodiment of the present invention, first, a plurality of groups are set based on the positions of RSUs at each intersection on a road, and packets are transmitted through data transmission between a plurality of RSUs included in the plurality of groups and a plurality of vehicles including the user vehicle.

[0056] The RSU group verification unit (311) of the source RSU broadcasts a QVRS (Q-Vector-based Road-Segment) control message to the RSU through the destination vehicle. The QVRS message includes a Q-vector including a time stamp, a sequence number of the source RSU, an address, and group information of the RSU of the destination vehicle, and when the destination vehicle receives the QVRS message, the location, speed, and direction of the destination vehicle are recorded, and an integrated path cost related to a propagation delay, a hop count, a minimum link quality, and a minimum expected expiration life are calculated and stored in a local link cost table, and a relationship with a neighboring RSU is established and maintained.

[0057] The RSU selection unit (312) of the above source RSU selects the next RSU from the group Q-table based on the information of the destination vehicle. To update group information through a Q-learning algorithm, the state of the RSU is encoded into a Q-vector using the group Q-table that stores the global Q-value and the local Q-table that stores the local Q-value of the RSU within the group.

[0058] The RSU selection unit (312) of the source RSU measures the expected total reward received by performing a specific action in a specific state representing a group of destination vehicles or an RSU within a group of destination vehicles according to the current location of the source RSU.

[0059] The RSU selection unit (312) of the above source RSU evaluates the V2X metrics including the Q-vector and transmission delay, number of hops, link quality, and minimum expected expiration lifetime, and uses the V2X metrics as a compensation function to select the next RSU from the group Q-table.

[0060] The Q-table management unit (313) of the source RSU receives an ACK message including information on the next RSU with the maximum Q-value in the group Q-table and updates the local Q-table of the source RSU.

[0061]

[0062] FIG. 4 is a block diagram of a hierarchical reinforcement learning-based routing algorithm supporting grouped RSUs in an urban VANET according to one embodiment of the present invention.

[0063] Referring to FIG. 4, each RSU acts as an agent (410) and individually maintains two types of Q-tables. The Group Q-table stores the global Q-value to update destination group information, and the Local Q-table updates the local Q-value of the RSU of the destination vehicle (420) within the group. Based on the Group and Local Q-tables, a Q-vector is generated for each RSU, which is included in the control packet to evaluate V2X metrics between adjacent RSUs in the Environment (430), such as transmission delay, number of hops, link quality, and expected expiration life. For data routing of the vehicle (420), the next RSU is selected based on the maximum Q-value, and the V2X metric is used as a reward function to update the Q-table. This process is repeated until the data is successfully transmitted to the destination. In the scenario of the present invention, the RSU is selected as the agent, and the environment is the topology of the entire network. The state is determined based on the current location of the RSU. or target RSU within the target group The state is defined as follows [Mathematical Formula 2]:

[0064] [Equation 2]

[0065]

[0066] Your current location is If outside of , the state is the destination group ID. Since any group can be a destination group, the state set contains all group IDs. The current location is If inside, the state is the destination group All RSU IDs within, for all possible destinations. All RSU IDs within the state set are included. The task of the RSU to select the next road segment to forward the packet is defined in several directions at each intersection. These directions are is defined as . Here is RSU is the number of neighboring RSUs.

[0067]

[0068] FIG. 5 is a diagram for explaining a group Q-table, a local Q-table, and a Q-vector according to one embodiment of the present invention.

[0069] In the Group Q-table shown in Fig. 5(a), the rows of the table represent all possible destination groups, and the columns are diagrams showing neighboring RSUs that an RSU can connect to with possible actions.

[0070] In the local Q-table shown in Fig. 5(b), the rows of the table represent all possible destination RSUs within a destination group, and the columns represent neighboring RSUs within the same group to which the RSU can connect with possible actions.

[0071] The Q-vector shown in Fig. 5(c) is a diagram that collects key information from the group Q-table and the local Q-table and represents values ​​shared with neighboring RSUs through broadcast messages.

[0072] Therefore, the Q-value, which is the value in each cell of the group Q-table and the local Q-table, represents the expected reward for the destination group / destination RSU that the RSU will receive when it takes that action.

[0073] The format of the Q-vector is shown in Fig. 5(c) and the following equations [Mathematical Formula 3] and [Mathematical Formula 4] are used.

[0074] [Equation 3]

[0075]

[0076] [Equation 4]

[0077]

[0078] Here All groups except the destination group Corresponding to Indicates the maximum Q-value among all maximum Q-values ​​of neighboring RSUs in the group Q-table. silver represents the neighboring RSU of , and the index am. Is Destination RSU in the local Q-table Represents the average of all maximum Q-values ​​for . is in the destination group represents the neighboring RSU of , and the index am.

[0079]

[0080] FIG. 6 is a diagram for explaining a V2X broadcasting mechanism according to one embodiment of the present invention.

[0081] Broadcasting is required within RSUs (611, 612) and vehicles to share information about the network environment and link status and to ensure that packets are delivered to their destinations reliably and in a timely manner. To enable efficient and effective communication in the network, a Q-vector-based broadcasting mechanism is proposed. Each RSU (611) broadcasts a designed QVRS control message to neighboring RSUs (612) through the vehicle. To reduce network overhead, broadcasting occurs independently on each road segment. The QVRS message includes a Q-vector that contains a timestamp, the sequence number of the transmitting RSU (611), its address, and the group information of the corresponding RSU. When a vehicle receives a broadcast message on this road segment, it first records its own position, speed, and direction, calculates an integrated V2X path cost related to the QVRS propagation delay, hop count, minimum link quality, and minimum expected expiration lifetime, and then stores it in a local V2X link cost table to establish and maintain relationships with neighboring RSUs (612). Finally, the vehicle rebroadcasts this QVRS message until it reaches the neighboring RSU (612). The metrics used to calculate the integrated V2X path cost are described below.

[0082] First, delivery delay Silver Sender RSU From previous intermediate node Recipient via This means the time it takes for a packet to be transmitted. at Delivery delays until class at One-time delivery delay to It consists of the sum of . This can be expressed as [Mathematical Formula 5].

[0083] [Equation 5]

[0084]

[0085] One hop delivery delay represents the time it takes for a packet to pass through a single network hop, from the sender to the receiver. Recipient from Transmission delay up to , propagation delay , at Queue delay to and If there is no adjacent node Store-forward delay It is determined by several factors including . These delays can be expressed as [Equation 6], [Equation 7], [Equation 8], and [Equation 9], respectively.

[0086] [Equation 6]

[0087]

[0088] [Equation 7]

[0089]

[0090] [Equation 8]

[0091]

[0092] [Equation 9]

[0093]

[0094] Here Is The length of the message, is the transmission rate, is the distance between two nodes, is the propagation speed in the medium, Is is the queue buffer size.

[0095] Next is the hop count is the sender RSU Current recipient in This represents the number of intermediate nodes, or 'hops', to the destination. Each time a vehicle forwards a packet, the hop count increases by 1. Generally, a lower hop count indicates a more direct and efficient route, with a lower chance of congestion or data loss.

[0096] Next, link quality silver class It represents the reliability and stability of the communication link between them, and can be expressed as [Mathematical Formula 10] below.

[0097] [Equation 10]

[0098]

[0099] Here is the minimum decryptable power (dBm) based on encoding rate, channel bandwidth and other factors. is the previous node of the message It is the received power.

[0100] Next, we will explain the expected lifetime. If the receiving node moves out of the transmitter's transmission range, the communication link is disconnected. The expected lifetime represents the duration of the communication connection, which is the period during which the connection link can exist. The expected lifetime is determined by the vehicle's position, speed, and movement. At time t, class The distance between is displayed as, are each at time t class Indicates the speed of . are each Represents the velocity components in the X and Y directions, silver represents the velocity components in the X and Y directions. After a certain amount of time, If, is calculated using the following equation [Equation 11] class Indicates the expected lifespan of the liver.

[0101] [Equation 11]

[0102]

[0103] Here,

[0104]

[0105]

[0106]

[0107] Communication range is expressed using the following equation [Mathematical Formula 12] in a path loss model using log-normal shadowing.

[0108] [Equation 12]

[0109]

[0110] Here,

[0111]

[0112] Next, the Integrated V2X Path Reward is a comprehensive indicator that considers four factors: long delivery delays and hop counts are disadvantageous, while link quality and long expected expiration times are advantageous. It is calculated using the following equation (Mathematical Equation 13).

[0113] [Equation 13]

[0114]

[0115] Here

[0116]

[0117]

[0118] Here class silver at Represents the minimum link quality and expected lifetime among all links to the variable and represent the maximum possible transmission delay and hop count values, respectively. Similarly, the variables and represent the minimum link quality and the maximum possible values ​​of the expected lifetime, respectively. Weighting factors , , , and is used to balance the importance of the four factors in a metric. Keep the value negative and this value is sent to the sender RSU and recipient Because it represents the path cost of the communication link between vehicles. During QVRS message broadcast, each vehicle updates and stores the information of its neighbors in the link cost table. Is at The maximum path compensation achieved through other vehicles is represented by [Mathematical Formula 14] below.

[0119] [Equation 14]

[0120]

[0121] At this time, it should be noted that the path compensation in the two directions on the same road segment has different values. When a QVRS message arrives at the surrounding RSUs, Is Update the entire column of the corresponding group Q-table using the following [Mathematical Formula 15].

[0122] [Equation 15]

[0123]

[0124] Here, represents the learning rate of the global Q-table, is for all groups Represent the Q-vector of .

[0125] When a vehicle generates and transmits a data packet, it first sends it to the source RSU ( ) and forward it to one of the neighboring vehicles based on the link table. After that, the source RSU must determine the nearest RSU to the destination vehicle (destination RSU, ) and identify the group to which the RSU belongs ( , in other words, Check the group to which it belongs. All RSUs ( ) must decide the next RSU to forward the data packet. If Go to destination group If it does not belong to the following RSU can be determined using the group Q-table of the corresponding RSU. If Go to destination group If it belongs to , it uses the local Q-table of the RSU as in [Equation 16] to find the next RSU. Decide.

[0126] [Equation 16]

[0127]

[0128]

[0129] FIG. 7 is a diagram for explaining a data routing process according to one embodiment of the present invention.

[0130] Figure 7 illustrates the data routing process of the present invention. When a vehicle generates a data packet to be transmitted, the data routing process first determines the source RSU and forwards the packet to a neighboring vehicle based on the link cost table (710). The source RSU is the RSU of the nearest destination vehicle. Identify (720) The group that belongs to . Based on the information of the destination vehicle, the next RSU is selected from the Q-table (730), and data is transmitted through the destination vehicle (740). The actual V2X cost is sent back to the sender RSU (i.e., the source RSU) using an ACK message that includes the neighbor RSU with the maximum Q-value (750), through which the Q-table of the sender RSU is updated (760). RSU Go to the next RSU When data is transmitted and rewards are observed, the Q-value is updated. The next RSU and destination group Maximum Q-value along the location of Calculate using [Equation 17].

[0131] [Equation 17]

[0132]

[0133] if Go to destination group If it is out of range, then use the following [Equation 18] Group Q-value of Update.

[0134] [Equation 18]

[0135]

[0136] if Go to destination group If you're inside, Local Q-value of is updated using [Mathematical Formula 19] below.

[0137] [Equation 19]

[0138]

[0139] represents the learning rate of the local Q table, and using different learning rates between group and local Q table updates allows the agent to better update the Q-value based on new experience. The reward function R is defined as follows [Mathematical Formula 20].

[0140] [Equation 20]

[0141]

[0142] Here is the end-to-end compensation provided only when the data packet reaches the destination RSU. represents the actual V2X path cost calculated using [Equation 14] during data transmission. Over time, the Q-table is updated with more information about the expected reward for each action in each state. As the Q-table becomes more accurate, the RSU can use it to select the action that maximizes the expected long-term reward. Meanwhile, the epsilon-exploration technique is used to balance exploration and exploitation. This technique initially gives a higher It starts with a value that promotes exploration and decreases exponentially as the Q-table accumulates more information about the environment.

[0143]

[0144] FIG. 8 is a diagram showing a data and ACK packet structure according to one embodiment of the present invention.

[0145] Figure 8(a) shows the data packet structure, and Figure 8(b) shows the ACK packet structure.

[0146]

[0147] FIG. 9 is a drawing for explaining an example of a scenario according to one embodiment of the present invention.

[0148] Fig. 9(a) is a diagram for explaining Q-vector generation, Fig. 9(b) is a diagram for explaining V2X metric calculation during broad test, Fig. 9(c) is a diagram for explaining Q-table update with Q-vector and path compensation, and Fig. 9(d) is a diagram for explaining Q-table update of RSU selection and data routing.

[0149] Referring to Figure 9, the example topology assumes three groups and eight RSUs. At each intersection, a Q-vector is generated based on the current group Q-table and local Q-table using Equations 3 and 4, respectively. As shown in Figure 9(a), R1 of G2 and G3 1 The Q-vector value of R1 captures the maximum Q-value in the G2 and G3 states in the group Q-table, while 1 The G1Q-vector value of R1 here 1 is calculated by summing the maximum Q-values ​​from all states in the local Q-table and then taking the average. The Q-vector is included in the QVRS message and broadcasted to neighboring RSUs. As shown in Fig. 9(b), R1 1 records V2X metrics and transmits QVRS messages to R2 via multi-hop vehicles. 1 The hop count and propagation delay are accumulated across the vehicles, but the link quality and expected expiration lifetime are the minimum values ​​among all links in the path. At the same time, each vehicle calculates the path compensation using Equation 13 with a weight factor of 0.25 and a maximum normalization value of 10. The path compensation is stored in the vehicle's link table to make V2X routing decisions. In QVRS, R2 1 Path rewards are updated during the broadcast until reaching R2. 1 The maximum path compensation is obtained by comparing multiple income QVRS messages using Equation 13. Select .

[0150] When learning in Fig. 9(c), R1 is obtained by using Equation 14. 1 Q-vector and R1 1 Corresponding to the action Based on R2 1 The group Q-table update is shown. The ratio is 0.1 and the discount factor is 0.9. In Fig. 9(d), a data packet is sent to R1 with destination information G2. 1 When I arrived at In this state G2, the maximum Q-value is stored. The data packet is stored in R2 1 The data packet is transmitted by the vehicle based on the link table for R2 1 When you arrive at R2 in G2 state 1 Maximum Q value and data path compensation R1 is included in the ACK 1 is sent back to R1. Finally, 1 silver Update the Q-value.

[0151]

[0152] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0153] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0154] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0155] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0156] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A step of generating a data packet to be transmitted by a user vehicle, determining a source RSU (roadside unit), and transmitting the packet to adjacent vehicles based on a link cost table; The above source RSU identifies the RSU of the nearest destination vehicle among adjacent vehicles and checks the group to which the RSU of the destination vehicle belongs; A step of selecting the next RSU from the group Q-table based on the information of the destination vehicle; A step of retransmitting an ACK message containing information of the next RSU with the maximum Q-value in the above group Q-table to the source RSU; and Steps to update the local Q-table of the source RSU A hierarchical reinforcement learning-based routing method including:

2. In paragraph 1, The step of generating a data packet to be transmitted by the user vehicle, determining a source RSU, and transmitting the packet to adjacent vehicles based on a link cost table is as follows. A plurality of groups are set based on the location of the RSU at each intersection on the road, and packets are transmitted through data transmission between a plurality of RSUs included in the plurality of groups and a plurality of vehicles including the user vehicle. A routing method based on hierarchical reinforcement learning.

3. In paragraph 1, The step of identifying the RSU of the nearest destination vehicle among the adjacent vehicles and checking the group to which the RSU of the destination vehicle belongs is as follows. The source RSU broadcasts a QVRS (Q-Vector-based Road-Segment) control message to the RSU through the destination vehicle, and the QVRS message includes a Q-vector including a time stamp, a sequence number of the source RSU, an address, and group information of the RSU of the destination vehicle, and when the destination vehicle receives the QVRS message, records the location, speed, and direction of the destination vehicle, calculates an integrated path cost related to a propagation delay, a hop count, a minimum link quality, and a minimum expected expiration life, and stores the calculated integrated path cost in a local link cost table, and establishes and maintains a relationship with a neighboring RSU. A routing method based on hierarchical reinforcement learning.

4. In paragraph 1, The step of selecting the next RSU from the group Q-table based on the information of the above destination vehicle is: To update group information through the Q-learning algorithm, the state of the RSU is encoded as a Q-vector using a group Q-table that stores the global Q-value and a local Q-table that stores the local Q-value of the RSU within the group. Measure the expected total reward received by performing a specific action in a specific state representing a group of destination vehicles or an RSU within a group of destination vehicles, based on the current location of the source RSU; The above source RSU evaluates the V2X metrics including the Q-vector and forwarding delay, number of hops, link quality and minimum expected expiration lifetime, and uses the V2X metrics as a reward function to select the next RSU from the group Q-table. A routing method based on hierarchical reinforcement learning.

5. An RSU group confirmation unit that generates a data packet to be transmitted by a user vehicle, determines a source RSU (roadside unit), and transmits the packet to adjacent vehicles based on a link cost table, identifies the RSU of the nearest destination vehicle among the adjacent vehicles, and confirms the group to which the RSU of the destination vehicle belongs; An RSU selection unit that selects the next RSU from the group Q-table based on the information of the destination vehicle; and A Q-table management unit that receives an ACK message containing information of the next RSU with the maximum Q-value from the group Q-table and updates the local Q-table of the source RSU. A hierarchical reinforcement learning-based routing system including:

6. In paragraph 5, The above RSU verification unit, The source RSU broadcasts a QVRS (Q-Vector-based Road-Segment) control message to the RSU through the destination vehicle, and the QVRS message includes a Q-vector including a time stamp, a sequence number of the source RSU, an address, and group information of the RSU of the destination vehicle, and when the destination vehicle receives the QVRS message, records the location, speed, and direction of the destination vehicle, calculates an integrated path cost related to a propagation delay, a hop count, a minimum link quality, and a minimum expected expiration life, and stores the calculated integrated path cost in a local link cost table, and establishes and maintains a relationship with a neighboring RSU. Hierarchical reinforcement learning-based routing system.

7. In paragraph 5, The above RSU selection unit, To update group information through the Q-learning algorithm, the state of the RSU is encoded as a Q-vector using a group Q-table that stores the global Q-value and a local Q-table that stores the local Q-value of the RSU within the group. Measure the expected total reward received by performing a specific action in a specific state representing a group of destination vehicles or an RSU within a group of destination vehicles, based on the current location of the source RSU; The above source RSU evaluates the V2X metrics including the Q-vector and forwarding delay, number of hops, link quality and minimum expected expiration lifetime, and uses the V2X metrics as a reward function to select the next RSU from the group Q-table. Hierarchical reinforcement learning-based routing system.

Images