System and method for cooperative lane assignment using reinforcement learning

Abstract

To provide a system.SOLUTION: A disclosed system and a method are relevant to a system and a method for collaborative lane assignment using reinforcement learning. According to an embodiment, a segment manager communicating with multiple connected vehicles within a segment of a road receives target lateral flows on two or more lanes of a road within a segment of the road from a regional manager. The segment manager receives pieces of traffic condition information from multiple connected vehicles. The segment manager processes a series of traffic condition information and target lateral flow using reinforcement learning-based models and determines lane change actions relevant to multiple connected vehicles. Here, the reinforcement learning base model is based on a single neural network that shares parameters for multiple connected vehicles relevant to multiple connected vehicles. The lane change action is transmitted to multiple connected vehicles from the segment manager.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The subject matter described herein generally relates to vehicle traffic flow adjustment systems, and more specifically, to systems and methods for cooperative lane assignment using reinforcement learning. [Background technology]

[0002] Cooperative lane allocation strategies offer promising strategies for improving vehicle flow and mitigating traffic congestion. These strategies can be applied to both connected (network-enabled) manual-drive vehicles and connected autonomous vehicles. By predicting potential congestion downstream and repositioning vehicles accordingly, such systems can significantly improve the safety and efficiency of traffic flow, even if connected vehicles constitute only a small percentage of the vehicles on the road. However, designing systems that can achieve such coordination under real-world conditions remains a challenging task. For example, a single-dimensional lane allocation strategy may not be scalable due to its dimensionality. Furthermore, execution speed and communication latency can be significant problems in a single-dimensional system. [Overview of the project]

[0003] In this specification, embodiments of a system for cooperative lane assignment are presented. In one embodiment, the system includes a processor and a memory storing machine-readable instructions that, when executed by the processor, cause the processor to receive, from a regional manager, a target lateral flow for two or more lanes of a road within a section of the road. The memory also stores machine-readable instructions that, when executed by the processor, cause the processor to receive traffic state information from a plurality of connected vehicles within the section of the road. The memory also stores machine-readable instructions that, when executed by the processor, cause the processor to use a reinforcement learning-based model to process the traffic state information and the target lateral flow to determine lane change actions for the plurality of connected vehicles. The reinforcement learning-based model is based on a single neural network that shares parameters for the plurality of connected vehicles. The memory also stores machine-readable instructions that, when executed by the processor, cause the processor to send the lane change actions to the plurality of connected vehicles.

[0004] Another embodiment is a non-transitory computer-readable medium for cooperative lane assignment that stores instructions that, when executed by a processor, cause the processor to receive, from a regional manager, a target lateral flow for two or more lanes of a road within a section of the road. The instructions also cause the processor to receive traffic state information from a plurality of connected vehicles within the section of the road. The instructions also cause the processor to use a reinforcement learning-based model to process the traffic state information and the target lateral flow to determine lane change actions for the plurality of connected vehicles. The reinforcement learning-based model is based on a single neural network that shares parameters for the plurality of connected vehicles. The instructions also cause the processor to send the lane change actions to the plurality of connected vehicles.

[0005] Another embodiment is a cooperative lane assignment method that includes receiving, at an in - segment manager communicating with a plurality of connected vehicles within a road segment, a target lateral flow for two or more lanes of the road from a regional manager. The method also includes receiving, at the in - segment manager, traffic state information from the plurality of connected vehicles. The method also includes determining, at the in - segment manager, a lane - change action for the plurality of connected vehicles by processing the traffic state information and the target lateral flow using a reinforcement - learning - based model. The reinforcement - learning - based model is based on a single neural network that shares parameters for the plurality of connected vehicles. The method also includes transmitting the lane - change action from the in - segment manager to the plurality of connected vehicles.

[0006] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the component boundaries (e.g., boxes, groups of boxes, or other shapes) illustrated in the drawings represent one embodiment of the boundaries. In some embodiments, one component may be designed as multiple components, or multiple components may be designed as one component. In some embodiments, a component shown as an internal component of another component may be implemented as an external component, and vice versa. Further, the components may not be drawn to scale.

Brief Description of the Drawings

[0007] [Figure 1] FIG. 1 is an architecture diagram of a hierarchical traffic flow adjustment system according to an exemplary embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram of a regional manager according to an exemplary embodiment of the present invention. [Figure 3] FIG. 3 is a diagram of a road segment, lanes, and traffic flow parameters according to an exemplary embodiment of the present invention. [Figure 4] Figure 4 shows a reinforcement learning-based Markov decision process model within a regional manager, according to an exemplary embodiment of the present invention. [Figure 5] Figure 5 shows the state of the Markov decision process model within a regional manager according to an exemplary embodiment of the present invention. [Figure 6] Figure 6 shows how the boundary on a transverse flow is determined according to an exemplary embodiment of the present invention. [Figure 7] Figure 7 is a flowchart of a traffic flow adjustment method via unified lateral flow control according to an exemplary embodiment of the present invention. [Figure 8] Figure 8 is a block diagram of a section manager according to an exemplary embodiment of the present invention. [Figure 9A] Figure 9A illustrates an exemplary embodiment of the present invention relating to a connected vehicle and the measured distances to a following vehicle in a target lane. [Figure 9B] Figure 9B is a diagram illustrating a ranking process based on each measured distance from a connected vehicle to a following vehicle in a target lane, according to an exemplary embodiment of the present invention. [Figure 10A] Figure 10A illustrates a connected vehicle and the measured distances to a following vehicle in a target lane, according to another exemplary embodiment of the present invention. [Figure 10B] Figure 10B is a diagram illustrating a ranking process based on each measured distance from a connected vehicle to a following vehicle in a target lane, in the case of bidirectional lateral flow, according to an exemplary embodiment of the present invention. [Figure 11] Figure 11 is a flowchart of a cooperative lane allocation method according to an exemplary embodiment of the present invention. [Figure 12] Figure 12 is a block diagram of a section manager according to another exemplary embodiment of the present invention. [Figure 13A] Figure 13A is a diagram of a Markov decision process model within an interval manager according to an exemplary embodiment of the present invention. [Figure 13B]Figure 13B shows the state of a Markov decision process model within an interval manager according to an exemplary embodiment of the present invention. [Figure 13C] Figure 13C shows a connected vehicle action in a Markov decision process model within a section manager, according to an exemplary embodiment of the present invention. [Figure 14A] Figure 14A shows a neural network employed in a Markov decision process model within an interval manager, according to an exemplary embodiment of the present invention. [Figure 14B] Figure 14B shows a scenario in an exemplary embodiment of the present invention where the number of connected vehicles in a section exceeds the number of observation nodes in the neural network. [Figure 14C] Figure 14C shows a scenario in an exemplary embodiment of the present invention where the number of observation nodes in the neural network exceeds the number of connected vehicles in the interval. [Figure 15] Figure 15 is a flowchart of a cooperative lane allocation method according to another exemplary embodiment of the present invention. [Modes for carrying out the invention]

[0008] The various embodiments of systems and methods for hierarchical traffic flow adjustment described herein overcome the limitations of conventional traffic flow adjustment systems. These embodiments employ a hierarchical paradigm that combines the ease of use of macroscopic lane allocation models with the safety and vehicle-level considerations of microscopic lane allocation models. An example of this hierarchical paradigm is shown in Figure 1.

[0009] Figure 1 is an architectural diagram of a hierarchical traffic flow control system according to an exemplary embodiment of the present invention. As shown in Figure 1, the architecture 100 includes a road 130 having a plurality of lanes 150. The three lanes 150 shown in Figure 1 are illustrative. In other embodiments, the road 130 may have only two lanes, or it may have four or more lanes. The road 130 is divided longitudinally into a plurality of sections, such as sections 140a to 140e shown in Figure 1. Although Figure 1 shows that the length of each section is 400m, this is only an example. Section 140 may be shorter or longer than 400m, depending on the embodiment. The illustration of the five sections (140a to 140e) in Figure 1 is also merely illustrative. In some embodiments, a given regional manager 110 (described later) can provide services to more than five sections 140, or to fewer sections.

[0010] As shown in Figure 1, a section manager 120 provides services to each of the road sections 140a to 140e. Collectively, the section managers 120a to 120e constitute a section level of architecture 100. The section manager 120 of a given section 140 communicates with one or more connected vehicles 160 within that section via a communication link 190. As shown in Figure 1, a given section 140 may also include one or more conventional (unconnected) vehicles 170. The section managers 120a to 120e receive vehicle movement information and traffic condition information from the connected vehicles 160 within their respective sections 140. Such data may include reports on vehicle position, speed, and measured distance between vehicles on the road 130. In some embodiments, at least some of the connected vehicles 160 are autonomous (self-driving) vehicles operating in a manner referred to by the automotive industry as autonomy levels 3 to 5.

[0011] At the regional level of architecture 100, the regional manager 110 determines the target lateral flow (leftward movement and / or rightward movement) for lanes 150 within sections 140a to 140e of road 130 by processing aggregated macroscopic traffic state information received from section managers 120a to 120e via communication links 180. In some embodiments, the regional manager 110 generates the target lateral flow by processing the aggregated macroscopic traffic state information using a reinforcement learning (RL)-based model. When determining the target lateral flow, the RL-based model within the regional manager 110 may consider information relating to slow-moving or stationary vehicles 195 ("incidents"). Such incidents can adversely affect traffic flow and may cause traffic congestion. The regional manager 110 sends targeted lateral flow to the section managers 120a to 120e, and the section managers 120a to 120e (1) convert the targeted lateral flow into lane change actions for their respective sections 140a to 140e, and (2) send the lane change actions to the connected vehicles 160 within their respective sections 140a to 140e. The autonomous connected vehicles 160 can automatically execute the received lane change actions. The drivers of the manually operated connected vehicles 160 can voluntarily comply with the received lane change actions based on their prior consent to cooperate with the lane change actions received from the hierarchical traffic flow adjustment system.

[0012] How the section managers 120a to 120e translate the target lateral flow received from the regional manager 110 into lane change actions for specific connected vehicles 160 varies depending on the embodiment. In some embodiments, the section managers 120a to 120e employ a hierarchical algorithm that ranks the connected vehicles 160 in a given section 140 according to the measured distance to each following vehicle in the target (new) lane 150. In other embodiments, the section managers 120a to 120e employ an RL-based model to determine specific lane change actions for connected vehicles in their respective sections 140a to 140e.

[0013] As shown in Figure 1, the architecture 100 is hierarchical in that high-level actions (target lateral flow) are determined by the regional manager 110 and assigned to the section manager 120, thereby ensuring coordination between sections 140. Section managers 120, which are at lower levels of hierarchy and closer to the connected vehicles 160, convert the lateral flow into specific lane change actions and send these lane change actions to the connected vehicles 160 within each section 140. This hierarchical architecture 100 overcomes communication concerns / delay concerns by assigning target lateral flow to the section manager 120 at a relatively lower frequency than the frequency with which the section manager 120 assigns lane change actions to the connected vehicles 160. For example, in one embodiment, the regional manager 110 sends an assigned target lateral flow to the section manager 120 every 30 seconds, and the section manager 120 sends lane change actions to connected vehicles 160 within each section 140 every 5 seconds. In other embodiments, these time intervals may differ.

[0014] In some embodiments, the regional manager 110 provides services to a geographical area corresponding to a neighborhood or sub-lot. In other embodiments, the regional manager 110 may provide services to a geographical area or segment of road 130 that is larger or smaller than a neighborhood / sub-lot.

[0015] Compared to conventional centralized control methods, the architecture 100 described above offers significant improvements in medium and high demand settings, and the hierarchical traffic flow adjustment system provides particular advantages for the mobility of connected vehicles 160 in high demand settings where dense traffic congestion is likely to form.

[0016] The remainder of this detailed description of the invention is as follows: First, one embodiment of the regional manager 110 will be described in detail. Next, an embodiment of the hierarchical ranking algorithm-based interval manager 120 will be described in detail. Then, one embodiment of the RL-based interval manager 120 will be described in detail.

[0017] Regional Manager Figure 2 is a block diagram of a regional manager 110 according to an exemplary embodiment of the present invention. In Figure 2, the regional manager 110 includes one or more processors 205, with a memory 210 communicably coupled to the processors 205. More specifically, in this embodiment, the memory 210 stores an input module 215, a reinforcement learning (RL) module 220, an output module 225, and a training module 228. The memory 210 is random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable non-transient memory for storing modules 215, 220, 225, and 228. Modules 215, 220, 225, and 228 are, for example, machine-readable instructions that, when executed by one or more processors 205, cause one or more processors 205 to perform various functions disclosed herein.

[0018] As shown in Figure 2, the regional manager 110 can store various types of data in the database 230. For example, the regional manager 110 can store target lateral flow 235, aggregated traffic condition information 240, and model data 245. The model data 245 can include various different types of data related to the RL-based model, such as hyperparameters, parameters, neural network weights, attractiveness terms (described later), intermediate calculation results, etc.

[0019] As shown in Figure 2, the regional manager 110 can communicate with other network nodes 250 (section managers 120, servers, road infrastructure, etc.) via the network 255. In some embodiments, the network 255 includes the internet. When communicating with other network nodes 250, the regional manager 110 can utilize communication technologies such as high-speed Ethernet, fiber optic connections, cellular data (LTE, 5G, 6G, etc.), dedicated short-range communication (DSRC), Bluetooth® Low Energy (LE), etc.

[0020] The input module 215, when executed by one or more processors 205, generally includes commands to cause one or more processors 205 to receive aggregated macroscopic traffic condition information 240 from a section manager 120 that is communicating with one or more connected vehicles 160 within a section 140 of road 130. The specific components constituting the aggregated traffic condition information 240 are identified and described below in relation to Figure 3.

[0021] Figure 3 shows a road section 140, lane 150, and traffic flow parameters according to an exemplary embodiment of the present invention. Figure 3 illustrates two sections 140 of road 130 to illustrate aggregated traffic state information 240. In accordance with the macroscopic traffic flow formulation in question, in this embodiment the aggregated traffic state information 240 includes (1) the number of vehicles in section i and lane j at time k, and section length L iThe density ρ within section i and lane j is defined as the result of dividing by ρ. i,j (2) The longitudinal flow q for section i and lane j is defined as the traffic volume that leaves section i and enters section i+1, and therefore remains in lane j, during the time interval (k, k+1) i,j (3) The net lateral flow f from lane j to lane j+1 within section i, defined as the traffic volume that remains within the same section 140 while moving from lane j to lane j+1 during the current time interval. i,j This includes ρ. In Figure 3, as an example, with respect to lane 1 of section 1, 1,1 (320) and q 1,1 (330) and f 1,1 (310) and are displayed. Transverse flow f i,j It should be noted that the above definition and the scenario illustrated in Figure 3 assume that the lateral flow is to the left (from the lower numbered lane to the higher numbered lane in Figure 3). In other embodiments, as will be described in detail later, both leftward and rightward lateral flow (from the higher numbered lane to the lower numbered lane in Figure 3) are considered.

[0022] Generally, when executed by one or more processors 205, the RL module 220 includes instructions that cause one or more processors 205 to process aggregated macroscopic traffic condition information 240 using an RL-based model and to determine target lateral flow 235 for two or more lanes 150 of road 130 within a section 140 of road 130. In this embodiment, the problem of determining target (desired) lateral flow is formulated as a Markov decision process (MDP). In some embodiments, the MDP includes (1) a system state in terms of macroscopic flow / density, particularly with respect to both the entire road network and the connected vehicle 160; (2) actions defined in terms of desirable lateral flow between lanes 150 within each section 140; (3) a reward function based on the average speed of the connected vehicle 160 and the average speed of all vehicles (connected vehicles and conventional vehicles); and (4) transition dynamics including vehicle tracking, lane changes, and section-level control. The MDP model will be described in more detail below with reference to Figure 4.

[0023] Figure 4 shows an RL-based MDP model 400 (hereinafter, "RL-based model 400") within a regional manager 110 according to an exemplary embodiment of the present invention. As shown in Figure 4, the RL-based model 400 includes one or more neural network-based agents 410 and associated policies, an MDP environment 420, actions 430, states and rewards 440. In this embodiment, the MDP is mathematically a tuple

number

number

number

number

Number

Number

Number

Number

Number

Number

[0024] Regarding the state S, in the embodiment of FIG. 4, as described above, a macroscopic illustration of the state space is considered. This is the density ρ i,j regarding section i and lane j, and the longitudinal flow q i,jThis includes the observed transverse flow f. i,j This is omitted to avoid problems involving overfitting and / or causal confusion. However, the aggregated traffic state information 240 does not reveal the state of individual connected vehicles 160, making it difficult to determine how the connected vehicles 160 are operating (on average) and which sections are actually under the control of the regional manager 110 at a particular time. This is resolved by including the macroscopic state of the connected vehicles 160 in the observation space, resulting in the final observation space shown in Figure 5. As shown in Figure 500 in Figure 5, the final observation space includes both the total macroscopic state 510 and the macroscopic state (520) of the connected vehicles 160.

[0025] Regarding Action A, learning pure lateral flow can be difficult because the system must learn what flow is reasonable, and assigning large / impossible flows can flatten the gradient during the optimization procedure. To solve this problem, in some embodiments, a specific desirable lateral flow boundary is defined, as shown in Figure 6. As shown in Figure 600 in Figure 6, the minimum value of the lateral flow 610 is

number

number

[0026] Regarding the reward function r, in some embodiments, the objective is to generate positive social impacts through a hierarchical traffic flow adjustment system. Therefore, the reward function balances the social impact with the individual benefits of the connected vehicle 160. In some embodiments, the reward function is r t =v avg,CV +γ·v avg Defined as, where, v avg,CV This is the average speed of connected vehicles 160, and v avg γ is the average speed of all vehicles (connected vehicles and conventional vehicles), and γ (the discount factor mentioned above) captures the relevance of the social impact in a particular implementation. In some embodiments, γ is set to 1, which corresponds to all vehicles being treated as equally important, regardless of whether they are connected or conventional. In different embodiments, by setting γ to zero, the RL module 220 focuses entirely on improving the performance of connected vehicles 160. In yet another embodiment, γ is set to 0.99.

[0027] Regarding the transition probability function P, as described above, it is based on factors such as vehicle following and lane change dynamics and section manager dynamics. Regarding the initial state distribution ρ0, it can take into account the initial position / speed of both the connected vehicle 160 and the conventional vehicle 170, and the position / lane of the incident 195 (e.g., a slow-moving vehicle or a stopped vehicle). The incident 195 may also include lane closures (e.g., due to road construction or an accident) or other conditions that disrupt the normal flow of vehicle traffic. The time range T varies depending on the embodiment. In one embodiment, a total simulation time of 1200 seconds is used in increments of 0.25 steps / second in the simulation conditions (e.g., during training of the RL base model 400 and subsequent testing).

[0028] The training module 228, when executed by one or more processors 205, generally includes instructions that cause one or more processors 205 to train the RL-based model 400. The training algorithm to be selected depends on the particular embodiment, but several options known to those skilled in the art, which are published in the literature, include the DQN algorithm, the TRPO algorithm, and the DDPG / TD3 algorithm. In one embodiment, the training module 228 uses TD3 as the training algorithm, which has standard hyperparameters and two hidden layers, each of size 256.

[0029] The output module 225, when executed by one or more processors 205, generally includes instructions that cause one or more processors 205 to send a target lateral flow 235 to the section manager 120, which then converts the target lateral flow 235 into a lane change action and sends the lane change action to one or more connected vehicles 160 within the applicable section 140. Two different embodiments of the section manager 120 are described in more detail below.

[0030] For simplicity, the above description is sometimes in terms of a single section manager 120 that sends aggregated traffic condition information 240 to a regional manager 110, and in terms of the processing that the RL module 220 performs in support of that single section manager 120. In most embodiments, it will be understood that the regional manager 110 serves (covers) a geographic area that includes at least one additional section 140 of the road 130. In such embodiments, the RL module 220 determines at least one additional target lateral flow 235 for two or more lanes 150 within at least one additional section 140 by processing additional aggregated macroscopic traffic condition information 240 received from at least one additional section manager 120 within at least one additional section 140 using the RL base model 400. In other words, in most embodiments, the regional manager 110 operates within an architecture such as that shown in Figure 1, in which case there are multiple sections 140 and corresponding section managers 120. In such an embodiment, the regional manager 110 receives aggregated traffic condition information 240 from each of the section managers 120 within the corresponding section 140, and the RL module 220 processes the aggregated traffic condition information 240 to provide each of the section managers 120 with a target lateral flow 235.

[0031] Figure 7 is a flowchart of a traffic flow adjustment method 700 via centralized lateral flow control, according to an exemplary embodiment of the present invention. The method 700 will be described with reference to Figures 1, 3 to 6, from the perspective of the regional manager 110 in Figure 2. Although the method 700 will be described in combination with the regional manager 110, it should be understood that the method 700 is not limited to being implemented within the regional manager 110, but rather the regional manager 110 is an example of a system in which the method 700 can be implemented as an alternative.

[0032] In block 710, the input module 215 receives aggregated macroscopic traffic condition information 240 from a section manager 120 that is communicating with one or more connected vehicles 160 within section 140 of road 130 (see Figure 1). As described above, the aggregated traffic condition information 240 includes (1) the density ρ in section i and lane j. i,j (2) Longitudinal flow q for section i and lane j i,j , and (3) net lateral flow f from lane j to lane j+1 within section i i,j It may include information such as,

[0033] In block 720, the RL module 220 uses the RL base model 400 to process aggregated macroscopic traffic condition information 240 to determine target lateral flow 235 for two or more lanes 150 of road 130 within a section 140 of road 130. As described above, the RL base model 400 is based on MDP in some embodiments. Embodiments of the RL base model 400 and the parameters of the MDP are described in detail above.

[0034] In block 730, the output module 225 sends the target lateral flow 235 to the section manager 120, which converts the target lateral flow 235 into a lane change action and sends the lane change action to one or more connected vehicles 160. As described above, two different embodiments of the section manager 120 will be described in more detail below.

[0035] In some embodiments, method 700 may include actions not shown in Figure 7. For example, in some embodiments, the RL module 220 outputs an attractive term, and a target lateral flow 235 for lane 150 of two or more lanes 150 of road 130 is calculated from a minimum lateral flow 610, a maximum lateral flow 620, and the attractive term. As described above, in some embodiments, the target lateral flow 235 is generally leftward. In other embodiments, the target lateral flow 235 can be leftward, rightward, or both (a mixture). Also, as described above, in some embodiments, the RL base model 400 includes a discount factor γ which can function as a social impact parameter determining how advantageous one or more connected vehicles 160 are over conventional vehicles 170 in achieving a given traffic flow objective. As described above, in some embodiments, the predetermined traffic flow objective on which the reward function r is based is the average speed (for example, the average speed of connected vehicles 160, and the average speed of all vehicles, including both connected vehicles 160 and conventional vehicles 170).

[0036] Furthermore, as described above, in most embodiments, the regional manager 110 operates within an architecture such as that shown in Figure 1, in which case there are multiple sections 140 and corresponding section managers 120. In such embodiments, the regional manager 110 receives aggregated traffic state information 240 from each of the section managers 120 within the corresponding section 140, and the RL module 220 processes the aggregated traffic state information 240 to provide a target lateral flow 235 to each of the section managers 120. As described above, in some embodiments, at least some of the connected vehicles 160 are autonomous vehicles.

[0037] 1 Embodiment of a Section Manager Figure 8 is a block diagram of a section manager 120 according to an exemplary embodiment of the present invention. In Figure 8, the section manager 120 includes one or more processors 805, with a memory 810 communicably coupled to the processors 805. More specifically, in this embodiment, the memory 810 stores an input module 815, a lane allocation module 820, and an output module 825. The memory 810 is random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable non-transient memory for storing modules 815, 820, and 825. Modules 815, 820, and 825 are, for example, machine-readable instructions that, when executed by one or more processors 805, cause one or more processors 805 to perform various functions disclosed herein.

[0038] As shown in Figure 8, the section manager 120 can store various types of data in the database 830. For example, the section manager 120 can store target lateral flow 835, the number of connected vehicle targets 840, lane change actions 845, vehicle dynamics data 848, and selected vehicles 850. These various types of data will be described further below.

[0039] As shown in Figure 8, the section manager 120 can communicate with other network nodes 855 (e.g., the regional manager 110 and the connected vehicle 160) via the network 860. When communicating with other network nodes 855, the section manager 120 can utilize communication technologies such as cellular data (LTE, 5G, 6G, etc.), DSRC, Bluetooth® LE, etc.

[0040] The input module 815, when executed by one or more processors 805, generally includes instructions to cause one or more processors 805 to receive from the regional manager 110 a target lateral flow 835 relating to two or more lanes 150 of the road 130 within a section 140 of the road 130 that includes one or more connected vehicles 160. Embodiments of the regional manager 110 and the manner in which the regional manager 110 generates the lateral flow 835 (a subset of the target lateral flow 235 in Figure 2 relating to a single section 140) have been described in detail above.

[0041] In the embodiment shown in Figure 8, the section manager 120 includes a hierarchical model that performs vehicle-level control based on the target lateral flow 835 received from the regional manager 110. This hierarchical model controls the lateral flow to Δt in an manner that minimizes faults and maximizes safety. section This includes converting to lane change actions 845 every second and selecting a specific connected vehicle 160 for the lane change. In some embodiments, possible lane change actions that a given connected vehicle 160 may receive from the section manager 120 are to move left, to move right, or "no operation," which is short for "no operation" and means that the vehicle simply remains in its current lane (no lane change). As will be described in more detail below, in the embodiment of Figure 8, the section manager 120 employs a ranking-based algorithm to convert aggregated target lateral flow 835 obtained from the area manager 110 into section-level connected vehicle lane change actions 845.

[0042] The ranking algorithm can be summarized as follows: (1) converting the target lateral flow 835 into the number of target connected vehicles 160; (2) ranking the connected vehicles 160 based on their distance from following vehicles in the target lane (the new lane, or the lane to which they are moving); (3) selecting the connected vehicles 160 that should change lanes based on the ranking, and sending lane change actions 845 to the connected vehicles 160 in section 140; (4) updating the number of missed vehicles; and (5) repeating steps 1 to 5. These steps are described in detail below.

[0043] Regarding step 1, the target lateral flow 835 is given by equation n veh =f lat ·Δt section +n missing Therefore, the number of target vehicles n veh It can be converted to, and here, f lat This is the target lateral flow 835, and Δt section n is the interval control time interval, missing n is the number of missed connected vehicles 160 (i.e., connected vehicles 160 that, for some reason, were unable to change lanes in response to a lane change action 845 received from the section manager 120 during a previous iteration of the algorithm). In some embodiments, n missing It is initialized to zero at the first iteration. Note that in the embodiment of Figure 8, the connected vehicle 160 does not move in the opposite direction to the specified direction of the target lateral flow 835. In one embodiment, Δt section Although this parameter is 5 seconds, it can be changed depending on the embodiment.

[0044] Steps 2 and 3 will be described in relation to Figures 9A to 10B. Figures 9A and 9B relate to an embodiment in which all target lateral flows 835 are directed to the left (unidirectional target lateral flows 835). Figures 10A and 10B relate to another embodiment in which the target lateral flows 835 are directed to the left and to the right, and it is necessary to resolve the competition between these bidirectional target lateral flows 835.

[0045] Figure 9A is an illustration showing a connected vehicle 160 and their respective measured distances to a following vehicle in a target lane (new lane) 150, according to an exemplary embodiment of the present invention. The connected vehicle 160 detects other vehicles on the road 130 (e.g., nearby and surrounding vehicles, in the same lane 150 or in different lanes 150) by using its on-board sensors (e.g., cameras, LiDAR, radar, sonar). Based on the sensor data, the connected vehicle 160 can measure the distance to other vehicles, whether they are connected vehicles 160 or conventional vehicles 170. These distance measurements can be sent to the section manager 120 (specifically to the input module 815) as part of vehicle motion data 848 that the connected vehicle 160 reports to the section manager 120. Of particular interest to the section manager 120 are the distances 905a to 905d shown in Figure 9A. These distances are the respective distances between a given connected vehicle (shown as "1", "2", "3", and "4" in Figure 9A for distinction) and a following vehicle in the target lane (new lane) 150. The target lane or new lane 150 is the lane on the road 130 that the connected vehicle 160 may be potentially directed to move in a lane change. Note that in the embodiment shown in Figure 9A, as described above, all target lateral flows 835 are directed to the left.

[0046] Figure 9B is a diagram illustrating a ranking process based on the measured distances 905a to 905d from connected vehicle 160 to following vehicles in the target lane 150, according to an exemplary embodiment of the present invention. As shown in the example in Figure 9B, the vehicle ranking 910 lists connected vehicles 1 to 4 in descending order of their corresponding measured distances 905a to 905d. Connected vehicle 160, with the largest following gap, is desirable from a safety standpoint because instructing those vehicles to change lanes is less likely to disrupt traffic or lead to a collision. In the example in Figure 9B, the number of vehicles to be targeted for lane changes, n, is calculated in step 1. veh Since the ratio is 2, the top two connected vehicles 160 (connected vehicles 3 and 1, respectively) are selected as targets for lane changes (leftward). These two connected vehicles 160 may be referred to as "selected vehicles" 850. In some embodiments, the distance from the connected vehicle 160 to the vehicle ahead in the target lane (preceding vehicle) is also included in the analysis.

[0047] Figure 10A illustrates a connected vehicle 160 and the respective measurement distances 905a to 905h relative to a following vehicle in the target lane, according to another exemplary embodiment of the present invention. In the bidirectional lane change embodiment of Figure 10A, the number of vehicles for a leftward lane change is n veh,l , and the number of vehicles n making a rightward lane change. veh,r For illustrative purposes, we assume that both are 2. In the embodiment of Figure 10A, the following vehicle distances 905a to 905d relate to the leftward lateral flow 835a to 835b (similar to the embodiment of Figure 9A), and the following vehicle distances 905e to 905h relate to the rightward lateral flow 835c to 835d.

[0048] Figure 10B is a diagram illustrating the ranking process based on the measured distances 905a to 905h for following vehicles in the target lane 150 from a connected vehicle 160 in the case of a bidirectional target lateral flow 835 according to an exemplary embodiment of the present invention. As shown at the top of Figure 10B, vehicle rankings 910 and 1005 list vehicles in descending order of their corresponding measured distances 905a to 905d (leftward lateral flow) and 905e to 905h (rightward lateral flow), respectively. Herein lies the problem of how to resolve the conflict between leftward and rightward rankings. In this embodiment, the conflict is resolved by using a multi-objective selection algorithm. For example, in this embodiment, a gray relationship optimization algorithm is used, which minimizes the distance from the optimal value for both leftward and rightward directions and equalizes the weights for both directions. The central portion of Figure 10B shows one possible solution (the row highlighted in bold in lists 910 and 1005) under the gray relation optimization algorithm. In this example, connected vehicles 1 and 4 are selected to change lanes to the left, and connected vehicles 2 and 3 are selected to change lanes to the right.

[0049] Continuing with the examples in Figures 10A and 10B, the first step of the gray relation optimization algorithm is as shown in the lower part of Figure 10B (component 1010), n veh This involves pairing lane change options to match the target, thereby,

number

number

number

number

number

number

number

[0050] After completing Step 3 (selecting the connected vehicle 160 to change lanes), the lane change action 845 is determined, and the section manager 120 can send the lane change action 845 (left, right, or no operation) to the connected vehicles 160 within section 140. Naturally, only the lane change action 845 instructed to the selected connected vehicle 160 (850) will perform the lane change. The remaining connected vehicles 160 within section 140 will receive a "no operation" response.

[0051] Regarding step 4, this step is for all n veh Consider a situation where the vehicle cannot be moved (i.e., is not properly instructed to change lanes). This can happen for various reasons, but two main reasons are (1) n veh However, this is greater than the number of connected vehicles 160 in the section, and (2) not all connected vehicles 160 can safely move to the adjacent lane 150 if desired. In the embodiment of Figure 8, n missing =n missing +f lat ·Δt section -n actual And here, n actualThis represents the number of connected vehicles (160) that successfully changed lanes. The above calculation has the effect of (1) remembering the number of vehicles that changed lanes less than the target and adjusting it in subsequent iterations of the hierarchical ranking algorithm, and (2) adjusting the number to reflect the number of vehicles that changed lanes more than the target (because the lane change was missed earlier).

[0052] Step 5 is as described above, at a predetermined time interval (for example, Δt section The process involves periodically (every second) repeating the ranking algorithm and sending out lane change action 845.

[0053] Considering the above step-by-step description of the ranking algorithm and the gray relationship optimization algorithm, the functions performed by the lane assignment module 820 can be summarized here. Referring again to Figure 8, the lane assignment module 820 generally includes instructions that, when executed by one or more processors 805, cause one or more processors 805 to convert the target lateral flow 835 to the target number of connected vehicles N (840), and, if the direction of the lane change is uniform among the groups of N connected vehicles 160, to select, with respect to the lane change, a group of N connected vehicles 160 whose ranked distance from a following vehicle in the target lane 150 is the largest among the one or more connected vehicles 160 in the section of road 140. For example, in the embodiments described above in relation to Figures 9A and 9B, the direction of the lane change is uniform (the same) with respect to all N selected connected vehicles 160 (selected vehicles 850). In that example, all lane changes are to the left. In a different embodiment, all lane changes for N selected connected vehicles 160 can alternatively be to the right, in which case the direction of the lane changes is also uniform (the same) for all N selected connected vehicles 160.

[0054] Referring again to step 5 of the ranking algorithm described above, in some embodiments, when the lane assignment module 820 is executed by one or more processors 805, one or more processors 805 are instructed to perform a predetermined interval (e.g., Δt section The system further includes instructions to convert a target lateral flow 835 into a specific lane change action 845 relating to a connected vehicle 160, and to send the lane change action 845 to the connected vehicle 160, by periodically repeating the ranking algorithm described above (every second).

[0055] As described above in relation to Figures 10A and 10B, in other embodiments, the direction of lane changes can differ among a set of N selected connected vehicles 160(850). In the example of Figures 10A and 10B, some of the selected connected vehicles 160 are chosen to change lanes to the left, while others are chosen to change lanes to the right within the same section 140. In such embodiments, the lane assignment module 820 selects a set of N connected vehicles 160 to change lanes according to a multi-objective selection algorithm. As described above, in some embodiments, such a multi-objective selection algorithm used is a gray-relational optimization algorithm.

[0056] Referring again to Figure 8, the output module 825, when executed by one or more processors 805, generally includes an instruction to one or more processors 805 to send a lane change action 845 to a group of N connected vehicles 160 via the network 860. The manner in which the lane change action 845 is determined is described in detail above. The lane change action 845 sent to the group of N connected vehicles 160 involves an actual lane change. As described above, the section manager 120 also sends the lane change action 845 to the remaining connected vehicles 120 in section 140, but those remaining connected vehicles 160 receive no operation (i.e., remain in their current lane).

[0057] Figure 11 is a flowchart of a cooperative lane allocation method 1100 according to an exemplary embodiment of the present invention. Method 1100 will be described with reference to Figures 9A to 10B, from the perspective of the section manager 120 in Figure 8. Although Method 1100 will be described in combination with the section manager 120, it should be understood that Method 1100 is not limited to being implemented within the section manager 120, and that the section manager 120 is an example of a system in which Method 1100 can be implemented as an alternative.

[0058] In block 1110, the input module 815 receives from the regional manager 110 a target lateral flow 835 relating to two or more lanes 150 of the road 130 within a section 140 of the road 130 that includes one or more connected vehicles 160. Embodiments of the regional manager 110 and the manner in which the regional manager 110 generates the lateral flow 835 have been described in detail above.

[0059] In blocks 1120 and 1130, the lane assignment module 820 converts the target lateral flow 835 into a target number of connected vehicles 160 within the applicable section 140 by implementing a ranking algorithm (described in detail above), and ultimately selects a specific connected vehicle 160 for the lane change, along with the corresponding assigned lane change action 845. More specifically, the lane assignment module 820 converts the target lateral flow 835 into a target number of connected vehicles N(840), and, if the direction of the lane change is uniform among the set of N connected vehicles, selects a set of N connected vehicles 160 within the road section 140 that has the largest ranked distance 190 from the following vehicle in the target lane 150 for the lane change. Embodiments of this type are described above in relation to Figures 9A and 9B.

[0060] As described above in relation to Figures 10A and 10B, in other embodiments, the direction of lane changes can differ among a set of N selected connected vehicles 160. In the example of Figures 10A and 10B, some of the selected connected vehicles 160 are chosen to change lanes to the left, while others are chosen to change lanes to the right within the same section 140. In such embodiments, the lane assignment module 820 selects a set of N connected vehicles 160 with respect to lane changes according to a multi-objective selection algorithm. Also, as described above, in some embodiments, the multi-objective selection algorithm used is a gray-relationship optimization algorithm.

[0061] In block 1140, the output module 825 sends a lane change action 845 via the network 860 to a group of N connected vehicles 160. The manner in which the lane change action 845 is determined is described in detail above. As described above, the section manager 120 may also send the lane change action 845 to the remaining connected vehicles 160 in section 140, but these remaining connected vehicles 160 receive no operation (i.e., remain in their current lane).

[0062] In some embodiments, method 1100 may include operations not shown in Figure 11. For example, in some embodiments, the lane assignment module 820 considers missed vehicles that were unable to change lanes in response to a previously sent lane change action 845, as described above. Also, as described above, the distance measurement 905 can be sent to the section manager 120 (specifically to the input module 815) as part of the vehicle movement data 848 that the connected vehicle 160 reports to the section manager 120. Also, as described above, in some embodiments, the lane assignment module 820 may send data at predetermined intervals (e.g., Δt section The ranking algorithm described above is repeated periodically (every second) to convert the target lateral flow 835 into a specific lane change action 845 for the connected vehicle 160, and to send the lane change action 845 to the selected vehicle 850 (N selected connected vehicles 160).

[0063] Section Manager Embodiment 2 Figure 12 is a block diagram of a section manager 1200 according to another exemplary embodiment of the present invention. In Figure 12, the section manager 1200 includes one or more processors 1205, with a memory 1210 communicably coupled to the processors 1205. More specifically, in this embodiment, the memory 1210 stores an input module 1215, a lane allocation module 1220, an output module 1225, and a training module 1230. The memory 1210 is random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable non-transient memory for storing modules 1215, 1220, 1225, and 1230. Modules 1215, 1220, 1225, and 1230 are, for example, machine-readable instructions that, when executed by one or more processors 1205, cause one or more processors 1205 to perform various functions disclosed herein.

[0064] As shown in Figure 12, the section manager 1200 can store various types of data in the database 1235. For example, the section manager 1200 can store target lateral flow 835, traffic condition information 1240, lane change actions 1245, and model data 1250. The model data 1250 may include, for example, hyperparameters, parameters, neural network weights, intermediate calculation results, etc., related to the RL-based MDP model that the section manager 1200 uses to convert target lateral flow 835 into lane change actions 1245 for connected vehicles 160 in section 140 of road 130. The RL-based model 1300 will be described in more detail below in relation to Figures 13A to 14C.

[0065] As shown in Figure 12, the section manager 1200 can communicate with other network nodes 1255 (e.g., the regional manager 110 and the connected vehicle 160) via the network 1260. When communicating with other network nodes 1255, the section manager 1200 can utilize communication technologies such as cellular data (LTE, 5G, 6G, etc.), DSRC, Bluetooth® LE, etc.

[0066] The input module 1215 generally includes instructions, when executed by one or more processors 1205, to cause one or more processors 1205 to receive target lateral flow 835 for two or more lanes 150 of road 130 within section 140 of road 130 from the regional manager 110, and to receive traffic condition information 1240 from multiple connected vehicles 160 within section 140 of road 130. Embodiments of the regional manager 110 and the manner in which the regional manager 110 generates the lateral flow 835 (a subset of the target lateral flow 235 in Figure 2 with respect to a single section 140) are described in detail above.

[0067] In the embodiment shown in Figure 12, the section manager 1200 includes an RL framework that performs vehicle-level control based on the target lateral flow 835 received from the area manager 110. This is in an manner that minimizes traffic disruptions and maximizes safety, Δt section This includes converting lateral flow into lane change actions 1245 for the connected vehicle 160 every second. In some embodiments, possible lane change actions 1245 that a given connected vehicle 160 may receive from the section manager 1200 are moving left, moving right, or no operation (remaining in the current lane). The RL framework is described in more detail below.

[0068] The lane assignment module 1220, when executed by one or more processors 1205, generally includes instructions that cause one or more processors 1205 to process traffic condition information 1240 and target lateral flow 835 using an RL-based model to determine lane change actions 1245 for multiple connected vehicles 160. In the embodiment of Figure 12, the RL-based model is based on a single neural network that shares parameters for multiple connected vehicles 160 within the application section 140. In the embodiment of Figure 12, the problem of translating the target lateral flow 835 received from the regional manager 110 into lane change actions 1245 is formulated as an MDP. In some embodiments, the MDP includes (1) states and microscopic traffic states defined in terms of lateral flow, including the position and speed of each connected vehicle 160, as well as the positions and speeds of preceding and following vehicles in the current lane and adjacent lanes 150; (2) actions for each connected vehicle 160, including moving to the right, moving to the left, and remaining in the current lane (no operation); (3) a reward function that balances efficiency and safety while satisfying an assigned target lateral flow 835; and (4) transition dynamics, including vehicle tracking, lane changes, and section-level control. The RL-based model will be described in more detail below in relation to Figures 13A to 14C.

[0069] Figure 13A is a diagram of an RL-based MDP model 1300 (hereinafter, "RL-based model 1300") within a section manager 1200 according to an exemplary embodiment of the present invention. As shown in Figure 13A, the RL-based model 1300 includes an agent 1310 including a neural network and associated policies, an MDP environment 1320, actions 1330, and states and rewards 1340. In this embodiment, the MDP is mathematically a tuple

number

number

number

number

number

number

number

number

number

number

[0070] Regarding state S, as described above, the RL base model 1300 uses lateral flow and microscopic traffic conditions (traffic condition information 1240), including position and speed data for each connected vehicle 160 itself, as well as position and speed data for preceding and following vehicles in the current lane and adjacent lanes 150. This is illustrated as system state 1350 in Figure 13B. In Figure 13B (and other drawings here), "CV" represents "connected vehicle". In some embodiments, the traffic condition information 1240 may include information about the spatial relationship (e.g., distance measured using on-board vehicle sensors) between the connected vehicle 160 in the applicable section 140 and other vehicles (connected vehicles or conventional vehicles) on the road 130.

[0071] Regarding Action A, in the embodiment shown in Figure 12, for each connected vehicle 160 within the applicable section 140, there are three possible lane change actions 1245 as described above: (1) moving to the left, (2) moving to the right, and (3) remaining in the current lane 150 (no operation). This is illustrated as Action 1330 in Figure 13C.

[0072] Regarding the reward function r, in the embodiment shown in Figure 12, the reward function is:

number

[0073] The transition probability function P is based on vehicle follow / lane change dynamics and section manager dynamics. The initial state distribution ρ0 includes the initial position and speed of both the connected vehicle 160 and the conventional vehicle 170, as well as information regarding the position and lane of the incident 195 (e.g., a slow-moving or stopped vehicle, or other condition that disrupts the flow of traffic on road 130).

[0074] One of the key features of the section manager 1200 is that the RL base model 1300 includes a single neural network for controlling multiple connected vehicles 160 via parameter sharing. This is illustrated in Figures 14A to 14C.

[0075] Figure 14A shows a neural network 1400 employed in the MDP model within the section manager 1200 according to an exemplary embodiment of the present invention. As shown in Figure 14A, the input to the neural network 1400 (combined observations 1405) includes information relating to all connected vehicles 160 that the model addresses. The output (combined actions 1410) includes a group of three nodes for each connected vehicle 160. Each of these three nodes corresponds to one of the actions 1330 described above, namely, (1) moving left, (2) moving right, and (3) remaining in the same lane (no operation). For each modeled connected vehicle 160, the lane assignment module 1220 employs a softmax algorithm 1415 to probabilistically select from the three output nodes. In this embodiment, the number of hidden layers and the number of nodes are adjusted based on performance.

[0076] The significant advantages of the architecture described above are 3 15 Instead of learning individual different Q-functions, one function can be designed for each possible action 1330. In this case, the same model can use all sampled information, thereby improving the efficiency of the learning procedure. One difficulty that arises in practical implementation is that the number of connected vehicles 160 within the application interval 140 varies. In the embodiment of Figure 12, the lane assignment module 1220 can determine the maximum possible number N of connected vehicles 160 within the interval 140 at any given time. CV This assumes that the number of connected vehicles is N CV A situation greater than or equal to N CV Techniques for handling situations smaller than this will be discussed later in relation to Figures 14B and 14C.

[0077] Figure 14B shows a situation according to an exemplary embodiment of the present invention where the number of connected vehicles 160 in section 140 exceeds the number of observation nodes 1405 in the neural network 1400. As shown in Figure 14B, NCV Any additional connected vehicles 160 (connected vehicles 160a) beyond this limit are ignored in this embodiment.

[0078] Figure 14C shows a situation according to an exemplary embodiment of the present invention where the number of observation nodes 1405 in the neural network 1400 exceeds the number of connected vehicles 160 in interval 140. As shown in Figure 14C, unused observation nodes 1405 are assigned zeros, and the corresponding output (action) nodes 1410 are ignored.

[0079] In some embodiments, when the lane assignment module 1220 is executed by one or more processors 1205, it instructs one or more processors 1205 at predetermined intervals (for example, Δt section The system further includes a command that periodically repeats the above RL-based processing (every second) to convert the target lateral flow 835 into a lane change action 1245 relating to the connected vehicle 160, and to send the lane change action 1245 to the connected vehicle 160.

[0080] Referring again to Figure 12, the training module 1230 generally includes instructions that, when executed by one or more processors 1205, cause one or more processors 1205 to train the RL-based model 1300, including the neural network 1400, as described above. In some embodiments, the training module 1230 employs an output activation function, i.e., the Gumbel softmax algorithm, so that the softmax algorithm used for training is differentiable. In some embodiments, the training module 1230 uses a cross-entropy loss function and gradient masking is applied to handle variations in the number of vehicles during training. Furthermore, in some embodiments, the training module 1230 warms up the training process by using imitation learning. For example, the neural network 1400 can be trained using expert demonstrations. In such embodiments, an existing lane change model, such as MOBIL, can be used as the expert, thereby generating simulated data for training purposes. To improve the robustness of weights that are not frequently trained, in some embodiments, the training module 1230 employs vehicle placement randomization. Without this technique, due to the changing number of vehicles, some nodes (e.g., bottom node 1405 of the input layer) would often lack input data. As a result, some parts of the neural network 1400 would not receive sufficient gradient flow. Randomizing the vehicle placement during training overcomes this difficulty.

[0081] Referring again to Figure 12, the output module 1225, when executed by one or more processors 1205, generally includes instructions that cause one or more processors 1205 to send lane change actions 1245 to multiple connected vehicles 160 via the network 1260. The lane change actions 1245 are generated and output by the RL base model 1300 described above.

[0082] Figure 15 is a flowchart of a cooperative lane allocation method 1500 according to another exemplary embodiment of the present invention. Method 1500 will be described with reference to Figures 13A to 14C, from the perspective of the section manager 1200 in Figure 12. Although Method 1500 will be described in combination with the section manager 1200, it will be understood that Method 1500 is not limited to being implemented within the section manager 1200, and that the section manager 1200 is an example of a system in which Method 1500 can be implemented as an alternative.

[0083] In block 1510, the input module 1215 receives from the regional manager 110 a target lateral flow 835 relating to two or more lanes 150 of road 130 within a section 140 of road 130. The embodiment of the regional manager 110 and the manner in which the regional manager 110 generates the lateral flow 835 (a subset of the target lateral flow 235 in Figure 2 relating to a single section 140) are described in detail above.

[0084] In block 1520, the input module 1215 receives traffic condition information 1240 from multiple connected vehicles 160 within a section 140 of the road 130. As described above, the traffic condition information 1240 may include position and speed data for each connected vehicle 160 itself, as well as position and speed data for preceding and following vehicles within the current lane and adjacent lanes 150. In some embodiments, the traffic condition information 1240 may include information regarding the spatial relationship (e.g., distance measured using on-board vehicle sensors) between the connected vehicles 160 within the applicable section 140 and other vehicles (connected vehicles or conventional vehicles) on the road 130.

[0085] In block 1530, the lane assignment module 1220 determines lane change actions 1245 for multiple connected vehicles 160 by processing traffic condition information 1240 and target lateral flow 835 using the RL base model 1300. As described above, the RL base model 1300 is based on a single neural network 1400 that shares parameters with respect to multiple connected vehicles 160. The RL base model 1300 and the neural network 1400 that implements it are described in detail above. As described above, in some embodiments, the RL base model 1300 is based on MDP.

[0086] In block 1540, the output module 1225 sends a lane change action 1245 to multiple connected vehicles 160 via the network 1260. The manner in which the lane change action 1245 is determined is described in detail above.

[0087] In some embodiments, method 1500 may include operations not shown in Figure 15. For example, in some embodiments, the lane assignment module 1220 selects from leftward, rightward, and no-operation lane change actions 1245 for a given connected vehicle 160 by employing a softmax algorithm. As described above, in some embodiments, the training module 1230 improves the training of the neural network 1400 by using one or more of the Gumbel softmax algorithm, cross-entropy loss function, gradient masking, imitation learning, and vehicle placement randomization. Also, as described above, the lane assignment module 1220 selects from leftward, rightward, and no-operation lane change actions at predetermined intervals (e.g., Δt section The above RL-based processing can be repeated periodically (every second), thereby converting the target lateral flow 835 into a specific lane change action 1245 relating to the connected vehicle 160, and sending the lane change action 1245 to the connected vehicle 160.

[0088] conclusion This specification discloses detailed embodiments. However, it should be understood that the disclosed embodiments are intended only as examples. Accordingly, the specific structural and functional details disclosed herein should not be construed as limitations, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to adapt the embodiments herein to substantially any suitable detailed structure. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an easily understandable description of possible implementations. Although various embodiments are illustrated in Figures 1 to 15, the embodiments are not limited to the exemplified structures or uses.

[0089] The components described above can be implemented in hardware or as a combination of hardware and software, and can be implemented centrally in a single processing system or in a distributed manner with different components distributed across multiple interconnected processing systems. A typical combination of hardware and software may be a processing system having computer-readable program code that, when loaded and executed, controls the processing system to implement the methods described herein. The systems, components, and / or processes may also be embedded in computer-readable storage, such as a machine-readable computer program product or other data program storage device, which is a tangible embodiment of a program consisting of multiple machine-executable instructions for performing the methods and processes described herein. These components may also be embedded in application products that have all the functions necessary to implement the methods described herein and, when loaded into a processing system, are capable of performing these methods.

[0090] Furthermore, the configurations described herein may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied therein, for example, stored therein. Any combination of one or more computer-readable media may be used. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. The term "computer-readable storage medium" means a non-transient storage medium. The computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (a non-exclusive list) of computer-readable storage media include portable computer diskettes, hard disk drives (HDDs), solid-state drives (SSDs), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), digital general-purpose discs (DVDs), optical storage devices, magnetic storage devices, or any appropriate combination of the above. In the context of this document, computer-readable storage media may also be any tangible media that may contain or store programs for use by, or in connection with, instruction execution systems, apparatus, or devices.

[0091] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, cable, RF, etc., or any suitable combination thereof. Computer program code for performing the operations relating to this configuration may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk, C++, or similar, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The program code may be executed entirely on the user's computer, or partially as an autonomous software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider).

[0092] In general, as used herein, “module” includes routines, programs, objects, components, data structures, or similar items that perform a particular task or implement a particular data type. In further embodiments, memory generally stores the indicated module. Memory associated with a module may be a buffer or cache embedded within a processor, RAM, ROM, flash memory, or other suitable electronic storage medium. In further embodiments, the modules envisioned in this disclosure are implemented as application-specific integrated circuits (ASICs), as hardware components of a system-on-a-chip (SoC), as programmable logic arrays (PLAs), or as other suitable hardware components embedded with a defined set of configurations (e.g., instructions) for performing the disclosed functions.

[0093] As used herein, the terms “one (a)” and “one (an)” are defined as one or more. As used herein, the term “plural” is defined as two or more. As used herein, the term “another” is defined as at least the second or subsequent. As used herein, the terms “contain” and / or “have” are defined as containing (i.e., open language). As used herein, the phrase “at least one of ~ and ~” refers to and encompasses any combination and all possible combinations of the one or more items listed in relation. For example, the phrase “at least one of A, B, and C” includes A only, B only, C only, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0094] As used herein, “cause” or “make happen” means to bring about, command, direct, and / or enable an event or action, thereby causing or making possible such event or action, either directly or indirectly.

[0095] Aspects of this specification can be embodied in other forms without departing from its spirit or essential attributes. Therefore, the following claims, rather than the above specification, should be used to illustrate the scope of this specification.

Claims

1. A system for cooperative lane allocation, Processor and Includes memory that stores machine-readable instructions, When the machine-readable instruction is executed by the processor, the processor will be able to: The regional manager will receive target lateral flow data for two or more lanes of the road within a section of the road. Receiving traffic condition information from multiple connected vehicles within the aforementioned section of the road, By using a reinforcement learning-based model based on a single neural network that shares parameters with respect to the aforementioned multiple connected vehicles, the traffic condition information and the target lateral flow are processed to determine the lane change action for the aforementioned multiple connected vehicles. A system that causes the lane change action to be sent to the multiple connected vehicles and to perform the action.

2. The system according to claim 1, wherein the reinforcement learning-based model is based on a Markov decision process.

3. The system according to claim 1, wherein, with respect to one of the plurality of connected vehicles, the machine-readable instruction for determining a lane change action includes, when executed by the processor, an instruction that causes the processor to use a softmax algorithm to select from a leftward lane change action, a rightward lane change action, and a no-operation lane change action.

4. The system according to claim 1, wherein the traffic condition information includes one or more of the following: the position of each connected vehicle in the plurality of connected vehicles; the speed of each connected vehicle in the plurality of connected vehicles; and information relating to the spatial relationship between each connected vehicle in the plurality of connected vehicles and other vehicles on the road.

5. The system according to claim 1, wherein the machine-readable instruction, when executed by the processor, includes a further instruction causing the processor to repeatedly receive the target lateral flow, receive the traffic state information, process the traffic state information and the target lateral flow, and periodically send out the lane change action at predetermined time intervals.

6. The system according to claim 1, wherein the machine-readable instructions, when executed by the processor, include further instructions causing the processor to train the reinforcement learning-based model using one or more of the Gumbel softmax algorithm, a cross-entropy loss function, gradient masking, imitation learning, and vehicle placement randomization.

7. The system according to claim 1, wherein at least one of the plurality of connected vehicles is an autonomous vehicle.

8. A non-transient computer-readable medium for cooperative lane allocation, It stores instructions, and when an instruction is executed by the processor, the processor, The regional manager will receive target lateral flow data for two or more lanes of the road within a section of the road. Receiving traffic condition information from multiple connected vehicles within the aforementioned section of the road, By using a reinforcement learning-based model based on a single neural network that shares parameters with respect to the aforementioned multiple connected vehicles, the traffic condition information and the target lateral flow are processed to determine the lane change action for the aforementioned multiple connected vehicles. A non-transient computer-readable medium that causes the lane change action to be sent to the multiple connected vehicles and to perform the action.

9. The non-transient computer-readable medium according to claim 8, wherein the reinforcement learning-based model is based on a Markov decision process.

10. The non-transient computer-readable medium according to claim 8, wherein, with respect to one of the plurality of connected vehicles, the instruction for determining a lane change action includes, when executed by the processor, an instruction causing the processor to use a softmax algorithm to select from a leftward lane change action, a rightward lane change action, and a no-operation lane change action.

11. The non-transient computer-readable medium according to claim 8, wherein the traffic condition information includes one or more of the following: the position of each connected vehicle in the plurality of connected vehicles; the speed of each connected vehicle in the plurality of connected vehicles; and information relating to the spatial relationship between each connected vehicle in the plurality of connected vehicles and other vehicles on the road.

12. The non-transient computer-readable medium according to claim 8, wherein the instruction, when executed by the processor, includes a further instruction causing the processor to repeatedly receive the target lateral flow, receive the traffic state information, process the traffic state information and the target lateral flow, and periodically send out the lane change action at predetermined time intervals.

13. The non-transient computer-readable medium according to claim 8, wherein the instruction, when executed by the processor, includes further instructions causing the processor to train the reinforcement learning-based model using one or more of the Gumbel softmax algorithm, a cross-entropy loss function, gradient masking, imitation learning, and vehicle placement randomization.

14. It is a method, A section manager communicating with multiple connected vehicles within a section of road receives target lateral flow data for two or more lanes within that section of road from a regional manager. At the aforementioned section manager, traffic condition information is received from the aforementioned multiple connected vehicles, At the section manager, the lane change action for the multiple connected vehicles is determined by processing the traffic condition information and the target lateral flow using a reinforcement learning-based model based on a single neural network that shares parameters for the multiple connected vehicles. A method comprising sending the lane change action from the section manager to the plurality of connected vehicles.

15. The method according to claim 14, wherein the reinforcement learning-based model is based on a Markov decision process.

16. The method according to claim 14, wherein determining a lane change action for one of the plurality of connected vehicles includes selecting from a leftward lane change action, a rightward lane change action, and a no-operation lane change action by using a softmax algorithm.

17. The method according to claim 14, wherein the traffic condition information includes one or more of the following: the position of each connected vehicle in the plurality of connected vehicles; the speed of each connected vehicle in the plurality of connected vehicles; and information relating to the spatial relationship between each connected vehicle in the plurality of connected vehicles and other vehicles on the road.

18. The method according to claim 14, further comprising repeatedly receiving the target lateral flow, receiving the traffic condition information, processing the traffic condition information and the target lateral flow, and sending out the lane change action at predetermined time intervals.

19. The method according to claim 14, further comprising training the reinforcement learning-based model using one or more of the Gumbel softmax algorithm, a cross-entropy loss function, gradient masking, imitation learning, and vehicle placement randomization.

20. The method according to claim 14, wherein at least one of the plurality of connected vehicles is an autonomous vehicle.

Images