Fine path scheduling method for automatic driving vehicle at intersection without traffic light based on double learning network

Abstract

The application discloses a kind of based on double learning network's automatic driving vehicle fine path scheduling method of intersection without signal light.It takes automatic driving vehicle as research object, carries out grid discretization processing to intersection without signal, obtains path learning dataset according to path planning algorithm.Based on the dataset, the coarse-grained quick path scheduling decision model is constructed by CNN network, and the real-time path of automatic driving vehicle at intersection without signal is quickly scheduled.For the fine path scheduling of automatic driving vehicle, on the basis of coarse-grained quick path scheduling decision model, DQN reinforcement learning network is added to calculate the evaluation value of model decision behavior, the reward function is constructed, and the adaptive learning rate is used to improve the accuracy of model prediction, balance the real-time and accuracy of path planning and scheduling of automatic driving vehicle at intersection without signal.The application can improve the traffic efficiency of automatic driving vehicle when passing through intersection without signal light.

Description

Technical Field

[0001] This invention relates to a path planning method for autonomous vehicles at intersections without traffic lights, specifically a refined path scheduling method for autonomous vehicles at intersections without traffic lights based on a dual learning network of CNN and DQN. Background Technology

[0002] Path planning for autonomous vehicles at intersections without traffic lights has become a hot topic in both the autonomous driving industry and academia. However, most path planning algorithms in the current autonomous driving field are used under stable network topology conditions. These algorithms cannot guarantee robustness in dynamic environments (such as intersections without traffic lights), which has become a bottleneck in the development of dynamic path planning for autonomous vehicles.

[0003] Intersection path planning schemes can be divided into two categories: centralized and distributed. Centralized vehicle path planning aims for global optimization. Typically, Roadside Units (RSUs) collect global information and perform path planning, then send the planned path to each autonomous vehicle. While simple, centralized vehicle path planning requires significant computing power from the Roadside Units, resulting in excessively concentrated network load and poor system reliability. In contrast, distributed vehicle path planning employs a decentralized system architecture, where each autonomous vehicle plans its path using only its own locally perceived information. Although distributed vehicle path planning faces challenges in global optimization, it does not rely on a central node (such as a Roadside Unit), offering advantages such as lower computational load and relatively balanced load, demonstrating promising application prospects.

[0004] The current mainstream distributed vehicle routing schemes can be mainly divided into the following three categories:

[0005] (1) Grid weighting method: The basic principle is to describe the intersection as a grid graph, with each grid having a different weight. Each vehicle selects the target grid to move to next based on the grid weight, thus obtaining the optimal path planning strategy, such as the improved Dijkstra algorithm, the Rapid Expanding Random Tree (RRT) algorithm, and the A* algorithm. Liu Zihao et al. combined jump point search theory and replaced the points in the Openlist and Closelist of the traditional A* algorithm with selected key points, improving the optimization speed of the path planning algorithm; Jiang Chenkai et al. proposed an improved Dijkstra algorithm based on time windows to realize dynamic path planning for multiple AGVs. This algorithm can avoid conflicts and deadlocks under the optimal path, making the system more robust. Wu Wei et al. proposed a vehicle path planning optimization model and time optimization allocation algorithm under gridded intersection, with the minimum total delay of the intersection as the control objective, and used the branch and bound method to determine the best driving path, speed, and entry time of the vehicle at the intersection. Although the models of these algorithms are simple, they are all static algorithms and are not suitable for intersection scenarios with a large number of moving vehicles.

[0006] (2) Bionic path planning method, which is to optimize the path planning strategy at intersections by simulating the behavior of organisms in nature. It has the characteristics of self-learning and self-determination, such as optimized particle swarm algorithm, improved genetic algorithm and artificial fish swarm algorithm. Zhuofei Li et al. made a decision on the vehicle passage sequence based on the genetic algorithm and calculated the optimal vehicle trajectory at the same time, which reduced the average travel time at the intersection. Qin Lei et al. proposed a vehicle optimized path planning method based on improved artificial fish swarm algorithm. First, the vehicle and the path are encoded by the three-dimensional particle encoding method to construct an effective vehicle path. Then, the original algorithm is iterated to find the optimal solution. Simulation shows that the algorithm has a fast convergence speed and converges to the optimal solution of artificial fish swarm algorithm with a high probability. Chen Qiulian et al. proposed an improved particle swarm algorithm based on neural network and cubic spline curve function in this patent. It unifies the static and dynamic obstacle environment representation and collision detection model, and can quickly plan a smooth collision-free robot path, so that the path length is shorter and the number of algorithm iterations is less. Jian Wang et al. proposed a hybrid ant colony algorithm for solving the vehicle routing problem. Based on an analysis of its advantages and disadvantages, they proposed an improved strategy for the ant colony algorithm that considers real-time traffic conditions, thereby improving the performance of the algorithm for solving dynamic VRP. This type of algorithm has strong applicability and flexibility to changes in environment, parameters, and tasks, but its global search capability is weak, and the optimization time is significantly affected by the distribution of individuals.

[0007] The core of artificial intelligence algorithms lies in the trial-and-error learning process of autonomous vehicles interacting with their external environment. For example, when an autonomous vehicle perceives new environmental features, it makes judgments and plans a path based on historical data, compares it with the best path, calculates the error, and updates the historical data accordingly. This process is the training process. After training, the algorithm can plan a path that is similar to the optimal path for new environments. Commonly used artificial intelligence algorithms include neural network algorithms, Sarsa algorithms, Q-learning algorithms, and deep reinforcement learning. Gu Xun et al. proposed a traffic flow scheduling method based on the Detroit model and deep learning. They used the Detroit model to predict the distribution of vehicle demand, while using a deep learning model to obtain the actual vehicle flow on different roads, thus achieving optimized traffic flow scheduling. Wang Chengbo et al. established a path planning model for unmanned ships based on Q-learning. They designed the activation function based on the Markov process and planned the optimal strategy, which can plan the optimal path in unknown environments and successfully avoid multiple obstacles. Su Min Jeon et al. proposed an automated vehicle routing method for port terminals based on Q-learning technology. This method utilizes Q-learning and constructs a shortest-time routing matrix for a given location of a terminal crane to estimate the waiting time for each vehicle due to interference between vehicles, thus finding the shortest route. This type of algorithm can obtain the shortest path in complex environments and is well-adapted to intersection scenarios with many moving vehicles. However, the solution obtained by the artificial intelligence algorithm is an approximation of the optimal solution but not the optimal solution itself. It is sensitive to parameter settings, and the error compared to the optimal solution is highly dependent on the parameters. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a fine-grained path scheduling method for autonomous vehicles at unsignalized intersections based on a dual-learning network, which addresses the shortcomings of the prior art. This fine-grained path scheduling method for autonomous vehicles at unsignalized intersections based on a dual-learning network can improve the traffic efficiency of autonomous vehicles when passing through unsignalized intersections and is suitable for unsignalized intersection scenarios with a large number of moving vehicles. Moreover, compared with other existing path planning algorithms, this invention has higher path prediction accuracy and lower model system overhead.

[0009] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0010] A refined path scheduling method for autonomous vehicles at traffic-free intersections based on dual learning networks includes:

[0011] (1) Define the rectangular area inside the intersection as the conflict area and the lane area entering the intersection from different directions as the buffer area. Grid the conflict area. Each grid can only accommodate one autonomous vehicle at a time. Specify the start and end points of the autonomous vehicle and obtain the initial path of each autonomous vehicle at the intersection without traffic lights.

[0012] (2) Select an autonomous vehicle at an intersection without traffic lights as the target vehicle, select one of the initial paths as the path of the target vehicle, modify and update the path of the target vehicle according to the actual situation during the driving process of the target vehicle, and obtain the new path of the target vehicle; wherein the target vehicle can jump one grid at a time.

[0013] (3) Take all the autonomous vehicles at the intersection without traffic lights as target vehicles in sequence, and obtain the new path of the target vehicles at the intersection without traffic lights according to the method in step (2).

[0014] (4) Encode the environmental state of the target vehicle at each moment in the new path to obtain the feature data corresponding to the environmental state at each moment.

[0015] (5) In the new path of the target vehicle, the feature data corresponding to the environmental state of the target vehicle at time t is called the state s of the target vehicle at time t. t The action taken by the target vehicle at time t is called the action a of the target vehicle at time t. t , where a t For s t The tag value;

[0016] (6) Combine multiple sets of states s t Sum and state s t The corresponding label values ​​are used as the path learning dataset for the CNN fast decision network. The CNN fast decision network is then trained to obtain a trained fast path scheduling decision model.

[0017] (7) Initialize the DQN network. The DQN network includes the current Q network Q(s). t a t ;θ), target Q-network Q(s) t+1 a t+1 ;θ - ) and experience pool, where s t Let a represent the state of the target vehicle at time t. t Let θ represent the action taken by the target vehicle at time t, and s represent the current Q-network parameters. t+1 Let a represent the state of the target vehicle at time t+1. t+1 Let θ represent the action taken by the target vehicle at time t+1. -This represents the target Q-network parameters; the DQN network uses an adaptive learning rate.

[0018] (8) Set the greedy value ε;

[0019] (9) The state s of the target vehicle at time t t The data is sent to the fast path scheduling decision model, which outputs the probability distribution of actions. The action with the highest probability value is selected as its candidate action, and the probability value of this candidate action is denoted as ρ(a). t The state s of the target vehicle at time t. t Send to the current Q network Q(s) t a t ;θ), output the Q-values ​​of all legal actions, and select the action with the highest Q-value as its candidate action. The Q-value of this candidate action is denoted as Q(a t );

[0020] (10) When the random number x is less than the greedy value ε, randomly select an action from all legal actions as the action a to be taken at the current time t. t When the random number is greater than the greedy value ε, then from ρ(a t ) and Q(a t Select the action with the highest value as the action to be taken at time t. t ;

[0021] (11) Execute action a t Receive reward r t and the state s at the next moment t+1 ;

[0022] (12) will (s) t a t r t s t+1 (Integrate these experiences into an experience pool;)

[0023] (13) The experience pool will have (s) at time t. t a t Input to the current Q network Q(s) t a t ;θ), s t+1 Input to the target Q network Q(s) t+1 a t+1 ;θ - ), will the reward value r t The input is fed into the loss function, the loss function value is calculated, and stochastic gradient descent is performed based on the loss function value to update the current Q-network parameters θ.

[0024] (14) After C rounds of training, update the target Q network parameters θ -←θ, when the value obtained from the loss function converges to 0, the optimal Q-network is obtained;

[0025] (15) Input the current state of the vehicle to be predicted into the optimal Q network to obtain the action to be taken at the current time.

[0026] As a further improvement of the present invention, in step (2), the path of the target vehicle is modified and updated according to the actual situation during the driving process of the target vehicle to obtain a new path for the target vehicle, specifically as follows:

[0027] (2.1) Select one of the initial paths as the path of the target vehicle AV, and denote this initial path as L. AV L AV It includes the path coordinates of the target vehicle AV at different times. During the driving process of the target vehicle AV, a warning area is defined with the target vehicle AV as the center. The warning area includes an inner warning area and an outer warning area. The single-hop area of ​​the target vehicle AV is the inner warning area, and the two-hop area of ​​the target vehicle AV is the outer warning area.

[0028] (2.2) Taking the path coordinates of the target vehicle AV at time t as the center, if other vehicles OV are detected in the outer warning area at time t, then extract the path coordinates of the vehicle OV on path L. OV The path coordinates of the target vehicle AV at time t+1 are extracted and updated to the obstacle point ob at time t. t ;

[0029] (2.3) If other vehicles OV are detected in the inner warning area at time t, then extract the vehicle OV along path L. OV The path coordinates at time t and time t+1 are extracted and then used to update the obstacle point ob of the target vehicle AV at time t. t ;

[0030] (2.4) Connect the outer warning area with the original initial path L AV The intersection point is updated to the local endpoint a'. Then, the target vehicle AV calls the A-star algorithm to calculate the local path from the path coordinates at time t to the local endpoint a', and then updates the local path to L. AV middle;

[0031] (2.5) Traverse the target vehicle path L according to the methods in steps (2.2) to (2.4). AV The round ends after finding the path coordinates of all time points, thus obtaining a new path for the target vehicle AV.

[0032] Where path L OVThis is the initial path for the vehicle (OV).

[0033] As a further improved technical solution of the present invention, the feature data corresponding to the environmental state at each moment in step (4) includes the current position of the target vehicle, the current position of the other vehicles at OV, the endpoint of the target vehicle in the conflict area, the area of ​​the target vehicle's two jump points centered on the target vehicle at the current moment, the area of ​​the target vehicle's single jump point centered on the target vehicle at the current moment, the obstacle point of the target vehicle at the current moment, the target vehicle's position at the previous moment, and the intersection of the outer warning area and the target vehicle's initial path.

[0034] As a further improved technical solution of the present invention, the CNN fast decision network includes 8 convolutional layers, 8 ReLU activation layers, flattening layers, 2 fully connected layers and softmax activation layers;

[0035] The loss function of the CNN fast decision network is:

[0036] Loss = -log(y i );

[0037] Where y i This is the output of the CNN fast decision network;

[0038] The formula for updating the parameters W of a CNN fast decision network is:

[0039]

[0040] Among them, W + Here, W represents the updated parameters, G is a diagonal matrix, α is the global learning rate of the CNN fast decision network, and δ is the latest gradient contribution value. Let W be the gradient of parameter W.

[0041] As a further improvement of the present invention, the reward function r of the DQN network t for:

[0042]

[0043]

[0044]

[0045] in, This represents the distance between the target vehicle AV and the endpoint E of the conflict zone. in Let x represent the position coordinates of the target vehicle AV at time t, (x) E yE The coordinates of the target vehicle AV at the endpoint E of the conflict zone are shown. This represents the distance between the target vehicle AV and the other vehicles OV at time t. COV represents the position coordinates of the remaining vehicles OV at time t. t COV represents the number of other vehicles (OV) within the outer warning area centered on the target vehicle (AV) at time t. min COV represents the number of other vehicles (OV) within the inner warning area centered on the target vehicle (AV) at time t. max This represents the number of all other vehicles (OVs) within the warning area centered on the target vehicle (AV) at time t.

[0046] Cumulative reward value R t for:

[0047]

[0048] Where T represents the time when the target vehicle AV arrives at the destination E, and γ is a decay factor with a value range of 0≤γ≤1;

[0049] The loss function of the DQN network is:

[0050] L(θ)=E[(Q target -Q(s t ,a t ;θ)];

[0051] Where E represents the expected value, Q target Indicates that s t+1 Input to the target Q network Q(s) t+1 ,a t+1 ;θ - ) target Q network Q(s) t+1 ,a t+1 ;θ - The output value of Q(s) t ,a t ;θ) represents the time t (s t ,a t Input to the current Q network Q(s) t ,a t When θ), the current Q-network Q(s) t ,a t The output value of θ);

[0052] The gradient descent formula for the current Q-network parameter θ is:

[0053]

[0054] θt This represents the value of the current Q-network parameter θ at time t. t+1 η represents the value of the current Q-network parameter θ at time t+1; t This represents the learning rate at time t;

[0055] The DQN network uses an adaptive learning rate, and the formula for the adaptive learning rate is:

[0056]

[0057] Where η t-1 L represents the learning rate at time t-1; t-1 L represents the loss function value at time t-1. t Let t represent the loss function value at time t, and τ represent a very small positive number.

[0058] The beneficial effects of this invention are as follows:

[0059] This invention addresses the problems of long waiting times and low traffic efficiency when vehicles pass through unsignalized intersections, as well as the low prediction accuracy and poor model performance of existing path planning algorithms. It proposes a refined path scheduling method for autonomous vehicles at unsignalized intersections based on a dual-learning network of CNN and DQN. This invention focuses on autonomous vehicles, discretizing unsignalized intersections into a grid and obtaining a path learning dataset based on existing path planning algorithms. Based on this dataset, a coarse-grained fast path scheduling decision model is first constructed using a CNN network to quickly decide and schedule the real-time path of the autonomous vehicle at the unsignalized intersection. Simultaneously, to achieve refined path scheduling for autonomous vehicles, this invention adds a DQN reinforcement learning network to calculate the evaluation value of the model's decision behavior, constructs a reward function, and uses an adaptive learning rate to improve the model's prediction accuracy, thus balancing the real-time performance and accuracy of path planning and scheduling for autonomous vehicles at unsignalized intersections. The algorithm of this invention can improve the traffic efficiency of autonomous vehicles when passing through intersections without traffic lights, and is suitable for intersections without traffic lights with a large number of moving vehicles. Compared with other existing path planning algorithms, this invention has higher path prediction accuracy, lower model system overhead, and better model performance. Attached Figure Description

[0060] Figure 1 This is a path planning model diagram for autonomous vehicles at intersections without traffic lights.

[0061] Figure 2 This is a timetable for autonomous vehicles.

[0062] Figure 3This is a diagram of a weighted grid path.

[0063] Figure 4 This is a schematic diagram of the CNN fast decision network structure.

[0064] Figure 5 This is a schematic diagram of the DQN value network structure.

[0065] Figure 6 This is a comparison chart of the three models using SMAPE.

[0066] Figure 7 This is a comparison chart of the average completion times of the three models.

[0067] Figure 8 This is a comparison chart of the average rewards of the three models.

[0068] Figure 9 This is a comparison chart of average number of conflicts.

[0069] Figure 10 This is a comparison chart of effective passage times.

[0070] Figure 11 This is a comparison chart showing the percentage of average dwell time in the total travel time. Detailed Implementation

[0071] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0072] This embodiment proposes a refined path scheduling method for autonomous vehicles at unsignalized intersections based on a dual learning network of CNN and DQN. Taking autonomous vehicles as the research object, this embodiment performs grid discretization processing on unsignalized intersections and obtains a path learning dataset based on existing path planning algorithms. Based on this dataset, a coarse-grained fast path scheduling decision model is first constructed using a CNN network to quickly decide and schedule the real-time path of the autonomous vehicle at the unsignalized intersection. Simultaneously, to achieve refined path scheduling for autonomous vehicles, this embodiment adds a DQN reinforcement learning network to calculate the evaluation value of the model's decision behavior, and constructs a reward function and uses an adaptive learning rate to improve the accuracy of model prediction, balancing the real-time performance and accuracy of path planning and scheduling for autonomous vehicles at unsignalized intersections. In the experimental section, this embodiment compares and analyzes the performance of models with different hyperparameters and the adaptive learning rate model, obtaining a model accuracy of 96.5%. Furthermore, this embodiment proposes three evaluation metrics and compares and analyzes the performance of the model with three different types of algorithms based on these metrics. The conclusion is that the performance of the model in this embodiment is improved by 61.54% to 85.37% compared with other path planning algorithms.

[0073] 1. System Model:

[0074] like Figure 1 As shown, consider a traffic light-free intersection with six lanes in each direction. Assume each autonomous vehicle is equipped with an onboard sensor unit to acquire current state parameters such as location, real-time speed, and heading angle. Each autonomous vehicle is also equipped with a wireless transceiver with a broadcast coverage radius of r meters, interacting with other vehicles through periodic broadcasts. A Roadside Unit (RSU) with wireless transceiver capabilities is installed at the center of the intersection. Assume the RSU's broadcast coverage is large enough to cover all autonomous vehicles within the intersection area, and that reliable data sharing between autonomous vehicles is achieved through cooperative forwarding. Define the rectangular area inside the intersection as the conflict zone, and the lane areas entering the intersection from different directions as buffer zones. According to relevant traffic regulations, autonomous vehicles are not allowed to change lanes after entering the buffer zone.

[0075] Establish a rectangular coordinate system with the center of the conflict area as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis. Define V = {v i |i≤n,i∈N +} represents the set of all autonomous vehicles at the current intersection. Let be the set of instantaneous steering angles of all autonomous vehicles at time t, and d be the safe distance between vehicles. Let vehicle v... i For example, define Let its velocity at time t be... Let be the turning angle at time t. For vehicle v i The geographical location at time t. Furthermore, define... For autonomous vehicles v i The initial moment of entering the conflict zone, For vehicle v i The final moment of leaving the conflict zone, For vehicle v i The travel time consumed from entering the conflict zone to leaving the conflict zone.

[0076] Define ΔT as the total scheduling time at the intersection, which is the total time interval between the moment all autonomous vehicles enter the conflict zone and the moment they all leave the conflict zone. Figure 2 Describes each autonomous vehicle v i The travel time t from entering the conflict zone to leaving the conflict zone i The logical relationship between the intersection passage time ΔT and the time ΔT. Where, set L = {l1, l2, l...} 3,l4,…,l n-1 ,l n The projection of each horizontal segment onto the horizontal axis represents the travel time of the corresponding autonomous vehicle within the conflict zone. Define vehicle v. i The effective section is vehicle v i With vehicle v i-1 Time difference Δt between leaving the conflict zone i ,Right now Let v1 be the first autonomous vehicle to enter the conflict zone. Then, the effective segment of vehicle v1's travel time is Δt1. i The overlapping section is vehicle v i With v i-1 The intersection of travel times δ(t) i ∩t i-1 Let ti represent the earliest time that autonomous vehicles in set V enter the conflict zone, and tv represent the latest time that vehicles in set V leave the conflict zone. Then, the relationship between the intersection travel time ΔT of autonomous vehicles in set V from the time they first enter the conflict zone to the time they all leave the conflict zone and the travel time of each autonomous vehicle can be expressed as:

[0077]

[0078] ΔT = t” - t’; (2);

[0079]

[0080]

[0081] Based on the above definition, the path planning process of autonomous vehicles at intersections can be described as a mathematical optimization problem as shown in equation (5).

[0082]

[0083]

[0084]

[0085]

[0086] Equation (5) is the objective function for autonomous vehicles to pass through intersections; constraint (5a) requires that autonomous vehicles cannot stop during driving; constraint (5b) requires that the vehicle steering angle cannot change abruptly; constraint (5c) requires that vehicles maintain a minimum safe distance.

[0087] 2. Enhanced Dijkstra's Algorithm:

[0088] For the optimal path problem, commonly used methods include Dijkstra's algorithm, A-Star algorithm, Floyd's algorithm, and Bellman-Ford algorithm. Many improvements to these algorithms have been developed, and these improved algorithms have been widely applied in path optimization, node scheduling, and other fields. A brief introduction will focus on an enhanced Dijkstra algorithm, whose core idea is to add direction weights, safety weights, and priority weights to Dijkstra's algorithm.

[0089] The direction weight α represents the deviation between the current vehicle's direction of travel and the target destination direction, constraining the vehicle to move towards the target destination. The safety weight β represents the relative positional relationship between the current vehicle and other vehicles, constraining the vehicle to choose the safest grid as its next target, avoiding collisions at intersections. The priority weight γ represents the vehicle's priority in choosing the next target grid, constraining the vehicle's passage order within the grid, and preventing conflicts during path planning. When the autonomous vehicle is at a certain position, the weights w of the surrounding path grids can be calculated, i.e., w = α * β * γ. Then, based on the basic idea of ​​Dijkstra's algorithm, the shortest path from the vehicle's position to the destination is recalculated.

[0090] like Figure 3 As shown, taking vehicle A as an example, vehicle A needs to travel from the starting grid a to the ending grid a′. Within the adjacent grids of vehicle A's turning range, the direction weights α of grids 11, 16, and 15 are 0.7, 1, and 1, respectively. Grid 11 is unaffected by other vehicles. Grid 15 is considered a dangerous and warning grid for vehicle A, and grid 16 is also considered a dangerous grid for vehicle A. Therefore, their safety weights β are 0.9, 0.05, and 0.01, respectively. Vehicle v a At grid 12, with vehicle v e Simultaneously, grid 15 is selected, and grid 16 is selected simultaneously by vehicles E and D. At this point, vehicle A enters grids 15 and 16, which are for going straight and turning left, respectively. After multiplying the priority weights, the priority weights γ of grids 11, 15, and 16 are 0.8, 0.8, and 0.16, respectively. According to the definition of grid weight w, the weights of grids 11, 15, and 16 are 0.504, 0.04, and 0.0016, respectively. Grid 11 has the highest weight, so it is selected as the next target grid. After vehicle A reaches grid 11, the above algorithm is repeated to find the next grid until the destination is reached, at which point the scheduling ends.

[0091] While this method partially adapts to the dynamic topology environment of intersections, the parameters of the three weights are manually set, lacking a theoretical and systematic analysis in the assignment of weights. To address these issues, a dual-network prediction model based on historical trajectories—a CNN fast decision network and a DQN reinforcement learning network—is proposed.

[0092] 3. CNN Fast Decision Network:

[0093] This section first introduces the dataset and proposes a coarse-grained fast path scheduling decision model based on this dataset, constructed using a CNN network, to quickly make real-time path decisions and schedule autonomous vehicles at unsignalized intersections. Then, to achieve fine-grained path scheduling for autonomous vehicles, a DQN reinforcement learning network is added to the coarse-grained fast path scheduling decision model to calculate and evaluate the model's decision actions, thereby improving the accuracy of the decision actions.

[0094] 3.1 Dataset:

[0095] This section introduces the composition of the dataset and briefly discusses each part. Unsignalized intersections present a dynamic and complex topological environment. To realistically simulate the driving path of autonomous vehicles at unsignalized intersections, this paper uses a hybrid dataset of four different types of driving paths. The dataset includes the following path types: (i) grid-weighted paths: Dijkstra's algorithm and A-star algorithm; (ii) biomimetic paths: ant colony algorithm; (iii) artificial intelligence paths: neural RRT algorithm; and (iiii) human-driven paths.

[0096] exist Figure 1 In this method, the conflict area is divided into a 20x20 grid, with the center of the grid as the origin, the horizontal X-axis as the horizontal axis, and the vertical Y-axis as the vertical axis, establishing a Cartesian coordinate system. Each grid can only accommodate one autonomous vehicle at a time. Each autonomous vehicle can hop to one grid at a time. The starting and ending points of the vehicles are defined, and different paths are traversed using the algorithm described above. Then, the autonomous vehicle is simulated to travel along the paths in the grid. In this method, one AV is selected as the target vehicle, and the path is modified according to its driving process, as shown in the following example.

[0097] First, assuming there are i autonomous vehicles passing through the conflict area, and each vehicle generates a complete driving path, then list their corresponding paths. Store them in the form of n initial paths L. n They are then reproduced sequentially on the grid disk according to their index order, and one of the paths is selected as the initial path for the target vehicle AV, denoted as L. AV ,like Figure 3 As shown, during the movement of the target vehicle AV, two warning zones are defined centered on the target vehicle AV. The single-hop area of ​​the target vehicle AV is the inner warning zone, and the two-hop area of ​​the target vehicle AV is the outer warning zone. If the outer warning zone detects another vehicle OV (vehicle E in the figure) at time t, then L is extracted.oV Find the path point at time t+1 and update this path point to the obstacle point ob at time t. t If there are other vehicles (OV) in the inner warning area (such as vehicle D), then the position of this vehicle at time t and the path point at time t+1 are updated together as the obstacle point ob. t The intersection of the outer warning area and the original path is updated to the local endpoint a'. Then, the target vehicle AV calls the A-star algorithm to calculate the local path from the waypoint at time t to the local endpoint a'. Then update the local path to L. AV In the middle. After traversing the target vehicle path L... AV The round ends after all elements have been processed, and the next target vehicle is selected to start a new round.

[0098] To transform the aforementioned path data into an object that can be processed by a CNN neural network, the path data needs to be encoded. This paper employs one-hot encoding to encode the path data generated by the target vehicle, encoding the environmental state of each step in the target vehicle's path into a 20*20*8 feature tensor. The encoder used in this paper is called octal encoding, which contains eight feature planes. The interpretation of the feature planes is shown in the table below, which shows the feature data corresponding to the eight feature planes.

[0099] Table 1, Characteristic planes:

[0100]

[0101]

[0102] After encoding the data as described above, the characteristic data generated by the target vehicle at the current moment (i.e., the 8 features in Table 1 above) can be defined as state s. t The action taken by the target vehicle at the current moment can be called action a. t This article will discuss action a. t As the corresponding feature data s t The label values ​​are used. The feature data generated by the target vehicle at each time step and the corresponding action taken at that time step are used as the path learning dataset for the CNN fast decision network.

[0103] 3.2 CNN Fast Decision Network:

[0104] In complex scenarios at intersections, in order for the model to adapt to state s t Quickly make the corresponding strategic actions a t This paper uses a convolutional neural network to extract the state s. t The characteristics of this are used to output the strategy action a. tThe probability distribution is determined, and the action with the highest probability value is selected as the candidate action for the CNN fast decision network.

[0105] like Figure 4 As shown, the CNN fast decision-making network structure presented in this paper is illustrated. The CNN fast decision-making network includes 8 convolutional layers, 8 ReLU activation layers, a flattening layer, 2 fully connected layers, and a softmax activation layer. The neural network operates based on the state s of the target vehicle. t As input, the data is encoded by an eight-plane encoder and then fed into a Conv2D layer for convolution. This network structure contains eight Conv2D layers, each extracting eight feature values. Each Conv2D layer uses a 3x3x1 filter, and the convolution stride is set to 1. To maintain the final output probability matrix size at 20x20, this paper employs SamePadding to ensure the data retains its original size after passing through the convolutional layers.

[0106] Given that this paper uses an 11-layer neural network, its ability to adapt to input data is good, but it may lead to overfitting. In more extreme cases, the model may completely memorize the data it has learned, but fail to predict accurately for slightly different new data. To avoid this and improve the model's generalization ability, this paper adds a dropout layer for regularization after each convolutional layer, setting the dropout rate to 50%. The dropout layer randomly selects a certain proportion of neurons and discards them completely in each training step, then passes the remaining neurons to the next layer, and finally participates in backpropagation. This technique prevents a single layer (and therefore the entire neural network) from over-focusing on the given data, thus preventing overfitting.

[0107] In this neural network architecture, a Rectified Linear Unit (ReLU) is used as the activation function. This activation function ignores negative inputs by setting them to 0, while positive inputs remain unchanged. Therefore, the stronger the positive signal, the stronger the ReLU activation, which increases the efficiency of gradient descent and backpropagation, and avoids the problems of gradient explosion and gradient vanishing.

[0108] After the data undergoes convolutional computation through eight Conv2D layers, the data tensor needs to be flattened into a vector using a Flatten layer before being input into the last two dense (fully connected) layers. Since the final output of the fast decision model is the probability value of the action to be taken at the current moment, this structure uses a softmax activation function at the end and outputs the probability distribution ρ of the legal action 'a'. w (a|s), where w are the parameters of the fast prediction network.

[0109] In this network architecture, parameter w is updated using stochastic gradient descent (SGD). To accommodate SGD, the parameter update is proportional to the difference between the predicted and true values. This paper introduces a classification cross-entropy loss function, defined as follows:

[0110]

[0111] in Here are the label values ​​for the model, where i is the index and y is the value of the label. i For the output of the model, in this application, this formula can be reduced to one term, namely... That term. Therefore, formula (6) can be updated to:

[0112] Loss = -log(y i (7);

[0113] Because this network uses a large amount of data, a global learning rate α cannot be directly set. Therefore, an adaptive gradient method is used in this network, which adjusts the learning rate individually for each parameter w. The parameter weight update formula is as follows:

[0114]

[0115] Among them, W + Let W be the updated parameters, α be the global learning rate of the CNN fast decision network, G be a diagonal matrix, and after each parameter update, G is updated by adding the diagonal terms to the latest gradient contribution value δ. θW is the gradient of parameter W.

[0116] Compared to other network structures, the network structure presented in this paper can make decisions quickly while maintaining an accuracy of 60%-70%. Its first 8 Conv2D layers can effectively extract features of state S, enabling the model to detect the spatial relationship between the target vehicle AV and state S. Then, two Dense layers perform non-linear transformations on the extracted features to improve the correlation between features. Finally, the softmax activation function is used to output the probability distribution of all possible actions.

[0117] Based on the above path learning dataset, a coarse-grained fast path scheduling decision model is constructed using a CNN network.

[0118] 4. DQN Value Network:

[0119] 4.1 Reward Function Setting:

[0120] In the previous section, the fast path scheduling decision model can be applied to the current state s of the target vehicle AV. t Output the probability distribution ρ of all the next legal actions. w(a|s), and then the model selects the action with the highest probability from it as the candidate action a for the fast path scheduling decision model. t However, this candidate action a t This action may not be optimal for target vehicle i. Therefore, this paper incorporates a DQN network to optimize the model's decision-making.

[0121] Suppose there is a policy function π representing state s t With action a t The relationship between a t =π(s) t When the target vehicle AV is in state s t Action a was performed t The vehicle then needs to receive a reward to learn and improve its behavior, and to make the optimal decision π to obtain the maximum future reward. Therefore, the design of a suitable reward function is crucial for effectively evaluating the policy π and guiding the target vehicle to achieve better learning performance.

[0122] When AV passes through the conflict zone, the distance between the target vehicle AV and the destination E can be represented by the Manhattan distance. At time t, the distance from AV to the destination E can be expressed as... in With (x) E y E Let represent the position coordinates of AV at time t and the destination coordinates of AV, respectively. The distance between AV and the other vehicles OV at time t can be expressed as: Assuming all vehicles within the conflict zone have a constant speed, then even if the reward function r t It can be described as follows:

[0123]

[0124]

[0125]

[0126] COV t COV represents the number of other vehicles (OV) within the outer warning area centered on the target vehicle (AV) at time t. min COV represents the number of other vehicles (OV) within the inner warning area centered on the target vehicle (AV) at time t. max This represents the number of all other vehicles (OVs) within the warning area centered on the target vehicle (AV) at time t. The action 'a' is required for the target vehicle (AV) to take... tTo enable AV to get closer and closer to the endpoint E, rather than moving away from it or keeping the distance constant, this paper designs a reward function based on formula (9). To encourage safe and efficient driving behavior, this paper designs a reward function (10), the significance of which is that action a... t Is COV for the next state? t+1 It will have an impact. If action a t If the number of OVs in the outer warning area of ​​the next state increases, a penalty is imposed accordingly; conversely, if the number of OVs in the outer warning area of ​​the next state decreases, a reward is imposed accordingly. In order to incentivize AVs to avoid letting OVs enter the inner warning area, formula (10) only penalizes the case where the number of OVs in the inner warning area increases, and no reward is given otherwise.

[0127] The cumulative reward value can then be derived from formula (11), and the formula is as follows:

[0128]

[0129] Where T represents the time when AV reaches the endpoint E, and γ is a decay factor with a value range of 0≤γ≤1. The further away from time t, the more significant the reward decay.

[0130] 4.2 Definition of Q function:

[0131] Q-learning is the core idea of ​​DQN. This method can evaluate action a at time t. t The value of Q is used to learn an optimal driving decision that maximizes the sum of future rewards in the current state. The optimal decision can be made by choosing a with the highest Q value. t The Q-value is used to evaluate the value of an action in a given state. π (s t ,a t ) represents the state s based on decision π. t Perform action a t The Q value is defined as follows:

[0132] Q π (s t ,a t )=E[R t+k |s t+k =s,a t+k =a,π] (13);

[0133] Where E represents the expected value, the Q function starts from state s, uses a as the maximum accumulated reward value for the first action, and k represents the time step after time t. π (s t ,a t ) and Q π(s t+1 ,a t+1 The recursive relationship between them can be described using the Bellman equation:

[0134] Q π (s t ,a t )=E[r t +γmaxQ π (s t+1 ,a t+1 (14)

[0135] In this paper, because the dynamic topology environment generates a large number of states, the Q-value of the action must be approximated using a parameterized method. Therefore, DQN is used to nonlinearly fit the Q-network, and a convolutional neural network (CNN) is used to build the current Q-network with parameters θ. The parameters of the current Q-network are updated in each iteration, and stochastic gradient descent (SGD) is used to minimize the loss function. The following formula L(θ) is the loss function:

[0136] Q target =r t +γmaxQ(s t+1 ,a t+1 ;θ - (15);

[0137] L(θ)=E[(Q target -Q(s t ,a t ;θ)] (16);

[0138]

[0139] Where E represents the expected value, maxQ(s) t+1 ,a t+1 ;θ - ) indicates that s t+1 Input to the target Q network Q(s) t+1 ,a t+1 ;θ - ) target Q network Q(s) t+1 ,a t+1 ;θ - The output value of Q(s) t ,a t ;θ) represents the time t (s t ,a t 0 is input to the current Q network Q(s) t ,a t ;θ) is the current output value of the Q network. Formula (17) is the gradient descent formula for parameter θ, η t Let be the learning rate for parameter θ.t Let a represent the state of the target vehicle at time t. t Let represent the action taken by the target vehicle at time t, θ represent the current Q-network parameters, and s represent the action taken by the target vehicle at time t. t+1 Let a represent the state of the target vehicle at time t+1. t+1 Let θ represent the action taken by the target vehicle at time t+1. - This represents the target Q-network parameters; the DQN network uses an adaptive learning rate.

[0140] 4.3 DQN Value Network Structure:

[0141] The DQN value network structure proposed in this paper is as follows: Figure 5 As shown, the DQN value network includes the current Q network Q(s) t ,a t ;θ), target Q-network Q(s) t+1 ,a t+1 ;θ-) and experience pool. Figure 5 The intermediate environment refers to the intersection where the target vehicle (AV) is located. If the target vehicle is in a state s at time t... t In this network, the environment will be in state s t The fast path scheduling decision model is sent to the current Q-network, and the fast path scheduling decision model determines the state based on the current state. t The probability distribution ρ of the output action w (a t |s t The action with the highest probability value is selected as the candidate action, and the probability value of the candidate action is denoted as ρ(a). t The current Q network is based on s. t The Q-values ​​of all legal actions are calculated using formula (13), and the action with the highest Q-value is selected as its candidate action. The Q-value of this candidate action is denoted as Q(a). t ) = argmaxQ(s t ,a t ;θ).

[0142] To enable the model to explore unknown actions and maintain a balance between the optimal action and the undesirable action, this paper sets an ε value to control this balance; this value is called the greedy value. Figure 5 In this algorithm, a judgment logic is defined for the greedy value ε. After both the fast path scheduling decision model and the current Q-network have selected candidate actions, a random number x is given, which is between [0, 1]. The algorithm checks whether the random number x is less than the greedy value ε. If it is, an action Ra is randomly selected from all legal actions. t As the current action a t If not, then in {ρ(a t ),Q(a tChoose the action with the highest estimated value from the two candidate actions as the action to be taken. t In the early stages of training, a relatively high greedy value ε is selected; in this paper, 0.5 is used. As training continues to improve, the model learns more and more new actions, and the loss function in the DQN value network converges to 0. At this point, the greedy value ε is gradually reduced. The model will then rely more on the current Q-network evaluation value and select the action with the highest Q-value.

[0143] Assuming action a has already been selected t Then first, action a t Return to the environment and output the state s at time t+1. t+1 To ensure stable convergence of the loss in the DQN value network, a memory replay technique is used, that is, the loss at time t (s t ,a t ,r t ,s t+1 (e) becomes the memory at time t t All memories {e1,e2,…,e} from time 1 to t t All memories are stored in the Replaymemory. During each training iteration, a set of memories is uniformly sampled from the Replaymemory, which avoids over-coupling caused by using adjacent memories.

[0144] During training, the experience pool Replaymemory will store the (s) at time t. t ,a t The input is given to the current Q-network. t+1 The input is fed into the Target Q Network, and the reward value r is... t The input is given to the loss function, which is significant because it allows us to calculate the loss function value using formula (16). The current Q-network obtains the Q-value with respect to parameter θ at time t, and the target Q-network calculates the maxQ(s) at time t+1. t+1 ,a t+1 ;θ - The function L(θ) returns the loss function value and then uses gradient descent to update the parameters θ in the target Q network.

[0145] After C rounds of training, the current Q-network copies all its parameters to the target Q-network and updates the parameters θ in the target Q-network. - When the value of the loss function converges to 0, the optimal Q-function, or the optimal Q-network model, is considered to have been obtained.

[0146] 4.4 Proof of convergence of adaptive learning rate in dual networks:

[0147] During DQN training, the current Q-network is used to fit the target Q-network. Although neural networks have strong nonlinear mapping capabilities, the addition of a greedy strategy in the dual networks can cause convergence oscillations in L(θ) after selecting actions with poor predictions, increasing model training time without significantly improving training accuracy.

[0148] To address the aforementioned issues, this paper proposes an improved adaptive learning rate that can adaptively update the learning rate value based on changes in the loss function L(θ). The convergence of this learning rate during the iterative process is mathematically proven. Specifically, η is defined as follows: t L is the learning rate at time t; t Let L be the value of the loss function at time t; t-1 -L t Let be the change in the loss function between time t-1 and time t. Then the adaptive learning rate formula is as follows:

[0149]

[0150] Here, τ is a very small positive number, used to prevent the learning rate from decreasing to zero in the later stages of neural network training and to stabilize the learning rate at a certain value. This learning rate can adaptively decrease when the loss function increases and overfitting occurs during training, effectively reducing the possibility of model non-convergence due to oscillations.

[0151] This paper further uses mathematical formulas to derive and prove that the loss function L(θ) converges under adaptive learning rate. It is generally believed that P... i =L t-1 -L t The iteration number T satisfies Right now:

[0152]

[0153] Similarly, it is assumed that the gradient of the function is equal to 0 without considering saddle points, and the algorithm converges as shown in equation (20):

[0154]

[0155]

[0156] Under the non-convex condition in this paper, the convergence condition can be updated to the following formula:

[0157]

[0158] Wherein: g t This represents the gradient during backpropagation, and its basic principle is based on the sequence {‖g}.t The algorithm is considered to have converged when it finds its minimum value and this value approaches 0.

[0159] Lipschitz continuity is an important tool in nonconvex optimization analysis. In practical research, Lipschitz gradient continuity is more common, and its function needs to satisfy:

[0160]

[0161] Definition 1: When function L: All satisfied There exists a constant B such that:

[0162]

[0163] That is, the function L is said to be in The top is smooth (L).

[0164] To verify that Lipschitz gradient continuity is equivalent to L-smoothness, by definition, when a function f(x) satisfies Lipschitz gradient continuity, it must satisfy equation (23), which yields the following equation:

[0165]

[0166] Right now

[0167] Therefore, it can be concluded that when the second gradient exists, the continuity of the Lipschitz gradient actually means that the second gradient (i.e., the Hessian matrix) of the function L is consistent with the second gradient of the function L. satisfy:

[0168]

[0169] Where matrix I represents the identity matrix, the above equation represents the matrix positive semidefinite matrix It is also a semi-positive definite.

[0170] Performing a Taylor expansion on the function L(x) and simplifying, we get:

[0171]

[0172] Therefore, equation (23) holds true, and thus the two are equivalent.

[0173] Proof: According to equation (23), the existence constant B satisfies:

[0174]

[0175] According to formula (17) θ t The update rules are as follows:

[0176]

[0177] Both sides of the inequality take the expected value of the random variable, i.e.:

[0178]

[0179] For a function f(θ), its gradient The unbiased estimate of g is the stochastic gradient g. t Right now:

[0180]

[0181] for stochastic gradient g t There exists a constant D that satisfies:

[0182] ||(gt)||<D (32);

[0183] Combining equations (30), (31), and (32), we can simplify to obtain:

[0184]

[0185] Definition 2: Learning rate {η} t} is a non-increasing positive term sequence that satisfies the following conditions:

[0186]

[0187] Theorem 1: When a function L(x) satisfies Definition 1, its stochastic gradient satisfies equations (32) and (33), its learning rate η t Satisfying Definition 2 and the function L(x) in the domain L inf =L * Then we have:

[0188]

[0189] According to Theorem 1, superimposing equation (34) from t=1 to T, we have:

[0190]

[0191] Since the learning rate satisfies Definition 2, we have:

[0192]

[0193] Then we can derive the loss function L(θ) at the adaptive learning rate η. t The contraction is downward.

[0194] 5. Simulation Analysis:

[0195] This section explains how to evaluate the proposed vehicle routing prediction model. First, models with different hyperparameters are presented, and then the metrics are evaluated and their relevant parameter curves are compared and analyzed. Next, the model curves under adaptive learning rate are given, and the above analysis is proven. Then, a performance analysis comparing this model with other algorithms is presented.

[0196] 5.1 Comparing model performance under different hyperparameters:

[0197] All network model architectures are used as described in Section 3. Furthermore, these models are implemented using Python and PyTorch libraries. Since the DQN network uses an L1 loss function, and the Q-value gradually increases during training, the value of the L1 loss function increases. To address this issue, we use the symmetric mean absolute percentage error (SMAPE) as the evaluation metric. The equation is shown below:

[0198]

[0199] in, Q represents the maximum Q-value of the target Q-network output for the i-th episode in the current loop. i This represents the Q-value of the current Q-network output for the i-th episode in the current loop.

[0200] Furthermore, this paper compares models with manually selected hyperparameters, models with hyperparameters optimized using grid search, and AVP-RLN. AVP-RLN represents the optimal Q-network model in this paper (i.e., the model with adaptive learning rate), which is illustrated by analyzing smape, the time spent per episode during training, and the average reward value. The hyperparameter settings are shown in Table 2.

[0201] Table 2: Hyperparameter settings;

[0202] parameter Parameter settings Optimized parameter settings Learning rate 0.002 0.004 Learning Strarts 40960 40960 Batch size 1024 1024 Gradient descent step size 10 10 Cache size 400000 200000 Gammaγ 0.96 0.99

[0203] As shown in Table 2, the optimized parameter settings are based on grid search technology and have been cross-validated. Figure 6 The comparison between two different hyperparameter models and the smape of AVP-RLN is shown.

[0204] In the SMAPE error plot, it can be observed that the model reaches its maximum value around the 110th iteration. This is because training parameters need to be initialized at the beginning of the training period, during which the parameters are randomly selected. When 40,960 parameters are selected, the model begins learning and training. Among these parameters, the maximum SMAPE value of the ordinary hyperparameter model is close to that of the optimized hyperparameter model, around 0.42. This is because both models randomly select parameters during the initialization phase of training. The optimized hyperparameter model gradually converges to a minimum value after the 110th iteration, with the minimum value between 0.08 and 0.06. The ordinary hyperparameter model also converges to a minimum value after the 100th iteration, but its minimum value is around 0.13-0.11. It can be seen that compared to the ordinary model, the optimized hyperparameter model improves prediction accuracy by about 5%. The SMAPE curve trend of AVP-RLN is roughly the same as the previous two models, reaching its maximum value around the 110th iteration and gradually converging in subsequent training. This is also due to the parameter initialization at the beginning of training. However, after 150 epochs, it gradually converges to a minimum of approximately 0.035, which corresponds to approximately 96.5% accuracy. Compared to the SMAPE values ​​of general hyperparameter models and models with optimized hyperparameters, AVP-RLN shows a 10%–5.2% improvement in SMAPE values. This is because, according to formula (18), when the SMAPE error value is high during training, the adaptive learning rate increases, making the model more dependent on experience; conversely, when overfitting occurs during training, the learning rate decreases.

[0205] Figure 7 The average reward value obtained by the model in each episode is displayed. It can be seen that the model begins to converge around episode 6000, and the reward value oscillates upwards. Within a certain range, due to the different environments of each training group, the model converges within this range, as shown by the curve trend. The average reward value convergence range of a typical hyperparameter model is [0.4, 0.45], while that of an optimized hyperparameter model is [0.50, 0.55]. Therefore, it can be seen that compared with the typical hyperparameter model, the optimized hyperparameter model can obtain higher reward values ​​for different environments during training. This indicates that the model can choose a better strategy to control the AV (Active Path), selecting a shorter path without conflicting with the OV (Original Path). From the reward value curve of AVP-RLN, it can be seen that the model obtains the highest average reward value around episode 3000, at which point the model's performance is optimal, with its highest average reward significantly exceeding that of the other two models by approximately 0.25-0.34. However, as the number of training sets increases, the learning rate gradually decreases and approaches zero, the reward value gradually reaches its peak, and then drops to approximately 0.46.

[0206] Figure 8This displays the time, in seconds, for the model to complete one episode during training. This parameter can be considered the time from the start to the end of the AV (Average Volume). From the graph, we can see that the time to complete each episode gradually converges, and the convergence interval is the same as the reward value. This is because... Figure 7 Similarly, each episode represents a different environment. The past time of the general hyperparameter model gradually converges between [70, 65], while the past time of the optimized hyperparameter model converges between [60, 55]. It can be seen that, compared to the general hyperparameter model, the optimized hyperparameter model reduces the time required to complete an episode by approximately 10 seconds. AVP-RLN has the shortest average time to complete an episode around 3000 episodes, because at this point, AVP-RLN can select the policy with the highest reward value while maintaining 96.5% accuracy.

[0207] 5.2 Performance comparison with other algorithms:

[0208] This paper describes three metrics used to evaluate and compare the performance of path prediction methods. In experiments, the proposed algorithm is compared with three other typical algorithms: Dijkstra's algorithm, ant colony optimization, and RNN model. The following three metrics are used to compare the performance results.

[0209] Average number of collisions: The average number of collisions for AVs controlled by the four algorithms under different traffic densities.

[0210] Effective transit time: Referring to the effective transit time ΔT proposed in Section 1, the performance of the four algorithms under different traffic densities is compared.

[0211] Average dwell time as a percentage of travel time: The percentage of dwell time in the AV during driving out of the total travel time.

[0212] Average number of conflicts Figure 9As shown in the figure, it can be concluded that when the number of vehicles is 20-30, both AVP-RLN and RNN neural networks can maintain 0 collisions, indicating that both algorithms are effective at low traffic densities. As the number of vehicles increases, the environment becomes more complex, and the number of collisions also increases. However, compared with the other three algorithms, AVP-RLN can maintain a lower number of collisions at higher traffic densities, with a maximum average number of collisions of 6, while Dijkstra's algorithm has a maximum average number of collisions of 41. Compared with Dijkstra's algorithm, the model performance of the proposed algorithm is improved by 85.37%. The improvements compared to RNN neural networks and ant colony algorithms are 61.54% and 81.43%, respectively. This is because Dijkstra's algorithm is a static grid algorithm and cannot adapt to dynamic and complex environments, while ant colony algorithms have long computation times and are not suitable for scenarios with high real-time requirements. RNN neural networks have poor adaptability in high traffic density environments and cannot guarantee optimal solutions for complex and changing network topologies.

[0213] Figure 10 The graph shows the effective transit time (ΔT) of four algorithms under different traffic densities. As can be seen from the graph, the effective transit time of autonomous vehicles is affected by traffic density. Compared to the other three algorithms, AVP-RLN maintains a lower effective transit time. Dijkstra's algorithm and the ant colony algorithm maintain a low ΔT at low traffic densities of 20-40. This is because the greedy strategy requires these two algorithms to always calculate the shortest path, but as traffic density increases, the shortest path is clearly not the optimal path, thus leading to a longer effective transit time under high traffic density. In contrast, AVP-RLN can adapt to changes in the environment, considering both the shortest path and the number of conflicts, and select the strategy with the highest evaluation value, keeping the effective transit time between 45s and 60s under high traffic density.

[0214] exist Figure 11 In this paper, we compare the percentage of average dwell time to travel time for AVs controlled by four algorithms under different traffic densities. It can be seen that AVP-RLN and RNN neural networks adapt better to dynamic environments. Furthermore, since the RNN neural network is trained primarily to avoid moving obstacles, its percentage of dwell time under low traffic density is slightly better than our algorithm model, but the overall trend is similar, with values ​​ranging from 8% to 24%. Dijkstra's algorithm and ant colony algorithm have poor adaptability to dynamic topology environments, resulting in higher relative dwell time percentages, which are significantly affected by traffic density, ranging from 15% to 48%.

[0215] 6. Conclusion:

[0216] This invention investigates the path optimization problem for autonomous vehicles (AVs) at unsignalized intersections. First, a system model for the unsignalized intersection is established, and an objective function is proposed accordingly. Based on the unsignalized intersection scenario, a fast path scheduling decision model is established, enabling the AV to quickly select a path based on the environment. Then, a DQN value network is added to evaluate all executable path actions of the AV and calculate the Q-value. To obtain a more accurate prediction model, an adaptive learning rate is proposed, and its convergence is proven. In the experimental section, the performance of different hyperparameter models and the adaptive learning rate model is compared. Numerical results demonstrate that the model under the adaptive learning rate has superior prediction accuracy. Finally, three evaluation metrics are proposed, and based on these metrics, the AVP-RLN algorithm of this invention is compared and analyzed with three other different types of algorithms, confirming that the algorithm model of this invention can effectively solve the path optimization problem in the dynamic and complex environment of unsignalized intersections.

[0217] The scope of protection of this invention includes, but is not limited to, the above embodiments. The scope of protection of this invention is defined by the claims. Any substitutions, modifications, or improvements to this technology that are easily conceived by those skilled in the art fall within the scope of protection of this invention.

Claims

1. A method for fine path scheduling of autonomous vehicles at a signal-free intersection based on a dual learning network, characterized in that: Comprise: (1) define the intersection inside the rectangular region as the conflict area, the lane area of different directions entering the intersection as the buffer area, grid the conflict area, each grid can only accommodate one autonomous vehicle at the same time, specify the starting point and end point of the autonomous vehicle, and obtain the initial path of each autonomous vehicle under the intersection without signal lights; (2) select an autonomous vehicle under the intersection without signal lights as the target vehicle, select one of the initial paths as the path of the target vehicle, modify and update the path of the target vehicle according to the actual situation in the driving process of the target vehicle, and obtain the new path of the target vehicle; Wherein the target vehicle can single hop one grid every time; (3) all autonomous vehicles under the intersection without signal lights are taken in turn as the target vehicle, and the new path of the target vehicle under the intersection without signal lights is obtained according to the method of step (2); (4) encode the environmental state of each time in the new path of the target vehicle to obtain the feature data corresponding to the environmental state of each time; (5) In the new path of the target vehicle, the target vehicle is in the state corresponding to the feature data of the environment state at the time The state of the target vehicle at the time The action made by the target vehicle at the time The action of the target vehicle at the time The action of the target vehicle at the time Wherein The label value of ; (6) Combine multiple states Sum and State The corresponding label values ​​are used as the path learning dataset for the CNN fast decision network. The CNN fast decision network is then trained to obtain a trained fast path scheduling decision model. (7) Initialize the DQN network, which includes the current Q network. Target Q network and experience pool, among which, Indicates the target vehicle is in The state at any given moment, Indicates the target vehicle is in Actions performed at all times Indicates the current Q network parameters. Indicates the target vehicle is in The state at any given moment, Indicates the target vehicle is in Actions performed at all times This represents the target Q-network parameters; the DQN network uses an adaptive learning rate. (8) Set the greed value ; (9) Place the target vehicle in state of time The data is sent to the fast path scheduling decision model, which outputs the probability distribution of actions. The action with the highest probability value is selected as its candidate action, and the probability value of this candidate action is denoted as... The target vehicle is state of time Send to the current Q network Output the Q-values ​​of all legal actions, and select the action with the highest Q-value as its candidate action. The Q-value of this candidate action is denoted as . ; (10) random number less than the greed value , a random action is selected from all legal actions as the action made at the current time , if the random number is greater than the greed value , the action with the highest value is selected from the and as the action made at the current time ; (11) performing an action , obtaining a reward and a state at the next time ; (12) to integrate into experience, stored in the experience pool; (13), the experience pool will the moment input to the current Q network , input to the target Q network , the reward value input to the loss function, calculate the loss function value, according to the loss function value, random gradient descent, update the current Q network parameters ; (14) After C rounds of training, update the target Q network parameters When the value derived by the loss function converges to 0, the optimal Q network is obtained; (15) input the state of the vehicle to be predicted at the current time into the optimal Q network to obtain the action to be taken at the current time; In the step (2), the path of the target vehicle is modified and updated according to the actual situation in the driving process of the target vehicle, and the new path of the target vehicle is obtained, specifically: (2.1) selecting one of the initial paths as a target vehicle's path, the initial path is denoted as , , containing path coordinate points of the target vehicle at different time points, the target vehicle , in the process of driving, defining a warning area with the target vehicle as the center, the warning area includes an inner warning area and an outer warning area, the target vehicle , a single-hop area is the inner warning area, the target vehicle , a two-hop area is the outer warning area; (2.2), the target vehicle at the time t0 is centered on the path coordinate point at the time t0 if the remaining vehicles in the outer warning area are detected at the time t0 , the path coordinate point in the vehicle at the time t0 is extracted, and the extracted path coordinate point is updated to the path coordinate point of the target vehicle at the time t0 ; the obstacle point at the time t0 (2.3) If in Other vehicles are constantly detected within the inner warning area. Then retrieve the vehicle In China Path coordinates at time and The path coordinates at time points are extracted, and the two extracted path coordinates are used to update the target vehicle. exist Time barrier ; (2.4), updating the intersection of the outer warning area and the original initial path as a local end point Then the target vehicle invokes the A-star algorithm and calculates the path coordinate points of the target vehicle from the moment to the local end point of the local path, and then updates the local path to the center.​​ (2.5) traversing all time point path coordinate points in the target vehicle path according to the method of step (2.2) to step (2.4) to end this round, and obtaining a new path of the target vehicle . ​ wherein is the initial path of the vehicle .

2. The dual learning network-based automatic driving vehicle fine path scheduling method for a signal-free intersection according to claim 1, characterized in that: The feature data corresponding to the environment state of each moment in the step (4) includes the target vehicle position at the current moment, the positions of the remaining vehicles at the current moment, the end point of the target vehicle in the conflict area, the area of the two-hop point range of the target vehicle centered on the target vehicle at the current moment, the area of the single-hop point range of the target vehicle centered on the target vehicle at the current moment, the obstacle point of the target vehicle at the current moment, the position of the intersection of the outer warning area and the initial path of the target vehicle, and the position of the target vehicle at the previous moment. The feature data corresponding to the environment state of each moment in the step (4) includes the target vehicle position at the current moment, the positions of the remaining vehicles at the current moment, the end point of the target vehicle in the conflict area, the area of the two-hop point range of the target vehicle centered on the target vehicle at the current moment, the area of the single-hop point range of the target vehicle centered on the target vehicle at the current moment, the obstacle point of the target vehicle at the current moment, the position of the intersection of the outer warning area and the initial path of the target vehicle, and the position of the target vehicle 3. The dual learning network-based automatic driving vehicle fine path scheduling method for a signal-free intersection according to claim 1, characterized in that: The CNN fast decision network comprises 8 convolution layers, 8 ReLU activation layers, a flattening layer, 2 fully connected layers and a softmax activation layer; The loss function of the CNN fast decision network is: ； wherein is the output of the CNN fast decision network; Parameters of the cnn fast decision network The update formula is: ； wherein, is the updated parameter, is the original parameter, is a diagonal matrix, is the global learning rate of the CNN fast decision network, is the latest gradient contribution value, is the gradient of the parameter .

4. The dual learning network-based automatic driving vehicle fine path scheduling method for a signal-free intersection according to claim 1, characterized in that: The reward function of the DQN network is: ； ； ； wherein represents the target vehicle and the remaining vehicles at the end of the conflict zone , wherein , represents the position coordinates of the target vehicle at the time instant , represents the position coordinates of the target vehicle at the end of the conflict zone , represents the distance between the target vehicle and the remaining vehicles at the time instant , wherein represents the position coordinates of the remaining vehicles at the time instant , represents the number of the remaining vehicles inside the outer warning zone centered on the target vehicle at the time instant , represents the number of the remaining vehicles inside the inner warning zone centered on the target vehicle at the time instant , represents the number of all the remaining vehicles inside the warning zone centered on the target vehicle at the time instant ; Accumulated reward value is: ； wherein represents a target vehicle arrives at the end point , is a decay factor, which takes a value in the range 1; The loss function of the DQN network is: ； wherein denotes taking the expectation, denotes feeding to the target Q-network the target Q-network the output value, denotes feeding the input at time to the current Q-network the output value of the current Q-network at time Current Q network parameters The gradient descent formula for the current Q network parameters is: ； representing current Q-network parameters at the time instant , representing current Q-network parameters at the time instant ; representing learning rate at the time instant; The DQN network adopts an adaptive learning rate, and the formula of the adaptive learning rate is: ； wherein denotes the learning rate at the time instant denotes the loss function value at the time instant denotes the loss function value at the time instant denotes a small positive number.

Images