Dynamic vehicle path planning method based on deep reinforcement learning

Abstract

The application relates to the technical field of intelligent traffic and path planning, and particularly discloses a dynamic vehicle path planning method based on deep reinforcement learning, which collects historical traffic flow data of a road network to construct implicit continuous space-time coding, outputs predicted passing speeds and first and second derivatives thereof and assigns the values to graph edge features; a differential attention kernel is defined according to the second derivative, neighbor node information aggregation weights are adjusted, and updated node states are output; after the node states and vehicle information are spliced, the spliced result is input into a deep reinforcement learning strategy network, and path decision actions are output; in a training stage, a double-scale evolution constraint is introduced, so that future space-time prediction and node state evolution maintain exponential convergence; when online execution is performed, speed residuals are calculated based on real-time data, single-step gradient descent update is performed on the implicit continuous space-time coding, and then the node state update and decision process are repeated, and a dynamically adjusted vehicle path is output; the application can synchronize congestion propagation dynamics and realize lightweight online self-adaptation.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and route planning technology, specifically to a dynamic vehicle route planning method based on deep reinforcement learning. Background Technology

[0002] Dynamic vehicle routing refers to the technology of adjusting vehicle routes online based on real-time changes in traffic conditions and new order requests during vehicle operation. Traditional methods such as tabu search and genetic algorithms are difficult to handle high-frequency dynamic changes. In recent years, deep reinforcement learning has been introduced into this field due to its end-to-end decision-making capabilities.

[0003] Existing dynamic vehicle path planning methods based on deep reinforcement learning cannot match the actual dynamics of congestion propagation by updating graph edge features at fixed time intervals. This causes the graph neural network to repeatedly amplify outdated information during message passing, leading to secondary congestion in vehicle detour decisions. At the same time, the lack of online lightweight adaptive mechanisms makes it difficult to track road network changes in real time and continuously optimize paths under limited computing resources. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic vehicle path planning method based on deep reinforcement learning to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions: The dynamic vehicle path planning method based on deep reinforcement learning includes the following steps: S1: Collect historical traffic flow data of each road segment in the road network, including road segment identification, continuous timestamps and corresponding traffic speeds, and construct an implicit continuous spatiotemporal code that uses road segment identification and continuous timestamps as inputs and outputs to predict traffic speeds; S2: Assign the predicted passage speed output by the implicit continuous spatiotemporal coding and its first and second derivatives with respect to time as edge features of the graph structure. Define the differential attention kernel based on the second derivative in the edge features. Adjust the weight when aggregating neighbor node information through the kernel and output the updated node state. S3: The updated node state is concatenated with the vehicle's current position, remaining capacity, and list of unfinished tasks. The concatenation result is input into the deep reinforcement learning policy network to output the vehicle's path decision action at the next moment. S4: During the training phase, a dual-scale evolution constraint is introduced to ensure that the future spatiotemporal predictions of the implicit continuous spatiotemporal coding output and the updated node state evolution maintain an exponential convergence relationship in the feature space, and output an implicit expression that satisfies the consistency of congestion propagation dynamics. S5: During online execution, the implicit continuous spatiotemporal code is continuously updated with real-time traffic flow data. The updated implicit continuous spatiotemporal code is then re-executed to output the updated node status and path decision actions, resulting in dynamically adjusted vehicle paths.

[0006] As a further aspect of the present invention: the implicit continuous spatiotemporal coding specifically includes: The continuous timestamps in the collected historical traffic flow data are mapped into high-dimensional temporal embedding vectors through a set of Fourier basis functions. The road segment identifiers are converted into road segment feature vectors through an embedding layer. The high-dimensional temporal embedding vectors and road segment feature vectors are concatenated and input into a multilayer perceptron. The training objective is to minimize the mean square error between the predicted traffic speed output by the multilayer perceptron and the corresponding collected traffic speed. After iterative optimization, an implicit continuous spatiotemporal code is obtained.

[0007] As a further aspect of the present invention: the updated node state output specifically includes: The absolute value of the second derivative on each edge is taken and converted into an adjustment coefficient through a monotonically increasing mapping function. When the second derivative is positive, the adjustment coefficient is greater than 1, and when it is negative, the adjustment coefficient is less than 1. The adjusted score is obtained by multiplying the adjustment coefficient by the original dot product score calculated from the node state features. The adjusted score is normalized, and the normalized result is used as the weight when aggregating neighbor node information. The weighted updated node state is then output.

[0008] As a further aspect of the present invention: the process of obtaining the adjustment coefficient is as follows: Obtain the absolute value of the second derivative and multiply it by the preset congestion propagation time constant to obtain the intermediate value; The intermediate quantity is used as the exponent of an exponential function with the natural constant as the base to perform exponential operations, and the exponential result is obtained. The index result is compared with the preset upper and lower limits of the coefficient. If the index result exceeds the upper limit, the upper limit is used; if it is lower than the lower limit, the lower limit is used; otherwise, the index result itself is used. The index result after amplitude limiting is output as the adjustment coefficient.

[0009] As a further aspect of the present invention: the splicing result is as follows: First, calculate the spatial distance between the vehicle's current position and each node, select the node with the smallest distance, and use the state of the node as the spatial anchor point state. The vehicle's current position coordinates are mapped to a position embedding vector using a set of radial basis functions; The spatial anchor point state is added element-wise to the position embedding vector to obtain the node state with fused position information; The node status, remaining capacity value, and time windows and location coordinates of each task in the unfinished task list are arranged in order and then concatenated to output the spliced ​​comprehensive feature vector.

[0010] As a further aspect of the present invention: the output process of the position embedding vector is as follows: First, select multiple reference points in the road network, and each reference point is assigned a preset width parameter; Calculate the Euclidean distance between the vehicle's current position coordinates and each reference point; Divide the square of the Euclidean distance by the square of the corresponding width parameter, take the negative value, and then perform an exponential operation with the natural constant as the base to obtain the radial basis function value of the corresponding reference point. Arrange the radial basis function values ​​corresponding to all reference points in a fixed order and output the position embedding vector.

[0011] As a further aspect of the present invention: maintaining the exponential convergence relationship specifically includes: The future spatiotemporal prediction output by implicit continuous spatiotemporal coding is mapped to the same feature dimension as the updated node state to obtain the prediction feature vector; Calculate the cosine distance between the predicted feature vector and the current node's state evolution direction vector; The squared difference between the cosine distance and the preset exponential decay target value is used as the constraint loss; The constraint loss is multiplied by a weight that decays exponentially with time step and added to the total training loss of the deep reinforcement learning policy network. This forces the feature space distance between the two networks to converge at an exponential rate through backpropagation.

[0012] As a further aspect of the present invention: the step of forcing the feature space distance between the two to converge at an exponential rate through backpropagation specifically includes: Get the current training iteration number; Preset an initial decay coefficient and a decay half-life step, calculate the exponent with base 1 / 2, where the exponent is the current training iteration step divided by the decay half-life step, to obtain the decay coefficient for the current step; Multiply the constraint loss by the decay factor to obtain the weighted constraint loss for the current step. The weighted constraint loss is added to the original training loss of the deep reinforcement learning policy network to form the updated total training loss, which is then used for backpropagation.

[0013] As a further aspect of the present invention: the output of the dynamically adjusted vehicle path specifically includes: Within each decision cycle, speed observations of each road segment are collected within the last three seconds. The difference between the observed values ​​and the prediction output of the implicit continuous spatiotemporal coding at the current timestamp is calculated to obtain the speed residual. After multiplying the velocity residual by a preset learning rate, backpropagation is performed to update the implicitly continuous spatiotemporally encoded internal parameters using single-step gradient descent. The updated implicit continuous spatiotemporal code is re-outputted to predict the travel speed and its first and second derivatives. The node state update and path decision actions are then executed sequentially. The newly output path decision action is time-sequentially merged with the sequence of unvisited nodes in the current vehicle path, and the order of subsequent nodes in the original path is replaced to output the dynamically adjusted complete vehicle path.

[0014] As a further aspect of the present invention: the backpropagation of multiplying the velocity residual by a preset learning rate and then performing single-step gradient descent update on the internal parameters of the implicit continuous spatiotemporal encoding specifically includes: Obtain the absolute value of the speed residual for each road segment within the current decision-making period, and calculate its maximum value; The maximum value is compared with the preset gradient clipping threshold. If the maximum value exceeds the threshold, the preset learning rate is divided by the ratio of the maximum value to the threshold to obtain the adjusted learning rate. If the learning rate is not exceeded, the preset learning rate will remain unchanged. Multiply the velocity residual by the adjusted learning rate to obtain the parameter update amount; The parameter update amount is subtracted layer by layer along the backpropagation path from the internal parameters of the implicit continuous spatiotemporal encoding to complete the single-step update.

[0015] The beneficial effects of this invention are: (1) By combining implicit continuous spatiotemporal coding with differential attention kernels, the traffic speed and its second derivative of each road segment in the road network are directly integrated into the message transmission process of the graph structure, so that the dynamic characteristics of congestion acceleration and deceleration are explicitly incorporated into the node state update. Compared with the method that only relies on discrete time steps to update edge weights, this invention can transmit the changing trend to adjacent nodes more quickly when sudden congestion occurs, such as traffic accidents, thereby reducing the risk of paths repeatedly falling into congestion due to the delay in the propagation of congestion information.

[0016] (2) During the online execution phase, only the real-time traffic flow data of the most recent few seconds is used to perform a single-step gradient descent update on the implicit continuous spatiotemporal code, without retraining the entire deep reinforcement learning policy network. This mechanism enables the deployed vehicle path planning system to automatically fine-tune the predictive code based on the actual observed speed residuals, thereby continuously adapting to the slowly drifting traffic pattern changes in the road network and maintaining the stability of the decision-making strategy under different time periods and different levels of congestion. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating the construction process of implicit continuous spatiotemporal coding in this invention; Figure 3 This is a flowchart illustrating the process of outputting the updated node status in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, this invention is a dynamic vehicle path planning method based on deep reinforcement learning, comprising the following steps: S1: Collect historical traffic flow data of each road segment in the road network, including road segment identification, continuous timestamps and corresponding traffic speeds, and construct an implicit continuous spatiotemporal code that uses road segment identification and continuous timestamps as inputs and outputs to predict traffic speeds; S2: Assign the predicted passage speed output by the implicit continuous spatiotemporal coding and its first and second derivatives with respect to time as edge features of the graph structure. Define the differential attention kernel based on the second derivative in the edge features. Adjust the weight when aggregating neighbor node information through the kernel and output the updated node state. S3: The updated node state is concatenated with the vehicle's current position, remaining capacity, and list of unfinished tasks. The concatenation result is input into the deep reinforcement learning policy network to output the vehicle's path decision action at the next moment. S4: During the training phase, a dual-scale evolution constraint is introduced to ensure that the future spatiotemporal predictions of the implicit continuous spatiotemporal coding output and the updated node state evolution maintain an exponential convergence relationship in the feature space, and output an implicit expression that satisfies the consistency of congestion propagation dynamics. S5: During online execution, the implicit continuous spatiotemporal code is continuously updated with real-time traffic flow data. The updated implicit continuous spatiotemporal code is then re-executed to output the updated node status and path decision actions, resulting in dynamically adjusted vehicle paths.

[0021] Please see Figure 1 As shown, in S1, historical traffic flow data of each road segment in the road network is collected, including road segment identifiers, continuous timestamps, and corresponding traffic speeds. An implicit continuous spatiotemporal code is constructed with road segment identifiers and continuous timestamps as inputs and outputs to predict traffic speeds. Specifically, this includes: Historical traffic flow data for each road segment in the road network is collected. The specific collection method involves embedding induction coil detectors beneath key road segments in the urban road network, with each coil corresponding to a unique road segment identifier. When a vehicle passes a coil, the detector records the moment of passage and calculates the instantaneous speed, generating a data set every 30 seconds, including the road segment identifier, consecutive timestamps (accurate to 0.1 seconds), and the average travel speed during that time period. Simultaneously, floating cars equipped with GPS (such as taxis and buses) transmit their location and speed information once per second, which is then supplemented to the corresponding road segments after map matching. The two data sources are merged, and outliers are removed to form a historical traffic flow dataset.

[0022] For each consecutive timestamp in the dataset, a set of Fourier basis functions is constructed to map it. Specifically, 10 different frequency values ​​are preset, each frequency corresponding to a pair of sine and cosine functions. The timestamp is multiplied by each frequency, and the sine and cosine values ​​are calculated to obtain a 20-dimensional high-dimensional time embedding vector. This mapping transforms discrete time points into continuous, smooth periodic features, facilitating subsequent processing.

[0023] For each road segment identifier, it is converted into a road segment feature vector through an embedding layer. This embedding layer is a trainable lookup table, where each row corresponds to a vector representation of a road segment identifier, and the vector dimension is preset to 16 dimensions. During the lookup, the corresponding row is directly retrieved based on the road segment identifier to obtain the 16-dimensional road segment feature vector.

[0024] The aforementioned high-dimensional temporal embedding vector (20-dimensional) and the road segment feature vector (16-dimensional) are concatenated sequentially end-to-end to form a 36-dimensional combined feature vector. This combined feature vector is then input into a multilayer perceptron with two hidden layers: the first hidden layer contains 128 neurons, the second hidden layer contains 64 neurons, and the output layer contains 1 neuron. All hidden layers use rectified linear units as activation functions, and the output layer uses a linear activation function.

[0025] The training objective is to minimize the mean square error between the predicted traffic speed output by the multilayer perceptron and the corresponding collected traffic speed. Specifically, for each set of input data (a combination of road segment identifier and timestamp), the square of the difference between the predicted value and the actual speed value is calculated, and the average is obtained within a training batch to obtain the loss value.

[0026] An adaptive moment estimation algorithm is used to iteratively optimize all trainable parameters in the multilayer perceptron and the embedding layer. In each iteration, a batch of combined feature vectors are input into the multilayer perceptron, the loss value is calculated, and the parameters are updated via backpropagation. This iteration is repeated until the loss value converges. The resulting multilayer perceptron and embedding layer together constitute the implicit continuous spatiotemporal code. This code can receive arbitrary road segment identifiers and continuous timestamps, and output the corresponding predicted traffic speed.

[0027] Please see Figure 3 As shown, in S2, the predicted passage speed output by the implicit continuous spatiotemporal coding and its first and second derivatives with respect to time are assigned as edge features of the graph structure. A differential attention kernel is defined based on the second derivative of the edge features. The kernel is used to adjust the weights when aggregating neighbor node information, and the updated node state is output. Specifically, this includes: The predicted travel speed of each edge output by the implicit continuous spatiotemporal coding is used as the initial speed value of that edge. The first and second derivatives of this speed with respect to time are calculated using the finite difference method: the speed value of the previous sampling time (1 second interval) is taken, the current speed is subtracted from the speed of the previous time, and the result is divided by the sampling interval to obtain the first derivative; then the difference between the current first derivative and the first derivative of the previous time is taken and divided by the sampling interval to obtain the second derivative. The predicted travel speed, the first derivative, and the second derivative are collectively assigned as the three feature components of each directed edge in the graph structure.

[0028] The second derivative of each edge is defined as the congestion acceleration, with its sign indicating whether congestion is accelerating or slowing down. The absolute value of this second derivative is taken and multiplied by a preset congestion propagation time constant to obtain a dimensionless intermediate value. The congestion propagation time constant is set to 5 seconds, representing the average time required for congestion to spread from one road segment to an adjacent road segment. In the specific calculation, the absolute value of the second derivative is first obtained; for example, if the second derivative is negative 0.3 meters per cubic second, the absolute value is 0.3, multiplied by 5 seconds to obtain an intermediate value of 1.5.

[0029] Using the natural constant as the base, the aforementioned intermediate quantity is used as the exponent of the exponential function to perform exponential operations, yielding the exponential result. That is, the intermediate quantity of the natural constant is raised to the power of itself. For example, when the intermediate quantity is 1.5, the exponential result is approximately 4.48. To control the range of the adjustment coefficient, the upper limit is preset to 3.0, and the lower limit to 0.33. The exponential result is compared with the upper and lower limits: if the exponential result is greater than 3.0, it is taken as 3.0; if the exponential result is less than 0.33, it is taken as 0.33; otherwise, the exponential result itself is taken. The value after amplitude limiting is output as the adjustment coefficient for that side. According to the above rules, when the second derivative is positive, the intermediate quantity is positive, the exponential result is greater than 1, and the adjustment coefficient is greater than 1; when the second derivative is negative, the intermediate quantity is negative, the exponential result is less than 1, and the adjustment coefficient is less than 1.

[0030] For each node in the graph structure, calculate the original dot product score by combining its current node state feature vector (64-dimensional) with the node state feature vectors of each neighboring node. Specifically, multiply each component of the current node's feature vector with the corresponding component of each neighboring node's feature vector, and then sum all the products to obtain the original dot product score. For example, the dot product of two 64-dimensional vectors results in a scalar.

[0031] The adjusted score is obtained by multiplying the adjustment coefficient corresponding to each edge by the original dot product score of that edge. For example, if the original dot product score is 0.8 and the adjustment coefficient is 1.5, the adjusted score is 1.2. The adjusted score is obtained for each of the node's neighbors.

[0032] Normalize the adjusted scores of all neighbors. The normalization process is as follows: First, calculate and sum the natural exponent values ​​of the adjusted scores of all neighbors, using this sum as the denominator. Then, divide the natural exponent value of each neighbor's adjusted score by this denominator to obtain the normalized weight for that neighbor. Multiply the node state feature vector of each neighbor by its normalized weight, and then sum the weighted vectors of all neighbors element-wise to obtain the updated node state. Perform the above process on all nodes and output the updated states of all nodes.

[0033] In S3, the updated node state is concatenated with the vehicle's current position, remaining capacity, and list of unfinished tasks. The concatenated result is then input into a deep reinforcement learning policy network, which outputs the vehicle's path decision action for the next time step, specifically including: Retrieve the updated node states of all nodes, with each node state being a 64-dimensional floating-point vector. Simultaneously, obtain the real-time information of the current vehicle: the vehicle's current location coordinates are represented by longitude and latitude, e.g., 39.9042 degrees North latitude and 116.4074 degrees East longitude; the vehicle's remaining capacity is in kilograms, rounded to the nearest integer, e.g., 150 kilograms remaining; the unfinished task list contains multiple tasks, each including the corresponding customer location coordinates (longitude and latitude) and the required delivery time window (start time and end time, in seconds, calculated from midnight of the current day).

[0034] Calculate the spatial distance between the vehicle's current location and the geographical location represented by each node in the graph structure. Specifically, obtain the longitude and latitude of the nodes. Multiply the difference between the vehicle's longitude and the node's longitude by 111320 meters (corresponding to the distance per degree of longitude at the equator), then multiply by the cosine of the vehicle's latitude to obtain the east-west distance. Multiply the difference between the vehicle's latitude and the node's latitude by 110540 meters (corresponding to the average distance per degree of latitude) to obtain the north-south distance. Add the squares of the east-west distance and the squares of the north-south distance, then take the square root to obtain the Euclidean distance between them. Calculate the distances between the vehicle and all nodes sequentially, select the node with the smallest distance, and extract its updated state as the spatial anchor point state.

[0035] The vehicle's current position coordinates are mapped to a position embedding vector using a set of radial basis functions. Specifically, 200 reference points are uniformly selected in the road network. The coordinates of each reference point are pre-determined, and a width parameter of 500 meters is assigned to each reference point. For the vehicle's current position coordinates, its position is calculated relative to the first reference point. The Euclidean distance between the reference points is denoted as . Calculate the first according to the following formula. Radial basis function values ​​corresponding to each reference point: ; in, This represents an exponential function with the natural constant as its base. For the vehicle's current location and the number Euclidean distance between reference points For the first The width parameter of each reference point (taken as 500 meters). Divide the square of d_i by 2. Square the value, take the negative value, and then perform an exponential operation to obtain φ_i. For 200 reference points, 200 radial basis function values ​​are calculated sequentially. These values ​​are arranged in a fixed order according to the reference point numbers to form a 200-dimensional vector, which is the position embedding vector.

[0036] The spatial anchor state (64-dimensional vector) and the location embedding vector (200-dimensional vector) are added element-wise. Since the two have different dimensions, the spatial anchor state needs to be expanded to 200 dimensions first through a linear transformation layer. This linear transformation layer is a trainable 200x64 weight matrix. The 64-dimensional spatial anchor state is multiplied by the weight matrix to obtain the 200-dimensional transformed vector. Then, the transformed vector is added to each corresponding element of the location embedding vector to obtain the node state (200-dimensional) with fused location information.

[0037] The node states with fused location information, the vehicle's remaining capacity value, and the information from the incomplete task list are sequentially concatenated into a comprehensive feature vector. The specific order is as follows: first, arrange the 200 elements of the node states with fused location information in sequence; then add one element, the remaining capacity value (e.g., 150); then, for each task in the incomplete task list, sequentially add the start time of the task's time window (1 value), the end time of the time window (1 value), the longitude of the task location (1 value), and the latitude of the task location (1 value). Assuming there are K tasks in the incomplete task list, each task contributes 4 values, for a total of 4K elements. All elements are then concatenated in the above order to form a comprehensive feature vector with dimensions 200 + 1 + 4K. This comprehensive feature vector is the concatenation result described in step S3 and is used as input to the deep reinforcement learning policy network. The policy network is a fully connected network with two hidden layers (256 neurons in the first hidden layer and 128 neurons in the second hidden layer). The output layer corresponds to the priority scores of all possible actions, and the action with the highest score is selected as the path decision action for the vehicle in the next moment.

[0038] In S4, during the training phase, a dual-scale evolution constraint is introduced to ensure that the future spatiotemporal predictions of the implicit continuous spatiotemporal encoding output and the updated node state evolution maintain an exponential convergence relationship in the feature space. The output is an implicit expression that satisfies the consistency of congestion propagation dynamics, specifically including: The updated states of all nodes are obtained, with each node's state represented as a 64-dimensional floating-point vector. Simultaneously, the predicted traffic speeds for the next five time points (each time point spaced 1 second apart) after the current moment are obtained using implicit continuous spatiotemporal encoding. For each edge in the road network, the implicit continuous spatiotemporal encoding outputs five predicted speed values. These predicted speed values ​​for the next five time points are arranged in edge order and time order, forming a one-dimensional vector with a length equal to the number of edges multiplied by 5. This vector is then input into a linear mapping layer, which is a trainable weight matrix with 64 rows and 5 columns equal to the number of edges. Multiplying the one-dimensional vector by this matrix yields a 64-dimensional predicted feature vector.

[0039] The node state evolution direction vector is calculated as follows: Obtain the state vectors (64-dimensional) of all nodes at the current time step and the state vectors of all nodes at the previous time step (1 second ago). Subtract the previous time step vector from the current time step vector to obtain the state change of each node. Concatenate all the state change vectors into a long vector with a dimension equal to the number of nodes multiplied by 64. Then, pass this long vector through another linear mapping layer (64 rows and 64 columns multiplied by the number of nodes) to map it back to a 64-dimensional vector. This vector is the node state evolution direction vector.

[0040] Calculate the cosine distance between the predicted feature vector and the node state evolution direction vector. The cosine distance is calculated as follows: first, calculate the dot product of the two vectors (multiply corresponding elements and sum them); then, calculate the magnitudes of the two vectors (square roots of the sum of the squares of their respective elements); divide the dot product by the product of the two magnitudes to obtain the cosine similarity; subtract the cosine similarity from 1 to obtain the cosine distance. For example, if the dot product is 0.96 and both magnitudes are 1, the cosine similarity is 0.96 and the cosine distance is 0.04.

[0041] A predefined sequence of exponentially decaying target values ​​is used, which decreases exponentially with each time step. Specifically, for the t-th training step (t starting from 1), the target value is 0.1 multiplied by 0.995 raised to the power of t. The currently calculated cosine distance is subtracted from this target value to obtain the difference, and the square of the difference is used as the constraint loss. For example, if the current cosine distance is 0.04 and the target value is 0.05, the difference is -0.01, which squares to 0.0001.

[0042] Obtain the current training iteration step number, denoted as the current step. The initial decay coefficient is preset to 1.0, and the decay half-life is 1000 steps. Calculate the decay coefficient for the current step: divide the current step number by 1000 to obtain the ratio; then, using 0.5 as the base, perform an exponentiation on this ratio, i.e., calculate 0.5 raised to the power of (current step number divided by 1000) to obtain the decay coefficient. For example, when the current step number is 500, the ratio is 0.5, and the decay coefficient is 0.5 to the power of 0.5, approximately 0.707; when the current step number is 1000, the decay coefficient is 0.5; and when the current step number is 2000, the decay coefficient is 0.25.

[0043] Multiplying the aforementioned constraint loss by the decay coefficient yields the weighted constraint loss for the current step. Adding this weighted constraint loss to the original training loss of the deep reinforcement learning policy network yields the updated total training loss. The original training loss is the sum of the policy gradient loss and the value loss, where the policy gradient loss optimizes action selection, and the value loss optimizes state value estimation.

[0044] The updated total training loss is used for backpropagation. During backpropagation, starting from the total training loss, the gradient of each trainable parameter (including the parameters of the multilayer perceptron, embedding layer, linear mapping layer, and all parameters of the deep reinforcement learning policy network within the implicit continuous spatiotemporal encoding) is calculated sequentially, and the parameters are updated according to a preset learning rate of 0.0001. By repeating the above process in each training step, the constraint loss forces the cosine distance between the predicted feature vector and the node state evolution direction vector to gradually approach the exponentially decaying target value, thereby achieving exponential convergence of the distance between the two in the feature space. After 20,000 training steps, this convergence relationship is stably established, at which point the future spatiotemporal prediction output by the implicit continuous spatiotemporal encoding and the node state evolution exhibit congestion propagation dynamics consistency.

[0045] In S5, during online execution, the implicit continuous spatiotemporal code is continuously updated with real-time traffic flow data. The updated implicit continuous spatiotemporal code is then re-executed to output the updated node states and path decision actions, resulting in dynamically adjusted vehicle paths. Specifically, this includes: The decision-making cycle for online execution is set to 1 second. Within each decision cycle, real-time traffic speed observations for each road segment are collected over the past 3 seconds using induction coil detectors deployed beneath each road segment and floating vehicles equipped with GPS. For each road segment, the arithmetic mean of all observations within these 3 seconds is taken as the measured speed for that segment in the current decision cycle. Simultaneously, the predicted speed for that road segment, output by implicit continuous spatiotemporal coding for the current timestamp (accurate to 0.1 seconds), is obtained. The measured speed of the road segment is subtracted from the predicted speed to obtain the speed residual. This same operation is performed on all road segments to obtain a speed residual vector with a length equal to the total number of road segments.

[0046] The velocity residual vector is processed before gradient clipping. Specifically, it iterates through all elements in the velocity residual vector, takes the absolute value of each element, and finds the maximum value, which is recorded as the maximum absolute residual. A preset gradient clipping threshold of 0.5 meters per second is set. The maximum absolute residual is compared with this threshold: if the maximum absolute residual is greater than 0.5, the ratio of the maximum absolute residual to 0.5 is calculated, and then the preset learning rate (initially set to 0.001) is divided by this ratio to obtain the adjusted learning rate; if the maximum absolute residual is less than or equal to 0.5, the preset learning rate of 0.001 remains unchanged. For example, if the maximum absolute residual is 2.0 meters per second, the ratio is 4, and the adjusted learning rate is 0.001 divided by 4, which equals 0.00025.

[0047] Multiply each residual value in the velocity residual vector by the adjusted learning rate to obtain a parameter update vector with the same dimension as the velocity residual vector. For each trainable parameter within the implicit continuous spatiotemporal coding (including the weights and biases of the multilayer perceptron and the lookup table values ​​of the road segment identification embedding layer), calculate the partial derivative of that parameter with respect to the velocity residual according to the chain rule of backpropagation. Then multiply the parameter update vector by these partial derivatives to obtain the update amount for each parameter. Finally, subtract this update amount from the current value of each parameter to complete the single-step gradient descent update. For example, if the current value of a parameter is 0.5 and its corresponding update amount is 0.02, then it becomes 0.48 after the update.

[0048] The updated implicit continuous spatiotemporal encoding is reused in the current decision-making cycle. Specifically, the current timestamp and all road segment identifiers are input into the updated implicit continuous spatiotemporal encoding, and the predicted traffic speed for each road segment is output. The first and second derivatives of the speed are calculated using the finite difference method, in the same way as described above (the first derivative is obtained by subtracting the previous speed from the current speed and dividing by the sampling interval of 1 second; the second derivative is obtained by subtracting the previous first derivative from the current first derivative and dividing by 1 second). These speeds and derivatives are assigned as edge features of the graph structure, and the updated node states are output according to the method described in S2 (defining the differential attention kernel based on the second derivative and adjusting the aggregation weights of neighbor node information). Then, the path decision action for the vehicle at the next moment is output according to the method described in S3 (concatenating the updated node states with vehicle information and inputting it into the deep reinforcement learning policy network).

[0049] Obtain the complete path currently being executed by the vehicle. This path is a sequential sequence of node identifiers, including completed nodes, the current node, and unvisited nodes. Merge the newly output path decision action (i.e., the identifier of the next node to be visited) with the unvisited node sequence in the current vehicle's path. Specifically, search the unvisited node sequence for a node identical to the new decision action. If it exists, move it to the first position in the original sequence; otherwise, insert it at the beginning of the sequence. Replace the order of all nodes after the current node in the original path to form a new complete vehicle path. Output this new path to guide the vehicle's subsequent driving, thus achieving dynamic adjustment. Repeat the entire process every 1-second decision cycle.

[0050] The working principle of this invention is as follows: Historical traffic flow data of the road network is collected to construct an implicit continuous spatiotemporal code that takes road segment identifiers and continuous timestamps as inputs and predicts traffic speed as outputs. The predicted speed output by the code and its first and second derivatives are assigned as edge features of a graph structure. A differential attention kernel is defined based on the second derivative, and the aggregation weights of neighbor node information are adjusted through this kernel to output the updated node state. The updated node state is then concatenated with the vehicle's current position, remaining capacity, and list of unfinished tasks, and input into a deep reinforcement learning policy network to output the path decision action for the next time step. During the training phase, a dual-scale evolution constraint is introduced to ensure that the future spatiotemporal prediction of the implicit continuous spatiotemporal code and the evolution of the node state maintain an exponential convergence relationship in the feature space, obtaining an implicit expression consistent with the congestion propagation dynamics. During online execution, the speed residual is calculated using real-time traffic flow data, and a single-step gradient descent update is performed on the implicit continuous spatiotemporal code. The above node state update and path decision process is repeated to output a dynamically adjusted vehicle path.

[0051] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A dynamic vehicle path planning method based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Collect historical traffic flow data of each road segment in the road network, including road segment identification, continuous timestamps and corresponding traffic speeds, and construct an implicit continuous spatiotemporal code that uses road segment identification and continuous timestamps as inputs and outputs to predict traffic speeds; S2: Assign the predicted passage speed output by the implicit continuous spatiotemporal coding and its first and second derivatives with respect to time as edge features of the graph structure. Define the differential attention kernel based on the second derivative in the edge features. Adjust the weight when aggregating neighbor node information through the kernel and output the updated node state. S3: The updated node state is concatenated with the vehicle's current position, remaining capacity, and list of unfinished tasks. The concatenation result is input into the deep reinforcement learning policy network to output the vehicle's path decision action at the next moment. S4: During the training phase, a dual-scale evolution constraint is introduced to ensure that the future spatiotemporal predictions of the implicit continuous spatiotemporal coding output and the updated node state evolution maintain an exponential convergence relationship in the feature space, and output an implicit expression that satisfies the consistency of congestion propagation dynamics. S5: During online execution, the implicit continuous spatiotemporal code is continuously updated with real-time traffic flow data. The updated implicit continuous spatiotemporal code is then re-executed to output the updated node status and path decision actions, resulting in dynamically adjusted vehicle paths.

2. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 1, characterized in that, The implicit continuous spatiotemporal coding specifically includes: The continuous timestamps in the collected historical traffic flow data are mapped into high-dimensional temporal embedding vectors through a set of Fourier basis functions. The road segment identifiers are converted into road segment feature vectors through an embedding layer. The high-dimensional temporal embedding vectors and road segment feature vectors are concatenated and input into a multilayer perceptron. The training objective is to minimize the mean square error between the predicted traffic speed output by the multilayer perceptron and the corresponding collected traffic speed. After iterative optimization, an implicit continuous spatiotemporal code is obtained.

3. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 1, characterized in that, The updated node state output specifically includes: The absolute value of the second derivative on each edge is taken and converted into an adjustment coefficient through a monotonically increasing mapping function. When the second derivative is positive, the adjustment coefficient is greater than 1, and when it is negative, the adjustment coefficient is less than 1. The adjusted score is obtained by multiplying the adjustment coefficient by the original dot product score calculated from the node state features. The adjusted score is normalized, and the normalized result is used as the weight when aggregating neighbor node information. The weighted updated node state is then output.

4. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 3, characterized in that, The process of obtaining the adjustment coefficient is as follows: Obtain the absolute value of the second derivative and multiply it by the preset congestion propagation time constant to obtain the intermediate value; The intermediate quantity is used as the exponent of an exponential function with the natural constant as the base to perform exponential operations, and the exponential result is obtained. The index result is compared with the preset upper and lower limits of the coefficient. If the index result exceeds the upper limit, the upper limit is used; if it is lower than the lower limit, the lower limit is used; otherwise, the index result itself is used. The index result after amplitude limiting is output as the adjustment coefficient.

5. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 1, characterized in that, The splicing result is as follows: First, calculate the spatial distance between the vehicle's current position and each node, select the node with the smallest distance, and use the state of the node as the spatial anchor point state. The vehicle's current position coordinates are mapped to a position embedding vector using a set of radial basis functions; The spatial anchor point state is added element-wise to the position embedding vector to obtain the node state with fused position information; The node status, remaining capacity value, and time windows and location coordinates of each task in the unfinished task list are arranged in order and then concatenated to output the spliced ​​comprehensive feature vector.

6. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 5, characterized in that, The output process of the position embedding vector is as follows: First, select multiple reference points in the road network, and each reference point is assigned a preset width parameter; Calculate the Euclidean distance between the vehicle's current position coordinates and each reference point; Divide the square of the Euclidean distance by the square of the corresponding width parameter, take the negative value, and then perform an exponential operation with the natural constant as the base to obtain the radial basis function value of the corresponding reference point. Arrange the radial basis function values ​​corresponding to all reference points in a fixed order and output the position embedding vector.

7. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 1, characterized in that, Maintaining the exponential convergence relationship specifically includes: The future spatiotemporal prediction output by implicit continuous spatiotemporal coding is mapped to the same feature dimension as the updated node state to obtain the prediction feature vector; Calculate the cosine distance between the predicted feature vector and the current node's state evolution direction vector; The squared difference between the cosine distance and the preset exponential decay target value is used as the constraint loss; The constraint loss is multiplied by a weight that decays exponentially with time step and added to the total training loss of the deep reinforcement learning policy network. This forces the feature space distance between the two networks to converge at an exponential rate through backpropagation.

8. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 7, characterized in that, The process of forcing the feature space distance between the two to converge at an exponential rate through backpropagation specifically includes: Get the current training iteration number; Preset an initial decay coefficient and a decay half-life step, calculate the exponent with base 1 / 2, where the exponent is the current training iteration step divided by the decay half-life step, to obtain the decay coefficient for the current step; Multiply the constraint loss by the decay factor to obtain the weighted constraint loss for the current step. The weighted constraint loss is added to the original training loss of the deep reinforcement learning policy network to form the updated total training loss, which is then used for backpropagation.

9. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 1, characterized in that, The dynamically adjusted vehicle path output specifically includes: Within each decision cycle, speed observations of each road segment are collected within the last three seconds. The difference between the observed values ​​and the prediction output of the implicit continuous spatiotemporal coding at the current timestamp is calculated to obtain the speed residual. After multiplying the velocity residual by a preset learning rate, backpropagation is performed to update the implicitly continuous spatiotemporally encoded internal parameters using single-step gradient descent. The updated implicit continuous spatiotemporal code is re-outputted to predict the travel speed and its first and second derivatives. The node state update and path decision actions are then executed sequentially. The newly output path decision action is time-sequentially merged with the sequence of unvisited nodes in the current vehicle path, and the order of subsequent nodes in the original path is replaced to output the dynamically adjusted complete vehicle path.

10. The dynamic vehicle path planning method based on deep reinforcement learning according to claim 9, characterized in that, The backpropagation process, which multiplies the velocity residual by a preset learning rate and then performs single-step gradient descent updates on the implicitly continuous spatiotemporal encoded internal parameters, specifically includes: Obtain the absolute value of the speed residual for each road segment within the current decision-making period, and calculate its maximum value; The maximum value is compared with the preset gradient clipping threshold. If the maximum value exceeds the threshold, the preset learning rate is divided by the ratio of the maximum value to the threshold to obtain the adjusted learning rate. If the learning rate is not exceeded, the preset learning rate will remain unchanged. Multiply the velocity residual by the adjusted learning rate to obtain the parameter update amount; The parameter update amount is subtracted layer by layer along the backpropagation path from the internal parameters of the implicit continuous spatiotemporal encoding to complete the single-step update.

Images