Time window constraint path modeling and optimizing method based on graph reinforcement learning

Abstract

The invention belongs to the technical field of operation planning optimization and artificial intelligence, provides a time window constraint path modeling and optimization method based on graph reinforcement learning, abandons a popular encoder-decoder architecture, and designs a graph neural network modeling mode for visually expressing an MDP sequence process. Secondly, designing a strategy network which takes the graph neural network as a main body and is based on context information reasoning; according to the method, edge features are embedded into an attention mechanism through a FiLM affine technology, and a graph attention convolutional network for effectively solving a path planning problem is designed. In addition, the invention provides an integer programming model which fuses multiple reasoning information of the network and disrupts and reconstructs sub-paths among a plurality of solutions, and by approximately solving the integer programming model, on the premise of considering time efficiency, the difference between the reasoning solution and the optimal solution is effectively shortened.

Description

Technical Field

[0001] This invention belongs to the field of operations research and artificial intelligence technology, specifically relating to a method for time window constraint path modeling and optimization based on graph reinforcement learning. Background Technology

[0002] Time-constrained path decision-making is crucial in smart logistics and operations management. While traditional heuristic algorithms can achieve better solutions in a shorter time compared to exact algorithms, they still rely on a complete, zero-based solution process for new problems, making it difficult to utilize prior or intermediate results from past solutions. In recent years, reinforcement learning-based methods have gradually emerged as a new paradigm for solving time-constrained path problems due to their powerful capabilities in sequence decision-making. These methods utilize data-driven learning path generation strategies, effectively avoiding the reliance of traditional algorithms on massive amounts of domain knowledge and manually designed heuristic rules, and demonstrating significant generalization advantages on similar new problems. However, designing a network architecture with high expressive power and efficient training / inference optimization methods while strictly satisfying complex spatiotemporal constraints, and ensuring robust transferability and generalization capabilities when facing larger-scale instances with significantly expanded node sizes or changing task distributions, remains a significant challenge. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a time-window-constrained path modeling and optimization method based on graph reinforcement learning, thereby resolving the issues in the prior art. The technical solution adopted by this invention is as follows: A time-window-constrained path modeling and optimization method based on graph reinforcement learning includes: Step 1: For the time-window-constrained path planning problem, construct a Markov decision process model based on dynamic graph structure evolution; Step 2: Construct a graph policy network for reasoning based on contextual information; Step 3: Use the preference optimization algorithm to train and update the graph policy network; Step 4: Implement path optimization by fusing multiple sub-paths based on the multi-sub-path reconstruction model.

[0004] Furthermore, in step 1, the Markov decision process model adopts the dynamic programming principle of no aftereffect: through state... Real-time encoding node access status and resource constraints; From the source to the remittance point The path generation process divides the states into unrelated contexts. And the time-varying diagram below To make the decision to complete the current action; Background information above Between the predecessor node and the candidate node Cost matrix, where The number of candidate nodes; The following time-varying diagram After executing node decisions and considering node reachability, then... Evolutionary; Node selection decisions depend only on the current state. The independent background texts above With the time-varying diagram below .

[0005] Furthermore, in step 2, the graph policy network includes processing the static information above. The MLP architecture, and the time-varying graph below. Multi-layer graph attention convolution.

[0006] Furthermore, step 2 includes: Step 2.1, extract the static information from the above text. MLP architecture: Regarding the initial background information above Then the new static information above pass Transformation: ; in These are the weight matrices for the first and second layers, respectively; It is a linear activation function; These are the bias vectors for the first and second layers, respectively; Step 2.2, examine the time-varying diagram below. Multi-layer graph attention convolution: For the underlying potential information, forward reasoning is performed on the progressively expanding reachable time-varying graph. This is achieved through attention-based graph convolution, given a graph... With node features First, calculate the feature similarity between node pairs: ; in For nodes and eigenvectors; for Projection matrix; for Projection matrix; for The dimension of a vector; Then, using binary relations... right conduct Affine modulation incorporates the implicit edge structure information of the entire graph into the attention mechanism: ; ; ; in For learnable scaling parameters; These are learnable offset parameters; To dynamically calculate based on the existence of edges factor; To dynamically calculate based on the existence of edges factor; Normalized edge weights through Given: ; in For nodes For nodes attention; The modulated attention score; And perform message aggregation: ; in This is the normalized attention weight matrix; The node feature matrix; The projection matrix is ​​a value. For the multi-head scenario, the final output is: ; in For the first Attention weight matrix for size; For the first The projection matrix of the values ​​of each head; To output the projection matrix; Step 2.3, Message aggregation decision: For the candidate node set Take its corresponding sequence features Node features And stitch together the pooled features of the entire image. ; in The feature is a pooled feature across the entire image; For nodes The graph output features; The comprehensive representation of the candidate nodes is obtained: ; in For nodes Comprehensive feature representation; features For nodes The output features above; MLP Prediction: ; in The feature matrix of all candidate nodes; Finally passed The overall selection probability distribution of the candidate set is obtained: ; in Candidate nodes The probability of selection; For nodes The original score.

[0007] Furthermore, step 3 includes: setting Represents the policy distribution parameterized by a graph neural network, where This represents a complete solution. Let it be the value of its objective function; set up Let the score function be calculated by the policy network, then the trajectory Superior The probability is defined as: ; in for The function; this model transforms the comparison between trajectories into a binary classification task, discriminating... Is it superior to ; Its optimization objective is to minimize the log-likelihood loss: ; in express Superior Otherwise Its theoretical gradient form is:

[0008] Combination The expected value is approximated by the empirical average: ; in: ; For input Lower trajectory The objective function value.

[0009] Furthermore, in step 4, the multi-sub-path reconstruction model integrates multi-sub-path information by assigning customer nodes to designated sub-path services and simultaneously considering the removal and retention operations performed on the same customer node served by multiple enabled sub-paths, thereby optimizing the vehicle scheduling scheme; where: Objective function: ; Constraints on customer point allocation: ; ; Customer point assignment and its binding to the corresponding sub-path location: ; For chain-like jump edges and degree conservation: ; ; ; ; Paths that are not enabled are prohibited from using any jump edges: ; in, For the candidate original path set, each sub-path sequence; Gathering customers; For distance; For customer location aggregation; For the complete set of locations; To enable or disable the original subpath ; For customers Is it assigned to a subpath? Finish; For path Location Should it be retained? For path Upper Nodes at various locations; For path Upper Nodes at various locations; To remove in the path In the middle, from the position Jump to the next save point .

[0010] Furthermore, in step 4, an approximate solution method is used to solve the multi-subpath reconstruction model: first, by solving the path coverage nodes, that is, by solving the model first: The model selects the fewest quantum paths that meet the coverage requirements, and then determines which path's repeating nodes will reduce costs.

[0011] The present invention has the following beneficial effects: This invention collects raw data from the foundation pit in real time and constructs a predictive model for analysis, thereby providing suggested measures to achieve real-time control of the axial force of the steel supports. This solves the problem of the inability to promptly compensate for prestress loss in the steel supports, and significantly reduces the number of times the axial force of the steel supports needs to be manually adjusted in actual engineering projects. It also reduces errors from manual operation, achieving higher accuracy, and minimizes the risks associated with manual operation, thus providing a safety guarantee for foundation pit engineering. Attached Figure Description

[0012] Figure 1 This is a three-dimensional layout diagram of the system provided in the embodiment of the present invention; Figure 2 This is a diagram showing the internal structure of the support head assembly in an embodiment of the present invention; Figure 3 This is a diagram showing the internal structure of the CNC hydraulic device in an embodiment of the present invention; Figure 4 This is a system flowchart provided in an embodiment of the present invention; Figure 5 This is a flowchart of the structural safety intelligent prediction layer provided in the embodiments of the present invention; Figure 6 This is a flowchart of the security early warning application interaction layer provided in this embodiment of the invention. Detailed Implementation

[0013] The following will be based on embodiments of the present invention. Figures 1-6 The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0014] To address the problems in existing technologies, this invention designs a time-window-constrained path modeling and optimization method based on graph reinforcement learning. First, most existing studies are based on encoder-decoder network architectures, which rely on the encoder's perception of global features but fail to utilize the process information of graph structure changes caused by preceding actions. This invention abandons the popular encoder-decoder architecture and designs a graph neural network modeling method that intuitively represents the MDP sequence process. Second, a policy network based on graph neural networks and contextual information reasoning is designed. To address the problem of insufficient global perception in general graph neural networks, this invention embeds edge features into the attention mechanism using the FiLM affine technique, designing a graph attention convolutional network that effectively solves the path planning problem. To address the problem of large reward variance in cross-sample learning, this paper will modify the traditional advantage function... The optimized reconstruction is a comparison of paired action preferences. This mechanism makes policy updates focus more on relative advantages rather than absolute reward values, significantly improving generalization to large-scale heterogeneous instances. Finally, this invention proposes an integer programming model that integrates multiple inference information from the network. By shuffling and recombining the sub-paths between multiple solutions, and by approximating the solution to this integer programming model, the gap between the inferred solution and the optimal solution is effectively shortened while maintaining time efficiency.

[0015] This invention takes the Vehicle Routing Problem with Time Windows (VRPTW) as an example and proposes a method for modeling and optimizing time-window constrained paths based on graph reinforcement learning, specifically including the following steps: Step 1: For the time-window-constrained path planning problem, construct a Markov decision process model based on dynamic graph structure evolution; This invention models time-window-constrained path planning as a Markov decision process based on dynamic graph structure evolution, such as... Figure 1 As shown. This model follows the principle of no aftereffect in dynamic programming: through states... Real-time encoding of key information such as node access status and resource constraints; node selection decisions rely solely on the current state. The two independent perspectives in the text – given the background above With potential time-varying graph information .like Figure 1 As shown.

[0016] Figure 1 Showing from the source to the remittance point The path generation process divides the states into unrelated contexts. And the time-varying diagram below To make the decision to complete the current action. It is determined by the relationship between the predecessor node and the candidate node. Cost matrix (where (This refers to the number of candidate nodes). It is not a single value, but rather a combination of relevant cost characteristics from the predecessor node to the candidate node. Dimensional vectors are typically related to distance, fuel consumption, etc. (e.g., the distance from the previous node to the candidate node, the distance of the candidate node from the endpoint, etc.). This is based on the decision made by the execution node, taking into account the node's reachability. It comes from changes. Its methods are described as follows: 1. Input initial state ,Right now and .

[0017] 2. According to and Make node decisions from candidate nodes, and then... Add the preceding path.

[0018] 3. Based on reachability constraints (time, return to sink, etc.), dynamically prune the graph, removing unreachable edges (red dashed lines) and isolated nodes (yellow nodes), and update the graph to obtain... Simultaneously, calculate the transfer cost matrix from the current node to each candidate node. .

[0019] 4. Repeat steps 1, 2, and 3 until the complete path from s to t is obtained. The solution to VRPTW can be seen as consisting of multiple Elementary Shortest Path Problems with Resource Constraints (ESPPRCs), and therefore can be based on the above MDP sequence decision process.

[0020] Step 2: Construct a graph policy network for reasoning based on contextual information; This invention directly adopts "static information above" The MLP architecture + the time-varying graph below The lightweight architecture of "multi-layer graph attention convolution" abandons the traditional encoder-decoder architecture. Its overall architecture is as follows: Figure 2 As shown, the overall process is as follows: 1. Clearly define the input status. The above are the background information. With the time-varying diagram below .

[0021] 2. Input the initial background information into the MLP architecture, and output the static information through two layers of linear transformation. .

[0022] 3. The time-varying graph described below extracts features through multi-layer graph attention convolution to obtain candidate node features. and global graph features

[0023] 4. , , The three elements are combined to obtain a comprehensive representation of the candidate nodes. Then through MLP and The probability distribution of candidate nodes is obtained through processing.

[0024] Step 2 includes: Step 2.1, extract the static information from the above text. MLP architecture: Regarding the initial background information above Then the new static information above pass Transformation: ; in These are the weight matrices for the first and second layers, respectively; It is a linear activation function; These are the bias vectors for the first and second layers, respectively; Step 2.2, examine the time-varying diagram below. Multi-layer graph attention convolution: For the underlying potential information, forward reasoning is performed on a progressively expanding reachable time-varying graph. However, ordinary graph convolution methods lack a global perspective. Therefore, this paper designs an attention-based graph convolution method, which, for a given graph... With node features First, calculate the feature similarity between node pairs: ;

[0025] in For nodes and eigenvectors; for Projection matrix; for Projection matrix; for The dimension of a vector; Then, using binary relations... right conduct Affine modulation incorporates the implicit edge structure information of the entire graph into the attention mechanism in this way: ; in for The parameter corresponds to whether the edge exists. Indicates existence; for The parameter corresponds to whether the edge exists. Indicates existence; To dynamically calculate based on the existence of edges factor; To dynamically calculate based on the existence of edges factor; Normalized edge weights Given: ; in For nodes For nodes attention; The modulated attention score; And aggregate messages accordingly: ; in This is the normalized attention weight matrix; The node feature matrix; The projection matrix is ​​a value. For multi-head scenarios (each head has independent) The final output is: ; in For the first Attention weight matrix for size; For the first The projection matrix of the values ​​of each head; To output the projection matrix; Step 2.3, Message aggregation decision: The final context features are obtained through multilayer perceptron and multilayer graph convolution. Graph node features For the candidate node set Take its corresponding sequence features Node features And stitch together the pooled features of the entire image. ; in The feature is a pooled feature across the entire image; For nodes The graph output features; The comprehensive representation of the candidate nodes is obtained: ; in For nodes Comprehensive feature representation; features For nodes The output features above; MLP Prediction: ; in The feature matrix of all candidate nodes; Finally passed The overall selection probability distribution of the candidate set is obtained: ; in Candidate nodes The probability of selection; For nodes The original score.

[0026] Step 3: Use the preference optimization algorithm to train and update the graph policy network; To improve the learning efficiency of graph policy networks in combinatorial optimization problems, this invention employs the Preference Optimization (PO) algorithm for training. Unlike traditional reinforcement learning based on Reinforcement Learning, PO does not rely on numerical reward signals but guides policy updates through pairwise preference comparisons, thereby maintaining stable convergence even under noisy reward distributions; specifically: set up Represents the policy distribution parameterized by a graph neural network, where This represents a complete solution (or trajectory). Let it be the value of its objective function. The traditional REINFORCE method utilizes the dominant function. = Perform gradient updates: ; This method relies on the absolute value of the reward and is susceptible to scaling variations and high variance noise.

[0027] To overcome the above problems, PO introduced the Bradley-Terry (BT) model to model the preference relationship between trajectories: set up Let the score function be calculated by the policy network, then the trajectory Superior The probability is defined as: ; in for The function transforms the comparison between trajectories into a binary classification task, determining whether " Is it superior to ".

[0028] Its optimization objective is to minimize the log-likelihood loss: ; in express Superior Otherwise Its theoretical gradient form is:

[0029] Further integration The expected value is approximated by the empirical average: ; in: ; For input Next path The objective function value.

[0030] For the complete PO training process, please refer to Figure 3 pseudocode.

[0031] 1. First, define the parameters, including the training set. Number of training iterations Batch size for each iteration Batch size Learning rate Preferred temperature coefficient Graph Strategy Networks Objective function reward and output the trained model parameters .

[0032] 2. Iterative training begins, from 1 to... Execute each iteration sequentially, within each iteration according to 1 to... Advance training. In each iteration, from the training set... Uniform extraction 1 instance, composed of Training data .

[0033] 3. For each instance Based on the current strategy Independent sampling Solution of a complete trajectory This forms a set of trajectories. The objective function reward for each trajectory is calculated, and for the same instance... The trajectories are combined in pairs, if Then, preference pairs are constructed.

[0034] 4. Based on The model calculates the log-likelihood loss gradient for each preference pair, and then approximates the gradient for the policy update using a formula. .use Optimizer, based on learning rate Apply the calculated gradient ,renew To minimize preference loss, the probability of the strategy generating a better trajectory is increased.

[0035] 5. Until all are completed After training, the final optimized model parameters are output. Training is over.

[0036] Step 4: Implement path optimization by fusing multiple sub-paths based on the multi-sub-path reconstruction model; For the Vehicle Routing Problem with Time Windows (VRPTW), the complete path is composed of multiple sub-paths, each typically independent and satisfying time and load constraints. Therefore, a less-than-ideal inference result for all sub-paths does not necessarily mean the inference result for a single path is bad. Through multiple inferences, sufficient sub-path information can be obtained. By shuffling and reconstructing these sub-paths, a better solution can be found, which can be summarized as an integer programming model.

[0037] Multi-Path Reconstruction Model (MPR): MPR integrates multi-path information by assigning customer nodes to designated sub-path services and considering the removal and retention operations performed on the same customer node served by multiple enabled sub-paths, thereby producing a better vehicle scheduling solution.

[0038] ; ; In the above equations, Equation 4.1 is the objective function; Equations 4.2 and 4.3 are constraints on customer point allocation, requiring that each customer point can be assigned to one and only one enabled sub-path; Equation 4.4 ensures the binding of customer point allocation and its position in the corresponding sub-path; Equations 4.5, 4.6, 4.7, and 4.8 are based on the degree conservation principle (i.e., the inflow and outflow of any node are strictly equal to its retention state), constraining vehicles to flow out from the starting point, maintain flow balance at intermediate nodes through chain-like jump edges (i.e., skipping the removed nodes and directly connecting the preceding and following retained points), and finally merge into the destination; Equation 4.9 prohibits the use of any chain-like jump edges on disabled paths.

[0039] The parameters in the formula are referenced in Table 1: Table 1: Symbol Explanation

[0040] Approximate Solution of Multi-Sub-Path Reconstruction Model: MPR is essentially a set covering problem, and it also involves coupled constraints of node preservation and skip edges within the path, as well as path and node assignment, making it a typical NP-hard problem. Therefore, its computational difficulty increases significantly with the number of sub-paths. To address this challenge, this invention employs an approximate solution method to solve the model. Specifically, it first solves the model by covering the path nodes: This model selects the fewest quantum paths that meet the coverage requirements, and then determines which path's repeating nodes should be retained to minimize costs. This invention demonstrates the specific process of solving VRPTW (Virtual Time Window Constraint Path) using graph reinforcement learning through experiments. Specifically, it includes the sources of training and test set data, feature selection and network architecture, training environment, and parameter settings.

[0041] 1. Partitioning of the dataset, training solutions, and test set: The experiments in this invention are based on the classic Solomon dataset from VRPTW. The dataset contains 56 instances, divided into three main types: concentrated customer location distribution (C class), random distribution (R class), and mixed distribution (RC class). Each class is further divided into two groups based on the width of the time window and the vehicle load (e.g., C1 / C2, R1 / R2, RC1 / RC2), thus reflecting different service constraints and route complexities. Each instance in the Solomon dataset includes 100 customer points, and each customer has fixed service time, demand, and time window constraints.

[0042] This paper trains the network on a small-scale test case and tests it on a large-scale test case. The training data is generated from 56 standard instances of the Solomon dataset. Each instance contains 100 customers (numbered 1–100). Using a sample length of 50 customers, two sampling strategies are employed: 1) continuously slides across the entire customer sequence with a window length of 50 and a stride of 1, resulting in 51 samples; 2) slides across the first half (1–50) and the second half (51–100) with windows of length 25 (26 variations each), and then concatenates these windows in their original order to form 50 customer samples, for a total of 676 samples. The two strategies combined yield 726 training samples per instance (one duplicated), for a total of 40,656 training samples across the entire dataset. The testing phase uses 56 instances containing all 100 customer nodes to evaluate the model's generalization performance on larger-scale problems.

[0043] 2. Feature selection and network architecture: The above information of the sample It includes two-dimensional features: the Euclidean distance between the predecessor node and the candidate node, and the distance between the candidate node and the depot; time-varying graph. Each node feature contains The system has seven dimensions: left time window, right time window, service time, demand, and time from the preceding node to the current node.

[0044] The policy network mainly consists of three parts: (as described in the background above) The extraction is performed using a 3-layer MLP with an input layer dimension of 2 and a hidden layer dimension of 64; for time-varying graphs First, the node features are uniformly mapped from 7 dimensions to 64 dimensions, and then 3 layers are used. An affine graph convolutional architecture embedding edge structures (64 hidden layer dimensions, 3 attention heads; the code does not require the hidden layer dimension to be divisible by the number of attention heads, so it rounds down), with residual connections between each convolutional layer, followed by... The function normalizes node features; the message fusion module is a 3-layer MLP with an input layer dimension of 64*3 and a hidden layer dimension of 64. Activation functions throughout the entire network architecture use... .

[0045] 3. Training environment: The network model of this invention is in Implemented within a deep learning framework, the network model uses the Adam optimizer. The approximate solution for the multi-subpath reconstruction model is based on the commercial solver Gurobi.

[0046] 4. Important parameter settings: During the testing phase, the learning rate was set to 0.0001, and training lasted for 100 epochs. Each epoch contained 100 batches, and each batch contained 32 samples randomly drawn from 40,656 training samples in a uniform distribution. For the same case, the graph neural network performed 15 independent samplings to generate results and construct corresponding preference pairs, which were then used for preference learning and stability evaluation of the model. The preference temperature coefficient in the PO algorithm... Set it to 0.03.

[0047] Experimental results: 1. Model training: like Figure 4 This invention compares two traditional heuristic algorithms—genetic algorithm and simulated annealing algorithm—and three types of reinforcement learning methods based on encoder-decoder network architectures with the graph-based reinforcement learning method proposed in this invention. Table 2 shows their overall performance on 56 Solomon cases.

[0048] Table 2: Performance of different algorithms on 56 Solomon datasets

[0049] The GA population size was set to 200, with 6000 iterations and a total of 1,200,000 individual evaluations; the SA evaluation count was set to the same as GA. Although increasing individual evaluations may still yield better solutions, it requires more time. Reinforcement learning-based neural network methods have different inference modes: Greedy selects the node with the highest probability to generate the path during decoding; Aug8 performs eight rotation transformations on the coordinates, then performs independent Greedy decoding on each of the eight transformations, finally selecting the best solution as the result; Sample performs multiple samplings according to the network strategy and selects the best result (here, 100 samplings are set); Recompose approximates the Sample model with a multi-subpath reconstruction.

[0050] Experimental results show that the graph reinforcement learning method designed in this invention achieves the best performance in terms of mean and variance of the gap across 56 cases. Firstly, in any inference mode, the method of this invention outperforms both GA and SA; in the Recompose mode, the gap difference with the two methods is even greater than 10%.

[0051] In a comparison of four reinforcement learning methods: in Greedy mode, our method has an average gap of 20.89%; in Aug8 mode, the average gap decreases to 16.69%; in the Sample model, due to the larger number of samples, the average gap is further reduced to 14.09%; and in the Recompose model, which integrates multiple sub-path information, the gap is reduced by more than 4% with only an increase of 0.25 seconds. In all four different inference methods, our method outperforms other baseline methods. Regarding time complexity, graph convolutional feature extraction and decoding are slightly more time-consuming than the standard encoder-decoder, but this slight increase in time cost significantly improves model accuracy.

[0052] In the spatial feature dimension ( Figure 5 In this invention, the method achieves optimal performance across three regional structures: C, RC, and R, exhibiting a characteristic of regular variation as regional structure weakens: In C (strong clustering), the graph learning method fully utilizes spatial structure, demonstrating the lowest mean / median, narrowest bin, and fewest outliers; it maintains its leading position in RC, with a compact distribution and stable results; even in R (random), it remains optimal with minimal fluctuations, indicating robustness even in the absence of explicit structure. Therefore, the more significant the regional characteristics, the stronger the structure utilization capability of the graph learning method, and the more obvious its advantages.

[0053] In the time feature dimension ( Figure 6 In the first series (narrow time window, strong constraints) cases, other methods showed significant fluctuations (wider distribution, longer tail), while the method of this invention still maintained the lowest central statistic and a more compact distribution, showing a prominent advantage. In the second series (wide time window) cases, the problems were easier overall, but the graph learning method still maintained a stable lead, showing a lower gap and fewer outliers.

[0054] In summary, the stronger the spatial or temporal structure, the more significant the advantages of this invention; it remains optimal and robust even when the structure is weakened.

[0055] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, alterations, alterations, or substitutions made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for time-window constrained path modeling and optimization based on graph reinforcement learning, characterized in that, include: Step 1: For the time-window-constrained path planning problem, construct a Markov decision process model based on dynamic graph structure evolution; Step 2: Construct a graph policy network for reasoning based on contextual information; Step 3: Use the preference optimization algorithm to train and update the graph policy network; Step 4: Implement path optimization by fusing multiple sub-paths based on the multi-sub-path reconstruction model.

2. The method for time window-constrained path modeling and optimization based on graph reinforcement learning according to claim 1, characterized in that, In step 1, the Markov decision process model adopts the principle of no aftereffect in dynamic programming: through states Real-time encoding node access status and resource constraints; From the source to the remittance point The path generation process divides the states into unrelated contexts. And the time-varying diagram below To make the decision to complete the current action; Background information above Between the predecessor node and the candidate node Cost matrix, where The number of candidate nodes; The following time-varying diagram After executing node decisions and considering node reachability, then... Evolutionary; Node selection decisions depend only on the current state. The independent background texts above With the time-varying diagram below .

3. The method for time window-constrained path modeling and optimization based on graph reinforcement learning according to claim 1, characterized in that, In step 2, the graph policy network includes processing the static information above. The MLP architecture, and the time-varying graph below. Multi-layer graph attention convolution.

4. The method for time window constraint path modeling and optimization based on graph reinforcement learning according to claim 3, characterized in that, Step 2 includes: Step 2.1, extract the static information from the above text. MLP architecture: Regarding the initial background information above Then the new static information above pass Transformation: ； in These are the weight matrices for the first and second layers, respectively; It is a linear activation function; These are the bias vectors for the first and second layers, respectively; Step 2.2, examine the time-varying diagram below. Multi-layer graph attention convolution: For the underlying potential information, forward reasoning is performed on the progressively expanding reachable time-varying graph. This is achieved through attention-based graph convolution, given a graph... With node features First, calculate the feature similarity between node pairs: ； in For nodes and eigenvectors; for Projection matrix; for Projection matrix; for The dimension of a vector; Then, using binary relations... right conduct Affine modulation incorporates the implicit edge structure information of the entire graph into the attention mechanism: ； ； ； in For learnable scaling parameters; These are learnable offset parameters; To dynamically calculate based on the existence of edges factor; To dynamically calculate based on the existence of edges factor; Normalized edge weights through Given: ； in For nodes For nodes attention; The modulated attention score; And perform message aggregation: ； in This is the normalized attention weight matrix; The node feature matrix; The projection matrix is ​​a value. For the multi-head scenario, the final output is: ； in For the first Attention weight matrix for size; For the first The projection matrix of the values ​​of each head; To output the projection matrix; Step 2.3, Message aggregation decision: For the candidate node set Take its corresponding sequence features Node features And stitch together the pooled features of the entire image. ； in The feature is a pooled feature across the entire image; For nodes The graph output features; The comprehensive representation of the candidate nodes is obtained: ； in For nodes Comprehensive feature representation; features For nodes The output features above; MLP Prediction: ； in The feature matrix of all candidate nodes; Finally passed The overall selection probability distribution of the candidate set is obtained: ； in Candidate nodes The probability of selection; For nodes The original score.

5. The method for time window-constrained path modeling and optimization based on graph reinforcement learning according to claim 1, characterized in that, Step 3 includes: setting Represents the policy distribution parameterized by a graph neural network, where This represents a complete solution. Let it be the value of its objective function; set up Let the score function be calculated by the policy network, then the trajectory Superior The probability is defined as: ； in for The function; this model transforms the comparison between trajectories into a binary classification task, discriminating... Is it superior to ; Its optimization objective is to minimize the log-likelihood loss: ； in express Superior Otherwise Its theoretical gradient form is: ； Combination The expected value is approximated by the empirical average: ； in: ； For input Lower trajectory The objective function value.

6. The method for time window-constrained path modeling and optimization based on graph reinforcement learning according to claim 1, characterized in that, In step 4, the multi-sub-path reconstruction model integrates multi-sub-path information by assigning customer nodes to designated sub-path services and considering the removal and retention operations performed on the same customer node served by multiple enabled sub-paths, thereby optimizing the vehicle scheduling scheme; where: Objective function: ； Constraints on customer point allocation: ； ； Customer point assignment and its binding to the corresponding sub-path location: ； For chain-like jump edges and degree conservation: ； ； ； ； Paths that are not enabled are prohibited from using any jump edges: ； in, For the candidate original path set, each sub-path sequence; Gathering customers; For distance; For customer location aggregation; For the complete set of locations; To enable or disable the original subpath ; For customers Is it assigned to a subpath? Finish; For path Location Should it be retained? For path Upper Nodes at various locations; For path Upper Nodes at various locations; To remove in the path In the middle, from the position Jump to the next save point .

7. The method for time window-constrained path modeling and optimization based on graph reinforcement learning according to claim 6, characterized in that, In step 4, an approximate solution method is used to solve the multi-subpath reconstruction model: first, the model is solved by addressing the path coverage nodes. The model selects the fewest quantum paths that meet the coverage requirements, and then determines which path's repeating nodes will reduce costs.

Images