A Heterogeneous Rescue and Path Planning Method Based on Reinforcement Learning-Assisted Iterative Greedy Approach

Abstract

This invention belongs to the field of resource path planning technology, and particularly relates to a heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approaches. The method first loads rescue data to generate initial heterogeneous resource path schemes, and constructs a graph embedding state representation through an ensemble attention pooling mechanism. Relying on a hierarchical Dueling Double DQN architecture, the upper-layer network selects destruction and reconstruction operators and destruction ratios, executes operations to generate intermediate solutions, and then the lower-layer conditional network combines the upper-layer actions and states to refine and optimize the solution using local search operators to obtain a new solution. A two-stage credit allocation mechanism is used to calculate rewards to train network parameters, and a simulated annealing strategy based on the normalized Metropolis criterion is used to determine whether to accept the new solution. After iterating to the termination condition, the rescue path scheme with the optimal overall cost is output. This invention can optimize multiple objectives such as operating costs and expected mortality costs, effectively improving the efficiency of joint scheduling of heterogeneous rescue resources and adapting to actual disaster emergency rescue scenarios.

Description

Technical Field

[0001] This invention belongs to the field of heterogeneous resource path planning and task allocation technology in disaster relief and emergency logistics, and particularly relates to a heterogeneous relief and path planning method based on reinforcement learning-assisted iterative greedy approach. Background Technology

[0002] With the intensification of global climate change and the increasing frequency of extreme natural disasters, the efficiency and scientific nature of large-scale emergency rescue operations have become crucial to the safety of people's lives and property and social stability. Rapid response and efficient resource delivery within the "golden 72 hours" after a disaster are key to minimizing casualties and losses. In such highly dynamic and constrained complex scenarios, rescue missions typically involve multiple objectives, including the transfer of the injured, the distribution of medical supplies, and the dispatch of professional personnel, requiring the coordination of various heterogeneous transportation tools such as land vehicles and helicopters. This problem of planning joint emergency logistics routes involving multiple tool types and multiple optimization objectives has become a cutting-edge challenge at the intersection of operations research, artificial intelligence, and emergency management.

[0003] In the field of emergency rescue route planning, traditional methods mainly focus on exact algorithms and metaheuristic algorithms. While exact algorithms (such as branch and bound) can guarantee optimality of the solution, their computational complexity increases exponentially with the problem size, making them unsuitable for handling the large-scale, real-time scheduling demands of disaster scenarios. Metaheuristic algorithms, represented by genetic algorithms, simulated annealing, and large-scale neighborhood search, seek satisfactory solutions within an acceptable timeframe through heuristic rules and have been widely applied in practice. However, these methods heavily rely on domain knowledge and operators designed for specific problems, exhibiting weak generalization ability and difficulty in effectively handling the complex trade-offs between multiple competing objectives such as operating costs, time window violations, and especially the risk to the lives of the injured. Crucially, most existing research simplifies "heterogeneous resources" as an extension of the homogeneous convoy routing problem, failing to deeply model the fundamental differences between vehicles and helicopters in terms of speed, capacity, accessibility, and cost structure, and the resulting collaborative complexities. This leads to poor feasibility and efficiency of the planning schemes in actual collaborative operations.

[0004] In recent years, reinforcement learning technology has provided a new paradigm for solving complex combinatorial optimization problems due to its powerful environmental interaction and policy self-learning capabilities. It can avoid tedious manual rule design and has the potential to learn general policies from data. However, directly applying reinforcement learning to the high-dimensional discrete decision problem of heterogeneous rescue path planning still faces significant challenges: First, the solution space for rescue solutions is enormous, leading to low agent exploration efficiency; second, the action space for path optimization (i.e., various path destruction, reconstruction, and local search operators) is large and complex, making it difficult for traditional reinforcement learning models to make efficient decisions; third, the reward signals in the optimization process are sparse and have long-range delays, and designing an effective reward mechanism to simultaneously guide the optimization of multiple competing objectives and accurately allocate the contribution of each decision step is a core challenge. Existing attempts often oversimplify the problem or rely heavily on extensive simulation pre-training, making it difficult to achieve a good balance between solution quality, learning efficiency, and generalization ability.

[0005] In summary, existing methods fall short in terms of modeling accuracy, decision-making intelligence, multi-objective coordination, and computational efficiency when addressing large-scale, dynamic, and heterogeneous resource disaster relief path planning problems. Therefore, a new method that deeply integrates operations research optimization prior knowledge with reinforcement learning decision-making capabilities is urgently needed to achieve efficient, intelligent, and collaborative scheduling of heterogeneous relief resources. Conducting research in this area has significant theoretical value and urgent practical implications for improving the modernization level of the national emergency management system and the ability to respond to major emergencies. Summary of the Invention

[0006] To address the technical problems existing in the above-mentioned background art, this invention proposes a heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] Step 1, Data Loading and Initialization: Load rescue point data containing geographic coordinates, casualty information, time windows and accessibility constraints, and generate an initial heterogeneous resource path scheme containing rescue vehicles, medical helicopters, transport vehicles and transport helicopters based on cluster analysis and heuristic rules.

[0009] Step 2, State Representation Construction: Extract the features of all task nodes in the current path scheme, generate a graph embedding global vector with permutation invariance through an ensemble attention pooling network, and fuse this vector with dynamic features of search progress, improvement stagnation depth, cost improvement momentum, etc. to construct the state representation of the reinforcement learning agent;

[0010] Step 3, Hierarchical D3QN Decision Making and Candidate Solution Generation: The state representation is input into the hierarchical DuelingDouble DQN architecture; the upper-layer network selects a destruction operator, a reconstruction operator, and a destruction ratio action based on the current state; based on the selected action, destruction and reconstruction operations are performed on the current path scheme to generate intermediate solutions and calculate their first-stage costs; the lower-layer conditional network selects actions in the local search operator space based on the state and the encoding of the upper-layer actions; based on the selected action, local search operations are performed on the intermediate solutions to generate new candidate solutions and calculate their final costs.

[0011] Step 4, Two-stage reward calculation and network update: A two-stage credit allocation mechanism is adopted to calculate the destructive reconstruction stage reward based on the difference between the initial cost and the first stage cost, and to calculate the local search stage reward based on the difference between the first stage cost and the final cost; using the rewards, the parameters of the hierarchical D3QN network are trained and updated through the Double DQN algorithm and priority experience replay.

[0012] Step 5, Solution Acceptance and Iteration: Based on the normalized Metropolis criterion, the simulated annealing strategy calculates the acceptance probability of a new candidate solution according to the normalized difference between the final cost and the current solution cost and the dynamic temperature, and decides whether to update the current solution.

[0013] Step 6, Termination Judgment: Determine whether the algorithm iteration count or time has reached the termination condition. If it is met, output the historically optimal heterogeneous resource rescue path scheme; otherwise, update the relevant parameters and return to Step 2.

[0014] Preferably, the initial heterogeneous resource path scheme in step 1 adopts a clustering partitioning-greedy insertion heuristic strategy: first, K-means clustering is performed based on the spatial proximity of task points; for each task point in each cluster, combined with tool accessibility constraints and capacity constraints, the minimum incremental cost insertion method is used to insert the task point into the feasible position of the currently constructed path in turn; if insertion is not possible, a new path is created; all clusters are processed in turn to finally form an initial feasible solution that satisfies all kinds of constraints.

[0015] Preferably, the state representation construction in step 2 specifically involves:

[0016] For each task node in the path scheme, extract its feature vectors, including coordinates, number of wounded, injury severity, time window, and task type. Input these feature vectors into an attention pooling network, which calculates and aggregates the attention weights between nodes to output a fixed-dimensional embedding vector representing the global solution structure. Then, embed the graph into this global vector and synchronize it with the normalized iteration progress. Where t is the current iteration number, The total number of iterations, the depth feature reflecting search stagnation, and the cost momentum feature reflecting recent improvement trends are concatenated to form the state representation. .

[0017] Preferably, the hierarchical Dueling Double DQN architecture in step 3 includes an upper-layer network and a lower-layer conditionalization network. The upper-layer network adopts a Dueling architecture, and its output layer contains three parallel sub-output heads, which are used to estimate the value of selecting different destruction operators, reconstruction operators, and destruction ratio actions, respectively. The destruction operators are selected from a library of operators including relevance removal, random removal, time-oriented removal, life-critical removal, demand clustering removal, and historical frequency removal. The reconstruction operators are selected from a library of operators including greedy insertion, regret value insertion, life-aware insertion, and perturbation insertion. The lower-layer conditionalization network is represented by the state. With the action selected by the upper layer The concatenated vector of the one-hot encoded vector is used as input, and its output is used to estimate the value of selecting different local search operator actions.

[0018] Preferably, in step 3, the local search operator action space is composed of the Cartesian product of single-path optimization operators and multi-path optimization operators. The single-path optimization operators include a 2-opt edge exchange operator for optimizing path order, a life-priority relocation operator for prioritizing the relocation of high-life-loss tasks, an enhanced relocation operator for simultaneously optimizing multiple node positions, an Or-opt operator for moving continuous subsequences, and a cascaded adjustment operator for optimizing time window urgency. The multi-path optimization operators include a cross-type task transfer operator for transferring tasks between different types of transportation vehicles, a sub-path insertion operator for inserting a sub-path of one path into another, a CROSS-Exchange operator for exchanging sub-paths between two paths, and an operator for exchanging a single task between two paths. Node swapping operator.

[0019] Preferably, the criticality removal operator in the destruction operator assesses criticality based on the number of wounded, injury severity, and waiting time at the task point, calculates expected life loss using an injury-adaptive Logistic survival function model, and prioritizes the removal of nodes with high expected life loss; the life-aware insertion operator in the reconstruction operator comprehensively evaluates the impact of the insertion position on operational costs and life risk costs when inserting nodes, making high-life-risk tasks more likely to be inserted into path locations with faster response times.

[0020] As a preferred embodiment, the reward calculation method for the two-stage credit allocation mechanism in step 4 is as follows:

[0021] Destruction and Reconstruction Phase Rewards The calculation formula is: ,in, For the cost of the path plan, For the current solution, This is an intermediate solution. This is the historical optimal solution. For indicator functions, These are the weighting coefficients;

[0022] Local search phase reward The calculation formula is: ,in, As a candidate solution, To prevent extremely small constants that are zero, These are the weighting coefficients.

[0023] Preferably, the parameter training and updating of the network in step 4 specifically involves maintaining an online Q-network for both the upper and lower layers. , and a target Q network , And a priority experience replay buffer; to transfer upper-level state transition experience Store in the upper buffer and transfer the lower state experience. The experience is stored in the lower-level buffer, and the priority of each experience is determined by the absolute value of its temporal difference error. During training, mini-batch experiences are sampled from the buffer according to priority, the Double DQN objective is calculated, and the parameters of the online network are updated. The parameters of the online network are periodically synchronized to the target network using a soft update method. ,in This is the soft update coefficient.

[0024] Preferably, the specific execution process of the normalized Metropolis criterion in step 5 is as follows: calculate candidate solutions. Relative to the current solution Normalized cost difference: Calculate the probability of acceptance: ,in The current temperature; if If the candidate solution is found to be correct, then the candidate solution is accepted directly; otherwise, it is accepted based on probability. Accept candidate solutions; initial temperature The cost is adaptively set based on the initial solution cost and decays exponentially during iteration: ,in This is the cooling coefficient.

[0025] Compared with existing technologies, the advantages and positive effects of this invention are as follows: This invention designs a graph embedding state representation method that integrates attention mechanisms, effectively encoding the overall structure of heterogeneous and dynamic heterogeneous resource path schemes; it innovatively proposes a hierarchical Dueling Double DQN (D3QN) decision architecture, which achieves decomposition and efficient exploration of a vast space of combined actions through an upper-layer network intelligently selecting destruction and reconstruction strategies and a lower-layer conditional network intelligently selecting local search strategies; it proposes a two-stage credit allocation mechanism to accurately evaluate the contributions of the two sub-stages of destruction / reconstruction and local search, effectively solving the problem of efficient learning under sparse and delayed rewards; and it constructs a multi-objective cost model that adapts to injury conditions, particularly refining the modeling of expected mortality costs, making the optimization objectives more aligned with actual rescue situations. In summary, the application of this invention can effectively solve the complex decision-making problem of heterogeneous resource joint rescue path planning in large-scale disaster scenarios. Through intelligent learning and coordination of multiple heterogeneous land and air rescue resources, it has the positive effects of significantly reducing overall operating costs and expected loss of life, and greatly improving overall rescue efficiency and timeliness. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the implementation of the present invention; Figure 2 3 represents the comprehensive results of the orthogonal experiments of the parameters of this invention; Figure 4 This is a schematic diagram representing the set of heterogeneous resource path solutions in this invention; Figure 5 This is a diagram of the ensemble attention pooling network state coding architecture used in this invention; Figure 6 This is a diagram illustrating the hierarchical Dueling Double DQN (D3QN) decision-making mechanism architecture of the present invention. Figure 7 This is a schematic diagram illustrating the decoupling of the two-stage reward and acceptance mechanism of the present invention; Figure 8 This is a comparison chart of the ablation experiment results of the present invention; Figure 9 This is a distribution chart of the relative percentage deviation (RPI) of the comparison algorithm of the present invention; Figure 10 , 11 Figures 12 and 13 are comparison diagrams of convergence curves for representative examples of this invention; Figure 14 , 15 16 and 17 are representative spatial path visualization diagrams of the present invention; Figure 18 , 19 20 and 21 are representative time execution Gantt charts of the present invention. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0030] In this embodiment, the present invention provides a heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approaches. Its overall framework and execution flow are mainly achieved through… Figure 1 The following steps are implemented to form a complete closed loop from initialization and iterative optimization to final output.

[0031] Parameter and scenario initialization phase. Specifically, first, all operating parameters of the reinforcement learning-assisted iterative greedy algorithm and rescue scenario data are initialized. The key control parameters of the algorithm are determined through the system... The orthogonal experimental design was optimized and determined, specifically including: the learning rate of the deep Q-network. This is used to control the update step size of the value network parameters; discount factor. This is used to weigh the importance of current gains against long-term future gains; the number of hidden layer nodes. This determines the representational capacity of the network model; minimum exploration rate: This ensures that the algorithm retains a certain degree of random exploration capability in the later stages of the search; the capacity of the experience replay buffer is set to... It is used to store past decision-making experience; the mini-batch size sampled during each network update is [size missing]. ; Soft update coefficients of the target network Ensuring smooth changes in the target value network is crucial for stable training of the Double DQN architecture. For example... Figure 2 , Figure 3 As shown, by performing signal-to-noise ratio (SNR) analysis and main effect plotting on the orthogonal experimental results, the discount factor was clarified. Number of hidden layer nodes Target network soft update coefficient Priority Experience Replay Coefficient and initial temperature The optimal parameter combination, which has the most significant impact on algorithm performance, was thus determined. Initial annealing temperature. It is not a fixed value, but rather the total cost based on the first feasible solution generated in the initial stage of the algorithm. Adaptive calibration is performed, and the calculation formula is as follows: ,in Set as This design allows the acceptance probability baseline to be dynamically adjusted according to the size of different instances, enhancing the robustness of the algorithm.

[0032] Complete rescue scenario input data includes: the latitude and longitude coordinates of the rescue center; the geographical coordinates and land / air accessibility indicators of all mission points, with air accessibility indicators limited to cases where land accessibility is not possible; and the number of injured at each mission point. or material demand Injury severity levels, which indicate the urgency of the injury. ; Soft time windows that allow service provision ; and the physical capacity limit, cruising speed, and unit distance travel cost for each of the four types of transport vehicles (Rescue Vehicle RV, Medical Helicopter MH, Transport Vehicle TV, and Transport Helicopter TH). and fixed start-up costs This stage provides all the necessary initial conditions and data foundation for the entire optimization process.

[0033] Initial Solution Generation and Representation Stage. This invention employs an efficient "clustering partitioning-greedy insertion" heuristic strategy, aiming to quickly construct an initial solution with excellent spatial structure within the feasible region, laying a high-quality starting point for subsequent reinforcement learning-guided deep optimization. The first step in this process is to group task nodes using the K-means clustering algorithm based on the geographic spatial proximity of task points and the type of task requirements. Number of clusters. Set to the total number of available transportation vehicles.

[0034] After clustering, the algorithm proceeds to path construction. For each cluster, the algorithm uses the rescue center as the start and end point of all paths, sequentially inserting task points within the cluster into the partially constructed paths. For each task point v to be inserted, the algorithm performs a detailed feasibility check and cost assessment: it traverses all initially generated paths and calculates the overall cost of inserting v between any two consecutive nodes in the path (or at the beginning of the path). Increment This incremental calculation is accurate and comprehensive, taking into account not only the direct increase in Euclidean travel distance due to the insertion of new nodes. The algorithm also evaluated the impact of the insertion operation on the entire path timeline through simulation calculations, including the risk that subsequent nodes might violate their time window constraints, and the potential changes in life loss costs calculated based on the injury adaptive survival function model. Among all candidate insertion positions that satisfy the capacity and time window hard constraints and do not violate the injury safety threshold (i.e., the probability of injury survival upon arrival is higher than the critical value), the algorithm selects the position that... Insertion is performed at the minimum position. If none of the current paths can accommodate v without violating any constraints, the algorithm immediately starts creating a new path that begins at the rescue center, visits v, and then returns to the center. This greedy insertion process iterates until all task points are assigned. The final initial solution is generated. That is, a like Figure 4 The diagram shows a set consisting of multiple cross-modal paths. This set has a clear mathematical property: the order of the paths within the set is permutable (i.e., the path set as a whole is unordered), but the access order of the task nodes within each path is strict and fixed. For example... Figure 4 As shown, different types of tool paths (RV, MH, TV, TH) are distinguished by different line types and colors, collectively forming a complete collaborative scheduling scheme. This representation method simultaneously encodes the local path logic and the global heterogeneous resource joint structure, laying the data structure foundation for subsequent state representation.

[0035] Next, in the state representation construction phase, this invention designs a dedicated trainable neural network encoder—an ensemble attention pooling network—to enumerate variable-length, unordered path solutions. Mapped to fixed-dimensional, semantically rich reinforcement learning state vectors The core mechanism of this network is as follows: Figure 5 As shown, its detailed workflow is as follows: First, for the current solution... For each task node i, the algorithm extracts its multidimensional feature vector. These features include, but are not limited to, the two-dimensional coordinates of the nodes. Wounded personnel load or material needs Quantified injury levels The urgency of the time window (e.g.) The reciprocal of the first value and the time window relaxation relative to other nodes in the current path. Feature vector of each node. The vectors are then fed into a fully connected layer (MLP) with shared weights, where a nonlinear transformation is performed to generate the hidden embedding vectors of the nodes. This step maps the original heterogeneous features into a unified vector space with latent semantics.

[0036] Next comes the core attention aggregation step of the network. The algorithm calculates a scalar attention weight for each node. The specific formula is: Here, W is a trainable weight matrix, and v is a trainable weight vector. This design makes the attention weight calculation depend on the hidden state of each node itself, rather than the pairwise interactions between nodes, thus ensuring complete invariance to the order of the input nodes (permutation invariance). Finally, the hidden embeddings of all nodes are... According to its corresponding attention weight Performing a weighted sum yields the global solution embedding vector. This vector It captures the structured information of the entire rescue plan, and due to the symmetry of its generation method, regardless of the arrangement of the path set, as long as the node assignments are the same, the output will be consistent. They are all completely identical. However, a static path structure encoding is not complete for the dynamic optimization process. Therefore, during state construction, the algorithm also incorporates the aforementioned global embedding vectors. Together with the three dynamic search features, they form the final state representation. These dynamic characteristics include: 1) the normalized iteration progress. t is the current iteration number. 1) Total number of iterations, indicating the stage of the search process; 2) Improved stagnation depth through hyperbolic tangent function normalization. Where d is the number of iterations since the last successful improvement, and L is the set saturation constant, which reflects the depth of the current search trapped in a local optimum; 3) Exponentially weighted momentum of recent cost improvements It captures the trend and intensity of solution improvement over a period of time. By fusing static structural encoding with dynamic process features, the state vector... This provides an informative and highly discriminative input for the subsequent hierarchical D3QN network, enabling it to make more intelligent decisions based on the quality of the current solution and the search state.

[0037] Then, a hierarchical Dueling Double DQN decision-making and candidate solution generation stage is performed. This invention constructs a hierarchical, decoupled action decision-making and execution system, such as... Figure 6 As shown, this system decomposes the three core operations of iterative greediness into upper and lower layers, each controlled by a dedicated reinforcement learning network, thereby solving the learning difficulties caused by the high-dimensional action space. The upper layer network is responsible for the macro-strategy of "destruction-reconstruction," while the lower layer network is responsible for the micro-tactics of "local search."

[0038] Upper-level decisions are performed by Dueling Double DQN, whose input is the state representation. Its output is specially designed as three parallel sub-output heads, each responsible for outputting a discrete action: 1) disrupting operator selection. 1) Operator selection: Operators are selected from a library that includes Shaw correlation disruption (group removal based on geographical and temporal window similarity between nodes), random disruption, life-critical disruption (prioritizing the removal of nodes with high expected life loss), demand clustering disruption, and completion time bottleneck disruption. 2) Reconstruction operator selection. : Select from the operator library, which includes greedy insertion, strongly constrained insertion, regret value insertion, and life-aware insertion. 3) Selection of violation ratio : Decide to start from the current solution The percentage of task nodes removed (e.g., 20%). Using a Dueling architecture decomposes action value into state value and action advantage, helping to learn more accurate value estimates in scenarios with many seemingly equivalent actions. (Received action) Then, the algorithm is executed sequentially: First, according to and Remove a corresponding proportion of task nodes to generate a partial solution. Subsequently, based on The removed nodes are then reinserted one by one to form an intermediate solution. And calculate its comprehensive cost. .

[0039] Lower-level decisions are executed by a conditional DQN network. Its input is the "context" of the upper-level actions: an expanded state vector, derived from the original state... With the actions already selected by the upper level It is composed of one-hot encodings. This means that all decisions of the lower-level network are based on the known current macroscopic perturbation strategy. This is based on the premise that hierarchical conditional decision-making is achieved, thus realizing true hierarchical conditional decision-making. The network outputs local search operator selection actions. This includes 2-opt edge swapping, node relocation, Or-opt subsequence movement, cross-path task transfer, and cross-exchange. Choose from a library of intra-path and cross-path collaborative optimization operators. Based on... The algorithm for intermediate solutions Perform an efficient local search to generate candidate solutions. And calculate its final comprehensive cost. This hierarchical decision-making mechanism not only greatly compresses the output dimension of the decision network, but also allows the upper-layer network to focus on learning when and how much structural disruption to perform, while the lower-layer network focuses on learning how to make the best refinement on a given new structure, thus achieving a clear separation of decision-making responsibilities and efficient collaboration.

[0040] Furthermore, this invention designs a precise "two-stage credit allocation" reward mechanism and combines it with the solution acceptance strategy of the "normalized Metropolis criterion" to form a closed-loop system that coordinates "learning" and "exploration." The complete logical flow of this stage is as follows: Figure 7 As shown.

[0041] Reward calculation follows a phased evaluation principle. Rewards for the destructive / reconstruction phase. Used to evaluate the actions of upper-layer networks The effectiveness is calculated using the following formula: ,in, For the cost of the path plan, For the current solution, This is an intermediate solution. This is the historical optimal solution. For indicator functions, These are the weighting coefficients; the first term measures the intermediate solutions. Relative to the current solution The relative cost improvement rate, using relative values ​​rather than absolute values, allows for reward scalarization across instances of different sizes. The second term is a breakthrough reward, awarded when an intermediate solution... The cost has surpassed the historical best solution. When the cost is, this indicator function Returning 1 provides additional positive incentives, encouraging the algorithm to pursue the global optimum. This is a reward during the local search phase. Used to evaluate the dynamics of lower-level networks The effectiveness is calculated using the following formula: ,in, As a candidate solution, To prevent extremely small constants that are zero, These are the weighting coefficients.

[0042] The reward focuses on local search solutions from the middle. to candidate solutions The resulting marginal improvements. This design extends the long decision chain. By decomposing it into two independently evaluable sub-segments, the problem of "credit allocation" of single-round reward signals in multi-step decision chains is effectively solved, enabling the upper-layer network to clearly perceive the contribution of its destruction-reconstruction strategy, and the lower-layer network to directly obtain feedback on its local optimization actions.

[0043] The core of the dynamic acceptance mechanism is the normalized Metropolis criterion. The algorithm first calculates candidate solutions. Relative to the current solution Normalized cost difference: This normalization process ensures that the algorithm maintains a consistent scale of acceptable behavior across instances with different objective function magnitudes. If the candidate solution is better, it is accepted directly; otherwise, it is determined by probability. Accept candidate solutions, where For index strategy The current temperature at which the decay occurs, where The coefficient is the cooling factor. This mechanism has a higher probability of accepting inferior solutions in the early stages of the search to encourage exploration, while in the later stages of the search it focuses on accepting improved solutions, achieving an adaptive balance between exploration and utilization.

[0044] The network update process runs in parallel with the steps described above. The algorithm incorporates upper-level decision-making experience. and lower-level decision-making experience The data are stored in two independent priority experience replay buffers. During the training phase, sampling is performed based on the temporal difference error (TD-error) of each experience entry as the priority. Subsequently, the Double DQN algorithm framework is used to utilize the calculated... and The target value is calculated separately, and then the parameters of the upper-layer D3QN network and the lower-layer conditional DQN network are updated using the gradient descent algorithm. The network parameters are periodically synchronized to their respective target networks using a soft update method. This is to ensure the stability of the learning process.

[0045] Finally, the algorithm continuously monitors the cumulative runtime or number of iterations. It terminates when a preset termination condition is met (e.g., computation time exceeds a threshold or the number of iterations reaches a certain threshold). If the search fails, the optimization process is immediately terminated. Then, the search with the lowest overall cost is selected from the entire search history. The solution is output as the final plan. The plan clearly specifies the detailed access sequence of each transport vehicle (RV, MH, TV, TH), the estimated service time window, the cost composition of each stage, and the overall rescue completion time, providing a complete and feasible decision-making basis for emergency command.

[0046] Furthermore, the overall cost of the route plan for: Among them: operating costs This encompasses the fixed operating costs of various modes of transportation and the variable costs calculated based on distance traveled, reflecting the economic efficiency of resource consumption. Expected mortality cost. This is the core of the model, and its calculation incorporates an adaptive injury coefficient. This is achieved through the aforementioned Logistic survival function model. Estimate the survival probability of wounded at each node at the expected service time point to obtain the expected number of deaths, then multiply this by the statistical value of life to obtain the cost. Model parameters. and According to the level of injury Personalized adjustments are key to accurately quantifying the urgency of different injuries. The penalty cost for urgency within a time window. A quadratic function is used to penalize the deviation of the actual service completion time from the ideal time window start time, thus encouraging on-time service and penalizing severe delays. Transportation completion cost. The time taken is determined by the time it takes for the last vehicle to complete all tasks and return to the rescue center. This time is used to measure the overall efficiency of the rescue mission and to incentivize parallel collaboration to shorten the total mission duration.

[0047] Furthermore, expected mortality cost The assessment is based not only on the number and severity of casualties at each mission point, but also integrates the current estimated waiting time for the mission. The operator utilizes a severity-adaptive Logistic survival function model. This allows for the dynamic calculation of the expected life loss at each node. In this model, the shape parameter characterizes the rate of decay of the survival probability. With critical time parameter All adjustments were made individually based on the average injury severity of the wounded at that point, i.e.: , This makes the survival probability decay curve of the severely injured group over time steeper, thus reflecting their time urgency earlier and more acutely in the model. Based on the expected loss of life calculated by this model, the operator prioritizes removing nodes with the highest estimated risk of loss of life from the current solution, creating opportunities for these most urgent tasks to be rescheduled more efficiently. Correspondingly, the life-aware insertion operator upgrades cost assessment during the reconstruction process. When a removed node is re-inserted into the path, the operator not only calculates the change in path distance (increase in operating cost) caused by the insertion, but also estimates the increase in expected death cost due to the change in the node's service time based on the survival model mentioned above. By comprehensively evaluating these two types of costs, the operator guides the algorithm to prioritize inserting high-risk task nodes into path positions that can minimize their total waiting time from the current moment, thus directly implementing the "life-first" optimization principle in local reconstruction.

[0048] To systematically verify the effectiveness and superiority of the method proposed in this invention, a comprehensive experimental evaluation was conducted based on the extended Solomon benchmark and randomly generated instances.

[0049] 1. Ablation experiment analysis: To clarify the contribution of each core component integrated in this invention, an ablation experiment was conducted, and the results are as follows: Figure 8 As shown. Experiments were conducted on 56 standard test cases, using relative percentage deviation (RPI) as the evaluation metric, comparing the average performance of the complete D3QNIG algorithm with seven key ablation variants. The results are as follows. Figure 8 As shown.

[0050] Experimental results show that removing the algorithm's basic optimization framework (the destruction-reconstruction module, D3QNIG_NDR) or key improvement steps (the local search module, D3QNIG_NLS) leads to the most significant performance degradation, with the median and distribution range of the RPI being far worse than the complete algorithm. This confirms that the iterative greedy skeleton combining "macro-perturbation" and "micro-tuning" is fundamental to the high performance of this method. Replacing the intelligent decision-making mechanism with random selection (D3QNIG_NRL) also results in a significant performance decrease, verifying the necessity of hierarchical D3QN networks in adaptively scheduling complex operators.

[0051] Furthermore, removing the two-stage reward design (D3QNIG_NHi) weakens the signal quality during network training; replacing the normalized Metropolis criterion with standard simulated annealing (D3QNIG_NAc) disrupts the balance between exploration and exploitation; and replacing attention pooling encoding with simple feature concatenation (D3QNIG_NSR) loses the ability to deeply perceive the solution structure. These variants all exhibit observable performance degradation, thus confirming the indispensable roles of the two-stage credit allocation mechanism, the dynamic adaptive acceptance strategy, and the ensemble attention state representation. In summary, the ablation experiments fully demonstrate that the D3QNIG algorithm is an organically synergistic system, with its core modules collectively contributing to its superior final solution performance.

[0052] 2. Algorithm Comparison and Convergence Analysis: The D3QNIG algorithm of this invention is compared with several advanced algorithms such as QALNS, AGA_HH, MOALNS_SA, Bandit_VNS, PBIGA, and IGPPO. Figure 9 As shown, the box plots of the relative percentage deviation (RPI) for each algorithm across all 56 test instances are presented. D3QNIG has the shortest box, the lowest median, and very few outliers, indicating that it significantly outperforms the comparison algorithms in average solution quality, the ability to find the optimal solution (achieving the optimal solution in 32 out of 56 instances), and cross-instance stability. Figure 10 , 11As shown in Figures 12 and 13, four typical examples with different distribution characteristics were further selected, and the convergence curves of the objective function values ​​of each algorithm within the same computation time were plotted. It can be observed that the D3QNIG algorithm exhibits the fastest descent rate in the early stages of iteration, indicating that its hierarchical decision-making mechanism can quickly identify effective search directions. In all examples, the convergence curve of D3QNIG consistently remains at the lowest position, indicating that it can continuously obtain and maintain the lowest cost level, resulting in the highest quality final solution.

[0053] 3. Path scheme visualization: To intuitively demonstrate the quality and interpretability of the schemes generated by the algorithm of this invention, Figure 14 , 15 Tables 16, 17, 18, 19, 20, and 21 present the final solutions planned by the algorithm on typical test instances from both spatial and temporal dimensions. For example... Figure 14 , 15 As shown in Figures 16 and 17, the different colors and line types of the paths clearly represent the task allocation of rescue vehicles (RVs), medical helicopters (MHs), transport vehicles (TVs), and transport helicopters (THs). It can be observed that helicopter paths (MH, TH) tend to connect spatially dispersed, remote, or time-critical task points, demonstrating their high-speed, direct access advantage; ground vehicle paths (RVs, TVs) are more concentrated on geographically clustered, heavily loaded task points, fully utilizing their high capacity and low cost characteristics; all paths form good spatial coordination and complementarity. Figure 18 , 19 The Gantt charts shown in Figures 20 and 21 clearly demonstrate that the timelines of each tool path are tightly arranged, tasks are smoothly connected, and idle and waiting times are minimal. This reflects that while optimizing the spatial order of paths, the algorithm also effectively coordinates global time synchronization. These two figures intuitively confirm that the scheme generated by D3QNIG not only has excellent objective function values ​​but also possesses good structural rationality and engineering feasibility.

[0054] In summary, through rigorous algorithm design, systematic ablation experiments, in-depth comparative and convergence analysis, and intuitive visualization of results, this invention fully demonstrates that the proposed reinforcement learning-based heterogeneous resource disaster relief path planning method significantly outperforms existing advanced methods in terms of solution quality, convergence speed, stability, and feasibility. This method provides an efficient, intelligent, and reliable solution to the complex resource scheduling and path planning problems in large-scale disaster scenarios, possessing significant theoretical value and broad practical application prospects.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach, characterized in that, Includes the following steps: Step 1, Data Loading and Initialization: Load rescue point data containing geographic coordinates, casualty information, time windows and accessibility constraints, and generate an initial heterogeneous resource path scheme containing rescue vehicles, medical helicopters, transport vehicles and transport helicopters based on cluster analysis and heuristic rules. Step 2, State Representation Construction: Extract the features of all task nodes in the current path scheme, generate a graph embedding global vector with permutation invariance through an ensemble attention pooling network, and fuse this vector with dynamic features of search progress, improvement stagnation depth, cost improvement momentum, etc. to construct the state representation of the reinforcement learning agent; Step 3, Hierarchical D3QN decision-making and candidate solution generation: Input the state representation into the hierarchical Dueling DoubleDQN architecture; The upper-layer network selects the destruction operator, reconstruction operator, and destruction ratio action based on the current state; according to the selected action, it performs destruction and reconstruction operations on the current path scheme, generates intermediate solutions, and calculates their first-stage costs; The lower-level conditional network selects an action in the local search operator space based on the encoding of the state and the upper-level action; and performs a local search operation on the intermediate solution based on the selected action to generate a new candidate solution and calculate its final cost. Step 4, Two-stage reward calculation and network update: A two-stage credit allocation mechanism is adopted to calculate the destructive reconstruction stage reward based on the difference between the initial cost and the first stage cost, and to calculate the local search stage reward based on the difference between the first stage cost and the final cost; using the rewards, the parameters of the hierarchical D3QN network are trained and updated through the Double DQN algorithm and priority experience replay. Step 5, Solution Acceptance and Iteration: Based on the normalized Metropolis criterion, the simulated annealing strategy calculates the acceptance probability of a new candidate solution according to the normalized difference between the final cost and the current solution cost and the dynamic temperature, and decides whether to update the current solution. Step 6, Termination Judgment: Determine whether the algorithm iteration count or time has reached the termination condition. If it is met, output the historically optimal heterogeneous resource rescue path scheme; otherwise, update the relevant parameters and return to Step 2.

2. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 1, characterized in that, The initial heterogeneous resource path scheme described in step 1 adopts a heuristic strategy of cluster partitioning and greedy insertion: First, K-means clustering is performed based on the spatial proximity of task points; for task points within each cluster, combined with tool accessibility constraints and capacity constraints, the minimum incremental cost insertion method is used to insert task points into feasible positions of the currently constructed paths in turn; if insertion is not possible, a new path is created; all clusters are processed in turn to form an initial feasible solution that satisfies all kinds of constraints.

3. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 1, characterized in that, The state representation construction in step 2 specifically involves: For each task node in the path scheme, extract its feature vectors, including coordinates, number of wounded, injury severity, time window, and task type. Input these feature vectors into an attention pooling network, which calculates and aggregates the attention weights between nodes to output a fixed-dimensional embedding vector representing the global solution structure. Then, embed the graph into this global vector and synchronize it with the normalized iteration progress. Where t is the current iteration number, The total number of iterations, the depth feature reflecting search stagnation, and the cost momentum feature reflecting recent improvement trends are concatenated to form the state representation. .

4. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 1, characterized in that, In step 3, the hierarchical Dueling Double DQN architecture comprises an upper-layer network and a lower-layer conditionalization network. The upper-layer network adopts a Dueling architecture, and its output layer contains three parallel sub-output heads, which are used to estimate the value of selecting different destruction operators, reconstruction operators, and destruction ratio actions. The destruction operators are selected from a library of operators including relevance removal, random removal, time-oriented removal, life-critical removal, demand clustering removal, and historical frequency removal. The reconstruction operators are selected from a library of operators including greedy insertion, regret value insertion, life-aware insertion, and perturbation insertion. The lower-layer conditionalization network is represented by the state. With the action selected by the upper layer The concatenated vector of the one-hot encoded algorithm is used as input, and its output is used to estimate the value of selecting different local search operator actions.

5. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 4, characterized in that, In step 3, the local search operator action space is composed of the Cartesian product of single-path optimization operators and multi-path optimization operators. The single-path optimization operators include a 2-opt edge-swapping operator for optimizing path order, a life-priority relocation operator for prioritizing high-life-loss tasks, an enhanced relocation operator for simultaneously optimizing multiple node positions, an Or-opt operator for moving continuous subsequences, and a cascaded adjustment operator for optimizing time window urgency. The multi-path optimization operators include a cross-type task transfer operator for transferring tasks between different types of transportation vehicles, a sub-path insertion operator for inserting a sub-path of one path into another, a CROSS-Exchange operator for exchanging sub-paths between two paths, and an operator for exchanging a single task between two paths. Node swapping operator.

6. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 4, characterized in that, The criticality removal operator in the destruction operator assesses criticality based on the number of wounded, injury severity, and waiting time at the task point. It calculates expected life loss using an injury-adaptive Logistic survival function model and prioritizes removing nodes with high expected life loss. The life-aware insertion operator in the reconstruction operator comprehensively evaluates the impact of the insertion position on operational costs and life risk costs when inserting nodes, making high-life-risk tasks more likely to be inserted into path locations with faster response times.

7. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 1, characterized in that, The reward calculation method for the two-stage credit allocation mechanism in step 4 is as follows: Destruction and Reconstruction Phase Rewards The calculation formula is: ,in, For the cost of the path plan, For the current solution, This is an intermediate solution. This is the historically optimal solution. For indicator functions, These are the weighting coefficients; Local search phase reward The calculation formula is: ,in, As a candidate solution, To prevent extremely small constants that are zero, These are the weighting coefficients.

8. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 1, characterized in that, The specific steps for training and updating the network parameters in step 4 are as follows: maintaining an online Q-network for both the upper and lower layers. , and a target Q network , and a priority experience replay buffer; Experience of transitioning upper-level states Store in the upper buffer and transfer the lower state experience. The experience is stored in the lower buffer, and the priority of each experience is determined by the absolute value of its temporal difference error. During training, mini-batch experience is sampled from the buffer according to priority, the Double DQN objective is calculated, and the parameters of the online network are updated; the parameters of the online network are periodically synchronized to the target network using a soft update method. ,in This is the soft update coefficient.

9. The heterogeneous rescue and path planning method based on reinforcement learning-assisted iterative greedy approach according to claim 1, characterized in that, The specific execution process of the normalized Metropolis criterion in step 5 is as follows: Calculate candidate solutions. Relative to the current solution Normalized cost difference: Calculate the probability of acceptance: ,in The current temperature; if If the candidate solution is found to be correct, then the candidate solution is accepted directly; otherwise, it is accepted based on probability. Accept candidate solutions; initial temperature The cost is adaptively set based on the initial solution cost and decays exponentially during iteration: ,in This is the cooling coefficient.

Images