Multimodal dynamic path planning and obstacle avoidance scheduling methods, intelligent agents and systems

Abstract

This invention discloses a method, agent, and system for dynamic path planning and obstacle avoidance scheduling in multimodal transport based on reinforcement learning. The method determines the activity route segment contract and its remaining constraints by acquiring the current confirmed location, time-varying transport network state, and time-indexed obstacle envelope; creates an evaluator snapshot bound to a decision identifier; predicts the transport state after the execution of a nominal scheduling action and calculates the predicted reachability value; maps the nominal scheduling action to a sendable mapping action or a safety rejection flag; forms a confirmation action and a confirmation local state based on execution feedback and state evidence, converting them into a confirmed route state; performs bounded residual path replanning when the calculated reachability value is lower than a judgment threshold or the search state is truncated to the upper limit of calculation; and only uses state transitions containing confirmation actions, confirmation local states, and actual rewards to update the reinforcement learning scheduling strategy, improving the credibility of the reinforcement learning strategy's credit allocation in a real multimodal transport execution environment.

Description

Technical Field

[0001] This invention relates to the fields of logistics transportation scheduling and artificial intelligence technology, specifically to a multimodal dynamic path planning and obstacle avoidance scheduling method, intelligent agent and system based on reinforcement learning. Background Technology

[0002] Multimodal transport refers to the use of at least two modes of transport—road, rail, waterway, air, or in-yard transshipment—between the origin and destination for the same transport task, with the mode of transport being switched at ports, railway stations, logistics parks, airport cargo terminals, or other transshipment nodes. In actual operation, the feasibility of an end-to-end route depends not only on the route distance and transport costs, but also on dynamic conditions such as schedules, route capacity, transshipment operation windows, terminal equipment, vehicle or vessel status, dangerous goods permits, cold chain energy supply, and local corridor occupancy. During the execution of a transport task, route closures, node unavailability, resource occupancy, capacity reduction, schedule changes, operation slot conflicts, or transshipment connection failures can render certain sections or the entire remaining route infeasible.

[0003] In existing technologies, multimodal transport route planning typically employs operations research optimization methods or reinforcement learning methods. Operations research optimization methods rely on static assumptions, making it difficult to characterize uncertainties such as weather and congestion; they are slow to adapt to new nodes or routes, and rolling replanning is costly. Reinforcement learning methods treat routes and scheduling as temporal decisions, with the agent learning strategies based on historical or simulation data to cope with changes. However, existing reinforcement learning methods face the following technical problems when applied to real multimodal transport execution environments: The output of a reinforcement learning policy is a nominally scheduled action based on current observations. This nominal action may be modified or replaced by security constraints before transmission, or it may not be received by the external system due to communication failure. Even if it has been transmitted, it may be delayed in execution, partially executed, executed with bias, refused to execute, or have missing feedback. The policy nominal action, security-mapped action, successfully transmitted action, and the actual executed action confirmed by feedback evidence belong to different levels of action facts. If they are uniformly treated as actual actions without distinction, it will lead to a mismatch between action and state facts, making the reinforcement learning state transition inconsistent with the real environment.

[0004] Furthermore, existing solutions often employ different evaluation rules or model versions during the prediction and execution confirmation phases. When the execution receipt arrives late, the system's evaluation rules may have changed. This means that actions evaluated as feasible during prediction may become infeasible during execution confirmation if re-evaluated using different rules, resulting in evaluation caliber drift and compromising the comparability and reliability of reinforcement learning credit assignment. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a multi-mode dynamic path planning and obstacle avoidance scheduling method, agent, and system based on reinforcement learning. This ensures the reliability of credit allocation for reinforcement learning policies under asynchronous execution feedback conditions, enabling prediction evaluation and confirmation evaluation to be based on the same computable metric framework. Furthermore, only confirmed action-state transitions are used for policy updates, while confirmed reachability drives bounded replanning, avoiding evaluation caliber drift, action-state fact mismatch, search truncation misjudgment, uncertain replanning boundaries, and unconfirmed data contaminating reinforcement learning experience.

[0006] In a first aspect, the present invention provides a multi-mode dynamic path planning and obstacle avoidance scheduling method based on reinforcement learning, comprising: Obtain the current confirmed location, time-varying transportation network status, and time-indexed obstacle envelope; construct a time-varying transportation map based on the time-varying transportation network status; determine the active route segment contract from the current confirmed location to the downstream exit anchor point, and the remaining constraints of the active route segment contract, including hard constraints. According to the preset reinforcement learning scheduling strategy, a nominal scheduling action is generated within the action space defined by the contract of the activity route segment; Assign a decision identifier to the current decision cycle and create an evaluator snapshot bound to the decision identifier. The evaluator snapshot includes the evaluator version, evaluation rules, judgment threshold, and search calculation limit. The evaluator snapshot remains unchanged within the current decision cycle. Predict the transportation state after the nominal scheduling action is executed, convert the transportation state into a predicted route state through state projection, and calculate the predicted reachability value and search state of the nominal scheduling action according to the evaluation rules in the evaluator snapshot. The search state includes at least the search space exhaustion state and the computational upper limit truncation state. Based on the hard constraints, the predicted reachability value, and the search state, the nominal scheduling action is mapped to a sendable mapping action or a security rejection flag; After sending the sendable mapping action and receiving a confirmation of receipt from the external interface, the sending action is recorded, and the sending action is associated with the decision identifier and the evaluator snapshot; Based on the execution feedback and status evidence matching the sending action, a confirmation action and a confirmation local state are formed, or a record to be confirmed is formed; The confirmed local state is converted into a confirmed route state through the state projection, and the confirmed reachability value is calculated based on the evaluator snapshot bound to the decision identifier; When the confirmed reachability value is lower than the judgment threshold or the search state is a calculation upper limit truncation, a bounded remaining path replanning is performed with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary; the bounded remaining path replanning refers to the replanning with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary. Only the state transitions containing the confirmation action, the confirmation local state, and the actual reward are used to update the preset reinforcement learning scheduling strategy.

[0007] Preferably, the time-varying transportation map is constructed by collecting transportation network nodes and edges, transportation modes, estimated travel time, capacity, available resource intervals, shifts or work slots, confirmed location of transported objects, historical confirmed path segments, and dynamic obstacle observations; The time index barrier envelope describes the unavailable intervals, capacity boundaries, or occupancy probability boundaries of nodes, edges, resources, or connections in the prediction time domain, and uses conservative boundaries, upper bounds of confidence intervals, or probability quantiles under a preset confidence level to form occupancy intervals for hard constraint judgment. The activity route segment contract includes: the currently confirmed starting anchor point, the downstream exit anchor point, the set of allowed nodes and edges, the latest departure time, the remaining time window, capacity and resource constraints, the remaining risk budget, and the maximum replanning expansion range; The activity route segment contract freezes the completed route segment only after confirming that the downstream exit anchor point has been reached. The downstream exit anchor point is used as the starting anchor point of the next route segment. The next contract version is generated based on the remaining route skeleton and the latest confirmed constraints. Nominal scheduling actions, mapping actions, or sending actions do not trigger contract version updates.

[0008] Preferably, the evaluation rules include path search and ranking rules, hard constraint types and their judgment rules, normalization scale and aggregation weight; the evaluator snapshot also includes route segment contract versions and secondary calculation budgets; The evaluator snapshot is created before the first nominal scheduling action enters the evaluation, and the evaluator version, path search and sorting rules, hard constraint type and its judgment rules, normalization scale, aggregation weight, judgment threshold, search calculation upper limit, route segment contract version and decision identifier are bound to the evaluator snapshot. The lifecycle states of the evaluator snapshot include active, confirmed, excluded, unconfirmable, and released. The active snapshot is converted to released only when the corresponding sending action has been confirmed, excluded due to security rejection or confirmed non-receipt, confirmed but not executed, or the retention period has been reached and it is closed as unconfirmable. After release, the snapshot identifier, parameter digest, and audit hash are retained. When a late receipt arrives, the original evaluator snapshot is loaded according to the original decision identifier to perform a confirmation reachability evaluation, without switching to the current default evaluator version.

[0009] Preferably, the prediction of the transportation status after the execution of the nominal scheduling action includes: based on the current confirmed local state, the nominal scheduling action, and the time index barrier envelope, predicting the expected time to leave the current segment, the expected next node or edge to enter, the expected resource occupation interval, the expected consumption time, and the risk budget after the action is executed; The process of converting the transportation state into a predicted route state through state projection includes: calling a local-global state projection function to form a predicted route state based on the predicted local state and the active route segment contract. The predicted route state includes at least the route location, the next connecting node, the remaining time window, the remaining risk budget, the availability of subsequent connections or services, and the confirmed path prefix. The step of calculating the predicted reachability value and search status according to the evaluation rules in the evaluator snapshot includes: advancing the current time-varying transportation graph from the decision time to the prediction time based on the predicted arrival time and time index obstacle envelope, updating the predicted availability status of nodes, edges, capacity, resources and time slots, and obtaining the prediction base graph; The prediction base map is trimmed according to the predicted route status and the predicted residual constraints to form the prediction residual map; In the predicted residual graph, acyclic paths are enumerated sequentially according to a fixed comprehensive cost, and hard constraints such as time window, risk budget, capacity, resources and node edge availability are verified. The search stops when the first K acyclic feasible paths are obtained, the search space is exhausted, or the upper limit of the search calculation is reached. K is a preset integer, and the state of exhaustion of the search space or truncation of the upper limit of the calculation is recorded. The predicted reachability value is set to zero only when the search space is exhausted and no feasible path is found; when the computational limit is reached but no feasible path is found, an evaluation uncertainty flag is returned. For the first K feasible paths obtained, the feasible path with the minimum comprehensive cost is taken as the baseline path. The time margin and risk margin are calculated respectively, and the predicted reachability value is calculated according to the aggregation weight, normalization scale and judgment threshold in the evaluator snapshot. When the predicted reachability value is not lower than the judgment threshold, it is determined as a sendable candidate.

[0010] Preferably, mapping the nominal scheduling action to a sendable mapping action or a security rejection flag based on the hard constraint, the predicted reachability value, and the search state includes: A set of safe actions is formed based on immediate hard constraints and predicted reachability thresholds. When the nominal scheduling action meets the admission conditions, it is retained as a mapping action. When the conditions are not met but other safe actions exist, it is modified or replaced as a mapping action. When no action meets the conditions, a safety rejection flag is output. The security rejection tag is formed before the action is sent, and is used to form a security intervention tag, but does not form the sending action or actual state transition; The step of recording the sending action after sending the sendable mapping action and receiving a reception confirmation from the external interface includes: Sending is done only when the mapping action is a sendable network scheduling action, via the control interface, API, message interface, RPA executor, or device adapter; the sending record also saves the decision identifier, route segment contract version, and evaluator snapshot identifier; The payload actually received by the external interface is recorded as a sending action only when the external interface returns a received result or verifiable evidence of reception exists; if communication fails and no external data is received, the sending action is not recorded, nor is it recorded as a security rejection.

[0011] Preferably, the step of forming a confirmation action and a confirmation local state based on the execution feedback and state evidence matching the sending action, or forming a record to be confirmed, includes: The sending action is associated with the task identifier, path version, route segment identifier, local decision identifier, sending sequence number, and idempotency identifier. Collect equipment receipts, business flow, location of transported objects, node entry or exit records, resource usage changes, obstacle observation, and operation completion status, and match them with the sent action facts; When the sending action, execution feedback and status evidence meet the consistency conditions, the confirmation action, confirmation of the next local state, action duration and actual reward are reconstructed based on the actual execution volume. The confirmation action may be the same as the sending action or different due to partial execution, execution deviation or external system correction. When the external system explicitly refuses to execute and the state after non-execution can be determined, a confirmation of non-execution is formed using a preset no-operation code, and the state after confirmation of non-execution and the recalcible reward constitute the confirmation fact. When a message has been sent but feedback is missing, evidence is insufficient, or evidence is conflicting, only a pending confirmation record is formed. The pending confirmation record does not define a confirmation action, confirmation of the next state, or actual reward, and does not form a reinforcement learning state transition. Subsequently, based on the late receipt, supplementary evidence, or manual verification results, the pending confirmation record will be promoted to the confirmed execution record, or when the status and reward after clear refusal to execute and non-execution can be determined, it will be promoted to the confirmed non-executed empty operation record. If it is still uncertain, it will be closed as an unconfirmable record. During partial execution, only when the executed portion can be broken down into identifiable atomic scheduling actions and the corresponding next state and actual reward can be determined based on evidence, is the execution amount confirmed and a confirmation transfer formed. The remaining unconfirmed portion is retained in the pending confirmation record.

[0012] Preferably, converting the confirmed local state into a confirmed route state through the state projection, and calculating the confirmed reachability value based on the evaluator snapshot bound to the decision identifier includes: The confirmed local next state is transformed into the confirmed route state using the same local-global state projection function and the same active route segment contract as in the prediction phase. A confirmation residual graph is formed based on the confirmed route status, the network availability envelope at the confirmation time, and the confirmation residual constraints. Call the original evaluator snapshot that is bound to the same decision identifier in the prediction stage before sending, and use the same path search and sorting rules, K value, hard constraint type and decision rules, normalization scale, aggregation weight and decision threshold to calculate and confirm the reachability value; The confirmation phase allows input of the latest confirmation location, confirmation time, confirmation resource usage, and confirmation obstacle facts, but does not change the evaluation rules in the snapshot; When the evaluation returns an uncertain flag because the search calculation limit has been reached and no feasible path has been found, the search continues according to the preset secondary calculation budget in the evaluation snapshot. If the situation remains uncertain after the second-level budget is completed, a conservative, bounded replanning or manual review will be triggered. The bounded residual path replanning includes: The initial boundary is the current confirmed node, the current edge endpoint, or the safe-to-switch anchor point corresponding to the confirmed route status in the time-varying transportation graph. The initial end boundary is the first downstream anchor point in the original remaining path that is not affected by the current obstacle envelope, still satisfies the hard constraints, and can connect with the original route; The initial search subgraph includes the starting boundary, the ending boundary, the affected nodes and edges, and their preset one-hop adjacency range; When there is no feasible path in the initial search subgraph, the search is expanded step by step according to the one-hop adjacency range, the preset time range, or the anchor point of the next route segment. When the maximum number of hops, the maximum number of nodes, the maximum computation time, or the maximum expansion range specified in the route segment contract is reached, and the expanded search space has been exhausted and there is still no feasible path, or the original downstream anchor point has become invalid, replanning is performed on all unconfirmed remaining paths from the current confirmed position. When the computational limit is reached but the search space is not exhausted, the evaluation is recorded as uncertain; confirmed path prefixes are not included in the search subgraph.

[0013] Preferably, the step of using only the state transition containing the confirmation action, the confirmation local state, and the actual reward to update the preset reinforcement learning scheduling policy includes: Actual action experience is generated and written into the actual action experience pool only when the action is confirmed, the next state is confirmed, the actual reward is determined, the duration of the action is determined, and the basis for confirmation is determined. The action field in the actual action experience stores the confirmed action; when the confirmed non-executed action can be determined by the no-operation code, the confirmation status after non-execution, the actual reward and the confirmation basis, it is transferred to the actual action experience pool as a confirmation. When a nominal scheduling action is retained, modified, replaced, or security denied, a security intervention label is formed. The security intervention label includes at least the nominal scheduling action, the mapped action or security denial flag, the intervention type, the constraint reason, the action modification amount, the predicted reachability value, and the reachability insufficiency amount. When security is denied, the action correction amount is represented by a preset denial code or a definite code from the nominal scheduled action to the denial action; the inadequate reachability amount is defined as the non-negative part of the difference between the predicted threshold and the predicted reachability value; communication failures are not written into the security intervention tag pool; sent pending confirmation records enter an independent pending confirmation area and do not enter the actual action experience pool or the security intervention tag pool; pending confirmation samples do not participate in any strategy loss, value loss, intervention classification loss, action correction loss or reachability calibration loss calculation; The preset reinforcement learning scheduling strategy adopts an actor-critic structure. The critic's temporal difference objective is calculated based on confirmation transition and action duration discount. The actor's objective is to maximize the confirmation action value and reduce the probability of nominally scheduled actions that are expected to require safety intervention. Safety intervention labels are used to calculate intervention type classification loss, action correction regression loss, or policy intervention penalty. The calibration loss for predicted accessibility and confirmed accessibility only uses records that have both numerical prediction evaluation and numerical confirmation results.

[0014] Secondly, corresponding to the method described above, the present invention also provides a multi-mode dynamic path planning and obstacle avoidance scheduling intelligent agent based on reinforcement learning, comprising: The network status and segment contract formation module is used to obtain the current confirmed location, time-varying transportation network status and time-indexed obstacle envelope, construct a time-varying transportation map based on the time-varying transportation network status, and determine the active route segment contract and its remaining constraints from the current confirmed location to the downstream exit anchor point. The remaining constraints include hard constraints. The reinforcement learning scheduling action generation module is used to generate nominal scheduling actions within the action space defined by the activity route segment contract according to a preset reinforcement learning scheduling strategy. The unified accessibility evaluation and snapshot management module is used to assign a decision identifier to the current decision cycle and create an evaluator snapshot bound to the decision identifier. The evaluator snapshot includes the evaluator version, evaluation rules, judgment threshold and search calculation limit, and remains unchanged within the current decision cycle. The local-global state projection module is used to predict the transportation state after the nominal scheduling action is executed, convert the transportation state into a predicted route state through state projection, and calculate the predicted reachability value and search state of the nominal scheduling action according to the evaluation rules in the evaluator snapshot. The search state includes at least the search space exhaustion state and the computational upper limit truncation state. The action mapping and sending module is used to map the nominal scheduling action to a sendable mapping action or a security rejection flag according to the hard constraint, the predicted reachability value and the search state. After sending the sendable mapping action and receiving the acceptance confirmation from the external interface, the sending action is recorded. The sending action is associated with the decision identifier and the evaluator snapshot. The execution status confirmation module is used to form a confirmation action and a confirmation local status, or to form a record to be confirmed, based on the execution feedback and status evidence that match the sending action. The confirmation evaluation module is used to convert the confirmed local state into a confirmed route state through the state projection, and calculate the confirmed reachability value based on the evaluator snapshot bound to the decision identifier; A bounded replanning module is used to perform bounded remaining path replanning when the confirmed reachability value is lower than the judgment threshold or the search state is a calculation upper limit truncation, with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary; the bounded remaining path replanning refers to replanning with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary. The reinforcement learning update module is used to update the preset reinforcement learning scheduling strategy by using the state transition that includes the confirmation action, the confirmation local state, and the actual reward.

[0015] Thirdly, the present invention also provides a multimodal dynamic path planning and obstacle avoidance scheduling system based on reinforcement learning, including the aforementioned scheduling agent, as well as a data acquisition interface and a tool execution adapter; the data acquisition interface is used to acquire time-varying transportation network status, time-indexed obstacle envelope, execution feedback and status evidence, and provide them to the scheduling agent; the tool execution adapter is used to send the sendable mapping actions output by the scheduling agent to an external system, and receive the reception confirmation and execution feedback from the external interface.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention assigns a decision identifier to the current decision cycle and creates an evaluator snapshot bound to that identifier. This allows the evaluation rules, judgment thresholds, and search computation limits from the same evaluator snapshot to be reused in both the pre-send prediction phase and the execution confirmation phase. Simultaneously, through lifecycle management of the evaluator snapshot, the activity reference is released and the audit hash is retained only after the sending action is confirmed, excluded, or closed as unconfirmable. When a late receipt arrives, the original evaluator snapshot is still loaded based on the original decision identifier to perform the confirmation evaluation, without switching to the current default evaluator version. This ensures that both phases use a recalculated and consistent evaluation standard, making reinforcement learning credit allocation comparable and reliable.

[0017] 2. This invention employs a four-stage fact isolation mechanism: nominal scheduling action, sendable mapping action, sending action, and confirmation action. Only state transitions where the confirmation action, the confirmed local state, and the actual reward are all determined are used to update the reinforcement learning scheduling policy. By excluding pending confirmation records from reinforcement learning loss calculations, data from communication failures, missing receipts, or insufficient evidence do not contaminate the actual action experience pool. A confirmation of non-execution action using a preset no-operation code is formed when the state after explicit refusal to execute is determinable, correctly distinguishing "confirmation of non-execution" from "pending confirmation," "communication failure," and "security rejection." During partial execution, confirmation transitions are formed only when the executed portion can be broken down into identifiable atomic scheduling actions and the corresponding next state and actual reward can be determined, avoiding the erroneous recording of uncertain partial executions as complete success samples.

[0018] 3. This invention performs bounded remaining path replanning when the confirmed reachability value is below a judgment threshold or the search state is truncated at the computational upper limit. The replanning scope is limited to the minimum necessary range, reducing the computational scale of unconditionally complete remaining path replanning. By distinguishing between the exhaustive search space state and the computational upper limit truncation state, inaccessibility is only identified when the search space is exhausted and no feasible path is found, preventing the erroneous generation of inaccessibility labels due to insufficient computational budget. When the evaluation returns an uncertain label due to reaching the search computational upper limit, a secondary computational budget is first executed; if uncertainty remains, conservative replanning or manual review is triggered, thus conservatively handling the uncertainty state of the evaluation.

[0019] 4. This invention prevents predicted or transmitted actions from prematurely altering the boundaries of subsequent actions by having the activity route segment contract roll only after confirming arrival at the downstream exit anchor point, and by ensuring that nominal scheduling actions, transmittable mapping actions, or transmit actions do not trigger contract version updates. Furthermore, by using a time-indexed obstacle envelope with a pre-set conservative boundary, upper bound of the confidence interval, or probability quantile value to form an occupied interval for hard constraint judgment, the uncertainty of dynamic obstacle prediction is explicitly incorporated into the safety judgment.

[0020] 5. This invention transforms the predicted local state and confirmed local state into predicted route state and confirmed route state respectively through a local-global state projection function. This ensures that the execution effect of local actions can be consistently evaluated in the global residual graph, preventing local actions from being executable in the current segment but becoming unreachable in subsequent paths after being projected onto the multimodal transport residual graph. By enumerating acyclic paths sequentially in the predicted residual graph according to a fixed comprehensive cost and verifying hard constraints, the search process stops when the first K acyclic feasible paths are obtained, the search space is exhausted, or the search computation limit is reached, making the path search process deterministic and recalcible. By calculating time margin and risk margin using the feasible path with the minimum comprehensive cost as the benchmark path, the conservatism and consistency of reachability evaluation are guaranteed.

[0021] 6. This invention, through the formation of a set of safe actions and the retention, modification, replacement, or safety rejection of nominal scheduling actions, ensures that hard-constraint violation actions are not allowed to proceed due to higher rewards; the safety rejection flag is formed before the action is sent, without resulting in a sent action or actual state transition. By simultaneously saving the decision identifier, route segment contract version, and evaluator snapshot identifier in the sending record, each sent action can be traced back to the corresponding decision cycle and evaluation rule version; when communication fails and no external reception occurs, the sent action is not recorded, nor is it recorded as a safety rejection, thus correctly distinguishing between communication failure and safety rejection.

[0022] 7. This invention associates sending actions with task identifiers, path versions, route segment identifiers, local decision identifiers, sending sequence numbers, and idempotency identifiers. It also collects and matches data such as device receipts, business flow records, transport object locations, node entry / exit records, resource usage changes, obstacle observations, and job completion status, ensuring sufficient factual basis for execution status confirmation. By not defining confirmation actions, confirming the next state, or providing actual rewards for pending confirmation records, it avoids reinforcement learning state transitions. Furthermore, subsequent promotion or closure is based on late receipts, supplementary evidence, or manual verification results, preventing unconfirmed data from polluting the actual action experience pool. Simultaneously, it preserves a channel for restoring confirmation for late receipts.

[0023] 8. This invention provides complete traceable information for security intervention tags by including at least nominal scheduling actions, mapping actions or security rejection flags, intervention type, constraint reason, action correction amount, predicted reachability value, and reachability insufficiency amount. This supports policy learning and reduces the probability of repeatedly proposing the same type of violation action. Furthermore, by not writing communication failures into the security intervention tag pool, placing sent pending confirmation records into an independent pending confirmation area, and excluding pending confirmation samples from any policy loss, value loss, intervention classification loss, action correction loss, or reachability calibration loss calculations, data of different confidence levels are strictly isolated, preventing low-confidence data from contaminating policy training.

[0024] 9. This invention employs an actor-commentator structure through a pre-defined reinforcement learning scheduling strategy. The commentator's temporal difference objective is calculated based on confirmation transitions and action duration discounts, while the actor's objective is to maximize the value of confirmed actions and reduce the probability of nominally scheduled actions requiring safety intervention. This allows the strategy update to simultaneously optimize task completion effectiveness and the probability of safety intervention. Safety intervention labels are used to calculate intervention type classification loss, action correction regression loss, or strategy intervention penalty, enabling the strategy to learn from safety intervention events. By using calibration losses for predicted and confirmed accessibility only with records possessing both numerical prediction evaluations and numerical confirmation results, the accessibility calibration model is trained solely on reliable data, improving the accuracy of predicted accessibility values. Detailed Implementation

[0025] The technical solution of the present invention will be described in detail below with reference to specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art can make various modifications and variations based on the technical solution of the present invention, and all such modifications and variations fall within the protection scope of the present invention.

[0026] Example 1: Scheduling Method This embodiment provides a multimodal dynamic path planning and obstacle avoidance scheduling method based on reinforcement learning, executed by a scheduling agent. The scheduling agent can be deployed on servers, edge computing nodes, vehicle terminals, station control equipment, or a distributed combination thereof. The scheduling agent interacts with the transportation management system, carrier system, station system, or execution equipment through a tool execution adapter.

[0027] The scheduling method in this embodiment includes steps S101 to S107.

[0028] Step S101: Construct time-varying transportation graph, obstacle envelope, and route segment contract Collect data on transportation network nodes and edges, transportation modes, estimated travel time, capacity, available resource ranges, shifts or work slots, confirmed locations of transported objects, historical confirmed path segments, and dynamic obstacle observations to construct a time-varying transportation map.

[0029] Time-varying transportation networks are represented as follows:

[0030] Where V is the set of nodes, Let be the set of edges available or to be evaluated at time t. For edges and nodes, the travel time, transportation mode, capacity, risk, resources, and time slot attributes are considered. This is the time index barrier or resource occupancy envelope from time t to the predicted time domain endpoint t+H.

[0031] The nodes can represent transportation origins, transportation destinations, transshipment nodes, terminal nodes, resource nodes, or connection anchors. The edges can represent transportation sections, connection sections, or operational sections. The transportation modes include at least two of the following: road, rail, waterway, air, or in-plant transshipment.

[0032] Dynamic obstacles are represented as time-indexed network availability envelopes. These envelopes at least describe the unavailability intervals, capacity boundaries, or occupancy probability boundaries of nodes, edges, resources, or connections within the prediction time domain. For obstacles with prediction uncertainty, conservative boundaries at a pre-set confidence level, upper bounds of confidence intervals, or probability quantiles can be used to form occupancy intervals for hard constraint determination.

[0033] The dynamic obstacles include, but are not limited to: geometric obstacles (such as road construction areas, accident vehicles, mobile equipment, ships, yard bridge operation areas, temporary restricted areas, and predicted obstacle trajectories); network obstacles (such as road closures, railway interruptions, waterway restrictions, weather-related navigation bans, service cancellations, and reduced line capacity); resource conflict obstacles (such as berths, tracks, gates, cranes, transshipment stations, yard access, charging / swapping facilities, or refrigerated sockets being occupied by other tasks at a specified time); and rule-based obstacles (such as prohibited periods for hazardous materials, driver working hour limits, load limits, temperature control continuity constraints, and equipment qualification mismatches).

[0034] Based on the currently confirmed location and the original route framework, a contract for the activity route segment is formed.

[0035] The activity route segment contract is represented as follows:

[0036] in, The current confirmed starting anchor point, As a downstream export anchor point, and To allow sets of nodes and edges, The latest departure time, For the remaining time window, Budgeting for remaining risks, Due to capacity and resource constraints, To maximize the replanning and expansion of boundaries, This is the route segment contract version.

[0037] The remaining constraints of the activity route segment contract include hard constraints. These hard constraints include at least: node edge availability, time window, capacity, resources, route segment contract, and risk budget. They may also include: compatibility of cargo with mode of transport, vehicle, equipment, and carrier qualifications; sufficient capacity for schedules, routes, and transshipment resources; continuity of cold chain power supply, hazardous materials isolation, and compliance with permits; compliance with driver working hours, energy endurance, load limits, and prohibited routes; and pre-set robust margins for subsequent transshipment connections.

[0038] The route segment contract is rolled only by the confirmed position:

[0039] Among them, RollContract freezes completed route segments, in order to It serves as the starting anchor for the next contract and generates incremental contract versions; unconfirmed actions are not included in this conditional judgment.

[0040] The current confirmation status is indicated as follows:

[0041] in, This is the current confirmed route location. The path prefix has been confirmed as complete. The remaining path is unconfirmed. For the already confirmed networks, resources, and enforcement facts.

[0042] Step S102: Reinforcement learning strategy generates nominal network scheduling actions. The route skeleton strategy generates a route skeleton with segment boundary anchors based on a time-varying transportation map, task origin and destination points, current confirmed location, and path hard constraints. The route skeleton can be generated by a reinforcement learning strategy, a graph search algorithm, or a combination of reinforcement learning and graph search. According to a preset reinforcement learning scheduling strategy, one or more nominal network scheduling actions are generated within the action space defined by the activity route segment contract.

[0043] The nominal action is generated by the reinforcement learning strategy:

[0044] Nominal actions include waiting, delaying execution, selecting the next transport segment, switching candidate connections, adjusting resources or job slots, bypassing unavailable nodes, or adjusting local paths in the current active segment.

[0045] The nominal scheduling action mentioned above only represents a strategy suggestion and does not indicate that it has passed security constraints, has been sent, or has been executed by the external environment.

[0046] The preset reinforcement learning scheduling strategy can be implemented using a value function, action-value function, policy function, actor-commentator structure, or a combination thereof. Rule engines, constraint search, shortest path, integer programming, knowledge bases, and prediction services can be used to generate finite candidates, encode states, or assist in evaluation, but cannot replace the reinforcement learning experience updated based on confirmed action state transitions.

[0047] The reinforcement learning scheduling strategy can adopt an actor-commentator structure. The commentator's temporal difference objective is calculated based on confirmation transitions and action duration discounts, while the actor's objective is to maximize the value of confirmation actions and reduce the probability of nominally scheduled actions that are expected to require safety intervention.

[0048] Step S103: Perform predictive projection and unified accessibility assessment Before the first candidate nominal scheduling action in this decision-making cycle enters the reachability evaluation, an evaluator snapshot is created and frozen. The evaluator version, path search and ranking rules, K value, hard constraint type and its decision rules, normalization scale, aggregation weight, decision threshold, search calculation upper limit, route segment contract version, and decision identifier are bound to this snapshot. Subsequent candidate action evaluations, late receipt confirmations, and confirmation phase re-evaluations all reference this snapshot, and new evaluation parameters published during the period are not read.

[0049] An evaluator snapshot is created before the first candidate action is evaluated. The evaluator snapshot is represented as follows:

[0050] in, For decision identification. The snapshot freezes the hard constraint type and its judgment rules, search rules and evaluation parameters; the route location, time, resource consumption, obstacle envelope and remaining budget in the prediction and confirmation phases are fact inputs and can vary with the prediction and confirmation results.

[0051] The lifecycle state of an evaluator snapshot is represented as follows:

[0052] In each lifecycle state, ACTIVE indicates activity, CONFIRMED indicates confirmation, EXCLUDED indicates exclusion, UNCONFIRMABLE indicates inability to confirm, and RELEASED indicates release.

[0053] An activity snapshot can only be converted to RELEASED after the corresponding action record has been confirmed, excluded due to security rejection or confirmed not to be received, confirmed not to be executed, or the retention period has been reached and closed as unconfirmable; the snapshot identifier, parameter summary, and audit hash are still retained after release.

[0054] For each candidate nominal scheduling action, the local state after the action is executed is first predicted based on the currently confirmed local state, the candidate action, and the time-indexed obstacle envelope. The predicted local state includes at least the estimated time to leave the current segment, the estimated next node or edge to enter, the estimated resource usage range, the estimated time consumption, and the risk budget.

[0055] The predicted local state after the candidate action is executed is represented as follows:

[0056] The predicted local state is transformed into the predicted route state through a local-global state projection function:

[0057] The projection result includes at least the current route location, the estimated or confirmed departure time from the current segment, the next connecting node, the resource occupancy interval, the remaining time window, the remaining risk budget, and the availability of subsequent connections. The prediction and confirmation phases share the same projection rules and field semantics, but the input data comes from the prediction model and confirmed facts, respectively. Before constructing the prediction residual graph, the current time-varying transportation graph is advanced from the decision time to the prediction time based on the predicted arrival time and time-indexed obstacle envelope of candidate actions, updating the predicted availability status of nodes, edges, capacity, resources, and time slots to obtain the prediction base graph. Then, the prediction base graph is trimmed according to the predicted route status and prediction residual constraints to form the prediction residual graph.

[0058] The prediction residual graph corresponding to the candidate action is represented as follows:

[0059] The prediction base graph shown in the first term of the formula is obtained by advancing the current time-varying transportation graph according to the predicted arrival time and execution time state of the candidate actions. During the advancement process, the predicted availability state of nodes, edges, capacity, resources and time slots is updated according to the time index obstacle envelope. Prune only prunes nodes and edges on this prediction base graph that are inconsistent with the projected route state, route segment contract or hard constraints.

[0060] in, and These are the predicted remaining time window and the predicted remaining risk budget after the candidate action is executed, respectively. It is a set of hard constraints for node edge availability, time window, capacity, resources, route segment contracts, and risk budget.

[0061] In the predicted residual graph, acyclic paths are enumerated sequentially from the predicted route position to the target node according to a fixed comprehensive cost, and path-level hard constraints are checked immediately after each path is generated; the search stops when K feasible paths are accumulated, the candidate search space is exhausted, or the preset search calculation limit is reached.

[0062] The search returns the set of the top K acyclic feasible paths and the search status.

[0063] Where K is a preset integer greater than or equal to 1. The overall cost of sorting for a fixed path. A search computation upper limit greater than zero is set to prevent the computational cost of path enumeration from becoming unbounded. according to Generate acyclic paths in ascending order, and only include those that satisfy... The path is added to the set. This indicates that K feasible paths have been obtained. This indicates that the candidate search space has been exhausted. This indicates that the computational limit has been reached, but the search space has not yet been exhausted.

[0064] Only when the search space is exhausted and no feasible path is found can the remaining path corresponding to the candidate action be considered unreachable; when the computational limit is reached but no feasible path is found, the evaluation result is only considered uncertain and conservative rejection is performed. It should not be recorded as proven unreachable or directly generate a reachability result with a value of zero.

[0065] This indicates that path P satisfies All path-level hard constraints in the equation. The number of feasible paths is defined as:

[0066] For each feasible path P, calculate the time margin and risk margin respectively:

[0067] From the first set of K acyclic feasible paths, a baseline path is selected based on the same comprehensive cost as in the path search phase:

[0068] All margins are derived from the same baseline path. The optimal time margin and optimal risk margin of different paths must not be combined into a false path.

[0069] The unified remaining path reachability evaluator is represented as:

[0070] in, This is a snapshot of the evaluator with a fixed execution cycle for this decision. , , , , All three are not less than zero and their sum is 1. Evaluation value and threshold. All are located in the interval [0,1]. When the search state is... If at least one feasible path has been found, the result of the calculation of the obtained set is used as a conservative lower bound; if no feasible path has been found, an uncertain flag is returned. It must not be recorded as zero or used as a training label for "proven unreachable".

[0071] Step S104: Generate mapping action and record sending facts A set of safe actions is formed based on immediate hard constraints and predicted reachability thresholds.

[0072] The set of safety actions is represented as follows:

[0073] A deterministic implementation of the mapping action is as follows:

[0074] when When empty, .

[0075] If the nominal action meets the admission criteria, it is retained as the mapped action; if the nominal action does not meet the criteria but other safe actions exist, it is modified or replaced with the mapped action; if no action meets the criteria, a safety denial flag is output.

[0076] Mapping actions are formed based on the set of safety actions and predicted reachability. These mapping actions are represented as follows:

[0077] in, It can be a network scheduling action that meets the sending conditions, or a security rejection flag. .

[0078] The set of security action mapping parameters includes at least action difference metrics, latency costs, risk costs, predicted reachability weights, admission thresholds, and deterministic ranking rules for parallel candidate actions; a fixed version of this parameter set is used within the same decision cycle. Security denial occurs before action transmission and can generate a security intervention tag, but does not constitute a transmission fact or actual state transition. Transmission is only conducted through the control interface, API, message interface, RPA executor, or device adapter if the mapped action is a transmittable network scheduling action; security denial tags do not invoke external transmission interfaces. The transmission record simultaneously stores the decision identifier, route segment contract version, and evaluator snapshot identifier.

[0079] A send action is initiated after the external interface confirms receipt; if security denial or there is no evidence of receipt, the send action is undefined.

[0080] in, For receiving evidence through external interfaces, This indicates whether the received evidence is valid; Payload represents the payload actually received by the external interface, not the payload that the caller planned to send.

[0081] The "undefined" flag in the send action formula indicates that there is no verifiable send action fact in this decision cycle, and is not equivalent to a no-operation action, a security rejection flag, or a pending confirmation status after sending. When communication fails and no external reception occurs, the mapped action must not be recorded as a send action, nor should the communication failure be incorrectly marked as a security rejection.

[0082] Step S105: Generate a confirmation action or a record to be confirmed based on the execution feedback. The system associates the sending action with the task identifier, path version, route segment identifier, local decision identifier, sending sequence number, and idempotency identifier. The execution status confirmation unit collects equipment receipts, business flow data, transport object location, node entry or exit records, resource usage changes, obstacle observations, and job completion status, and matches them with the actual sending action.

[0083] Once the execution feedback and status evidence meet the confirmation conditions, a confirmation action and confirmation transfer are generated:

[0084] in, For execution receipts and status evidence. The above three items are only defined if the confirmation conditions are met.

[0085] The confirmation condition is expressed as follows:

[0086] When the sent action, execution feedback, and status evidence meet the consistency conditions, the confirmation action, confirmation of the next local state, action duration, and actual reward are reconstructed based on the actual execution volume.

[0087] The confirmation action can be the same as the sending action, or it can differ due to partial execution, execution deviation, or external system correction. A confirmation of non-execution is formed when feedback evidence proves that the external system explicitly refused to execute, and it can be determined that the status of the transport object and network resources has not changed due to the sending action.

[0088] Confirm that no action was performed using the preset no-operation code:

[0089] in, This indicates that no action was performed, but it is not the same as pending confirmation, communication failure, or security denial; the branch still needs to be confirmed. and Only then can a confirmed transfer be formed.

[0090] when When =0, only records awaiting confirmation are generated:

[0091] Undefined record for confirmation , or Therefore, it does not constitute a state transition in reinforcement learning.

[0092] Subsequently, based on the late receipt, supplementary evidence, or manual verification results, the pending confirmation record will be promoted to a confirmed execution record, or promoted to a confirmed non-execution empty operation record when the status and reward after clear refusal to execute and non-execution can be determined; if it still cannot be determined, it will be closed as an unconfirmable record.

[0093] During partial execution, a confirmed transfer is formed only when the executed portion can be broken down into identifiable atomic scheduling actions and the corresponding next state and actual reward can be determined based on evidence; the remaining unconfirmed portion remains in the pending confirmation record.

[0094] Step S106: Reuse the same evaluator snapshot and perform bounded replanning. The confirmed local next state is transformed into the confirmed route state using the same local-global state projection function and the same route segment contract as in step S103:

[0095] The projection results include at least the current route location, the estimated or confirmed departure time from the current segment, the next connecting node, the resource usage interval, the remaining time window, the remaining risk budget, and the availability of subsequent connections. The prediction and confirmation phases share the same projection rules and field semantics, but the input data comes from the prediction model and confirmed facts, respectively.

[0096] A confirmation residual graph is generated based on the confirmed route status, the network availability envelope at the time of confirmation, and the remaining constraints of confirmation.

[0097] Retrieve the original evaluator snapshot that is bound to the same decision identifier as the prediction stage before sending, and use the same path search and sorting rules, K value, hard constraint type and decision rules, normalization scale, aggregation weight and decision threshold to calculate and confirm the reachability value:

[0098] Within the same decision-execution cycle, the evaluator version Parameter set Each aggregation weight and threshold The upper limit for search calculations remains unchanged. Late receipts are used. Retrieve the original snapshot Do not use the default evaluator version when a late receipt arrives.

[0099] If the evaluation returns an uncertain flag because the search computation limit has been reached and no feasible path has been found, the search will continue according to the preset secondary computation budget in the evaluation snapshot. If the search is still uncertain after the secondary budget is completed, a conservative bounded replanning or manual review will be triggered, but the result must not be recorded as a confirmed reachability that has been proven unreachable or has a value of zero.

[0100] The replanning trigger condition is expressed as:

[0101] when In this case, the reason for replanning is recorded as evaluation uncertainty, rather than the remaining path being proven unreachable.

[0102] If the reachability value is confirmed to be no less than the threshold, update the local action mask and continue local obstacle avoidance scheduling; if the route position is confirmed to have reached the downstream exit anchor point of the current contract, freeze the completed segment and generate the next route segment contract version with that exit anchor point as the new starting anchor point. Contracts must not be rolled forward before the exit anchor point is reached.

[0103] If the reachability value is confirmed to be below the threshold or remains uncertain after the second-level calculation budget, the scope of replanning shall be determined according to the following rules: The initial boundary is the current confirmed node, the current edge endpoint, or the safe-to-switch anchor point corresponding to the confirmed route status in the time-varying transportation graph. The initial end boundary is the first downstream anchor point in the original remaining path that is not affected by the current obstacle envelope, still satisfies the hard constraints, and can connect with the original route; The initial search subgraph includes the starting boundary, the ending boundary, the affected nodes and edges, and their preset one-hop adjacency range; When there is no feasible path in the initial search subgraph, the search is expanded step by step according to the one-hop adjacency range, the preset time range, or the anchor point of the next route segment. When the maximum number of hops, the maximum number of nodes, the maximum computation time, or the maximum expansion range specified in the route segment contract is reached, and the expanded search space has been exhausted and there is still no feasible path, or the original downstream anchor point has become invalid, replanning is performed on all unconfirmed remaining paths from the current confirmed position. If the computational limit is reached but the search space is not exhausted, it should be recorded as an uncertain evaluation and should not be stated as having been proven to be unsolvable. Confirmed path prefixes are not included in the search subgraph and are not modified or rolled back due to replanning.

[0104] The initial replanning boundary is represented as:

[0105] in, This is the first downstream anchor point in the original remaining path that is unaffected by obstacles, satisfies hard constraints, and is still connectable. If there is no feasible path within the j-th level boundary, then the path is expanded according to the preset expansion operator:

[0106] When the maximum expansion level is reached Maximum computation time If the maximum range stipulated in the route segment contract is reached, and the extended search space has been exhausted and there are still no feasible paths, then replanning is performed on all remaining unconfirmed paths from the current confirmed position; if only the computational limit is reached and the search space has not been exhausted, the evaluation uncertainty state is retained and conservative processing is performed. Do not enter any replanning search subgraph.

[0107] Once the sending record corresponding to this decision identifier has been confirmed, excluded as definitely not executed, or reached the retention period and closed as unconfirmable, release the active reference of the evaluator snapshot; before release, at least the snapshot identifier, parameter digest, and audit hash should be retained to support sample traceability and recalculation.

[0108] Step S107: Generate learning samples and update the policy according to the closed admission rule. Actual action experience is formed and written to the actual action experience pool only when the confirmed action, confirmed next state, actual reward, action duration, and confirmation basis are all determined. The action field in the actual action experience must store the confirmed action; it must not store nominal actions, mapped actions, or actions that only send evidence.

[0109] Confirmed practical experience is expressed as follows:

[0110] Practical experience is required if and only if the admission criteria are:

[0111] in, Mark the end of a task or route segment.

[0112] As a condition for the completeness of the actual action experience tuple, the action duration, termination flag, and confirmation basis must also have definite values; among them, the confirmation basis must be able to be associated with the sending facts, execution feedback, and status evidence. If any field is missing, only the audit record is retained, and it is not included in the actual action experience pool.

[0113] An action that has not been executed is only transferred to the actual action experience pool if the no-operation code, the confirmation status after the action has not been executed, the actual reward, and the basis for confirmation are all definite. If the status or reward after the action has not been executed cannot be determined, the corresponding confirmation record is either closed or retained, and zero-reward samples must not be constructed.

[0114] When a nominal action is retained, modified, replaced, or denied security, a security intervention label can be generated.

[0115] Safety intervention labels are indicated as follows: when At that time, the above admission conditions still apply, and additionally require that evidence of refusal to execute, confirmation status after non-execution, and actual reward have all been determined; unknown execution results may not be rewritten as no operation or zero reward in order to close the sample.

[0116]

[0117] The intervention type field in the formula can be retained, modified, replaced, or rejected for safety; the constraint reason code is used to identify reasons such as immediate hard constraints, insufficient predictability, or evaluation uncertainty; the action difference code represents the deterministic difference between the mapped action and the nominal action, and the preset rejection code is used when rejecting for safety.

[0118] The definition of inadequate accessibility is:

[0119] when At that time, the intervention label retains the reason for "evaluation uncertainty", but does not use this field as input for numerical action-corrected loss or inaccessibility regression loss.

[0120] The basic access criteria for safety intervention labels are:

[0121] The safety evaluation completion indicator in the formula is determined according to the following rules: It is set to 1 only when the current nominal action has completed hard constraint judgment and predictive reachability evaluation using the bound evaluator snapshot, and formed a traceable mapping result and cause field; it is set to 0 when communication fails, safety evaluation is not performed, or evaluation input is incomplete. Records of communication failure, pending confirmation, and unconfirmed status are not automatically entered due to their communication or confirmation status. .

[0122] The complete admission criteria are as follows: a security intervention tag will only enter the security intervention tag pool if the nominal action is defined, the mapping result is a sendable action or a security denial tag, the fields of intervention type, constraint reason, action difference and insufficient reachability are defined, and the basic admission formula is met; if any of the above fields are missing, the tag will not enter the pool.

[0123] Communication failure does not constitute a security intervention and should not be added to the security intervention tag pool solely because of communication failure.

[0124] Records that have been sent for confirmation are placed in a separate confirmation area and will not be included in the actual action experience pool or the safety intervention tag pool.

[0125] Confirmed actual action experience and intervention labels that have completed safety assessments can be associated with the same decision identifier, but they are used separately for reinforcement learning state transition updates and safety intervention learning, and are not merged into the same sample.

[0126] Samples awaiting confirmation shall not be included in any calculation of strategy loss, value loss, intervention classification loss, action correction loss, or accessibility calibration loss.

[0127] The following is an example of a discrete motion actor-critic, used for illustration. and How to drive learning.

[0128] For from The sampling confirmation transition i, considering the discount factor for the action duration, is:

[0129] The critic's TD goal is:

[0130] in, Can be the target actor It will be generated in the next confirmation state. The commentator's mean square loss is:

[0131] The actor's goal is to maximize the confirmed value of the action and reduce the nominal probability of actions that are expected to require safety intervention:

[0132] in, Let be the intervention probability prediction function. For The intervention type label in the data uses cross-entropy loss:

[0133] The j-th safety intervention sample has a corresponding intervention type category code; the true label in the classification loss formula is the one-hot expansion of the category code, and the prediction term is the category probability output by the intervention type prediction model.

[0134] For samples that can be modified or replaced with a definable action correction amount, a regression loss is used:

[0135] The calibration loss for predicted and confirmed accessibility is determined using only records that simultaneously provide both numerical predictive and numerical confirmation results; the predicted or confirmation result is... Records are not eligible for this loss:

[0136] The evaluation context features of the i-th confirmed sample include at least the route segment contract version, obstacle envelope summary, search status, number of feasible paths, and remaining constraints; the accessibility calibration model is controlled by parameter η and is used to map the predicted accessibility value and the context features to an estimate of the confirmed accessibility value.

[0137] The calibration model is used only to evaluate prediction errors or adjust subsequent prediction models; it does not modify models already fixed within the same decision-making cycle. Snapshots, hard constraints, or thresholds.

[0138] When the route skeleton strategy needs to be updated using reinforcement learning, a route-level confirmation transition is formed after the route segment or task phase ends, based on the confirmation passage time, confirmation risk consumption, connection completion status, and target arrival status:

[0139] The TD objective and mean squared loss for route-level commentators are expressed as follows:

[0140] in, Only save route-level transfers that have been completed or confirmed in the task phase. Do not use unconfirmed local results to directly generate... , Or update the route skeleton strategy.

[0141] One implementation method for total training loss is:

[0142] The target network uses soft updates:

[0143] in, and All are non-negative loss weights; when route-level reinforcement learning is not enabled, the weights are taken as follows: . Records pending confirmation The sample set that does not belong to any of the above loss functions.

[0144] Example 2: Scheduling Agent This embodiment provides a multi-mode dynamic path planning and obstacle avoidance scheduling intelligent agent based on reinforcement learning, including the following modules: Network status and segment contract formation module: The network state and segment contract formation module is used to obtain the current confirmed location, time-varying transportation network state, and time-indexed obstacle envelope. Based on the time-varying transportation network state, a time-varying transportation graph is constructed to determine the active route segment contract and its remaining constraints from the current confirmed location to the downstream exit anchor point. The time-varying transportation graph is constructed by collecting transportation network nodes and edges, transportation modes, estimated travel time, capacity, resource availability intervals, shifts or work slots, confirmed locations of transported objects, historical confirmed path segments, and dynamic obstacle observations. The time-indexed obstacle envelope describes the unavailable intervals, capacity boundaries, or occupancy probability boundaries of nodes, edges, resources, or connections within the prediction time domain, and uses conservative boundaries, upper bounds of confidence intervals, or probability quantiles at a pre-set confidence level to form occupancy intervals for hard constraint judgment. The active route segment contract includes: the current confirmed starting anchor point, the downstream exit anchor point, the set of allowed nodes and edges, the latest departure time, the remaining time window, capacity and resource constraints, the remaining risk budget, and the maximum replanning expansion range. The activity route segment contract freezes the completed route segment only after confirming that the downstream exit anchor point has been reached. The downstream exit anchor point is then used as the starting anchor point for the next route segment, and the next contract version is generated based on the remaining route skeleton and the latest confirmed constraints.

[0145] Reinforcement learning scheduling action generation module: The reinforcement learning scheduling action generation module generates nominal scheduling actions within the action space defined by the activity route segment contract, based on a preset reinforcement learning scheduling strategy. The preset reinforcement learning scheduling strategy can be implemented using a value function, action-value function, policy function, actor-commentator structure, or a combination thereof. Nominal scheduling actions include waiting, delaying execution, selecting the next transport segment, switching candidate connections, adjusting resources or job slots, bypassing unavailable nodes, or adjusting local paths within the current activity segment.

[0146] Unified accessibility assessment and snapshot management module: The unified accessibility evaluation and snapshot management module is used to assign a decision identifier to the current decision cycle and create an evaluator snapshot bound to the decision identifier. The evaluator snapshot includes the evaluator version, evaluation rules, judgment threshold, and search computation limit, and remains unchanged within the current decision cycle. The evaluator snapshot also includes path search and ranking rules, hard constraint types and their judgment rules, normalization scale, aggregation weight, route segment contract version, and search computation limit. The lifecycle status of the evaluator snapshot includes active, confirmed, excluded, unconfirmable, and released. An active snapshot is converted to released only when the corresponding sending action has been confirmed, excluded due to security rejection or confirmed non-receipt, confirmed but not executed, or reaches its retention period and is closed as unconfirmable. After release, the snapshot identifier, parameter digest, and audit hash are retained. When a late receipt arrives, the original evaluator snapshot is loaded according to the original decision identifier to perform a confirmed accessibility evaluation, without switching to the current default evaluator version.

[0147] Local-Global State Projection Module: The local-global state projection module is used to predict the transportation state after the execution of the nominal scheduling action. It converts the transportation state into a predicted route state through state projection and calculates the predicted reachability value and search state of the nominal scheduling action according to the evaluation rules in the evaluator snapshot. The search state includes at least a search space exhaustion state and a computational upper limit truncation state. Predicting the transportation state after the execution of the nominal scheduling action includes: based on the currently confirmed local state, the nominal scheduling action, and the time-indexed obstacle envelope, predicting the estimated departure time from the current segment, the estimated next node or edge to enter, the estimated resource usage interval, the estimated time consumption, and the risk budget. Converting the transportation state into a predicted route state through state projection includes: calling the local-global state projection function to form a predicted route state based on the predicted local state and the active route segment contract. The predicted route state includes at least the route location, the next connecting node, the remaining time window, the remaining risk budget, the availability of subsequent connections or services, and the confirmed path prefix. The calculation of predicted reachability values ​​and search status based on the evaluation rules in the evaluator snapshot includes: advancing the current time-varying transportation graph from the decision time to the prediction time based on the predicted arrival time and time index obstacle envelope, updating the predicted availability status of nodes, edges, capacity, resources, and time slots to obtain a prediction base graph; trimming the prediction base graph according to the predicted route status and prediction residual constraints to form a prediction residual graph; enumerating acyclic paths in the prediction residual graph according to a fixed comprehensive cost and verifying hard constraints on time windows, risk budgets, capacity, resources, and node / edge availability, and obtaining the first K acyclic feasible paths and search status. The search stops when the search space is exhausted or the upper limit of the search computation is reached, and the state of exhaustion of the search space or truncation of the upper limit of the computation is recorded. The predicted reachability value is considered to be zero only when the search space is exhausted and no feasible path is found. When the upper limit of the computation is reached but no feasible path is found, an evaluation uncertainty flag is returned. For the first K feasible paths obtained, the feasible path with the minimum comprehensive cost is used as the baseline path, and the time margin and risk margin are calculated respectively. The predicted reachability value is calculated according to the aggregation weight, normalization scale and judgment threshold in the evaluator snapshot. When the predicted reachability value is not lower than the judgment threshold, it is determined to be a candidate that can be sent.

[0148] Action mapping and sending module: The action mapping and sending module is used to map the nominal scheduling action to a sendable mapping action or a security rejection flag based on the hard constraints, the predicted reachability value, and the search state. After sending the sendable mapping action and receiving a reception confirmation from the external interface, the sending action is recorded. The sending action is associated with the decision identifier and the evaluator snapshot. Mapping the nominal scheduling action to a sendable mapping action or a security rejection flag based on the hard constraints, the predicted reachability value, and the search state includes: forming a set of security actions based on the immediate hard constraints and the predicted reachability threshold; retaining the nominal scheduling action as a mapping action when it meets the admission conditions; modifying or replacing it with a mapping action when it does not meet the conditions but other security actions exist; and outputting a security rejection flag when no action meets the conditions. The security rejection flag is formed before the action is sent and is used to form a security intervention label; it does not form a sending action or an actual state transition. The step of recording the sending action after sending the sendable mapping action and receiving a reception confirmation from the external interface includes: sending via the control interface, API, message interface, RPA executor, or device adapter only when the mapping action is a sendable network scheduling action; the sending record simultaneously saves the decision identifier, route segment contract version, and evaluator snapshot identifier; the actual payload received by the external interface is recorded as a sending action only when the external interface returns a received result or verifiable reception evidence exists; if communication fails and no external reception is received, no sending action is recorded, nor is it recorded as a security rejection.

[0149] Execution status confirmation module: The execution status confirmation module is used to form a confirmation action and a confirmation local state, or to form a record to be confirmed, based on the execution feedback and status evidence matched with the sending action. Forming the confirmation action and confirmation local state, or forming a record to be confirmed, based on the execution feedback and status evidence matched with the sending action includes: associating the sending action with a task identifier, path version, route segment identifier, local decision identifier, sending sequence number, and idempotency identifier; collecting equipment receipts, business flow, transport object location, node entry or exit records, resource usage changes, obstacle observation, and job completion status, and matching them with the sending action facts; when the sending action, execution feedback, and status evidence meet the consistency condition, reconstructing the confirmation action, confirming the next local state, action duration, and actual reward based on the actual execution volume; the confirmation action can be the same as the sending action or different due to partial execution, execution deviation, or external system correction; when the external system explicitly refuses to execute and the state after non-execution can be determined, a preset no-operation state is formed. The coded confirmation of unexecuted actions constitutes a confirmation fact based on the state after confirmation of non-execution and the recalculated reward. When a message has been sent but feedback is missing, evidence is insufficient, or evidence conflicts, only a pending confirmation record is formed. This pending confirmation record does not define a confirmation action, the next state, or the actual reward, and does not form a reinforcement learning state transition. Subsequently, based on late receipts, supplementary evidence, or manual verification results, the pending confirmation record is promoted to a confirmed execution record, or promoted to a confirmed non-executed no-operation record when execution is explicitly refused and the state and reward after non-execution can be determined. If it still cannot be determined, it is closed as an unconfirmable record. During partial execution, only when the executed part can be broken down into identifiable atomic scheduling actions and the corresponding next state and actual reward can be determined based on evidence, is a confirmation transition formed based on the confirmed execution amount. The remaining unconfirmed part is retained in the pending confirmation record.

[0150] Confirmation and evaluation module: The confirmation evaluation module is used to convert the confirmed local state into a confirmed route state through the state projection, and calculate the confirmed reachability value based on the evaluator snapshot bound to the decision identifier. The process of converting the confirmed local state into a confirmed route state through the state projection and calculating the confirmed reachability value based on the evaluator snapshot bound to the decision identifier includes: converting the next confirmed local state into a confirmed route state using the same local-global state projection function and the same activity route segment contract as in the prediction phase; forming a confirmed residual graph based on the confirmed route state, the network availability envelope at the confirmation time, and the remaining constraints; calling the original evaluator snapshot bound to the same decision identifier as in the pre-send prediction phase, using the same path search and sorting rules, K value, hard constraint type and decision rules, normalization scale, aggregation weight, and decision threshold to calculate the confirmed reachability value; allowing input of the latest confirmed location, confirmation time, confirmed resource occupancy, and confirmed obstacle facts in the confirmation phase, but without changing the evaluation rules in the snapshot; when the confirmation evaluation returns an uncertain flag because it has reached the search calculation limit and has not yet obtained a feasible path, the search continues according to the preset secondary calculation budget in the evaluator snapshot; if it is still uncertain after the secondary budget is completed, a conservative bounded replanning or manual review is triggered.

[0151] Boundary replanning module: The bounded replanning module is used to perform bounded remaining path replanning when the confirmed reachability value is lower than the judgment threshold or the search state is a calculation upper limit truncation, with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary. The bounded remaining path replanning refers to replanning with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary. The bounded remaining path replanning includes: the starting boundary is the currently confirmed node, the current edge endpoint, or the anchor point that can be safely switched in the time-varying transportation graph corresponding to the confirmed route status; the initial ending boundary is the first downstream anchor point in the original remaining path that is not affected by the current obstacle envelope, still satisfies the hard constraints, and can connect with the original route; the initial search subgraph includes the starting boundary, the ending boundary, the affected nodes and edges, and their preset one-hop adjacency range; when there is no feasible path in the initial search subgraph, it is expanded step by step according to the one-hop adjacency range, the preset time range, or the next route segment anchor point; when the maximum number of hops, the maximum number of nodes, the maximum calculation time, or the maximum expansion range specified by the route segment contract is reached, and the expanded search space has been exhausted and there is still no feasible path, or the original downstream anchor point has failed, replanning is performed on all unconfirmed remaining paths from the current confirmed position; when only the calculation limit is reached but the search space has not been exhausted, it is recorded as uncertain; confirmed path prefixes are not included in the search subgraph.

[0152] Reinforcement learning update module: The reinforcement learning update module is used to update the preset reinforcement learning scheduling strategy only by using state transitions containing the confirmation action, confirmation local state, and actual reward. The step of using state transitions containing the confirmation action, confirmation local state, and actual reward only to update the preset reinforcement learning scheduling strategy includes: forming actual action experience and writing it into the actual action experience pool only when the confirmation action, confirmation next state, actual reward, action duration, and confirmation basis are all determined; the action field in the actual action experience stores the confirmation action; when the confirmation of an unexecuted action is determined, and the no-operation code, the confirmation state after non-execution, the actual reward, and the confirmation basis are all determined, it is entered into the actual action experience pool as a confirmation transition; when the nominal scheduling action is retained, modified, replaced, or security rejected, a security intervention label is formed, the security intervention label including at least the nominal scheduling action, the mapped action or security rejection label, the intervention type, the constraint reason, the action correction amount, the predicted reachability value, and the reachability insufficiency amount; when security is rejected, the action correction amount is changed from the preset rejection code or nominal scheduling action to... The deterministic encoding representation of the rejection action; the inadequacy of accessibility is defined as the non-negative part of the difference between the predicted threshold and the predicted accessibility value; communication failures are not written into the security intervention tag pool; sent records awaiting confirmation enter an independent awaiting confirmation area and do not enter the actual action experience pool or the security intervention tag pool; awaiting confirmation samples do not participate in any policy loss, value loss, intervention classification loss, action correction loss, or accessibility calibration loss calculation; the preset reinforcement learning scheduling strategy adopts an actor-commentator structure, the commentator's temporal difference objective is calculated based on confirmation transition and action duration discount, and the actor's objective is to maximize the confirmation action value and reduce the probability of the nominal scheduling action that is expected to require security intervention; security intervention tags are used to calculate intervention type classification loss, action correction regression loss, or policy intervention penalty; the calibration loss of predicted accessibility and confirmed accessibility only uses records that have both numerical prediction evaluation and numerical confirmation results. The agent can be deployed in servers, edge computing nodes, vehicle terminals, station control equipment, or a distributed combination thereof. In distributed deployment, task-level path planning can be performed on the server, and execution-level conflict avoidance and security shielding can be performed on edge nodes or vehicle terminals close to the execution device.

[0153] Example 3: Scheduling System This embodiment provides a multi-modal dynamic path planning and obstacle avoidance scheduling system based on reinforcement learning, including the scheduling agent described in the above-described agent embodiment, as well as a data acquisition interface and a tool execution adapter.

[0154] The data acquisition interface is used to acquire time-varying transportation network status, time-indexed obstacle envelopes, execution feedback, and status evidence, and provide them to the scheduling agent. The data acquisition interface can access heterogeneous data related to the transportation network topology and operational status through multi-source data interfaces, and complete format normalization and spatiotemporal alignment in a unified mode to generate standardized records with spatiotemporal keying.

[0155] The data acquisition interface can also perform rule-driven completion and uncertainty quantification for missing or uncertain entries under preset boundary conditions.

[0156] The tool execution adapter is used to send the sendable mapping actions output by the scheduling agent to an external system, and to receive reception confirmation and execution feedback from the external interface.

[0157] The tool execution adapter can send actions through the control interface, API, message interface, RPA executor, or device adapter. For the RPA executor, before execution, it reads the page's business object and current state, compares them with the preceding states in the tool's call plan; if they match, it fills in the fields and submits; if they don't match, it terminates the operation. After submission, RPA extracts the business serial number and changed state from the results page, list status, or business details, and compares them with the expected result. If the result cannot be confirmed, the call status is marked as "Result Unknown," and verification is performed through a read-only query before directly clicking submit again.

[0158] The scheduling system may also include the following units: The state and model storage unit is used to store time-varying transportation maps, obstacle envelopes, route segment contracts, evaluator version snapshots, action fact records, areas awaiting confirmation, actual action experience pools, and safety intervention label pools. The state and model storage unit can also store policy versions, training data snapshots, tool directories, permission policies, and hard constraint sets.

[0159] The strategy training and version management unit is used to sample from the actual action experience pool and the safety intervention tag pool in a hierarchical manner when the preset update conditions are met at the corresponding level, train candidate strategies, and perform preset verification and controlled release on the candidate strategies. The released candidate strategies are used for dynamic path actions or obstacle avoidance scheduling actions in subsequent tasks, and fall back to the previous stable strategy when the operating indicators deteriorate.

[0160] Candidate strategies are first compared against the same historical replay set, abnormal scenario set, and digital twin scenario as the current production strategy. During shadow operation, the candidate strategy receives the same state input as the production strategy and outputs the corresponding action, but does not send write operations. Only when the number of hard constraint violations of the candidate strategy is zero, and the task completion rate, on-time rate, equipment change success rate, security shielding intervention rate, plan disturbance, response latency, and anomaly recovery capability meet the release threshold, will it enter a small-scale gray-scale operation for low-risk tasks. When a strategy is released, the task-level strategy version, execution-level strategy version, state encoder version, action space version, reward function version, applicable domain, gray-scale range, effective time, and rollback version are recorded. When a transportation task is created, the strategy combination version is bound; in-transit tasks generally maintain the original strategy version, and switching to a new version is only allowed when the task-level transportation option boundary is reached, the input / output structure is compatible, and the security check passes.

[0161] The event detection and cross-layer replanning unit receives updated environmental situational information during the execution of the external execution system. This triggers the agent collaboration framework to invoke candidate path generation, resource verification, and decision-making for replanning, and outputs the replaced target solution. The replanning module triggers incremental recalculation with the affected subgraph as the boundary when the environmental situational representation is updated. It combines a safe switching point and a versioned replacement mechanism to complete the minimum-range solution update, ensuring stable reference relationships during execution, atomic external deployments, and rollback capability. The replanning frequency is controlled through throttling and debouncing strategies.

[0162] The completion confirmation and audit unit verifies the handover of destination nodes and the completion status of necessary stages, settles the final reward, and outputs the final plan, autonomous execution receipt, and full-link audit record. The completion confirmation and audit unit determines whether the transportation task is completed based on the destination node arrival receipt, handover or signed receipt, necessary resource release results, and a list of incomplete stages. Only when all necessary completion conditions are met and there are no necessary stages in progress, with unknown results, or awaiting compensation, is the task status updated to "transportation completed"; otherwise, the current status is maintained or an exception is handled. Upon completion, the final plan version is frozen, and the stage status, tool calls, external business serial numbers, and plan changes are linked together to form a full-link audit record.

[0163] The scheduling system can be deployed on a cloud-based scheduling platform, RPA execution nodes, site edge nodes, and vehicle-mounted terminals. Each unit communicates phase plans, execution contracts, tool call plans, execution ledgers, feasibility envelopes, and execution status through a unified task identifier and a data structure with version numbers and expiration dates.

[0164] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-modal dynamic path planning and obstacle avoidance scheduling method based on reinforcement learning, characterized in that, include: Obtain the current confirmed location, time-varying transportation network status, and time-indexed obstacle envelope; construct a time-varying transportation map based on the time-varying transportation network status; determine the active route segment contract from the current confirmed location to the downstream exit anchor point, and the remaining constraints of the active route segment contract, including hard constraints. According to the preset reinforcement learning scheduling strategy, a nominal scheduling action is generated within the action space defined by the contract of the activity route segment; Assign a decision identifier to the current decision cycle and create an evaluator snapshot bound to the decision identifier. The evaluator snapshot includes the evaluator version, evaluation rules, judgment threshold, and search calculation limit. The evaluator snapshot remains unchanged within the current decision cycle. Predict the transportation state after the nominal scheduling action is executed, convert the transportation state into a predicted route state through state projection, and calculate the predicted reachability value and search state of the nominal scheduling action according to the evaluation rules in the evaluator snapshot. The search state includes at least the search space exhaustion state and the computational upper limit truncation state. Based on the hard constraints, the predicted reachability value, and the search state, the nominal scheduling action is mapped to a sendable mapping action or a security rejection flag; After sending the sendable mapping action and receiving a confirmation of receipt from the external interface, the sending action is recorded, and the sending action is associated with the decision identifier and the evaluator snapshot; Based on the execution feedback and status evidence matching the sending action, a confirmation action and a confirmation local state are formed, or a record to be confirmed is formed; The confirmed local state is converted into a confirmed route state through the state projection, and the confirmed reachability value is calculated based on the evaluator snapshot bound to the decision identifier; When the confirmed reachability value is lower than the judgment threshold or the search state is a calculation upper limit truncation, a bounded remaining path replanning is performed with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary; the bounded remaining path replanning refers to the replanning with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary. The state transition, which includes the confirmation action, the confirmation local state, and the actual reward, is used to update the preset reinforcement learning scheduling strategy.

2. The method according to claim 1, characterized in that, The time-varying transportation map is constructed by collecting transportation network nodes and edges, transportation modes, estimated travel time, capacity, available resource intervals, shifts or work slots, confirmed location of transported objects, historical confirmed path segments, and dynamic obstacle observations. The time index barrier envelope describes the unavailable intervals, capacity boundaries, or occupancy probability boundaries of nodes, edges, resources, or connections in the prediction time domain, and uses conservative boundaries, upper bounds of confidence intervals, or probability quantiles under a preset confidence level to form occupancy intervals for hard constraint judgment. The activity route segment contract includes: the currently confirmed starting anchor point, the downstream exit anchor point, the set of allowed nodes and edges, the latest departure time, the remaining time window, capacity and resource constraints, the remaining risk budget, and the maximum replanning expansion range; The activity route segment contract freezes the completed route segment only after confirming that the downstream exit anchor point has been reached. The downstream exit anchor point is used as the starting anchor point of the next route segment. The next contract version is generated based on the remaining route skeleton and the latest confirmed constraints. Nominal scheduling actions, mapping actions, or sending actions do not trigger contract version updates.

3. The method according to claim 1, characterized in that, The evaluation rules include path search and ranking rules, hard constraint types and their judgment rules, normalization scale, and aggregation weights; the evaluator snapshot also includes route segment contract versions and secondary computational budgets. The evaluator snapshot is created before the first nominal scheduling action enters the evaluation, and the evaluator version, path search and sorting rules, hard constraint type and its judgment rules, normalization scale, aggregation weight, judgment threshold, search calculation upper limit, route segment contract version and decision identifier are bound to the evaluator snapshot. The lifecycle states of the evaluator snapshot include active, confirmed, excluded, unconfirmed, and released; the active snapshot is changed to released only when the corresponding sending action has been confirmed, excluded due to security rejection or confirmed not received, confirmed but not executed, or the retention period has been reached and it is closed as unconfirmed; after release, the snapshot identifier, parameter digest, and audit hash are retained; When a late receipt arrives, the original evaluator snapshot is retrieved based on the original decision identifier to perform a confirmation of reachability evaluation, without switching to the current default evaluator version.

4. The method according to claim 1, characterized in that, The prediction of the transportation status after the execution of the nominal scheduling action includes: based on the current confirmed local state, the nominal scheduling action, and the time index barrier envelope, predicting the expected time to leave the current segment, the expected next node or edge to enter, the expected resource occupation interval, the expected consumption time, and the risk budget after the action is executed; The process of converting the transportation state into a predicted route state through state projection includes: calling a local-global state projection function to form a predicted route state based on the predicted local state and the active route segment contract. The predicted route state includes at least the route location, the next connecting node, the remaining time window, the remaining risk budget, the availability of subsequent connections or services, and the confirmed path prefix. The step of calculating the predicted reachability value and search status according to the evaluation rules in the evaluator snapshot includes: advancing the current time-varying transportation graph from the decision time to the prediction time based on the predicted arrival time and time index obstacle envelope, updating the predicted availability status of nodes, edges, capacity, resources and time slots, and obtaining the prediction base graph; The prediction base map is trimmed according to the predicted route status and the predicted residual constraints to form the prediction residual map; In the predicted residual graph, acyclic paths are enumerated sequentially according to a fixed comprehensive cost, and hard constraints such as time window, risk budget, capacity, resources and node edge availability are verified. The search stops when the first K acyclic feasible paths are obtained, the search space is exhausted, or the upper limit of the search calculation is reached. K is a preset integer, and the state of exhaustion of the search space or truncation of the upper limit of the calculation is recorded. The predicted reachability value is set to zero only when the search space is exhausted and no feasible path is found; when the computational limit is reached but no feasible path is found, an evaluation uncertainty flag is returned. For the first K feasible paths obtained, the feasible path with the minimum comprehensive cost is taken as the baseline path. The time margin and risk margin are calculated respectively, and the predicted reachability value is calculated according to the aggregation weight, normalization scale and judgment threshold in the evaluator snapshot. When the predicted reachability value is not lower than the judgment threshold, it is determined as a sendable candidate.

5. The method according to claim 1, characterized in that, Mapping the nominal scheduling action to a sendable mapping action or a security rejection flag based on the hard constraint, the predicted reachability value, and the search state includes: A set of safe actions is formed based on immediate hard constraints and predicted reachability thresholds; When the nominal scheduling action meets the admission conditions, it is retained as a mapping action; when the conditions are not met but other security actions exist, it is modified or replaced as a mapping action; when no action meets the conditions, a security rejection flag is output. The security rejection tag is formed before the action is sent, and is used to form a security intervention tag, but does not form the sending action or actual state transition; The step of recording the sending action after sending the sendable mapping action and receiving a reception confirmation from the external interface includes: Sending is done only when the mapping action is a sendable network scheduling action, via the control interface, API, message interface, RPA executor, or device adapter; the sending record also saves the decision identifier, route segment contract version, and evaluator snapshot identifier; The payload actually received by the external interface is recorded as a sending action only when the external interface returns a received result or verifiable evidence of reception exists; if communication fails and no external data is received, the sending action is not recorded, nor is it recorded as a security rejection.

6. The method according to claim 1, characterized in that, Based on the execution feedback and status evidence matching the sending action, a confirmation action and a confirmation local state are formed, or a record to be confirmed is formed, including: The sending action is associated with the task identifier, path version, route segment identifier, local decision identifier, sending sequence number, and idempotency identifier. Collect equipment receipts, business flow, location of transported objects, node entry or exit records, resource usage changes, obstacle observation, and operation completion status, and match them with the sent action facts; When the sending action, execution feedback and status evidence meet the consistency conditions, the confirmation action, confirmation of the next local state, action duration and actual reward are reconstructed based on the actual execution volume. The confirmation action may be the same as the sending action or different due to partial execution, execution deviation or external system correction. When the external system explicitly refuses to execute and the state after non-execution can be determined, a confirmation of non-execution is formed using a preset no-operation code, and the state after confirmation of non-execution and the recalcible reward constitute the confirmation fact. When a message has been sent but feedback is missing, evidence is insufficient, or evidence is conflicting, only a pending confirmation record is formed. The pending confirmation record does not define a confirmation action, confirmation of the next state, or actual reward, and does not form a reinforcement learning state transition. Subsequently, based on the late receipt, supplementary evidence, or manual verification results, the pending confirmation record will be promoted to the confirmed execution record, or when the status and reward after clear refusal to execute and non-execution can be determined, it will be promoted to the confirmed non-executed empty operation record. If it is still uncertain, it will be closed as an unconfirmable record. During partial execution, only when the executed portion can be broken down into identifiable atomic scheduling actions and the corresponding next state and actual reward can be determined based on evidence, is the execution amount confirmed and a confirmation transfer formed. The remaining unconfirmed portion is retained in the pending confirmation record.

7. The method according to claim 1, characterized in that, The process of converting the confirmed local state into a confirmed route state through the state projection, and calculating the confirmed reachability value based on the evaluator snapshot bound to the decision identifier, includes: The confirmed local next state is transformed into the confirmed route state using the same local-global state projection function and the same active route segment contract as in the prediction phase. A confirmation residual graph is formed based on the confirmed route status, the network availability envelope at the confirmation time, and the confirmation residual constraints. Call the original evaluator snapshot that is bound to the same decision identifier in the prediction stage before sending, and use the same path search and sorting rules, K value, hard constraint type and decision rules, normalization scale, aggregation weight and decision threshold to calculate and confirm the reachability value; The confirmation phase allows input of the latest confirmation location, confirmation time, confirmation resource usage, and confirmation obstacle facts, but does not change the evaluation rules in the snapshot; When the evaluation returns an uncertain flag because the search computation limit has been reached and no feasible path has been found, the search continues according to the preset secondary computation budget in the evaluation snapshot; if it is still uncertain after the secondary budget is completed, a conservative bounded replanning or manual review is triggered. The bounded residual path replanning includes: The initial boundary is the current confirmed node, the current edge endpoint, or the safe-to-switch anchor point corresponding to the confirmed route status in the time-varying transportation graph. The initial end boundary is the first downstream anchor point in the original remaining path that is not affected by the current obstacle envelope, still satisfies the hard constraints, and can connect with the original route; The initial search subgraph includes the starting boundary, the ending boundary, the affected nodes and edges, and their preset one-hop adjacency range; When there is no feasible path in the initial search subgraph, the search is expanded step by step according to the one-hop adjacency range, the preset time range, or the anchor point of the next route segment. When the maximum number of hops, the maximum number of nodes, the maximum computation time, or the maximum expansion range specified in the route segment contract is reached, and the expanded search space has been exhausted and there is still no feasible path, or the original downstream anchor point has become invalid, replanning is performed on all unconfirmed remaining paths from the current confirmed position. When the computational limit is reached but the search space is not exhausted, the evaluation is recorded as uncertain; confirmed path prefixes are not included in the search subgraph.

8. The method according to claim 1, characterized in that, The step of using the state transition, which includes the confirmation action, the confirmation local state, and the actual reward, to update the preset reinforcement learning scheduling strategy includes: When the action, the next state, the actual reward, the duration of the action, and the basis for confirmation are all determined, actual action experience is formed and written into the actual action experience pool. The action field in the actual action experience stores the confirmed action; when the confirmed non-executed action can be determined by the no-operation code, the confirmation status after non-execution, the actual reward and the confirmation basis, it is transferred to the actual action experience pool as a confirmation. When a nominal scheduling action is retained, modified, replaced, or security denied, a security intervention label is formed. The security intervention label includes at least the nominal scheduling action, the mapped action or security denial flag, the intervention type, the constraint reason, the action modification amount, the predicted reachability value, and the reachability insufficiency amount. When security is denied, the action correction amount is represented by a predetermined code from a preset denial code or a nominally scheduled action to a denial action; The inadequacy of accessibility is defined as the non-negative part of the difference between the prediction threshold and the predicted accessibility value; The preset reinforcement learning scheduling strategy adopts an actor-critic structure. The critic's temporal difference objective is calculated based on confirmation transition and action duration discount. The actor's objective is to maximize the confirmation action value and reduce the probability of nominally scheduled actions that are expected to require safety intervention. Safety intervention labels are used to calculate intervention type classification loss, action correction regression loss, or policy intervention penalty. The calibration loss for predicted accessibility and confirmed accessibility only uses records that have both numerical prediction evaluation and numerical confirmation results.

9. A multi-modal dynamic path planning and obstacle avoidance scheduling intelligent agent based on reinforcement learning, characterized in that, include: The network status and segment contract formation module is used to obtain the current confirmed location, time-varying transportation network status and time-indexed obstacle envelope, construct a time-varying transportation map based on the time-varying transportation network status, and determine the active route segment contract and its remaining constraints from the current confirmed location to the downstream exit anchor point. The remaining constraints include hard constraints. The reinforcement learning scheduling action generation module is used to generate nominal scheduling actions within the action space defined by the activity route segment contract according to a preset reinforcement learning scheduling strategy. The unified accessibility evaluation and snapshot management module is used to assign a decision identifier to the current decision cycle and create an evaluator snapshot bound to the decision identifier. The evaluator snapshot includes the evaluator version, evaluation rules, judgment threshold and search calculation limit, and remains unchanged within the current decision cycle. The local-global state projection module is used to predict the transportation state after the nominal scheduling action is executed, convert the transportation state into a predicted route state through state projection, and calculate the predicted reachability value and search state of the nominal scheduling action according to the evaluation rules in the evaluator snapshot. The search state includes at least the search space exhaustion state and the computational upper limit truncation state. The action mapping and sending module is used to map the nominal scheduling action to a sendable mapping action or a security rejection flag according to the hard constraint, the predicted reachability value and the search state. After sending the sendable mapping action and receiving the acceptance confirmation from the external interface, the sending action is recorded. The sending action is associated with the decision identifier and the evaluator snapshot. The execution status confirmation module is used to form a confirmation action and a confirmation local status, or to form a record to be confirmed, based on the execution feedback and status evidence that match the sending action. The confirmation evaluation module is used to convert the confirmed local state into a confirmed route state through the state projection, and calculate the confirmed reachability value based on the evaluator snapshot bound to the decision identifier; A bounded replanning module is used to perform bounded remaining path replanning when the confirmed reachability value is lower than the judgment threshold or the search state is a calculation upper limit truncation, with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary; the bounded remaining path replanning refers to replanning with the current confirmed position as the starting boundary and the first connectable downstream anchor point in the original remaining path as the ending boundary. The reinforcement learning update module is used to update the preset reinforcement learning scheduling strategy by using the state transition that includes the confirmation action, the confirmation local state, and the actual reward.

10. A multi-modal dynamic path planning and obstacle avoidance scheduling system based on reinforcement learning, characterized in that, Includes the scheduling agent as described in claim 9, as well as the data acquisition interface and the tool execution adapter; The data acquisition interface is used to acquire time-varying transportation network status, time-indexed obstacle envelope, execution feedback, and status evidence, and provide them to the scheduling agent; The tool execution adapter is used to send the sendable mapping actions output by the scheduling agent to an external system, and to receive reception confirmation and execution feedback from the external interface.

Images