Method, device and medium for irregular ship scheduling based on ship-level agent collaborative learning

Abstract

The application relates to a kind of irregular ship scheduling method, equipment and medium based on ship level intelligent agent collaborative learning, comprising: constructing irregular ship scheduling snapshot data and initializing ship intelligent agent interactive decision environment;Construct the cooperative finite time domain Markov decision process of ship intelligent agent, configure local observation space, discrete action space and ship level strategy decision neural network for each ship intelligent agent;Each ship intelligent agent uses respective ship level strategy decision neural network, based on the joint action of the same decision moment of respective local observation output;When joint action is candidate cargo selection action and passes through constraint determination, the current node of ship, the loading port and unloading port of selected cargo are determined as fixed service fragment, speed and time feasibility calculation is carried out on fixed service fragment, and scheduling strategy is output when there is feasible arrangement.Compared with prior art, the application effectively improves the feasibility, economy and execution efficiency of complex irregular ship scheduling strategy.

Description

Technical Field

[0001] This invention relates to the field of shipping scheduling, and in particular to an irregular ship scheduling method, equipment and medium based on ship-level intelligent agent collaborative learning. Background Technology

[0002] Irregular shipping typically serves the transportation needs of bulk cargo, project cargo, and special-purpose cargo. Its routes, schedules, and port visit sequences are not fixed in advance but are determined based on cargo demand, vessel location, market contracts, and port operating conditions. During an irregular shipping dispatch, dispatchers need to simultaneously decide which vessel will carry which cargo, the order in which vessels will visit loading and unloading ports, when each service node will commence operations, and the navigation arrangements for each segment of the voyage. This type of problem involves multiple factors, including vessel-cargo matching, route continuity, service sequence, time windows, sailing time, fuel consumption, external carrier costs, and penalties for unserved cargo, making it a high-dimensional combinatorial optimization problem.

[0003] In existing technologies, mathematical programming methods can rigorously express ship flow balance, cargo service, time windows, and capacity constraints. However, as the number of ships, cargo, and port nodes increases, the scale of variables and constraints expands rapidly, and the solution time often fails to meet the needs of repeatedly generating scheduling schemes in rolling plans. Heuristic and metaheuristic methods have certain engineering applicability, but they usually rely on manually designed neighborhood structures, search rules, and parameter configurations. When cargo batches, fleet status, or time window structures change, it may be necessary to re-search or readjust parameters, affecting the quality and stability of the solution.

[0004] Therefore, it is necessary to provide an intelligent scheduling method for irregular ship business structures to improve the feasibility, economy and execution efficiency of scheme generation in complex irregular ship scheduling scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the defects of the prior art by providing a method, device and medium for irregular ship scheduling based on ship-level intelligent agent collaborative learning.

[0006] The objective of this invention can be achieved through the following technical solutions: According to a first aspect of the present invention, an irregular ship scheduling method based on ship-level intelligent agent cooperative learning is provided, comprising: Acquire parameters such as ships, cargo, ports, routes, time windows, and costs to generate snapshot data of irregular ship scheduling, and initialize the ship agent interactive decision-making environment for irregular ship route scheduling constraints; Each ship in the fleet is modeled as a ship agent, and a cooperative finite-time Markov decision process for the ship agents is constructed. Each ship agent is configured with a local observation space, a discrete action space, and a ship-level policy decision neural network. The parameters in the local observation space include the ship's current position, the ship's current time, and the cargo sharing status. Each ship intelligent agent uses its own pre-trained ship-level strategy decision neural network to output a joint action at the same decision moment based on its own local observations. The joint action includes idle action or candidate cargo selection action. When the joint action is a candidate cargo selection action and passes the constraint determination, the current node of the ship, the loading port and unloading port of the selected cargo are determined as fixed service segments. The speed and time feasibility of the fixed service segments are calculated, and the scheduling strategy is output when a feasible arrangement exists. If the constraint determination fails or there is no feasible arrangement that meets the speed and time requirements, the action is determined to be infeasible. For infeasible actions, the ship state and cargo state in the ship agent interaction decision environment remain unchanged and the current invalid action is recorded. Otherwise, the ship state and cargo state in the ship agent interaction decision environment are updated.

[0007] Preferably, the acquisition of ship, cargo, port, route, time window, and cost parameters forms irregular ship scheduling snapshot data, wherein the irregular ship scheduling snapshot data specifically includes: ship set, cargo set, port set, ship initial position, ship current available time, ship carrying capacity, ship speed boundary, cargo loading port, cargo unloading port, cargo demand, loading and unloading time window, port service time, route distance, transportation cost, and external carrier cost.

[0008] Preferably, the step of modeling each ship in the fleet as a ship agent, constructing a cooperative finite-time Markov decision process for the ship agents, and configuring a local observation space and discrete action space for each ship agent specifically involves: The cooperative finite-time Markov decision process corresponding to the ship's intelligent agent is represented as: , In the formula: Represents a collection of ship intelligent agents; The joint state space is represented by the state parameters in the joint state space, which include the current position, current time, service sequence, cumulative transportation cost, waiting time of all ships and the dynamic service status of all cargo. The cargo status categories include cargo not assigned, cargo already carried by an internal ship, and cargo already carried by an external ship or not served within the planning period. Represents ship intelligent agents The action space; Represents ship intelligent agents The local observation space; This represents the state transition function implemented by the interactive decision-making environment of the ship's intelligent agent; This represents the shared reward function at the fleet level, used for training the ship-level strategy decision neural network; Represents the finite scheduling time domain; Configure local observation space: Ship intelligent agent At any moment The local observation is represented as: ;in, Represents ship intelligent agents The current position of the corresponding vessel; Represents ship intelligent agents The current time of the corresponding ship; Represents the shared cargo state vector. The total number of goods; Represents ship intelligent agents The corresponding static attribute vector of the ship; Configuring Discrete Action Space: Ship Intelligent Agent The discrete action space is represented as: In this context, action 0 represents an idle action, and action... Indicates the selection of goods When ship intelligent agents Select goods And if the action is feasible, the action simultaneously determines the goods. By ship carriers, ships The next port of loading, the next port of discharge and the vessel A loading-sailing-unloading service segment within the service sequence.

[0009] Preferably, the constraint determination includes ship-cargo compatibility determination, cargo status consistency determination, whole-ship transportation determination, loading before unloading determination, time window constraint determination, and speed boundary constraint determination.

[0010] Preferably, when multiple ship agents select the same unallocated cargo, the unallocated cargo is assigned to the ship agent that first passes the constraint determination, and subsequent selections for the same cargo are determined as invalid actions due to inconsistent cargo status.

[0011] Preferably, the joint state of the entire fleet, consisting of the state vectors of all ship agents and shared cargo in the fleet, is used as input, and the fleet-level cumulative shared reward estimate is output according to the multi-component shared reward function to update the strategic decision neural network of each ship.

[0012] Preferably, the multi-component shared reward function includes a transportation cost saving reward item, a waiting time penalty item, an invalid action penalty item, an unserved cargo penalty item, an average cost item, and a global cost deviation item.

[0013] Preferably, the step of performing speed and time feasibility calculations on the fixed service segment and outputting a scheduling strategy when a feasible arrangement exists specifically involves: Based on the vessel speed boundaries, segment distances, port service times, and loading / unloading time windows in the irregular vessel scheduling snapshot data, calculate the sailing time, arrival time, departure time, and waiting time. Once a feasible arrangement is found, speed optimization is performed based on fixed loading-sailing-unloading service segments, outputting executable speed, arrival time, departure time, waiting time, fuel consumption, emissions, and transportation costs; The ship-cargo matching relationship, port access order, service timing, and speed optimization results are used as the output of the scheduling strategy.

[0014] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0015] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0016] Compared with the prior art, the present invention has the following advantages: (1) Starting from the business structure of the coupling of multiple elements of ship-cargo-port-time window-navigation execution in irregular ship scheduling, this invention constructs an intelligent scheduling technology framework for heterogeneous fleets, realizing the complete transformation from ship-level decentralized actions to fleet-level executable scheduling schemes. This enables the scheduling scheme to not only include ship-cargo matching relationships and port access order, but also execution sequence that meets the requirements of time window and speed boundary. It can generate executable scheduling schemes under the conditions of heterogeneous fleets, indivisible cargo and strict time window, thereby improving the collaborative decision-making ability, scheme feasibility and economy of irregular ship scheduling.

[0017] (2) Construct a ship-level distributed strategy decision-making mechanism based on local observation and shared cargo status. Each ship agent selects idle action or candidate cargo according to its current position, current time and shared cargo status vector. Then, combined with constraint judgment and feasibility calculation, the reliability of ship action selection is further guaranteed, and the feasibility of irregular ship scheduling scheme is guaranteed.

[0018] (3) In the training phase, the present invention takes the joint state of the entire fleet composed of all ship agents and shared cargo state vectors in the fleet as input, and outputs the fleet-level shared cumulative reward estimate according to the multi-component shared reward function to update the strategic decision neural network of each ship, comprehensively evaluate the impact of each ship's local actions on the overall cost, waiting, cargo service rate and scheduling feasibility of the fleet, and promotes fleet coordination by using global evaluation signals. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the irregular ship route scheduling architecture based on ship-level intelligent agent collaborative learning according to the present invention.

[0021] Figure 3 This is a schematic diagram of a one-to-one mapping structure from ships to ship intelligent agents.

[0022] Figure 4 A schematic diagram of the scheduling action execution and reward feedback mechanism for ship-class intelligent agents. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] In real-world, irregular ship scheduling scenarios, the scheduling system typically receives known data such as fleet status, cargo demand, port information, time windows, route distances, speed limits, and transportation costs at a given decision point. Each vessel has a different current location, available time, carrying capacity, speed range, and service cost; each shipment has a defined loading port, unloading port, demand, loading / unloading time windows, and external transportation costs. The scheduling objective is not merely to select general actions for multiple entities, but to simultaneously determine which vessel will carry which shipment, the service sequence for executing the loading-sailing-unloading process, and ensure that constraints such as vessel capacity, cargo indivisibility, time windows, route continuity, and speed feasibility are met.

[0025] Existing multi-agent reinforcement learning methods are typically geared towards general robot control or game-theoretic interaction tasks. Their agent partitioning, state representation, action space, and reward functions are often based on specific, general environments. These methods do not provide a specific modeling approach for the aforementioned irregular ship scheduling scenario; that is, they do not provide a unified mapping of ship state, cargo state, port time windows, ship-cargo compatibility, service sequence, external carriers, and speed-time feedback into interactively learnable states, actions, transitions, and reward mechanisms for agents. Therefore, without substantial adjustments to the agent construction method, scheduling environment state, action mapping, constraint consistency transitions, and centralized evaluation mechanisms to accommodate irregular ship scheduling business rules, existing multi-agent reinforcement learning frameworks cannot naturally generate executable routes and scheduling schemes that meet actual scheduling constraints.

[0026] Example In view of this, such as Figure 1 and Figure 2 As shown, this embodiment provides an irregular ship scheduling method based on ship-level intelligent agent cooperative learning, which includes: S1. Obtain parameters such as ships, cargo, ports, routes, time windows, and costs to form snapshot data of irregular ship scheduling, and initialize the ship intelligent agent interactive decision-making environment for irregular ship route scheduling constraints.

[0027] S101. At a certain decision point, receive confirmed data on ships, cargo, ports and routes to form snapshot data of irregular ship scheduling.

[0028] In one embodiment of the present invention, the irregular ship scheduling snapshot data includes at least: ship set, cargo set, port set, ship current location, ship current available time, ship carrying capacity, ship speed limit, cargo loading port, cargo unloading port, cargo demand, loading and unloading time window, port service time, route distance, transportation cost, and external carrier cost.

[0029] Specifically, each port or service node includes service hours. Each cargo includes at least the cargo number, demand, port of loading, port of discharge, loading time window, unloading time window, and external carrier cost. Each segment includes segment distance, reference travel time, travel cost, and fuel parameters.

[0030] The above data is fixed as a deterministic scheduling snapshot, used to describe a replanning node in the rolling schedule. Within this scheduling snapshot, ship status, cargo status, time window, cost parameters, and segment parameters serve as the input basis for this scheduling solution.

[0031] S102. Initialize the ship's intelligent agent interactive decision-making environment.

[0032] Based on scheduling data, a ship intelligent agent interactive decision-making environment is constructed to transform the business constraints in irregular ship scheduling into states, observations, actions, state transitions, and reward mechanisms that can be used for ship intelligent agents to make interactive decisions.

[0033] During environment initialization, the current position, current time, load capacity, speed boundary and service sequence of each ship are written into the ship status record; the loading port, unloading port, demand, loading and unloading time window, external transportation cost and service status of each cargo are written into the cargo status record; all cargo statuses are combined into a shared cargo status vector; at the same time, a ship-cargo compatible set and a ship feasible navigation arc set are established.

[0034] Specifically, in this embodiment, the ship intelligent agent interactive decision-making environment includes: 1) Vessel status records, including at least: each vessel's current position, current time, load capacity, speed limits, service sequence, and cumulative transportation costs.

[0035] 2) Cargo status records, including at least: port of loading, port of discharge, demand, loading and unloading time window, external carrier cost and dynamic service status for each cargo, wherein the dynamic service status is used to identify the cargo as being in one of the following states: unassigned, already carried by an internal vessel, already carried by an external carrier or not in service.

[0036] 3) Service node records.

[0037] 4) Action mapping unit: maps the discrete actions output by the ship agent to idle actions or candidate cargo selection actions; Constraint determination unit: determines whether the candidate cargo selection actions meet the constraints of ship-cargo compatibility, cargo status consistency, whole ship transportation, loading before unloading, time window and speed boundary.

[0038] 5) Constraint Decision Unit: Performs constraints on cargo status consistency, ship-cargo compatibility, whole ship transportation, loading before unloading, time window and speed boundary on the joint actions output by the ship agents; wherein, the joint actions are the joint actions output by each ship agent based on its local observations at the same decision moment, and the joint actions include idle actions or candidate cargo selection actions.

[0039] 6) State transition unit: used to update the ship's state (the ship's state includes at least the ship's current position and the ship's current time), cargo state, service sequence, waiting time and transportation cost when the action is feasible, and to keep the ship's physical state and cargo state unchanged and record invalid actions when the action is not feasible.

[0040] 7) Speed ​​Optimization Interface: Transmits the fixed loading-sailing-unloading service segment formed after the action is feasible to the speed optimization module, and receives the executable speed, arrival time, departure time, waiting time, fuel consumption, emissions and transportation cost returned by the speed optimization module.

[0041] 8) Reward Calculation Unit: Calculates multi-component shared rewards based on the outputs of the state transition unit and the speed optimization interface.

[0042] The ship intelligent agent interactive decision-making environment takes parameters such as fleet set, cargo set, port set, route distance, loading / unloading time window, ship capacity, speed boundary, and transportation cost as inputs to establish a scheduling state space composed of ship state records and cargo state records. The ship state records include at least the ship's current position, current available time, ship attributes, service sequence, and cumulative transportation cost; the cargo state records include at least the cargo loading port, unloading port, demand, time window, and external carrier cost. Through this environment, ship-cargo matching, loading-sailing-unloading sequence, time window constraints, capacity constraints, cargo indivisibility constraints, external carrier processing, and speed-time feedback in irregular ship scheduling are uniformly transformed into states, actions, transitions, and reward mechanisms that can be interactively learned by the intelligent agent.

[0043] S2. Model each ship in the fleet as a ship agent and construct a cooperative finite-time Markov decision process for the ship agents. Configure a local observation space, a discrete action space, and a ship-level policy decision neural network for each ship agent. The parameters in the local observation space include the ship's current position, the ship's current time, and the cargo sharing status.

[0044] S201. Each ship in the fleet is modeled as a ship agent. With ships One-to-one correspondence, and inheriting the ship's current position, current time, load capacity, speed limit, cargo set and service sequence; thus transforming the centralized fleet scheduling problem into a collaborative sequential decision-making problem among multiple ship agents in the same scheduling environment.

[0045] S202. Construct the cooperative finite-time Markov decision process corresponding to the ship's intelligent agent, represented as: , In the formula: Represents a collection of ship intelligent agents; The joint state space is represented by the state parameters in the joint state space, which include the current position, current time, service sequence, cumulative transportation cost, waiting time of all ships and the dynamic service status of all cargo. The cargo status categories include cargo not assigned, cargo already carried by an internal ship, and cargo already carried by an external ship or not served within the planning period. Represents ship intelligent agents The action space; Represents ship intelligent agents The local observation space; This represents the state transition function implemented by the interactive decision-making environment of the ship's intelligent agent; This represents the shared reward function at the fleet level, used for training the ship-level strategy decision neural network; This represents the finite scheduling time domain.

[0046] Ship intelligent agent The discrete action space is represented as: In this context, action 0 represents an idle action, and action... Indicates the selection of goods When ship intelligent agents Select goods And if the action is feasible, the action simultaneously determines the goods. By ship carriers, ships The next port of loading, the next port of discharge and the vessel This refers to a loading-sailing-unloading service segment within a service sequence. Therefore, discrete actions not only represent candidate cargo numbers but also correspond to actual business decisions in irregular vessel scheduling.

[0047] Ship intelligent agent At any moment The local observation is represented as: ;in, Represents ship intelligent agents The current position of the corresponding vessel; Represents ship intelligent agents The current time of the corresponding ship; Represents the shared cargo state vector. The total number of goods; Represents ship intelligent agents The corresponding static attribute vector of the ship may include at least one of the following: load capacity, speed limit, unit transportation cost parameter, or fuel consumption parameter.

[0048] When the ship From the current service node Transfer to the next service node The ship's state is based on the state transition function. Update, the update expression is: , in, Indicates a ship In the decision-making step Subsequent to the current service node Service start time; Indicates service node The loading / unloading or service operation time; Indicates a ship At the service node Complete the service and leave the service node Time; Indicates a ship By service node sail to service node The sailing time; Indicates a ship Arrival at service node Time; Indicates service node Upper bound of the time window; Indicates a ship At the service node Waiting time incurred due to arriving early; Indicates a ship At the next decision-making step; Indicates a ship The next step in the decision-making process is at the service node. The service start time.

[0049] The above state transition relationship means: Ship First, at the current service node Once loading, unloading, or service operations are completed, it leaves the current service node. The time is The ship then proceeded according to the route. sail to service node Its arrival time is If the vessel arrives at the service node The time is earlier than the service node lower bound of time window If the vessel is not in service, it will need to wait until the service window opens before it can begin service. The corresponding waiting time is [time period missing]. If the ship's arrival time is not earlier than the lower limit of the time window If the above calculations are completed, the waiting time will be zero. After completing the above calculations, the scheduling environment will update the ship's position to the service node. and will place the ship at the service node The service start time has been updated to .

[0050] In the feasibility assessment within the time window, if Exceeding service nodes The upper bound of the time window indicates the ship Unable to start service nodes within the specified time window The service, if the scheduling action can be determined as an inactive action or trigger external carrier processing; if the conditions are met... If so, the action is feasible under the time window constraint. The above state transition results are further used to update the ship service sequence, cargo service status, waiting cost, sailing cost, and reward function.

[0051] Shared cargo state vector ,in, Indicates goods Not yet allocated. Indicates goods It has already been transported by internal vessels. Indicates goods Cargo already outsourced or not yet served during the planning period. By sharing cargo state vectors, each ship's intelligent agent can perceive the cargo service status during decentralized decision-making, avoiding duplicate selection of already served cargo.

[0052] The shared cargo state vector in the above observation space is defined as follows: , in, Indicates the total number of goods. Indicates goods In the decision-making step The dynamic service status under this. Specifically, , Shared cargo state vector As a local observation of each ship intelligent agent A component of [the system], used to provide each ship's intelligent agent with the current service status of all cargo. Specifically, the [system / mechanism]... The local observations of a ship intelligent agent can be represented as: ,in, Represents ship intelligent agents Corresponding ships in the decision-making step Current location This indicates the current availability of the vessel. This represents the attribute vector of the ship.

[0053] During the action selection process, if a certain cargo status Then the cargo can be selected as a candidate cargo by the ship's intelligent agent; if or If the cargo has already been carried by an internal vessel, an external vessel, or is no longer considered a serviceable cargo during the planning period, the scheduling environment will recognize it as a duplicate selection or an inactive action when other vessel agents select the cargo again. By setting a shared cargo state vector, each vessel agent can perceive the cargo service status during decentralized decision-making, avoid repeatedly selecting already served cargo, and maintain state consistency during the ship-cargo matching process.

[0054] If an action violates ship-cargo compatibility, cargo status consistency, whole-ship transportation, loading before unloading, time window constraints, or speed boundary constraints, the action is deemed invalid. For invalid actions, the ship agent interaction decision environment does not update the ship's physical position and current time, does not change the service status of the selected cargo, and adds an invalid action penalty to the multi-component shared reward function.

[0055] Configuring ship-level strategy decision-making networks: ship intelligent agents The configured ship-level strategy decision network is represented as follows: Based on the local observations of the ship's intelligent agent Given the selection probabilities of each action in the action space as input, output the joint action. .

[0056] In one implementation, the multi-component shared reward function includes a transportation cost saving reward, a waiting time penalty, an invalid action penalty, an unserved cargo penalty, an average cost term, and a global cost deviation term. The multi-component shared reward function at time t is calculated as follows: , in, Represents ship intelligent agents Transportation cost savings incentives This indicates a waiting time penalty. Indicates a penalty for an invalid action. Indicates penalties for unserved goods. This represents the average cost item. This represents the global cost deviation term. to This represents the reward weighting coefficient.

[0057] In one implementation, each reward component can be calculated as follows: , , , , , , in, Indicates the waiting time. This represents the normalization constant for the waiting time. Indicates goods External carrier costs, This indicates the transportation costs incurred by the internal fleet in carrying the cargo. This represents the maximum value of the external transportation cost of goods. This represents the penalty constant for invalid actions. Indicates the quantity of unserved goods. This represents the total cost of the current scheduling scheme. Indicates the baseline cost. This represents a constant that prevents division by zero.

[0058] This reward function operates on all ship agents in a fleet-shared manner, enabling the local actions of individual ships to receive feedback from the overall scheduling effect. Through this reward mechanism, ship-level policy learning is simultaneously guided by economy, timeliness, feasibility, and service integrity.

[0059] Specifically, at the network structure level, such as Figure 3 As shown, this represents each ship's intelligent agent. Configure a ship-class strategy decision neural network The ship-level strategy decision-making neural network is based on the local observations of the ship's intelligent agent. Given the input and the selection probabilities of each action in the action space, output the joint action. This is used to output cargo selection or idle actions based on the ship's intelligent agent's local observations; simultaneously, a centralized evaluation neural network shared by multiple ship intelligent agents is configured. The centralized evaluation neural network evaluates the joint state of the entire fleet, which is composed of the states of all ships and the shared cargo state vector. It takes as input and outputs a shared cumulative return estimate or value estimate.

[0060] Thus, during the execution phase, each ship agent relies on its own local observations to make decentralized decisions, while during the training phase, it obtains evaluation signals at the overall fleet level through a fleet-level centralized evaluation neural network shared by multiple ship agents, thereby achieving a unity between ship local autonomous decision-making and overall fleet collaborative optimization.

[0061] In one embodiment of the present invention, during the training phase, the time difference error is calculated based on a lumped evaluation neural network. and sharing of advantageous signals : , , in, Indicates the discount factor. This represents the smoothing parameter for the shared advantage signal.

[0062] By sharing advantageous signals Update the strategic decision neural networks of each ship class so that each ship agent can learn action selection strategies that are consistent with the overall fleet scheduling objectives under local observation conditions.

[0063] During the execution phase, the fleet-level centralized evaluation neural network does not participate in action selection. Each ship agent only invokes its own ship-level policy decision neural network and makes action selections based on its own local observations. Thus, this invention utilizes global evaluation signals to promote fleet collaboration during the training phase and maintains ship-level decentralized decision-making during the execution phase.

[0064] This invention overcomes the inherent limitations of existing technologies that rely solely on neural network outputs for decision-making actions by embedding a consistency verification mechanism for irregular ship scheduling constraints during the environmental transition process. For joint actions output by multiple ship agents, the scheduling environment executes ship-cargo matching, cargo status updates, time window feasibility assessments, ship capacity adaptation assessments, service sequence continuity assessments, and identification of duplicate or ineffective actions according to preset rules. For decision-making actions that do not meet scheduling constraints, the physical scheduling state is not directly updated; instead, feedback is provided to the learning process through mechanisms such as invalid action penalties, cargo status maintenance, or external carrier processing. This approach ensures that the reinforcement learning process remains consistent with the irregular ship scheduling business rules, avoiding the generation of unexecutable solutions that violate capacity, time window, or service sequence constraints.

[0065] Furthermore, the multi-component shared reward function constructed in this invention for the purpose of irregular ship scheduling transforms factors such as cargo completion, transportation costs, waiting time, invalid actions, unserved cargo, external carrier costs, and average cost deviation into learning feedback signals at the fleet level. Compared to the simple, immediate reward calculation method used in general reinforcement learning, this invention comprehensively characterizes the economy, feasibility, and timeliness of irregular ship scheduling, enabling the centralized evaluation network to guide the updating of individual ship strategy networks from the perspective of the overall fleet operational effectiveness.

[0066] Therefore, the ship-cargo matching and service sequence generation in irregular ship scheduling is transformed into a collaborative decision-making process among multiple ship agents, which can reduce the problem of the centralized scheduling action space expanding sharply with the increase in the number of ships and cargo.

[0067] S3. Each ship agent uses its own pre-trained ship-level strategy decision neural network to output joint actions at the same decision moment based on its own local observations. The joint actions include idle actions or candidate cargo selection actions.

[0068] Specifically, at each decision moment, the first A ship's intelligent agent, based on its local observations Through the corresponding ship-level strategy decision neural network Output Action When the action When the action is idle, it means that the ship's intelligent agent will not select any new candidate cargo at the current decision-making moment; when the action... When selecting an action for a candidate cargo, it indicates that the ship's intelligent agent requests that the corresponding candidate cargo be added to the ship's subsequent service sequence.

[0069] Actions output by multiple ship intelligent agents at the same decision moment constitute a joint action. It then inputs the data into the ship's intelligent agent interactive decision-making environment to execute subsequent constraint judgments and state transitions.

[0070] S4. When the joint action is a candidate cargo selection action and passes the constraint judgment, the current node of the ship, the loading port and unloading port of the selected cargo are determined as fixed service segments. Speed ​​and time feasibility calculations are performed on the fixed service segments. If a feasible arrangement exists, a scheduling strategy is output. If the constraint judgment fails or no feasible arrangement satisfying the speed and time conditions does not exist, the action is determined to be infeasible. For infeasible actions, the ship state and cargo state in the ship agent interactive decision-making environment remain unchanged, and the current invalid action is recorded. Otherwise, the ship state and cargo state in the ship agent interactive decision-making environment are updated. Specific implementation details are as follows: Figure 4 As shown.

[0071] In one embodiment of the present invention, the constraint determination includes ship-cargo compatibility determination, cargo status consistency determination, whole ship transportation determination, loading before unloading determination, time window constraint determination, and speed boundary constraint determination.

[0072] Specifically, after receiving joint actions proposed by various ship agents, each action is evaluated sequentially according to a deterministic ship processing order. The evaluation criteria include: whether the selected cargo is still unassigned; whether the ship belongs to a compatible set for the cargo; whether the requirements for whole-ship transportation and loading before unloading are met; whether the ship's arrival at the loading and unloading ports meets the time window; and whether there is an executable time schedule for fixed service segments that meets speed boundaries. If all conditions are met, the ship and cargo states are updated, including the ship's current position, current time, service sequence, waiting time, cumulative transportation cost, and cargo service status. If any condition is not met, the action is deemed invalid, the ship's physical state and cargo state remain unchanged, and an invalid action penalty is generated in the reward function.

[0073] In one embodiment of the present invention, when multiple ship agents select the same unallocated cargo, the unallocated cargo is assigned to the ship agent that first passes the constraint determination, and subsequent selections for the same cargo are determined as invalid actions due to inconsistent cargo status.

[0074] In this embodiment, speed and time feasibility calculations are performed on fixed service segments. If a feasible arrangement exists, a scheduling strategy is output, specifically including: Based on the vessel speed boundaries, segment distances, port service times, and loading / unloading time windows in the irregular vessel scheduling snapshot data, calculate whether the vessel can arrive and start service within the specified time, including return voyage time, arrival time, departure time, and waiting time; Once a feasible arrangement is found, speed optimization is performed based on fixed loading-sailing-unloading service segments, outputting executable speed, arrival time, departure time, waiting time, fuel consumption, emissions, and transportation costs; The ship-cargo matching relationship, port access order, service timing, and speed optimization results are used as the output of the scheduling strategy.

[0075] Output a complete scheduling plan when all cargo is transported by internal vessels, external vessels, or when the planned time-domain termination conditions are met.

[0076] In this embodiment, the scheduling scheme includes each vessel's cargo carrier list, loading port visit order, unloading port visit order, arrival time, departure time, waiting time, speed and time feasibility calculation results, transportation cost, optional fuel consumption, optional carbon dioxide emissions, external carrier costs, and unserved cargo list.

[0077] In a preferred embodiment, the present invention further combines high-level ship-cargo matching with low-level speed feasibility calculation. Once the ship-level strategy network determines that a ship is selecting a specific cargo, it invokes speed optimization based on a fixed loading-sailing-unloading service segment, calculates the sailing time, waiting time, fuel consumption, and related costs that satisfy speed boundaries and time window constraints, and feeds this result back to state transition and reward calculation. Thus, the neural network strategy is responsible for high-level discrete scheduling decisions, while speed optimization performs low-level continuous time and speed feasibility verification; both together ensure the executability and economy of the scheduling scheme.

[0078] This invention provides a new solution to the problem of the lack of a dedicated interactive decision-making environment in existing technologies. Existing technologies do not provide a unified mapping of ship status, cargo status, port time windows, capacity constraints, service order, external carriers, and speed time feedback into interactively learnable states, actions, transitions, and reward mechanisms for intelligent agents. Nor do they provide technical insights into modeling each ship in an irregular fleet as an intelligent agent and designing a shared cargo status representation, a ship-level distributed strategy network, a fleet-level centralized evaluation network, a constraint consistency environment transition mechanism, a multi-component scheduling reward function, and a speed feasibility feedback mechanism around the interactive decision-making environment of this ship intelligent agent to solve the problem of irregular shipping routes and scheduling.

[0079] Through the above-mentioned structural modifications, this invention can effectively reduce the combined complexity of large-scale irregular ship scheduling, improve the collaborative decision-making ability among ships, reduce unmanageable actions and unserved cargo, and enhance the feasibility, economy, and stability of the scheduling scheme.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for irregular ship scheduling based on ship-level intelligent agent cooperative learning, characterized in that, include: Acquire parameters such as ships, cargo, ports, routes, time windows, and costs to generate snapshot data of irregular ship scheduling, and initialize the ship agent interactive decision-making environment for irregular ship route scheduling constraints; Each ship in the fleet is modeled as a ship agent, and a cooperative finite-time Markov decision process for the ship agents is constructed. Each ship agent is configured with a local observation space, a discrete action space, and a ship-level policy decision neural network. The parameters in the local observation space include the ship's current position, the ship's current time, and the cargo sharing status. Each ship intelligent agent uses its own pre-trained ship-level strategy decision neural network to output a joint action at the same decision moment based on its own local observations. The joint action includes idle action or candidate cargo selection action. When the joint action is a candidate cargo selection action and passes the constraint determination, the current node of the ship, the loading port and unloading port of the selected cargo are determined as fixed service segments. The speed and time feasibility of the fixed service segments are calculated, and the scheduling strategy is output when a feasible arrangement exists. If the constraint determination fails or there is no feasible arrangement that meets the speed and time requirements, the action is determined to be infeasible. For infeasible actions, the ship state and cargo state in the ship agent interaction decision environment remain unchanged and the current invalid action is recorded. Otherwise, the ship state and cargo state in the ship agent interaction decision environment are updated.

2. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 1, characterized in that, The process involves acquiring parameters such as ships, cargo, ports, routes, time windows, and costs to form irregular ship scheduling snapshot data. Specifically, this irregular ship scheduling snapshot data includes: ship sets, cargo sets, port sets, initial ship positions, current available time for ships, ship carrying capacity, ship speed limits, cargo loading ports, cargo unloading ports, cargo demand, loading and unloading time windows, port service hours, route distances, transportation costs, and external carrier costs.

3. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 1, characterized in that, The process involves modeling each ship in the fleet as a ship agent, constructing a cooperative finite-time Markov decision process for these agents, and configuring a local observation space and discrete action space for each agent. Specifically: The cooperative finite-time Markov decision process corresponding to the ship's intelligent agent is represented as: ， In the formula: Represents a collection of ship intelligent agents; The joint state space is represented by the state parameters in the joint state space, which include the current position, current time, service sequence, cumulative transportation cost, waiting time of all ships and the dynamic service status of all cargo. The cargo status categories include cargo not assigned, cargo already carried by an internal ship, and cargo already carried by an external ship or not served within the planning period. Represents ship intelligent agents The action space; Represents ship intelligent agents The local observation space; This represents the state transition function implemented by the interactive decision-making environment of the ship's intelligent agent; This represents the shared reward function at the fleet level, used for training the ship-level strategy decision neural network; Represents the finite scheduling time domain; Configure local observation space: Ship intelligent agent At any moment The local observation is represented as: ;in, Represents ship intelligent agents The current position of the corresponding vessel; Represents ship intelligent agents The current time of the corresponding ship; Represents the shared cargo state vector. The total number of goods; Represents ship intelligent agents The corresponding static attribute vector of the ship; Configuring Discrete Action Space: Ship Intelligent Agent The discrete action space is represented as: In this context, action 0 represents an idle action, and action... Indicates the selection of goods When ship intelligent agents Select goods And if the action is feasible, the action simultaneously determines the goods. By ship carriers, ships The next port of loading, the next port of discharge and the vessel A loading-sailing-unloading service segment within the service sequence.

4. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 1, characterized in that, When the ship From the current service node Transfer to the next service node The ship state update expression is then: ， in, Indicates a ship In the decision-making step Subsequent to the current service node Service start time; Indicates service node The loading / unloading or service operation time; Indicates a ship At the service node Complete the service and leave the service node Time; Indicates a ship By service node sail to service node The sailing time; Indicates a ship Arrival at service node Time; Indicates service node Upper bound of the time window; Indicates a ship At the service node Waiting time incurred due to arriving early; Indicates a ship At the next decision-making step; Indicates a ship The next decision step is at the service node. The service start time.

5. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 1, characterized in that, The system takes the joint state of the entire fleet, consisting of the state vectors of all ship agents and shared cargo in the fleet, as input, and outputs the fleet-level cumulative shared reward estimate based on the multi-component shared reward function to update the policy decision neural network of each ship class.

6. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 5, characterized in that, The process involves taking the joint state of the entire fleet, composed of the state vectors of all ship agents and shared cargo in the fleet, as input, and outputting a fleet-level estimate of the cumulative shared reward based on a multi-component shared reward function to update the policy decision neural networks of each ship class. Specifically, this includes: The joint state of the entire fleet, composed of the state vectors of all ship agents and shared cargo in the fleet. And a multi-component shared reward function, employing a centralized evaluation neural network. Calculate time difference error The expression is: ； Based on time difference error Calculate shared advantage signal The expression is: ， In the formula: Indicates the discount factor; Indicates the smoothing parameters of the shared advantage signal; By sharing advantageous signals Update the strategic decision neural networks of each ship class so that each ship agent can learn action selection strategies that are consistent with the overall fleet scheduling objectives under local observation conditions.

7. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 5, characterized in that, The multi-component shared reward function includes a transportation cost saving reward, a waiting time penalty, an invalid action penalty, an unserved cargo penalty, an average cost item, and a global cost deviation item.

8. The irregular ship scheduling method based on ship-level intelligent agent cooperative learning according to claim 1, characterized in that, The step of performing speed and time feasibility calculations on the fixed service segment, and outputting a scheduling strategy when a feasible arrangement exists, specifically involves: Based on the vessel speed boundaries, segment distances, port service times, and loading / unloading time windows in the irregular vessel scheduling snapshot data, calculate the sailing time, arrival time, departure time, and waiting time. Once a feasible arrangement is found, speed optimization is performed based on fixed loading-sailing-unloading service segments, outputting executable speed, arrival time, departure time, waiting time, fuel consumption, emissions, and transportation costs; The ship-cargo matching relationship, port access order, service timing, and speed optimization results are used as the output of the scheduling strategy.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images