Port multi-resource low-carbon cooperative scheduling optimization method based on reinforcement learning

Abstract

The invention discloses a port multi-resource low-carbon cooperative scheduling optimization method based on reinforcement learning, and the method comprises the steps: constructing a mixed integer nonlinear programming model comprising berths, quay cranes, storage yards and mobile shore power scheduling, and determining a target function and constraints; diversified initial solutions are generated according to the input data, and algorithm parameters are initialized; in the main iteration loop, a damage / repair operator pair is selected through a Q-learning agent, and ALNS damage-repair operation is executed to generate a candidate solution; according to an objective function and a simulated annealing acceptance criterion, determining whether to accept a candidate solution, and calculating and feeding back a reward to a Q-learning module to update a Q table and an operator weight; and gradually cooling the temperature, judging a termination condition, and outputting an optimal scheduling scheme and related performance indexes. According to the method, collaborative optimization of multiple resources such as ships, berths, quay cranes, mobile shore power and storage yards can be achieved, and the method has the advantages of being high in algorithm adaptability, high in solving efficiency, remarkable in carbon emission control effect and the like and is suitable for complex ports.

Description

Technical Field

[0001] This invention relates to the field of port scheduling and intelligent optimization technology, specifically to a port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning. Background Technology

[0002] In port operations, berth selection, quay crane configuration, yard layout, and freight truck scheduling are intertwined. Simultaneously, the energy supply during ship berthing (such as auxiliary engine diesel or shore power) significantly impacts carbon emissions in port. The port scheduling problem has expanded from a "single-objective, single-resource" approach to a complex optimization involving "multiple objectives, multiple resources, and policy constraints." Berth allocation directly affects the quay crane operation sequence and yard inflow / outflow; the allocation of sub-containers in the yard, in turn, affects in-yard transport distances, freight truck carbon emissions, and quay crane efficiency. The integration of mobile shore power transforms shore power supply from a fixed interface into a spatiotemporally mobile resource: providing shore power to different berths at different times can significantly reduce ship diesel consumption, but it requires scheduling the movement and dwell time of shore power in a temporal sequence, incurring additional costs due to the number of moves, and shore power can only serve a single berth at any given time or is limited by the number of devices. These factors significantly increase the problem's dimensionality and coupling strength, making traditional hierarchical optimization methods prone to getting trapped in local optima or neglecting key interaction effects.

[0003] Among existing solution methods, exact solutions can provide high-quality solutions for small-scale or well-structured problems. However, when faced with problems involving spatiotemporal variables of shore power and nonlinear terms of carbon trading, the model size and nonlinear characteristics often make it difficult for the solver to converge within a reasonable time or obtain a feasible solution. Traditional heuristic methods are scalable, but the selection of operators in implementation depends on experience or manual weight updates, lacking a systematic self-learning mechanism, which leads to a large amount of manual parameter tuning in different instances or policy scenarios. Few studies combine reinforcement learning with ALNS to achieve intelligent operator selection, especially when carbon tax / carbon trading policy scenarios, spatiotemporal scheduling of mobile shore power and joint problems of berth-quay crane-yard occur simultaneously. There is a lack of a large-scale scheduling method that can both ensure the realism of modeling and solve problems efficiently. Summary of the Invention

[0004] The purpose of this invention is to provide a port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning, comprising the following operational steps: Step S1: Construct a mixed-integer nonlinear programming model that includes berth, quay crane, yard and mobile shore power dispatch, and determine the objective function and constraints.

[0006] Preferably, determining the objective function and constraints includes: constructing an overall objective function that includes port operation-related costs, carbon emission costs based on carbon tax policies, and carbon emission costs based on carbon trading policies; and setting necessary constraints, including: berth space and time occupancy non-conflict constraints, quay crane resource capacity constraints, mobile shore power station legality constraints, ship arrival / departure time windows, minimum / maximum operation time constraints, and sub-container area allocation restrictions.

[0007] Step S2: Generate diverse initial solutions using the input data and initialize the algorithm parameters, including the Q-table, operator weights, and annealing temperature.

[0008] Preferably, the specific implementation process includes: Step S21: Collect input data and perform data preprocessing; Step S22: Generate several initial feasible solutions using heuristic rules, allocate berths according to arrival time, determine the operation time of each vessel, dispatch shore power, and repair solutions; Step S23: Initialize Q-Learning parameters, simulated annealing initial temperature, and cooling coefficient. The Q-Learning parameters include Q-table, learning rate, discount factor, exploration rate, and operator weights.

[0009] Step S3: In the main iteration loop, the Q-learning agent selects the destruction / repair operator pair and performs the ALNS destruction-repair operation to generate candidate solutions.

[0010] Preferably, the specific implementation process includes: Step S31: Define the states and actions in the Q-learning agent; Step S32: Define destruction operators, including random removal operator, cost-time hotspot removal operator, and cost-spatial region removal operator; Step S33: Define the repair operators, which include the berth translation operator and the minimum regret value repair operator; Step S34: Define non-destructive perturbation operators, which include greedy repair operators and operators that shorten operation time; Step S35: Select operator pairs as actions according to the adaptive operator selection mechanism.

[0011] Step S4: Determine whether to accept candidate solutions based on the objective function and simulated annealing acceptance criteria, calculate and feed back the reward to the Q-learning module to update the Q-table and operator weights.

[0012] Preferably, the specific implementation process includes: Step S41: Define the reward function, taking into account both global and local improvements, and introduce an opportunity cost mechanism; Step S42: Use the standard Q-Learning update formula as the Q-table update rule; Step S43: Incrementally update the operator weights based on the reward.

[0013] Step S5: Gradually cool down the temperature and determine the termination condition, then output the optimal scheduling scheme and related performance indicators.

[0014] Preferably, a certain probability of accepting degraded solutions is retained by using simulated annealing criteria to avoid getting trapped in local optima and to control search convergence; the algorithm terminates and outputs the optimal result when the maximum number of iterations is reached or when there is no improvement for several consecutive iterations.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are: (1) This invention introduces reinforcement learning (Q-Learning) to automatically learn the combination effect of operators, replacing the traditional heuristic method of manually adjusting operator weights, and realizing intelligent and dynamic optimization of operator selection; (2) This invention constructs a mixed integer nonlinear model that simultaneously considers operating costs, carbon emission costs and carbon policy constraints. It can adaptively solve under two policy scenarios: carbon tax and carbon trading, and has strong policy adaptability and versatility. (3) This invention can be applied to various types of ports, energy scheduling and industrial carbon emission optimization, etc. The algorithm can be encapsulated as a service interface and integrated into the port scheduling platform to realize online optimization and real-time decision support; (4) Under the dual-carbon policy background, this invention introduces carbon policy factors and mobile shore power time and air conditioning scheduling decisions. This invention can effectively reduce carbon emissions in port while ensuring operational efficiency, and provides technical support for the greening and intelligentization of port operations. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the convergence curve of the QL-ALNS algorithm based on a port case provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 Combination Figure 1 As shown, the present invention provides a technical solution: a port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning, comprising the following operational steps: Step S1: Construct a mixed-integer nonlinear programming model that includes berth, quay crane, yard and mobile shore power dispatch, and determine the objective function and constraints.

[0019] In this embodiment, the overall objective function is constructed. Solving the minimization problem: in: For port operation-related costs, The cost of carbon emissions based on carbon tax policy, The cost of carbon emissions based on carbon trading policies, and This is a binary variable for deciding whether to choose this carbon policy.

[0020] For example, port operation-related costs include vessel waiting costs, delay costs, penalty costs, and mobile shore power dispatch costs. The calculation formula is: in, For ship waiting costs, For ship delay costs, For the cost of penalties for ships, For ships Arrival time For ships The expected completion time, For ships The set, berth The set, For time points The set, Costs associated with shore power dispatch. To indicate ships The integer variable representing the berthing time. To indicate ships The integer variable representing the departure time. It is a binary auxiliary variable used to determine the ship Has the latest departure time been exceeded? , It is a binary variable used to determine mobile shore power. exist Is it in berth at the moment? superior, Similarly, this is used to determine mobile shore power. exist Is it in berth at the moment? When both values ​​are equal to 1, it indicates that a dispatch has occurred for the mobile shore power, thus incurring a dispatch fee.

[0021] For example, the carbon emission costs of both carbon policies are related to the amount of carbon emissions, while the carbon emission costs based on the carbon tax policy are... And carbon emission costs based on carbon trading policies The calculation formula is: in, For carbon tax rate, For carbon quotas, For carbon trading prices, This represents the total CO2 emissions, with carbon sources including ships, quay cranes, and transport trucks. The calculation formula is: in, A collection of original ships. A collection of converted boats, For mobile shore power The set, For sub-box area The set, For ships Number of shore power units available for allocation The feasible range, To provide ships The number of sub-boxes retained. This represents the amount of diesel fuel consumed per unit of time during a ship's berthing period. This refers to the power consumption per unit time of the modified ship after connecting to the mobile OPS. This represents the power consumption per unit time of the quay crane. The amount of CO2 produced per kilogram of diesel fuel. The amount of CO2 indirectly generated per kilowatt-hour of electricity. This refers to the carbon emissions generated per unit distance traveled by a container truck. For ships The number of containers to be loaded For ships The number of containers to be unloaded For the sub-box area to the yard berth Loading route length, From berth Average length of unloading routes to all sub-container areas in the yard. For ships Duration at the berth, a binary variable Used to determine the ship In time Does it occupy a berth? binary variables Used to determine the ship In time Use Taiwan embankment bridge, binary variables Used to determine shore power Is it in berth? Time binary variables Used to determine the ship Does it occupy a berth? binary variables Used to determine sub-box area Whether it was assigned to a ship .

[0022] In this embodiment, necessary constraints are set to ensure the feasibility of the solution, including the constraint that the berth's spatiotemporal occupancy does not conflict: Among them, binary variables Used to determine whether it is a ship and All are docked at the berths On, and the ship No later than the ship leave, It is a sufficiently large positive number.

[0023] Quay crane resource capacity constraints and vessel workload constraints (the number of quay cranes used at any given time shall not exceed the upper limit, and the number of quay cranes allocated to a vessel shall not be less than its workload): in, for shore bridge The set, This indicates the total number of quay cranes. Indicates ship The capacity requirements for quay cranes.

[0024] Legality constraints on mobile shore power stationing (only one shore power station can be located at the same berth, and a shore power station can only be located at one berth at the same time): Vessel arrival / departure time windows: Minimum / maximum job time constraints: Among them, binary variables Used to determine whether it is a ship At the point of time At least one quay bridge was assigned. and These respectively indicate that it is allowed to be a ship at the same time. Minimum and maximum number of quay cranes to be served; Sub-box allocation restrictions: Step S2: Generate diverse initial solutions based on the input data and initialize algorithm parameters (including Q-table, operator weights, and annealing temperature).

[0025] In this embodiment, generating diverse initial solutions and initializing algorithm parameters includes: Step S21: Collect and preprocess the input data.

[0026] For example, the input data includes: the number of sub-container areas allocated to the vessel. Transport distance matrix from storage yard to berth , Ship collection berth assembly shore bridge assembly shore power collection Sub-box area collection Time step set Time of arrival of the ship Expected end time Latest end time Ship operation volume Ships can allocate upper and lower boundaries of quay cranes , Operating cost parameters (ship waiting costs) Ship delay costs Ship penalty costs Shore power dispatch costs ), carbon parameters (carbon tax rate) ,quota Transaction price )wait.

[0027] Step S22: Generate several initial feasible solutions using heuristic rules.

[0028] For example, the specific implementation process includes: Step S221: Based on the vessel's arrival time The ships are sorted and berths are assigned to them sequentially. For each ship to be assigned, the earliest available time of each berth is calculated in relation to the ship's expected completion time. The berth corresponding to the smallest positive value is selected based on the time difference. If multiple berths have the same minimum value, a berth is randomly selected to increase solution diversity. The earliest available time of the berth is dynamically updated based on the operating time of the assigned vessels.

[0029] Step S222: After berth allocation, the specific operating time for each vessel at its assigned berth needs to be determined. For berths with only one vessel, the operating time is directly determined by the vessel's arrival time, expected completion time, and the minimum / maximum operating time constraints for the quay crane. For berths with multiple vessels, the following sequential optimization strategy is adopted: (a) For non-last vessels, the ideal operating time is determined based on the time interval between them and the next vessel; (b) Select an appropriate operation time while satisfying the minimum and maximum operation time constraints; (c) Pay special attention to cases where the operation time is less than the minimum value to ensure that the time constraint is met.

[0030] Step S223: The shore power dispatching process considers dispatching costs and adopts a strategy that minimizes the number of shore power moves. (a) First, construct a berth demand matrix to record the shore power demand of each berth at each time point; (b) Then, the shore power is allocated in chronological order, prioritizing the use of existing shore power at the current target berth, followed by the use of idle shore power, and finally, the shore power with the fewest moves is selected for scheduling. (c) It should be noted that during periods when there are no ship operations, shore power should be kept in its current position to minimize unnecessary movement.

[0031] Step S2224: The solution and repair process mainly addresses the conflict between QC quantity constraints and the ship's spatiotemporal constraints. (a) For QC over-limit use: First, identify all time points of limit violation and sort them by the severity of the over-limit; then select the most severe time point, determine the vessel that is berthed at that time and has the possibility of extending the operation time, prioritize the vessel that has not exceeded the expected completion time or berthing time, and extend its operation time by one unit; if the limit is still not met, repeat the above process until the constraint is met or the maximum number of attempts is reached.

[0032] (b) In cases where different vessels are operating at the same berth at the same time, the berthing time of subsequent vessels shall be postponed in sequence to ensure that there is no time conflict.

[0033] Step S23: Initialize Q-Learning parameters (Q table) Learning rate Discount Factor Exploration rate Operator weights (For ALNS operator priority), simulated annealing initial temperature and cooling coefficient .

[0034] Step S3: In the main iteration loop, the Q-learning agent selects the destruction / repair operator pair and performs the ALNS destruction-repair operation to generate candidate solutions.

[0035] In this embodiment, the specific implementation process includes: Step S31: Define the states and actions in the Q-learning agent: (a) State To characterize the current solution, a binary discrete state is defined, whose value depends entirely on whether an improved solution has emerged in the current iteration: (b) Actions Defined as a combination of a pair of destruction operators and repair operators. The action space consists of all operator pairs; Step S32: The destruction operators include the random removal operator, the cost-time hotspot removal operator, and the cost-spatial region removal operator: (a) The random removal operator randomly selects a certain proportion of ships for removal. This method ensures the randomness of the removal and helps the algorithm escape local optima; (b) The cost-time hotspot removal operator identifies the vessel with the highest operating cost and removes all other vessels whose operating times overlap with its; (c) The cost-space area removal operator identifies the berth area with the highest cost and removes all vessels docked in that area. The process is as follows: first, the overall cost of each berth is calculated; then, the berth with the highest overall cost is selected, and all vessels within it are removed.

[0036] Step S33: The repair operators include the berth translation operator and the minimum regret repair operator: (a) The berth translation operator identifies vessels that can be translated to berths and moves them randomly. The specific process is as follows: First, a time-berth occupancy table is constructed to record the usage of each berth in each time period; then, movable vessels are identified (i.e., vessels that do not conflict with other berths during the same operating period); then, some of these vessels are randomly selected, removed from their original berths, and randomly assigned to other berths while maintaining the time order to ensure the rationality of berth scheduling.

[0037] (b) The minimum regret repair operator is based on a regret-value strategy, which considers the cost differences between different insertion candidate schemes. The specific process is as follows: First, the berth assignments of the removed vessels are completely cleared. Then, the insertion cost of each vessel at all possible positions is calculated. Next, the cost of each vessel to be inserted at all possible berths and the insertion position is calculated, and its regret value is defined as the cost difference between the k-th best insertion position and the optimal insertion position. The vessel with the largest regret value is preferentially selected for insertion, thereby maximizing the cost improvement potential at the local level.

[0038] Step S34: Non-destructive perturbation operators include greedy repair operators and operators that shorten operation time: (a) The greedy repair operator sorts the ships by arrival time and inserts the removed ships into the optimal positions in turn. The specific process is as follows: First, sort the ships to be removed by arrival time and remove them from their original berths. Then, for each ship to be inserted, evaluate the cost of its insertion into all possible positions in all berths, select the lowest cost option, and execute the insertion, while resolving any potential time conflicts.

[0039] (b) The operation time reduction operator optimizes the scheme by reducing the operation time of the vessels. The specific process is as follows: First, the vessels are sorted according to the workload, and the vessels with the largest workload are selected for optimization first; for each vessel, try to reduce its operation time by one unit, and check whether the adjusted time still meets the constraint of the number of quay cranes; if it is feasible, update the operation time of the vessel and the quay crane allocation scheme.

[0040] Step S35: Select operator pairs as actions according to the adaptive operator selection mechanism.

[0041] For example, the QL-ALNS algorithm combines the decision-making ability of reinforcement learning with the neighborhood search ability of ALNS. During the iteration process, it simultaneously adopts the Q-table update mode of Q-learning and the operator weight update mode of ALNS. It is a random number between 0 and 1, when the greed rate... When choosing an action, select the action that maximizes the expected Q value. ,when When using the ALNS operator weighting model, the operator weights are selected probabilistically based on their magnitudes: in, Indicates the state Select action The policy probability defines the distribution of an agent's decisions from its current state to its action space. This represents the expected cumulative reward value for a state-action pair in Q-learning. Indicates the selection action The probability and its weight Proportional These are weights updated based on the historical performance of operators within the ALNS framework. It is the total weight of all actions. It is an action The probability of being selected is proportional to its weight, calculated as follows: in This represents the entire subset of optional operators.

[0042] Step S4: Determine whether to accept candidate solutions based on the objective function and simulated annealing acceptance criteria, calculate and feed back the reward to the Q-learning module to update the Q-table and operator weights.

[0043] In this embodiment, the specific implementation process includes: Step S41: Define a reward function that considers both global and local improvements, and introduce an opportunity cost mechanism.

[0044] For example, by defining the improvement amount Among them, the relative improvement relative to the historical best , The global improvement weight is used to balance global and local improvements. This is the best historical value. The objective value for the newly generated solution. This represents a local improvement relative to the current solution, where This is the target value of the current solution.

[0045] in, For instant rewards, This represents the current iteration number. This is a normalization constant, and its value is the initial fitness value. , This is an opportunity cost penalty. This segmented design both encourages actions that achieve significant improvement and applies negative incentives to actions that do not improve or deteriorate. Step S42: The Q-table update rule adopts the standard Q-Learning update formula.

[0046] For example, the update formula is as follows: in For learning rate, As a discount factor, For the first The immediate reward value of the next iteration. Indicates the next state Maximum estimated action value This represents the expected cumulative reward for state-action pairs in Q-learning. The implementation also maintains a discrete index or functional approximation of the Q-table.

[0047] Step S43: Based on the reward, pair the operator weights Perform incremental updates using the following formula: in, The learning rate parameter, For the first The weights of each operator and That is, the first The original and updated values ​​of each operator For the first The average score of each operator in the most recent evaluation period is calculated using the following formula: in For the first The cumulative score of each operator The number of times this operator is used is used, and the selection probability of each operator in ALNS is adjusted accordingly.

[0048] Step S5: Gradually cool down the temperature and determine the termination condition, then output the optimal scheduling scheme and related performance indicators.

[0049] In this embodiment, a simulated annealing criterion is used to retain a certain probability of accepting a degraded solution to avoid getting trapped in a local optimum, and to control search convergence. The acceptance probability is defined as: in To solve for the quality degradation value, Given the current temperature, the temperature is adjusted according to the cooling function during the iteration process. The algorithm gradually decreases. It terminates and outputs the optimal result when it reaches the maximum number of iterations or when there is no improvement after a number of consecutive iterations.

[0050] Example 2 Based on the technical solution of this invention, the following case scenario illustrates the implementation process of this invention in practical application. The specific application implementation scheme is as follows: In the case port, there are 10 vessels entering the port during a scheduling cycle (48 hours), numbered 1-10, of which vessels 2 and 5 are converted vessels. There are 5 berths, numbered 1-5. There are 15 quay cranes, 360 sub-container areas, and one mobile shore power supply. The berth utilization rate and quay crane utilization rate in this case are 75% and 75.8% respectively, which conforms to the actual port scenario. Other parameters related to the target value and vessels are shown in Tables 1 and 2. Table 1. Parameters related to the target value

[0051] Table 2. Ship-related parameters

[0052] After inputting the model and parameters, following steps 3, 4, and 5, the optimal target value and scheduling scheme are finally obtained as follows: Figure 2 And as shown in Table 3. Among them Figure 2 The line graph shows the historical best value changing with the number of iterations, reflecting the process of the input example continuously optimizing through the QL-ALNS algorithm until the optimal solution is finally found. Table 3 shows the data of the optimal solution for the example, as follows: Table 3: Optimal Solution Data Table for Case Study

[0053] Table 3 records the berth numbers assigned to 10 vessels and their berthing and departure times within the 0-48 hour scheduling period. The column "Number of Quay Cranes Assigned During Berthing" records the number of quay cranes used per unit of time from berthing to departure; therefore, the number of elements in this column for each vessel represents its operating time. The column "Reserved Sub-Container Area Number" records the specific numbers of the sub-container areas reserved for the vessels, and the number is the same as the number of reserved sub-container areas in Table 2 (…). )correspond.

[0054] This invention has significant application value in the field of multi-resource collaborative scheduling in ports. By constructing a unified port operation optimization model and introducing an adaptive large neighborhood search algorithm based on reinforcement learning, this invention can achieve joint optimization among multiple elements such as ship berths, quay crane configuration, mobile shore power dispatching, and yard sub-area allocation. Under the dual-carbon policy background, this invention can not only accurately characterize the carbon emission costs under the carbon tax and carbon trading mechanisms, but also automatically balance operational efficiency and carbon emission control requirements in scheduling decisions, thereby effectively supporting port decision-making in its green and low-carbon transformation. The algorithm framework of this invention has self-learning capabilities and can dynamically adjust the operator combination and search direction according to the port operation status, maintaining high robustness and solution efficiency in complex scenarios, and can provide ports with optimal or near-optimal low-carbon operation scheduling solutions.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0056] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning, characterized by: The following steps are included: Step S1: Construct a mixed-integer nonlinear programming model that includes berth, quay crane, yard and mobile shore power dispatch, and determine the objective function and constraints; Step S2: Generate diverse initial solutions using the input data and initialize the algorithm parameters, including the Q-table, operator weights, and annealing temperature; Step S3: In the main iteration loop, the Q-learning agent selects the destruction / repair operator pair and performs the ALNS destruction-repair operation to generate candidate solutions; Step S4: Determine whether to accept candidate solutions based on the objective function and simulated annealing acceptance criteria, calculate and feed back the reward to the Q-learning module to update the Q-table and operator weights; Step S5: Gradually cool down the temperature and determine the termination condition, then output the optimal scheduling scheme and related performance indicators.

2. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 1, characterized in that: Step S1, which involves determining the objective function and constraints, includes: Construct an overall objective function that includes port operation-related costs, carbon emission costs based on carbon tax policies, and carbon emission costs based on carbon trading policies; The necessary constraints include: berth space and time occupancy non-conflict constraints, quay crane resource capacity constraints, mobile shore power station stay legality constraints, ship arrival / departure time windows, minimum / maximum operation time constraints, and sub-container area allocation restrictions.

3. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 2, characterized in that: The implementation process of step S2 includes: Step S21: Collect input data and perform data preprocessing; Step S22: Generate several initial feasible solutions using heuristic rules, allocate berths according to arrival time, determine the operation time of each vessel, dispatch shore power, and repair solutions; Step S23: Initialize Q-Learning parameters, simulated annealing initial temperature, and cooling coefficient. The Q-Learning parameters include Q-table, learning rate, discount factor, exploration rate, and operator weights.

4. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 3, characterized in that: The implementation process of step S22 includes: Step S221: Sort the vessels according to their arrival time and assign them berths in sequence; Step S222: After the berth allocation is completed, determine the specific operating time of each ship in its assigned berth; Step S223: The dispatching process for moving shore power calculates the dispatching cost and adopts a strategy that minimizes the number of shore power moves; Step S2224: Resolve the QC quantity constraint and the ship's spatiotemporal conflict through the repair process.

5. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 4, characterized in that: The implementation process of step S3 includes: Step S31: Define the states and actions in the Q-learning agent; Step S32: Define destruction operators, including random removal operator, cost-time hotspot removal operator, and cost-spatial region removal operator; Step S33: Define the repair operator; Step S34: Define the non-destructive perturbation operator; Step S35: Select operator pairs as actions according to the adaptive operator selection mechanism.

6. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 5, characterized in that: The state defined in step S31 is a binary discrete state. The state value depends on whether an improved solution appears in the current iteration. The action is defined as a combination of a pair of destruction operators and repair operators. The action space consists of all operator pairs.

7. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 6, characterized in that: The repair operator in step S33 includes: Berth translation operator: Identifies vessels that can be translated into berths and performs random movement; Minimum regret repair operator: Repair based on regret value strategy; The non-destructive perturbation operators in step S34 include: Greedy repair operator: Sort ships by arrival time and insert the removed ships into the optimal positions in turn; Operation time reduction operator: Optimize the scheme by reducing the operation time of the vessel.

8. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 7, characterized in that: Step S35, which selects operator pairs as actions according to the adaptive operator selection mechanism, includes: the QL-ALNS algorithm simultaneously adopts the Q-table update mode of Q-learning and the operator weight update mode of ALNS during the iteration process. It is a random number between 0 and 1, when the greed rate... When, choose the action that maximizes the expected Q value. When using the ALNS operator weight mode, the operator weight is selected probabilistically based on the magnitude of the operator weight.

9. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 8, characterized in that: The implementation process of step S4 includes: Step S41: Define the reward function, taking into account both global and local improvements, and introduce an opportunity cost mechanism; Step S42: Use the standard Q-Learning update formula as the Q-table update rule; Step S43: Incrementally update the operator weights based on the reward.

10. The port multi-resource low-carbon collaborative scheduling optimization method based on reinforcement learning according to claim 9, characterized in that: Step S5 also includes: using simulated annealing criteria to retain a certain probability of accepting degraded solutions to avoid getting trapped in local optima, and controlling search convergence; the algorithm terminates and outputs the optimal result when it reaches the maximum number of iterations or when there is no improvement for several consecutive iterations.

Images