Harbor logistics-energy optimization scheduling method based on multi-agent reinforcement learning

Abstract

The invention discloses a seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning, and the method comprises the steps: constructing a port logistics-energy space-time cooperation model which comprises an AES berth distribution model, a QC distribution model, an AGV scheduling model, and an SPDN optimal power flow model; a logistics-energy space-time coordination model based on optimization is converted into a multi-agent fully observable Markov decision process (MDP), and a general multi-agent reinforcement learning (MARL) large model is constructed and trained to calculate a reward function, observe and act space and automatically generate scheduling strategies of various scenes. According to the method, an optimal scheduling strategy is directly obtained by considering a comprehensive logistics-energy space-time cooperation model of an all-electric ship (AES), a quay crane (QC), an automatic guided vehicle (AGV) and a port power distribution network (SPDN) and inputting system operation parameters through a multi-agent reinforcement learning large model, and the calculation efficiency is greatly improved while the optimality is ensured.

Description

Technical Field

[0001] The present invention relates to a technology in the field of power grid optimization, specifically a seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning. Background Art

[0002] Existing port logistics-energy collaborative optimization technology uses a mathematical optimization-based approach that requires updating model parameters and resolving the problem every operating day, resulting in a large computational burden. Summary of the Invention

[0003] In view of the fact that the existing technology ignores the physical constraints in space, resulting in the inability to implement the obtained scheduling strategy in actual production, the present invention proposes a seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning. By considering the comprehensive logistics-energy spatiotemporal collaborative model of all-electric ships (AES), quay cranes (QC), automatic guided vehicles (AGVs) and port distribution networks (SPDNs), and directly obtaining the optimal scheduling strategy by inputting system operating parameters through a large multi-agent reinforcement learning model, the optimality is ensured while greatly improving the computational efficiency.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning. The method constructs a port logistics-energy spatiotemporal coordination model comprising an AES berth allocation model, a QC allocation model, an AGV scheduling model, and an SPDN optimal power flow model. The optimization-based logistics-energy spatiotemporal coordination model is converted into a multi-agent fully observable Markov decision process (MDP). A general multi-agent reinforcement learning (MARL) large model is constructed and trained to calculate reward functions, observation and action spaces, and automatically generate scheduling strategies for various scenarios.

[0006] The MARL model includes: a hypernetwork, an agent, a cascade action modification layer, an environment unit, and a MIX network, wherein: the hypernetwork generates parameters for the agent strategy network based on the environmental data of the scene; each agent generates an original value function and outputs it to the cascade action modification layer to generate a modified action and a corresponding value function that meets the spatiotemporal constraints. The modified action is then output to the environment unit to obtain a reward and the next state, and the value function is output to the MIX network to calculate the joint action value function.

[0007] The present invention relates to a system for implementing the above-mentioned method, comprising: a model training module, an environmental data collection module, an agent decision calculation module, and a scheduling instruction execution module. The model training module trains the MARL large model and uses the trained large model for subsequent calculations; the environmental data collection module is used to collect environmental data for each operating day, which is used as input to the hypernetwork in the MARL large model; the agent decision calculation module is used to obtain scheduling decisions applicable to the environmental data of that operating day. The parameters of each neural network in the agent are generated by the hypernetwork, and then the scheduling decision of each agent is obtained by inputting the system environmental state. This process is a dynamic Markov process until the end of the scheduling cycle; and the instruction execution module issues scheduling instructions to the port control room to execute the obtained optimal scheduling plan. Technical Effects

[0008] This invention not only considers the involvement of AGVs but also the spatial coordination of these components. Through a large-scale multi-agent reinforcement learning model architecture for port logistics-energy spatiotemporal collaboration, it can adapt to the varying number of ships arriving daily and automatically generate a scheduling strategy for each operating day without requiring daily re-calculation. Compared to existing technologies, this invention not only further improves system operational flexibility and reduces scheduling costs, but also ensures the feasibility of scheduling solutions in practice. Once trained, the MARL model can directly obtain the optimal scheduling strategy by inputting system operating parameters, with computational time in milliseconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a schematic diagram of the seaport logistics system and its connection with the power system;

[0010] Figure 2 A flowchart for training and applying a large general-purpose multi-agent reinforcement learning model;

[0011] Figure 3 This is a schematic diagram of the architecture of a general multi-agent reinforcement learning model;

[0012] Figure 4 Schematic diagram of the agent strategy network structure;

[0013] Figure 5 Schematic diagram of the super network structure;

[0014] Figure 6 This is a topological diagram of the power system of Rizhao Port;

[0015] Figure 7 Schematic diagram of the training / fine-tuning process for the three methods;

[0016] Figure 8 Schematic diagram of the training / fine-tuning process for different methods on a large system. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, this is the application scenario of this embodiment, namely the port distribution network and logistics system. In the logistics system, the arriving AES first waits at the anchorage, and then sails to the assigned berth after receiving the port dispatch signal. This process is called berth allocation. At the same time, the port will allocate several QCs to each AES docked at the port for cargo loading and unloading. The goods are transported between the terminal and the yard by AGV. After the AGV is exhausted, it will go to the battery charging station (BCS) to charge. Throughout the logistics process, SPDN obtains electricity from the upper power grid and renewable energy sources (RES) to provide power for the AES, QC and charging AGV at the port of call.

[0018] The docking time of an AES affects the temporal distribution of its own power load and that of its assigned QCs, as well as the charging timing of AGVs. Due to time-varying grid electricity prices and RES output, the operating conditions of the SPDN vary depending on the temporal demand response of the logistics power load. Furthermore, the docking location of an AES influences which power nodes are loaded and the associated QC allocation, thus affecting the spatial distribution of their loads. AGV transport efficiency depends on the distance between the cargo yard and the docking AES. This further influences the docking time of the AES and the temporal distribution of its load. Consequently, the spatiotemporal characteristics of the logistics system and the SPDN are tightly coupled.

[0019] like Figure 2 As shown, this embodiment relates to a seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning, including:

[0020] Step 1: Construct a port logistics-energy spatiotemporal coordination model that includes the AES berth allocation model, the QC allocation model, the AGV scheduling model, and the SPDN optimal power flow model, where:

[0021] 1.1) The AES berth allocation model refers to the ship berthing status, that is, ship s berths at berth b at time t, then X bst is 1, otherwise it is 0, specifically: Constraints on AES berth allocation include: limiting berthing time and permitted range of berthing locations Each AES can only be assigned one berth, and each berth can accommodate at most one AES at a time: 1≤B s ≤B max ,as well as in: and They represent the arrival time, berthing start time and departure time of ship s respectively, is the maximum port stay of ship s, B s is the berth number of ship s, B max The total number of shore power berths at the port terminal.

[0022] 1.2) QC allocation model refers to: the operating status of QC, that is, ω q,s,t A value of 1 indicates that QC q serves AES s at time t. The time constraints for QC allocation include: each QC can only serve one AES at most at any time. Total number of QCs that can be assigned Number of QCs assigned to each AES QC cargo handling efficiency All QCs serving the same AES operate at the same efficiency The temporal and spatial constraints of QC allocation include: all QCs serving the same AES are spatially adjacent, and idle QCs must not be located between working QCs on the same AES -1≤ω q+1,s,t -ω q,s,t +ω q′,s,t ≤1, The location of each QC is the same as the location of the AES it serves All QCs cannot cross each other Where: Q min is the total number of QCs available at the port, and They represent the minimum and maximum QC numbers that ship s can withstand due to the limitation of hull length, is the container loading and unloading efficiency of QCq for AESs at time t, For its maximum value, B qt is the position of QCq at time t. If QCq serves AESs at time t (i.e. ω qst =1), then it is the same as the position of ship s; and the linearized form of the above constraints: B q,t ≤B s +M(1-ω q,s,t ), B q,t +M(1-ω q,s,t )≥B s ,

[0023] 1.3) AGV scheduling model refers to: AGV operating state includes idle state Service AES status Charging status The constraints of AGV scheduling include: each AGV can only operate in one state at any time AGV only serves AES at the port AGV transportation efficiency Allocate enough AGVs to complete cargo loading and unloading tasks before AES departs Limit the maximum transportation efficiency of AGV based on the maximum QC loading and unloading efficiency The QC of each AES and the overall efficiency of the AGV are consistent AGV overcharging restrictions in: and are binary variables, and when they are equal to 1, they respectively indicate that AGV v is idle, serving AES s, and charging at time t. is the container transportation efficiency of AGV v to AES s at time t. Different docking locations of AES will result in different transportation distances of AGV. represents the maximum efficiency of AGV v serving AES s at berth b, in TEU s is the number of standard containers on ship s, TEU is the unit of measurement for standard containers, E v,t For the energy of AGVv battery, and are their minimum and minimum values ​​respectively, and are the energy consumed and obtained by charging of AGV v in one scheduling step, respectively.

[0024] 1.4) The SPDN optimal power flow model is: the power loads of AES, QC, and AGV at SPDN node j are: Among them: AES load is determined by its basic load and operating status, QC load depends on its rated power and cargo handling efficiency, is the ratio of the total efficiency to the total number of QCs serving AES, that is, The AGV charging load depends on the battery charge. The linear DistFlow equation is used to model the SPDN, and its operating constraints include: power balance equation Calculation of nodal power load and available renewable energy Node voltage balance and amplitude in: are the rated powers of AESs and QCq, Π(j), Φ(j), Θ(j) are the sets of all shore power berths, QCs and charging stations electrically connected to node j, and P ij,t , Q ij,t and P jk,t , Q jk,tare the active and reactive power flows from node i to node j and the active and reactive power flows from node j to node k at time t, Ω(j) and θ(j) are the upstream and downstream nodes connected to node j, respectively. If and only if node j is connected to a grid node, are the active and reactive loads of node j at time t, U i,t is the voltage amplitude of node i at time t, r ij 、x ij are the resistance and reactance of line ij, U i,t is the voltage at node i, are the actual dispatch power and maximum available power of renewable energy unit m at time t, is the rated capacity of line ij, are the minimum and maximum voltage amplitudes at node j, respectively.

[0025] The objective function of the port logistics-energy spatiotemporal synergy model is to minimize the sum of the cost of purchasing electricity from the upstream power grid and the waiting and docking costs of all AESs, specifically: in: is the electricity price purchased from the main grid at time t, is the amount of electricity purchased from the main network at time t, and are AES waiting and docking costs, respectively.

[0026] Step 2: Transform the optimization-based logistics-energy spatiotemporal coordination model into a multi-agent fully observable Markov decision process (MDP)<A,S,O,U,r,P,γ> ,in:

[0027] 2.1) The agent set A includes: i) AES agent s∈S∈A; ii) QC agent q∈Q∈A; iii) AGV agent v∈V∈A; each agent group has a different action space but shares the same reward function.

[0028] 2.2) To reflect the current operating state of the system and guide the agent to make decisions, it is necessary to define the environment state s t ∈S, which specifically includes: public state AES Status and AGV status Among them: public status Including the RES output of the current time period t and the entire scheduling period The latter includes observations up to t and predictions from t+1 to T. This enables the agent to observe future information and decide whether to work now or later, thereby facilitating temporal coordination; AES state In binary Represents AES: Not reached Arrived but not docked Moored at berth during [t-1,t] and have left Indicates unprocessed goods. and Represent the time distance from the current time to the arrival and latest departure time of AES respectively. and Respectively, they represent the cumulative time since the arrival of the AES and the cumulative delay time after the latest departure time, so as to improve logistics efficiency by allocating berths and handling cargo in a timely manner; AGV status China E v,t It is the battery level.

[0029] The environmental state s t Specifically: In a fully observable MDP, the observation of each agent is the same as the state, i.e.

[0030] 2.3) Action space includes: AES action space is vector Indicates berth selection Or do not select a berth The action space of QC is vector for Serve or idle The action space of AGV is vector Representation Service or idle or charging

[0031] 2.4) State transition means: the state of the environment s is affected by the environment and the action of the agent. t middle Evolving naturally over time, Varies according to AES operation, After all the goods have been processed, it changes from 0 to 1. When the AES is parked and there are AGVs serving it, the unprocessed goods

[0032] 2.5) The state function is used to guide the agent to find the optimal scheduling strategy, specifically: Among them: the first item is the cost of purchasing electricity; the second item is the logistics cost, including waiting costs Berthing costs Delayed departure fine Fines for failure to complete loading and unloading tasks in: and are the penalty costs for delayed departure and unfinished loading and unloading tasks respectively.

[0033] 2.6) The environment includes: logistics system and SPDN, where: the joint actions of all agents are used to calculate the logistics rewards and logistics loads, and then the DistFlow model of the SPDN optimal power flow is solved using a commercial solver (such as Gurobi) to minimize As the optimization goal), the total reward can be obtained and the state of the system at the next moment can be updated.

[0034] Step 3: Construct and train a general multi-agent reinforcement learning (MARL) large model to calculate the reward function, observation and action space and automatically generate scheduling strategies for various scenarios without retraining.

[0035] like Figure 3 As shown, the MARL model includes: a hypernetwork, an intelligent agent, a cascade action modification layer, an environment unit, and a MIX network, wherein: the hypernetwork is based on the environment data of the scene Generate parameters for the agent policy network; each agent generates an original value function Q(τ,·) and outputs it to the cascade action modification layer to generate modified actions that meet the spatiotemporal constraints and the corresponding value function The modified action is output to the environment unit to obtain the reward and next state, and the value function is output to the MIX network to calculate the joint action value function The parameters of the MIX network also depend on

[0036] like Figure 4 As shown in Figure 2, the policy network parameters of the agent are shared among the same agent group, so the agent ID is included in the observation space to implement different strategies for each agent. In the observation space, the two types of vectors are The first type is Each The length of is fixed, but the total number is |S|. The second type includes the one-hot vector of the AES agent ID in the AES policy network, the previous action vector in the QC and AGV policy networks and These vectors have a count of 1 but have lengths and Similarly, the number of QC and AGV action vectors is 1, but the lengths are and In the AES policy network, an attention network is used to handle different numbers of Where query is AESs itself Embedded, while key and value are other AES Embedding. This allows AESs to pay attention to relevant information from other AESs and make more intelligent decisions. To process one-hot vectors of AES agent IDs of length S|, an FC network V-FC is established to convert the vectors into fixed-dimensional embeddings. V-FC has a fixed dimension in a given scene, but it will be regenerated in each scene through a hypernetwork to adapt to different lengths. The QC and AGV policy networks are basically similar. In the policy network, another type of network (point-wise FC / MLP) is introduced to process each input separately and generate the corresponding output. QC action output is a length of To generate this vector, we first use point-wise FC, V-FC, and standard FC networks to transform all state inputs into a unified dimension. This is to enable these inputs to be fed into W simultaneously. Q ,W K and W V Then, use the self-attention network to Input generation vectors. Each output vector has a length greater than 1, so point-wise MLP converts the length of these vectors to 1. Vectors of length 1 are concatenated into a vector of length vector, thus obtaining For the AGV policy network, an additional feedforward path is required (shown by the dotted line in the figure) to generate S+2 outputs, and then generate an AGV action output vector of length S+2

[0037] The agent policy network includes: standard FC / MLP, whose dimension is fixed and V-FC, whose dimension varies Changes, including: Standard FC / MLP through Figure 5 (a)-(b) are generated. Figure 4 (a) All environmental data are converted into fixed-length embedding vectors. To handle the varying number of AES parameters Construct another hypernetwork to generate V-FC, which will Convert to a fixed-length vector Each AES number is embedded in the vector e a(s) embeds the vector using the point-wise concatenate operator and the AES number concatenate. This operator outputs vector, and then transform it into matrices. The dimension of each matrix is ​​d p ×d emb , where d p is a vector The combination of these matrices forms a matrix of size The weight matrix of of Multiply them together to get a value of size 1×d emb The output vector of . Adding it to the deviation with the same dimension gives the AES parameter embedding vector In this way, the length Changing Then, after all the environment data are converted into fixed-length embedding vectors, they can be Figure 5 (b) Standard FC / MLP in the generative agent policy network.

[0038] The V-FC in the agent policy network has to process a length of and Therefore, the dimension of the resulting weight matrix should match the vector length. Figure 5 In (c), all AES parameters are first embedded by the self-attention network, outputting vectors. Since the V-FC in the AES policy network processes the one-hot vector of AESID, it is necessary to give a unique weight to each AES. This is done by a (s) with and This is achieved by combining point-wise concatenate operators. Then, the outputs of the self-attention network and point-wise concatenate are combined through another point-wise concatenate, and point-wise MLP is used to generate a network of size S×d. FC This can match the one-hot vector of AESID of size 1×S. Similarly, in Figure 5 (d) is the size of the matching The QC action vector of size is The weight matrix is ​​obtained by These vectors are generated by a point-wise MLP with The self-attention network generated by e a (s) and environmental data. For the AGV policy network, an additional feedforward path is added (indicated by the dotted line in the figure) to generate a The weight matrix of .

[0039] The super network is based on the environmental data of the scene Generate weights and biases for all neural networks in the agent's policy network, where the environment data of the scene Includes: Public Information That is, the predicted value of renewable energy output and the main grid electricity price; the amount reaching AES Parameters of each AES

[0040] like Figure 6 As shown, the cascade action modification layer includes:

[0041] 1) Action Masking (AM) filters invalid actions for each agent based on the system state. AESs that have not arrived or left cannot select any berth. AESs waiting can select any available berth. AESs parked can only select their current berth until their cargo handling task is completed. QCs and AGVs cannot provide services to AESs that have not arrived or left. AGVs with depleted batteries cannot operate. Invalid actions are filtered by replacing the logit of the corresponding action with -∞.

[0042] 2) Action Correction (AC) corrects erroneous actions that violate constraints. This is a cascaded action correction process. Specifically, the AES and QC actions are first corrected together. If the AES action is correct, only the QC action is modified based on the AES action; otherwise, both actions are modified. The AGV action is then adjusted and modified based on the correct AES and QC actions. This approach avoids solving a large-scale integer programming problem, thus accelerating the solution process.

[0043] Modifying the AGV action means making minimal changes to the original action, specifically: in: and u t are the modified action and the original action respectively. is the modified action of the reference agent Constructed action space.

[0044] The MARL large model is trained in the following way:

[0045] a) Use the QMIX algorithm to collect the optimal policies for all training scenarios. The parameters of these optimal policies are the output of the hypernetwork and stored in the training dataset.

[0046] b) Train the hypernetwork through supervised learning to establish a mapping from environmental data to the optimal policy.

[0047] In step a above, if the calculation is started from scratch for each scenario, it will result in a huge computational burden. Considering that similar scenarios usually share similar strategy parameters, the present invention provides two strategies to accelerate the collection of the optimal strategy for each scenario in step a:

[0048] 1) Strategy 1: Classify scenes with the same number S of AES into one category, and gradually start training from the class with the smaller number of AES to the class with the larger number of AES. The first scene in the class is initialized by the previously trained scene with the smaller number of AES. This method is suitable for standard FC / MLP, while the V-FC network needs to be retrained due to dimensionality changes. Specific operations: Step 1, classify all scenes, all scenes in each class have the same number of AES, and sort all classes from small to large according to the number of AES; Step 2, starting from the class with the smaller number of AES, randomly select a scene and train from scratch to obtain the optimal policy parameters for the scene; Step 3, select the next class, and the first scene in the class is not trained from scratch, but the optimal policy parameters of the scene that has been trained in Step 2 are used as the initialization parameters of the neural network, and then trained with the initialization parameters; Step 4, execute Steps 2 and 3 for each subsequent class in sequence until training for all classes is completed.

[0049] 2) Strategy 2: Because the optimal strategy for a scene is directly related to its environmental data, we first evaluate the distance between scenes (e.g., Euclidean distance) based on their environmental data. The optimal parameters of the trained scene are then used as the initialization parameters for the scene closest to it. This allows for rapid training of subsequent scenes without having to start from scratch. Furthermore, to achieve even faster training of the remaining scenes, scenes within the same category can be clustered based on their environmental parameters to obtain multiple clusters, and each cluster can then be trained in parallel. Specific operations: Step 1, use a clustering algorithm (such as K-MEANS) to cluster the scenes in each class and obtain multiple clusters. The clustering is based on the environmental data of each scene; Step 2, for each scene in the cluster, calculate the Euclidean distance between each scene and other scenes, and then sort all scenes from the center point of the cluster to the outside according to the distance. This sequence serves as the order of training scenes in subsequent steps; Step 3, use the first scene (trained) obtained in Step 1 to initialize the first scene to be trained in each cluster. After the scene is trained, it is used as the initialization for subsequent scenes; Step 4, repeat Step 3 for all clusters in all classes until the optimal parameters of all scenes are collected.

[0050] By adopting the above two strategies, the training process is significantly accelerated compared to training all scenarios from scratch, especially for scenarios with a large number of AES.

[0051] After specific practical experiments, the effectiveness of the proposed method is verified using the Rizhao Port system in China. The topology of the distribution network is as follows: Figure 6 As shown in Figure 1, there are 6 berths, 12 quay cranes, 20 automated guided vehicles (AGVs), 1 battery charging station, and 3 wind turbine units. The connections between these components and the distribution grid are shown in Table I. The dispatch period is 48 hours, i.e., T = 48, and the model temporal resolution is 1 hour. Both training and testing scenarios are based on historical port operation data. All MLP modules use Reinforced Luminance (ReLU) activation. The dispatch period is 24 hours, with a resolution of 1 hour. Numerical experiments are conducted using PyTorch 2.3.0 and Gurobi 10.0. The optimal policies for all scenarios are trained in parallel on five computing platforms equipped with an Intel(R) Xeon(R) Platinum 8338C CPU @ 2.60 GHz, an NVIDIA GeForce RTX4090 GPU, and 256 GB of RAM. Subsequently, one of these platforms was used to train the hypernetwork.

[0052] Three methods are set up for comparison with the present invention. Model-based optimization method: The logistics-energy spatiotemporal coordination model is optimized using the Gurobi solver. Baseline reinforcement learning method: A MARL model is trained from scratch using the QMIX algorithm, where all networks are standard FC / MLP. Transfer training method: A MARL model trained for a given scenario is fine-tuned on a test scenario using the QMIX algorithm, where an attention network is used to handle different numbers of AES states, while the other networks are standard FC / MLP networks that are rebuilt and retrained when the dimension changes.

[0053] Table 1 Correspondence between electrical nodes and connected devices

[0054] The present invention evaluates its performance on unseen test scenarios by comparing it to other methods. First, the training / fine-tuning process of the baseline reinforcement learning method, the transfer training method, and the proposed method are compared. The transfer training method fine-tunes the model trained on the scenario with |S|=6 on the test scenario, while the present invention intentionally continues to fine-tune the model generated directly by the hypernetwork for the test scenario. The training / fine-tuning process of the three methods is as follows: Figure 7As shown. Due to the changes in environmental data caused by different arrivals to the AES, the baseline reinforcement learning method needs to train the model independently for each scenario. This unique characteristic of logistics energy coordination means that the baseline reinforcement learning method must retrain the model every day, while a typical power system only needs to train the model once for daily operations. Therefore, the baseline reinforcement learning method is as time-consuming as the model-based optimization method. Since the optimal strategies of the training scenarios and the test scenarios have certain similarities, the fine-tuning process of the transfer training method is significantly faster than the baseline reinforcement learning method. This is particularly evident when the |S| of the training scenario is the same as the |S| of the test scenario, as shown in Figure 7 (b) shows that despite this, a large number of fine-tuning steps are still required to find the optimal policy for the test scenario. In addition, when the dimensions of the network used for AES ID, QC / AGV last action, and QC / AGV action output change, it needs to be rebuilt and retrained, which further slows down the fine-tuning process. For these reasons, fine-tuning the trained model every working day still imposes a computational burden. In contrast, the present invention uses a hypernetwork to directly generate policies for test scenarios with various |S|. The reward at the beginning of fine-tuning is almost the same as the final reward of the other two methods, demonstrating the effectiveness and adaptability of the present method.

[0055] Next, we evaluated the performance of these methods in online applications. Each |S| consists of 10 test scenarios, each with 10,000 renewable energy output curves. For our proposed method, we evaluated the performance of both the directly generated strategy (without fine-tuning) and the fine-tuned strategy. The comparative results are shown in Table 2.

[0056] It can be seen that the model-based optimization method has the lowest scheduling cost among all methods due to Gurobi's accurate solution. However, its computational cost becomes expensive as |S| increases. Compared to model-based methods, baseline reinforcement learning methods and transfer training methods have similar scheduling costs but require significant training / fine-tuning time. Typically, these two methods only need to be trained once and used daily. Despite the longer training / fine-tuning time, they still outperform model-based methods that are re-solved every operating day. However, for the problem addressed by the present invention, the situation changes. These two methods also require daily retraining, like model-based optimization methods, thus losing their advantages in economic and computational efficiency. In contrast, the proposed method (without fine-tuning) directly generates policies, regaining the computational efficiency advantage of MARL. Because fine-tuning is not required, this method has the fastest execution time. However, its economic performance is slightly worse than that of the baseline reinforcement learning method and transfer training method. To this end, the present invention proposes a fine-tuning method, which achieves the lowest scheduling cost among all MARL-based methods. Because the policy generated by the fine-tuning-free method is already near-optimal, the fine-tuning method only requires a few fine-tuning steps. Compared with baseline reinforcement learning methods and transfer training methods, the fine-tuning time is negligible. Therefore, the present invention can obtain high-quality solutions in a very short time, which is superior to other methods.

[0057] Table 2 Online application test results of different methods

[0058] Subsequently, in order to reflect the advantages of the proposed model in reducing operating costs, the following comparison scheme is set up, as shown in Table 3.

[0059] Table 3 Comparison of different scheduling methods

[0060] Table 4 Comparison of scheduling results of different methods

[0061] Table 4 compares the scheduling results of different methods. The expected cost is the cost directly obtained by the solver, while the actual cost is the cost obtained by recalculating the distribution network flow after modifying the infeasible logistics solution. As shown in Table 4, although Method 1 has a lower expected cost, it ignores spatial constraints and the involvement of AGVs, resulting in the highest scheduling cost among the four methods. By considering the spatial constraints of the AES and QC, Method 2 achieves a higher expected cost than Methods 1 and 3. However, due to ignoring the impact of space on AGV transportation efficiency, the AES may require more AGVs than expected, resulting in certain constraint violations. By considering the joint scheduling of AGVs and other components, Method 3 and the proposed method achieve lower expected and actual costs than Methods 1 and 2. However, similar to Method 1, Method 3 still has a higher comparison value due to ignoring spatial constraints. In contrast, all methods achieve the lowest scheduling cost and 100% solution feasibility (comparison value of 0), demonstrating the necessity and effectiveness of the proposed logistics-energy spatiotemporal coordination model.

[0062] Finally, the present invention verifies its scalability by training a large model on a large system. Considering that the area of ​​SPDN is usually small, the IEEE-33 bus system is adopted as its topology. The number of available berths, QCs, and AGVs is 15, 30, and 60 respectively. Figure 8 Figure 2 shows the training / fine-tuning process for the four test scenarios. Since training a large system from scratch is challenging, the figure also includes the two proposed acceleration strategies for comparison. It is obvious that the baseline reinforcement learning method cannot find the optimal policy within an acceptable training step due to the large state and action space. However, the optimal policy is crucial for training large models. In this regard, strategy 1 is effective because it transfers a well-trained model from the previous |S|, enabling the present invention to find the optimal policy for the first scenario of the new |S|, as shown in Figure 2. Figure 8 (a) and (c). Strategy 2 can then be used to find the optimal policy for other scenarios with the same |S|. Compared to the baseline reinforcement learning method, these two acceleration strategies significantly reduce the required training steps, allowing large models to be trained for large systems. Although the transfer training method is faster than the baseline reinforcement learning method, it is computationally more demanding when |S| is large. Finally, the large model trained by the present invention can still directly generate effective policies, demonstrating its good scalability for large-scale systems.

[0063] Compared with the existing technology, the present invention further reduces the system operating cost while ensuring the feasibility of the scheduling strategy through the logistics-energy spatiotemporal collaborative model. In addition, the trained large model can be directly applied online without re-solving, and the calculation cost is at the millisecond level, which is better than the existing methods based on mathematical optimization.

[0064] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning, characterized in that: By constructing a port logistics-energy spatiotemporal coordination model that includes the AES berth allocation model, the QC allocation model, the AGV scheduling model, and the SPDN optimal power flow model, the optimization-based logistics-energy spatiotemporal coordination model is transformed into a multi-agent fully observable Markov decision process (MDP). By constructing and training a general multi-agent reinforcement learning (MARL) large model, it is used to calculate the reward function, observation and action space and automatically generate scheduling strategies for various scenarios. The MARL large model includes: a hypernetwork, an agent, a cascade action modification layer, an environment unit and a MIX network, wherein: the hypernetwork generates parameters for the agent strategy network according to the environmental data of the scene; each agent generates an original value function and outputs it to the cascade action modification layer to generate a modified action and a corresponding value function that satisfies the time and space constraints. The modified action is then output to the environment unit to obtain a reward and the next state, and the value function is output to the MIX network to calculate the joint action value function.

2. The seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The port logistics-energy space-time coordination model is constructed in the following way: 1.1) The AES berth allocation model refers to: ship berthing status, that is, ship s berths at berth b at time t, then X bst is 1, otherwise it is 0, specifically: AES berth allocation constraints include: limiting berthing time and the permitted range of berthing locations Each AES can only be assigned one berth, and each berth can accommodate at most one AES at a time: 1≤B s ≤B max ,as well as in: and They represent the arrival time, berthing start time and departure time of ship s respectively. is the maximum port stay of ship s, B s is the berth number of ship s, B max is the total number of shore power berths at the port terminal; 1.2) QC allocation model refers to: the operating state of QC, that is, ω q,s,t =1 indicates that at time t, QC q serves AES s. The time constraints for QC allocation include: each QC can only serve one AES at most at any time. Total number of QCs that can be assigned The number of QCs assigned to each AES QC cargo handling efficiency All QCs serving the same AES operate at the same efficiency The temporal and spatial constraints of QC allocation include: all QCs serving the same AES are spatially adjacent, and idle QCs must not be located between working QCs on the same AES -1≤ω q+1,s,t -ω q,s,t +ω q′,s,t ≤1, The location of each QC is the same as the location of the AES it serves All QCs cannot cross each other Where: Q min is the total number of QCs available at the port, and They represent the minimum and maximum QC numbers that ship s can withstand due to the limitation of hull length, is the container loading and unloading efficiency of QC q for AES s at time t, For its maximum value, B qt is the position of QCq at time t. If QCq serves AESs at time t (i.e., ω qst =1), then it is the same as the position of ship s; and the linearized form of the above constraints: B q,t ≤B s +M(1-ω q,s,t ), B q,t +M(1-ω q,s,t )≥B s , 1.3) AGV scheduling model refers to: the operating state of AGV includes idle state Service AES Status Charging status The constraints of AGV scheduling include: each AGV operates in only one state at any time AGV only serves AES at the port AGV transportation efficiency Allocate enough AGVs to complete cargo loading and unloading tasks before AES departs Limit the maximum transportation efficiency of AGV according to the maximum QC loading and unloading efficiency The QC of each AES and the overall efficiency of the AGV are consistent AGV overcharging restrictions in: and are binary variables. When they are equal to 1, they indicate that AGV v is idle, serving AES s, and charging at time t. is the container transportation efficiency of AGV v to AES s at time t. Different docking locations of AES will result in different transportation distances of AGV. represents the maximum efficiency of AGV v serving AES s at berth b, TEU s is the number of standard containers on ship s, TEU is the unit of measurement for standard containers, E v,t The energy of the AGVv battery, and are their minimum and minimum values ​​respectively, and are the energy consumed by AGV v during operation and obtained by charging in one scheduling step, respectively; 1.4) The SPDN optimal power flow model means that the power loads of AES, QC, and AGV on SPDN node j are: Among them: AES load is determined by its basic load and operating status, QC load depends on its rated power and cargo handling efficiency, is the ratio of the total efficiency to the total number of QCs serving AES, that is The AGV charging load depends on the battery charge; the linear DistFlow equation is used to model the SPDN, and its operating constraints include: Power balance equation Calculation of node power load and available renewable energy Node voltage balance and magnitude in: are the rated powers of AESs and QCq, Π(j), Φ(j), Θ(j) are the sets of all shore power berths, QCs and charging stations electrically connected to node j, P ij,t , Q ij,t and P jk,t , Q jk,t are the active and reactive power flows from node i to node j and from node j to node k at time t, Ω(j) and θ(j) are the upstream and downstream nodes connected to node j, respectively. If and only if node j is connected to a grid node, are the active and reactive loads of node j at time t, U i,t is the voltage amplitude of node i at time t, r ij 、x ij are the resistance and reactance of line ij, U i,t is the voltage at node i, are the actual dispatch power and maximum available power of renewable energy unit m at time t, is the rated capacity of line ij, are the minimum and maximum voltage amplitudes at node j, respectively.

3. The seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning according to claim 1 or 2 is characterized in that: The objective function of the port logistics-energy spatiotemporal synergy model is to minimize the sum of the cost of purchasing electricity from the upstream power grid and the waiting and docking costs of all AES, specifically: in: is the electricity price purchased from the main network at time t, is the amount of electricity purchased from the main network at time t, and are AES waiting and docking costs, respectively.

4. The seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The optimization-based logistics-energy spatiotemporal coordination model is transformed into a multi-agent fully observable Markov decision process<A,S,O,U,r,P,γ> , specifically including: 2.1) The agent set A includes: i) AES agent s∈S∈A; ii) QC agent q∈Q∈A; iii) AGV agent v∈V∈A; each agent group has a different action space but shares the same reward function; 2.2) To reflect the current operating state of the system and guide the agent to make decisions, it is necessary to define the environment state s t ∈S, which specifically includes: public state AES Status and AGV status Among them: public status Includes the RES output for the current time period t and the entire scheduling period The latter includes observations up to t and predictions from t+1 to T; this enables the agent to observe future information and decide whether to work now or later, thus facilitating temporal coordination; the AES state Medium, binary Represents AES: Not reached Arrived but not docked Moored at the berth during [t-1,t] and have left Indicates unprocessed goods; and They represent the time distance from the current time to the arrival and latest departure time of AES respectively; and Respectively represent the cumulative time since the arrival of the AES and the cumulative delay time after the latest departure time, so as to improve logistics efficiency by allocating berths and handling goods in a timely manner; AGV status Medium E v,t is the battery level; The environmental status s t Specifically: In a fully observable MDP, the observation of each agent is the same as the state, that is, 2.3) Action space includes: The action space of AES is vector Indicate berth selection Or do not select a berth The action space of QC is the vector for Serve or Idle The action space of AGV is vector Indicates service or Idle or charging 2.4) State transition means: the state of the environment s is affected by the environment and the action of the agent. t middle Natural evolution over time, Varies according to AES operation, Changes from 0 to 1 after all goods have been processed; when the AES is parked and there are AGVs serving it, the unprocessed goods AGV status 2.5) The state function is used to guide the agent to find the optimal scheduling strategy, specifically: Among them: the first item is the electricity purchase cost; the second item is the logistics cost, including waiting cost Berthing cost Delayed departure fine Fines for failure to complete loading and unloading tasks in: and are the penalty costs for delayed departure and unfinished loading and unloading tasks respectively; 2.6) The environment includes: logistics system and SPDN, where: the joint actions of all agents are used to calculate the logistics reward and its logistics load, and then use a commercial solver to minimize To optimize the goal, the DistFlow model of the SPDN optimal power flow is solved to obtain the total reward and update the state of the system at the next moment.

5. The seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The MARL model includes: a hypernetwork, an agent, a cascade action modification layer, an environment unit, and a MIX network, wherein: the hypernetwork is based on the environment data of the scene Generate parameters for the agent policy network; each agent generates a primitive value function Q(τ,·) and outputs it to the cascade action modification layer to generate modified actions that satisfy spatiotemporal constraints and the corresponding value function The modified action is output to the environment unit to obtain the reward and next state, and the value function is output to the MIX network to calculate the joint action value function The parameters of the MIX network also depend on 6. The seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning according to claim 1 or 5 is characterized in that: The MARL large model is trained in the following way: a) Use the QMIX algorithm to collect the optimal strategies for all training scenarios, whose parameters are the output of the hypernetwork and stored in the training dataset; b) Train the hypernetwork through supervised learning to establish a mapping from environmental data to the optimal strategy.

7. The seaport logistics-energy optimization scheduling method based on multi-agent reinforcement learning according to claim 6 is characterized in that: Two strategies are used to accelerate the collection of the optimal strategy for each scenario in step a: 1) Strategy 1: The number of AES The scenarios are grouped into one category, and training is gradually started from the class with a smaller number of AES to the class with a larger number of AES, including: Step i, classify all scenes, all scenes in each class have the same number of AES, and sort all classes from small to large according to the number of AES; Step ii, starting from the class with a smaller number of AES, randomly select a scenario to train from scratch to obtain the optimal policy parameters for the scenario; Step iii, select the next class. The first scene in this class is not trained from scratch. Instead, the optimal strategy parameters of the scene that has been trained in step ii are used as the initialization parameters of the neural network, and then training is performed with the initialization parameters; Step iv: perform steps ii and iii for each subsequent class in sequence until the training of all classes is completed; 2) Strategy 2: Evaluate the distance between scenes based on their environmental data, use the optimal parameters of the trained scene as the initialization parameters of the scene closest to it, cluster the scenes in the same category according to their environmental parameters and obtain multiple clusters, and then train each cluster in parallel, including: Step a, clustering the scenes in each class using a clustering algorithm and obtaining multiple clusters, where the clustering is based on the environmental data of each scene; Step b, for each scene in the cluster, calculate the Euclidean distance between each scene and other scenes, and then sort all scenes from the center point of the cluster to the outside according to the distance. This sequence is used as the order of training scenes in the following steps; Step c, using the first trained scene obtained in step a to initialize the first scene to be trained in each cluster, and after the scene is trained, it is used as the initialization of subsequent scenes; Step d: Repeat step c for all clusters in all classes until the optimal parameters for all scenarios are collected.

8. A seaport logistics-energy optimization scheduling system based on multi-agent reinforcement learning that implements any of the methods described in claims 1-7, characterized in that: include: Model training module, environmental data collection module, agent decision calculation module and scheduling instruction execution module, among which: the model training module trains the MARL big model and uses the trained big model for subsequent calculations; the environmental data collection module is used to collect the environmental data of the day on each operation day, and these environmental data are used as the input of the super network in the MARL big model; the agent decision calculation module is used to obtain the scheduling decision suitable for the environmental data of the operation day. The parameters of each neural network in the agent are generated by the super network, and then the scheduling decision of each agent is obtained by inputting the system environment state. This process is a dynamic Markov process until the end of the scheduling cycle; The instruction execution module sends the dispatch instructions to the port control room to execute the best dispatch plan.

Images