Port operation resource dynamic allocation method based on multi-objective optimization algorithm

Abstract

The application discloses a wharf operation resource dynamic allocation method based on a multi-target optimization algorithm, which comprises the following steps: S1, collecting wharf operation data and pre-processing the collected data; S2, constructing a Dreamer reinforcement learning model, and predicting a wharf operation state through a world model of the Dreamer reinforcement learning model; S3, optimizing a prediction process of the world model by using a water circulation algorithm; S4, training an agent of the reinforcement learning module, and learning a wharf operation resource allocation strategy; S5, optimizing a training process of the agent of the reinforcement learning module by using the water circulation algorithm; S6, calculating a wharf operation resource allocation deviation and a wharf operation resource allocation scheme; and S7, updating parameters of the Dreamer reinforcement learning model by using incremental learning, and adjusting the wharf operation resource allocation scheme. The application fuses the Dreamer reinforcement learning model and the water circulation algorithm, realizes dynamic optimization allocation of wharf operation resources, and has the advantages of fast response, strong adaptability and strong multi-target coordination capability.

Description

Technical Field

[0001] This invention relates to the field of intelligent scheduling and resource optimization technology, and in particular to a method for dynamic allocation of port operation resources based on a multi-objective optimization algorithm. Background Technology

[0002] With the rapid development of global port operations and the increasing complexity of international logistics chains, terminal operation systems are facing unprecedented pressure to operate efficiently. As the core node of port logistics, the efficiency of terminal resource scheduling directly affects the overall port operation capacity, the smoothness of logistics channels, and port throughput. Traditional terminal operation resource allocation methods generally rely on empirical rules or heuristic algorithms. Although they have certain practicality in specific scenarios, their robustness, real-time performance, and adaptability are often limited when facing dynamically changing operating environments, complex and ever-changing operational tasks, and multi-objective coordination needs. They are difficult to meet the high-standard management requirements of current smart ports and automated terminals.

[0003] Current mainstream research on terminal resource scheduling mainly focuses on sub-tasks such as optimizing the operation sequence of equipment such as loading and unloading bridges, yard bridges, and trailers, berth allocation, and yard scheduling. Traditional methods widely used in related research include integer programming, linear programming, genetic algorithms, ant colony optimization, and particle swarm optimization. Although these methods have achieved certain results in static or semi-dynamic scenarios, they often face problems such as difficulty in parameter adjustment, easy getting trapped in local optima, and low computational efficiency when dealing with the highly dynamic and uncertain actual working conditions of terminal systems. In particular, they are difficult to deal with complex events such as equipment failure, ship schedule adjustments, and changes in task priorities in real time.

[0004] In recent years, with the development of artificial intelligence and reinforcement learning, some studies have attempted to introduce reinforcement learning models such as Q-learning, Deep QNetwork, and Actor-Critic into port operation scheduling to learn optimal scheduling strategies from the environment. However, the application of existing reinforcement learning methods in port operation scenarios still faces several bottlenecks: First, port operation states are complex and high-dimensional, making it difficult for reinforcement learning models to accurately model long-term state transition patterns, resulting in low sample efficiency during the policy learning process. Second, real-world port scheduling often involves multiple objectives (such as equipment utilization, energy balance, operation completion time, and operation fairness), while most reinforcement learning methods are more suitable for single-objective optimization and lack the ability to coordinate conflicting relationships between multiple objectives. Third, reinforcement learning agents typically rely on a large amount of interactive data during training, while actual port systems do not have sufficient safety space for extensive trial-and-error learning, limiting the deployability of models in real systems.

[0005] In terms of optimization algorithms, although traditional swarm intelligence optimization methods (such as genetic algorithms and particle swarm optimization) have been used in some scheduling optimization tasks, these algorithms generally suffer from slow convergence speed, easy getting trapped in local optima, and most of them are static structures, lacking a linkage mechanism with deep learning models. The water cycle algorithm, as a novel global optimization algorithm proposed in recent years, is inspired by the principle of water cycle in nature and introduces dynamic processes such as evaporation, precipitation, and flow. It has advantages such as strong global search ability, good convergence stability, and simple parameter control. However, it has not yet formed a standard method in joint optimization with reinforcement learning agents. In particular, the collaborative mechanism in core links of deep reinforcement learning, such as world model parameter optimization and policy network update, needs further in-depth research.

[0006] In summary, existing terminal operation resource allocation methods still have significant room for improvement in terms of multi-objective coordination, dynamic adaptation, and intelligent optimization. In particular, there is a lack of a systematic solution that can integrate world model prediction, reinforcement learning strategy generation, and swarm intelligence optimization algorithm for collaborative optimization. Existing technologies cannot effectively solve the problem of how to generate accurate, efficient, and stable resource scheduling strategies based on historical and real-time data in complex, dynamic, and multi-constrained terminal operation scenarios, and are therefore unable to support the high requirements of future smart terminals for "dynamic, real-time, and adaptive" resource scheduling systems.

[0007] Therefore, there is an urgent need to propose a dynamic allocation method for terminal operation resources based on a multi-objective optimization algorithm. This method integrates the prediction capabilities of the world model, the training capabilities of reinforcement learning strategies, and the water cycle optimization mechanism to achieve efficient, stable, and adaptive control of complex resource scheduling systems in multi-objective scenarios. Summary of the Invention

[0008] One objective of this invention is to propose a dynamic resource allocation method for terminal operations based on a multi-objective optimization algorithm. This invention integrates the Dreamer reinforcement learning model and the water cycle algorithm to make high-precision predictions of terminal operation status and dynamically optimize resource scheduling strategies, thereby achieving efficient and stable multi-objective resource allocation. It has both strategy self-learning ability and multi-objective collaborative optimization ability, and can adapt to complex and ever-changing terminal operation environments, improving the real-time performance, intelligence and overall system efficiency of scheduling.

[0009] A method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm according to an embodiment of the present invention includes the following steps:

[0010] S1. Collect dock operation data, preprocess the collected data, and generate a standardized dataset;

[0011] S2. Construct the Dreamer reinforcement learning model, train the world model of the Dreamer reinforcement learning model based on the standardized dataset, and predict the port operation status through the world model.

[0012] S3. Optimize the prediction process of the world model using the water cycle algorithm. Set water droplets to represent different combinations of training parameters, calculate the prediction error of the world model, adjust the parameters of the world model based on the mechanism of water droplet flow to the optimal solution, and generate optimized prediction data.

[0013] S4. Based on the predicted data, set the state space, action space and reward function of the reinforcement learning module agent, train the reinforcement learning module agent, learn the terminal operation resource allocation strategy, and generate a resource allocation scheme.

[0014] S5. Optimize the training process of the reinforcement learning module agent using the water cycle algorithm. Set water droplets to represent different combinations of reinforcement learning module parameters, calculate the fitness of the resource allocation scheme, and adjust the reinforcement learning module parameters based on the water droplet flow process to optimize the dock operation resource allocation scheme.

[0015] S6. Collect real-time dock operation data, calculate the deviation of dock operation resource allocation, and if the resource allocation deviation exceeds the set threshold, adjust the dock operation resource allocation scheme based on the evaporation-precipitation mechanism of the water cycle algorithm.

[0016] S7. Optimize the Dreamer reinforcement learning model based on historical terminal operation data, use incremental learning to update the parameters of the Dreamer reinforcement learning model, and adjust the terminal operation resource allocation scheme.

[0017] Optionally, the terminal operation data includes vessel arrival time, cargo loading and unloading schedule, equipment availability, operation progress, and environmental status. The preprocessing includes format conversion, outlier detection, time synchronization, and normalization.

[0018] Optionally, S2 specifically includes:

[0019] S21. Construct a Dreamer reinforcement learning model, which includes a world model and a reinforcement learning module. The world model includes a state encoder layer, a state transition layer, a reconstruction decoder layer, a reward predictor layer, and a termination predictor layer. The reinforcement learning module includes a policy network and a value network. The world model predicts the terminal operation status, reward information, and task termination information based on terminal operation data. The policy network is used to optimize resource allocation schemes in the world model, and the value network is used to evaluate the merits of different resource allocation schemes.

[0020] S22. Train a world model based on a standardized dataset, construct a state encoder layer, and process the input standardized dataset o.t Feature extraction is performed, and a variational autoencoder is used to map the standardized dataset into latent states z. t :

[0021] z t =W μ ·φ(o t )+b μ +softplus(W σ ·φ(o t )+b σ )·∈,∈~N(0,1);

[0022] Among them, z t W represents the current potential state. μ and W σ φ(o) represents the weight matrix of the state encoder layer. t ) represents the feature extraction vector, b μ and b σ The term represents the bias term of the state encoder layer, softplus represents the non-linear activation function, ∈ represents the noise variable following a standard normal distribution, N represents a standard normal distribution with a mean of 0 and a variance of 1, and o t This represents the standardized dataset of the input.

[0023] S23. Construct a state transition layer and use gated loop units to recursively model historical latent states and actions to obtain the current hidden state h. t And generate predicted potential states Prior distribution:

[0024]

[0025] Among them, h t Indicates the current hidden state, GRU represents the gated loop unit, h t-1 z represents the hidden state at the previous time step. t-1 Let a represent the potential state at the previous time step. t-1 Indicates the action at the previous time step. Let N represent the predicted potential state, and N represent the normal distribution. and This represents the weight matrix of the state transition layer. and represents the bias of the state transition layer, and softplus represents the non-linear activation function;

[0026] S24. Construct the reconstruction decoder layer to predict the latent states. Decoded into a reconstructed standardized dataset The reconstruction decoder layer uses a multilayer perceptron for decoding;

[0027] S25. Construct a reward predictor layer to predict the immediate reward under the current potential state and action. The reward predictor layer uses a fully connected network;

[0028] S26. Construct a termination predictor layer to determine whether the current potential state is a termination state and output the termination flag of the prediction. The termination predictor layer uses a fully connected network combined with the sigmoid activation function.

[0029] S27. Jointly train all modules in the world model, construct a comprehensive loss function, and minimize the reconstruction prediction error, reward prediction error, termination prediction error, and KL divergence:

[0030]

[0031] Where L represents the comprehensive loss function, λ1, λ2, λ3, and λ4 represent the weighting coefficients of each loss term, and o t This represents the standardized input dataset. Represents the standardized data for reconstruction, ||·|| 2 Denotes the square norm. r represents the predicted potential state. t This indicates a real reward. This represents the predicted current potential state and the immediate reward under the given action. The terminator represents the termination probability of the prediction, and d describes the termination marker of the prediction. t KL(·||·) represents the true termination probability, and KL(·||·) represents the Kullback-Leibler divergence. The probability distribution representing the true potential state, and the ideal distribution describing the world model. μ represents the distribution of potential states predicted by the world model, describing the world model's current prediction of the environmental state. t and The mean and variance represent the true potential states. and This represents the mean and variance of the world model's predictions;

[0032] S28. Based on the trained world model, input the current task data to generate a sequence of potential states for future time steps. Predicting reward sequences and termination probability sequence

[0033] Optionally, S3 specifically includes:

[0034] S31. Based on the trained world model, set the optimization objective of the water cycle algorithm. The optimization objective is to minimize the prediction error of the world model and optimize the parameters θ of the state transition layer and the parameters of the state encoder layer of the world model.

[0035] S32. Initialize the water cycle algorithm, set the water droplet population size P, and randomly generate P sets of candidate parameter combinations within the parameter search space. The parameters of the state transition layer and state encoder layer corresponding to each water droplet are represented respectively. The initial fitness for each parameter combination is calculated. The fitness is used to measure the performance of the current parameter combination on the optimization objective, and is obtained through prediction error. express;

[0036] S33. For each set of parameter combinations In the world model, forward prediction is performed, the corresponding prediction error is calculated, and the error function is defined:

[0037]

[0038] in, Represents the error function, o t This represents the standardized input dataset. This represents the reconstructed standardized data, and KL(·||·) represents the Kullback-Leibler divergence. The probability distribution representing the true potential state, and the ideal distribution describing the world model. μ represents the distribution of potential states predicted by the world model, describing the world model's current prediction of the environmental state. t and The mean and variance represent the true potential states. and The values ​​represent the mean and variance of the world model predictions, α1 and α2 represent the loss weights, and ||·|| 2 Represents the square norm;

[0039] S34. Execute the water droplet flow process, guide the search direction among the water droplets according to the fitness value, and update the parameter combination of each group of water droplets:

[0040]

[0041] in, and Let these represent the parameters of the state transition layer and the state encoder layer for the i-th water droplet in round t+1, respectively. and Let ξ1 and ξ2 represent the parameters of the state transition layer and state encoder layer for the i-th droplet in the t-th round, respectively. α represents the learning rate, β represents the perturbation factor, and ξ1 and ξ2 represent independent random perturbation terms. and This represents another set of droplet parameters used for reference in round t, which has better fitness than the i-th droplet.

[0042] S35. Calculate the prediction error for each updated parameter combination. If the convergence condition is met... Then the droplet parameters are marked as converged, where ∈ represents the convergence threshold;

[0043] S36. For non-converged water droplet combinations, execute the evaporation-precipitation mechanism of the water cycle algorithm, wherein the evaporation-precipitation mechanism resets the parameters of the M groups of water droplets with the worst fitness:

[0044]

[0045] in, and θ represents the new parameter combination for the k-th water droplet that has been reset. rand and denoted as the combination of water droplet parameters randomly generated again in the parameter search space, γ represents the perturbation intensity coefficient in the evaporation-precipitation process, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism;

[0046] S37. Repeat steps S34 to S36 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination, update the world model based on the optimized parameter combination, and recalculate the predicted values ​​of the operation status for future time steps.

[0047] Optionally, S4 specifically includes:

[0048] S41. Construct a reinforcement learning module and define a policy network for optimizing the allocation of terminal operation resources. Value Network V ψ Initialize the parameters of the policy network and value network, set the input of the agent to the dock operation status, reward information and task termination information provided by the world model, and set the output of the agent to the allocation strategy of dock operation resources.

[0049] S42. Construct the state space, action space, and reward function for terminal operation resource allocation, and define the state vector. in This represents the potential state predicted by the world model at time t. Indicates the immediate reward for the prediction. Indicates the probability of prediction termination. ρ represents the change in the potential state. t This represents the Euclidean distance between the current potential state and the average potential state.

[0050] S43. Based on the world model, sample the resource allocation strategy for terminal operations and set the action strategy distribution:

[0051]

[0052] Among them, a t This indicates that in the state vector s t The action obtained by downsampling This represents the policy network, where Softmax represents the normalization operation, and W... π and b π Let σ represent the weight matrix and bias vector of the policy network, and σ represent the nonlinear activation function.

[0053] S44. Calculate the cumulative reward of the terminal operation resource allocation strategy and define the reward function:

[0054]

[0055] Among them, R t Let γ represent the reward function, describing the cumulative reward starting from time t, where T represents the total number of time steps and γ represents the discount factor. Let ρ represent the instantaneous reward predicted at time t+k in the future, λ represent the state offset penalty coefficient, and ρ represent the instantaneous reward predicted at time t+k. t+k This represents the state distance at time t+k.

[0056] S45. Calculate the advantage function and adjust the value deviation based on the execution of terminal operation resource allocation:

[0057]

[0058] Among them, A t Represents the dominance function. V represents the predicted immediate reward. ψ Represents the value network, s t and s t+1 These represent the current state and the next state, respectively, and γ represents the discount factor. a represents the motion disturbance coefficient. t Indicates the current action. This represents the average of historical actions. V represents the L2 norm between the current action and historical actions, k' represents the category, and V ψ (s t ) and V ψ (s t+1 ) indicates that in state s t Below, the value estimation of the state is based on the parameter ψ output by the value network;

[0059] S46. Optimize the policy network, calculate the policy loss function, optimize the value network, calculate the value error of resource allocation decisions, and define the value loss function:

[0060]

[0061] Among them, L π L represents the policy loss function. V Let E represent the value loss function, E represent the expected value, and D represent the experience pool. Let A represent the logarithmic probability of the current action. t Let κ represent the advantage function, and let κ represent the entropy regularization coefficient. V represents the entropy value of the policy network output distribution. ψ (s t R represents the value estimate of the current state. t Let δ represent the reward function, δ represent the value difference adjustment coefficient, and ΔV represent the value difference adjustment coefficient. t This represents the difference between the predicted values ​​of adjacent states;

[0062] S47. Perform parameter optimization by adjusting the parameters of the policy network and value network based on gradient descent.

[0063] S48. Output the optimized terminal operation resource allocation strategy, store the optimized strategy network parameters and value network parameters, apply the optimized resource allocation strategy to the terminal operation scheduling system, and adjust the resource allocation scheme in the actual operation process.

[0064] Optionally, S5 specifically includes:

[0065] S51. Initialize the reinforcement learning module optimization process for the water cycle algorithm, and optimize the policy network parameters in the reinforcement learning module. Using the value network parameter ψ as a joint optimization variable, and setting the droplet population size to G, G individual droplets are randomly generated in the parameter space.

[0066] S52, Based on the current individual water droplet Perform terminal operation resource allocation tasks within the optimized world model, utilizing a policy network. With Value Network V ψ Conduct simulated interactions to obtain resource allocation strategies and generate experience trajectories D. i ={(s t ,a t ,R t ,s t+1 )};

[0067] S53. Calculate the fitness function for each water droplet based on the empirical trajectory:

[0068]

[0069] in, This represents the fitness value of the i-th water droplet. L represents the policy loss corresponding to the policy network. V (ψ i ) represents the value loss corresponding to the value network, and δ represents a positive constant to prevent division by zero overflow;

[0070] S54. Set the update rules for water droplets in the parameter space, execute the water droplet flow process, and select water droplet pairs based on relative fitness. and :

[0071]

[0072] in, and Let these represent the policy network parameters and value network parameters of the i-th droplet in round t+1, respectively. and Let ᵢ represent the policy network parameters and value network parameters of the i-th droplet in round t, respectively, and let ι represent the learning rate. This represents the disturbance factor. and Let ξ1 and ξ2 represent another set of droplet parameters used for reference in round t, where ξ1 and ξ2 represent random perturbation terms under uniform distribution.

[0073] S55. After updating all water droplets, re-execute the resource allocation simulation in the world model, generate policy trajectories based on the new parameters, and recalculate the fitness function. Determine whether the convergence condition is met;

[0074] S56. If the convergence condition is not met, the evaporation-precipitation mechanism of the water cycle algorithm is executed, wherein the evaporation-precipitation mechanism randomly resets the M water droplets with the lowest fitness ranking:

[0075]

[0076] in, and Let represent the policy network parameters and value network parameters of the k-th water droplet that has been reset, and η represent the perturbation intensity coefficient in the evaporation-precipitation mechanism. and ψ rand ξ1′ and ξ2′ represent the newly generated random parameters, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism;

[0077] S57. Repeat steps S54 to S56 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination of the reinforcement learning module and replace the original parameters to optimize the dock operation resource allocation scheme.

[0078] The beneficial effects of this invention are:

[0079] First, this invention provides a dynamic allocation method for terminal operation resources based on a multi-objective optimization algorithm. This method overcomes the problems of static operation scheduling, lagging strategy updates, poor adaptability to complex operation scenarios, and lack of multi-objective coordination capabilities in existing technologies. By constructing a comprehensive intelligent scheduling framework that integrates a world model, a reinforcement learning module, and a water cycle optimization mechanism, this invention improves the intelligence level and operational efficiency of terminal operation resource allocation. Based on standardized terminal operation data and combined with the Dreamer reinforcement learning model, this invention achieves dynamic prediction of future operation states. It completes iterative optimization of operation strategies without requiring extensive interaction with real systems, thereby improving the utilization rate of training samples and the speed of strategy generation.

[0080] Secondly, the world model of this invention not only includes state transition and state encoding modules, but also introduces a reward predictor layer and a termination predictor layer, which can simultaneously model the dock operation status, immediate benefits, and task termination information. This enables the policy learning process to have stronger environmental modeling and target perception capabilities in the potential state space. At the same time, this invention introduces the water cycle algorithm into the world model parameter optimization and reinforcement learning policy network training process. During the optimization process, by simulating water droplet flow, evaporation, and precipitation mechanisms, the optimal parameter path is explored, avoiding the problems of parameter selection relying on experience and optimization easily falling into local optima in traditional methods. This improves the global search capability and convergence stability of the overall scheduling model.

[0081] Finally, this invention incorporates the core idea of ​​multi-objective optimization. In the resource scheduling process, it not only focuses on a single performance indicator, but also incorporates multiple scheduling objectives (such as energy consumption balance, operational fairness, and yard balance rate) into the reward function system. In reinforcement learning, it achieves a comprehensive improvement in multi-objective scheduling performance through multi-factor fusion design. The method also sets up a real-time resource allocation deviation monitoring mechanism. By calculating the difference between the actual operation action and the strategy output, the strategy adjustment process is automatically triggered when the deviation exceeds a set threshold. Combined with the perturbation mechanism in the water circulation algorithm, the parameters are reset, enabling the model to have rapid adaptation and continuous optimization capabilities in the dynamically changing terminal environment. Attached Figure Description

[0082] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0083] Figure 1 This is a flowchart of a dynamic allocation method for dock operation resources based on a multi-objective optimization algorithm proposed in this invention. Detailed Implementation

[0084] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0085] refer to Figure 1 A method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm includes the following steps:

[0086] S1. Collect dock operation data, preprocess the collected data, and generate a standardized dataset;

[0087] S2. Construct the Dreamer reinforcement learning model, train the world model of the Dreamer reinforcement learning model based on the standardized dataset, and predict the port operation status through the world model.

[0088] S3. Optimize the prediction process of the world model using the water cycle algorithm. Set water droplets to represent different combinations of training parameters, calculate the prediction error of the world model, adjust the parameters of the world model based on the mechanism of water droplet flow to the optimal solution, and generate optimized prediction data.

[0089] S4. Based on the predicted data, set the state space, action space and reward function of the reinforcement learning module agent, train the reinforcement learning module agent, learn the terminal operation resource allocation strategy, and generate a resource allocation scheme.

[0090] S5. Optimize the training process of the reinforcement learning module agent using the water cycle algorithm. Set water droplets to represent different combinations of reinforcement learning module parameters, calculate the fitness of the resource allocation scheme, and adjust the reinforcement learning module parameters based on the water droplet flow process to optimize the dock operation resource allocation scheme.

[0091] S6. Collect real-time dock operation data, calculate the deviation of dock operation resource allocation, and if the resource allocation deviation exceeds the set threshold, adjust the dock operation resource allocation scheme based on the evaporation-precipitation mechanism of the water cycle algorithm.

[0092] S7. Optimize the Dreamer reinforcement learning model based on historical terminal operation data, use incremental learning to update the parameters of the Dreamer reinforcement learning model, and adjust the terminal operation resource allocation scheme.

[0093] This invention constructs a dynamic scheduling framework that integrates the Dreamer reinforcement learning model and the water cycle algorithm, thereby achieving high-precision modeling of terminal operation status and adaptive optimization of resource allocation strategies. This enhances the intelligence, real-time performance, and multi-objective coordination capabilities of resource scheduling, meeting the practical needs of efficient resource allocation in complex environments of automated terminals.

[0094] In this embodiment, the terminal operation data includes ship arrival time, cargo loading and unloading plan, equipment availability, operation progress and environmental status, and the preprocessing includes format conversion, outlier detection, time synchronization and normalization.

[0095] This invention collects heterogeneous operational data from multiple sources, such as ship arrival times, loading and unloading plans, and equipment availability. It then performs format conversion, anomaly detection, time synchronization, and normalization on the data to construct a high-quality, standardized dataset with a unified format. This provides a stable and reliable data foundation for subsequent world model training and policy learning, thereby improving the accuracy and generalization ability of model training.

[0096] In this embodiment, S2 specifically includes:

[0097] S21. Construct a Dreamer reinforcement learning model, which includes a world model and a reinforcement learning module. The world model includes a state encoder layer, a state transition layer, a reconstruction decoder layer, a reward predictor layer, and a termination predictor layer. The reinforcement learning module includes a policy network and a value network. The world model predicts the terminal operation status, reward information, and task termination information based on terminal operation data. The policy network is used to optimize resource allocation schemes in the world model, and the value network is used to evaluate the merits of different resource allocation schemes.

[0098] S22. Train a world model based on a standardized dataset, construct a state encoder layer, and process the input standardized dataset o. t Feature extraction is performed, and a variational autoencoder is used to map the standardized dataset into latent states z. t :

[0099] z t =W μ ·φ(o t )+b μ +softplus(W σ ·φ(o t )+b σ )·∈,∈~N(0,1);

[0100] Among them, z t W represents the current potential state. μ and W σφ(o) represents the weight matrix of the state encoder layer. t ) represents the feature extraction vector, b μ and b σ The term represents the bias term of the state encoder layer, softplus represents the non-linear activation function, ∈ represents the noise variable following a standard normal distribution, N represents a standard normal distribution with a mean of 0 and a variance of 1, and o t This represents the standardized dataset of the input.

[0101] S23. Construct a state transition layer and use gated loop units to recursively model historical latent states and actions to obtain the current hidden state h. t And generate predicted potential states Prior distribution:

[0102]

[0103] Among them, h t Indicates the current hidden state, GRU represents the gated loop unit, h t-1 z represents the hidden state at the previous time step. t-1 Let a represent the potential state at the previous time step. t-1 Indicates the action at the previous time step. Let N represent the predicted potential state, and N represent the normal distribution. and This represents the weight matrix of the state transition layer. and represents the bias of the state transition layer, and softplus represents the non-linear activation function;

[0104] S24. Construct the reconstruction decoder layer to predict the latent states. Decoded into a reconstructed standardized dataset The reconstruction decoder layer uses a multilayer perceptron for decoding;

[0105] S25. Construct a reward predictor layer to predict the immediate reward under the current potential state and action. The reward predictor layer uses a fully connected network;

[0106] S26. Construct a termination predictor layer to determine whether the current potential state is a termination state and output the termination flag of the prediction. The termination predictor layer uses a fully connected network combined with the sigmoid activation function.

[0107] S27. Jointly train all modules in the world model, construct a comprehensive loss function, and minimize the reconstruction prediction error, reward prediction error, termination prediction error, and KL divergence:

[0108]

[0109] Where L represents the comprehensive loss function, λ1, λ2, λ3, and λ4 represent the weighting coefficients of each loss term, and o t This represents the standardized input dataset. Represents the standardized data for reconstruction, ||·|| 2 Denotes the square norm. r represents the predicted potential state. t This indicates a real reward. This represents the predicted current potential state and the immediate reward under the given action. The terminator represents the termination probability of the prediction, and d describes the termination marker of the prediction. t KL(·||·) represents the true termination probability, and KL(·||·) represents the Kullback-Leibler divergence. The probability distribution representing the true potential state, and the ideal distribution describing the world model. μ represents the distribution of potential states predicted by the world model, describing the world model's current prediction of the environmental state. t and The mean and variance represent the true potential states. and This represents the mean and variance of the world model's predictions;

[0110] S28. Based on the trained world model, input the current task data to generate a sequence of potential states for future time steps. Predicting reward sequences and termination probability sequence

[0111] This invention constructs a Dreamer world model that includes a state encoder, a state transition layer, a reward predictor, and a termination predictor. It can model the job environment based on the potential state and predict future job states, rewards, and termination signals. This allows reinforcement learning agents to complete policy learning without relying on a large number of real interactions, effectively improving training efficiency and the accuracy and stability of scheduling policies.

[0112] In this embodiment, S3 specifically includes:

[0113] S31. Based on the trained world model, set the optimization objective of the water cycle algorithm. The optimization objective is to minimize the prediction error of the world model and optimize the parameters θ of the state transition layer and the parameters of the state encoder layer of the world model.

[0114] S32. Initialize the water cycle algorithm, set the water droplet population size P, and randomly generate P sets of candidate parameter combinations within the parameter search space. The parameters of the state transition layer and state encoder layer corresponding to each water droplet are represented respectively. The initial fitness for each parameter combination is calculated. The fitness is used to measure the performance of the current parameter combination on the optimization objective, and is obtained through prediction error. express;

[0115] S33. For each set of parameter combinations In the world model, forward prediction is performed, the corresponding prediction error is calculated, and the error function is defined:

[0116]

[0117] in, Represents the error function, o t This represents the standardized input dataset. This represents the reconstructed standardized data, and KL(·||·) represents the Kullback-Leibler divergence. The probability distribution representing the true potential state, and the ideal distribution describing the world model. μ represents the distribution of potential states predicted by the world model, describing the world model's current prediction of the environmental state. t and The mean and variance represent the true potential states. and The values ​​represent the mean and variance of the world model predictions, α1 and α2 represent the loss weights, and ||·|| 2 Represents the square norm;

[0118] S34. Execute the water droplet flow process, guide the search direction among the water droplets according to the fitness value, and update the parameter combination of each group of water droplets:

[0119]

[0120] in, and Let these represent the parameters of the state transition layer and the state encoder layer for the i-th water droplet in round t+1, respectively. and Let ξ1 and ξ2 represent the parameters of the state transition layer and state encoder layer for the i-th droplet in the t-th round, respectively. α represents the learning rate, β represents the perturbation factor, and ξ1 and ξ2 represent independent random perturbation terms. and This represents another set of droplet parameters used for reference in round t, which has better fitness than the i-th droplet.

[0121] S35. Calculate the prediction error for each updated parameter combination. If the convergence condition is met... Then the droplet parameters are marked as converged, where ∈ represents the convergence threshold;

[0122] S36. For non-converged water droplet combinations, execute the evaporation-precipitation mechanism of the water cycle algorithm, wherein the evaporation-precipitation mechanism resets the parameters of the M groups of water droplets with the worst fitness:

[0123]

[0124] in, and θ represents the new parameter combination for the k-th water droplet that has been reset. rand and denoted as the combination of water droplet parameters randomly generated again in the parameter search space, γ represents the perturbation intensity coefficient in the evaporation-precipitation process, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism;

[0125] S37. Repeat steps S34 to S36 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination, update the world model based on the optimized parameter combination, and recalculate the predicted values ​​of the operation status for future time steps.

[0126] This invention utilizes a water cycle algorithm to globally optimize the parameters of the state encoder layer and state transition layer in the world model. By adjusting the model parameters through water droplet flow and evaporation-precipitation mechanisms, it effectively avoids the problems of getting stuck in local optima or the difficulty in parameter convergence in traditional training, improves the prediction accuracy and robustness of the world model, and provides more reliable prediction support for subsequent policy optimization.

[0127] In this embodiment, S4 specifically includes:

[0128] S41. Construct a reinforcement learning module and define a policy network for optimizing the allocation of terminal operation resources. Value Network V ψ Initialize the parameters of the policy network and value network, set the input of the agent to the dock operation status, reward information and task termination information provided by the world model, and set the output of the agent to the allocation strategy of dock operation resources.

[0129] S42. Construct the state space, action space, and reward function for terminal operation resource allocation, and define the state vector. in This represents the potential state predicted by the world model at time t. Indicates the immediate reward for the prediction. Indicates the probability of prediction termination. ρ represents the change in the potential state. t This represents the Euclidean distance between the current potential state and the average potential state.

[0130] S43. Based on the world model, sample the resource allocation strategy for terminal operations and set the action strategy distribution:

[0131]

[0132] Among them, a t This indicates that in the state vector s t The action obtained by downsampling This represents the policy network, where Softmax represents the normalization operation, and W... π and b π Let σ represent the weight matrix and bias vector of the policy network, and σ represent the nonlinear activation function.

[0133] S44. Calculate the cumulative reward of the terminal operation resource allocation strategy and define the reward function:

[0134]

[0135] Among them, R t Let γ represent the reward function, describing the cumulative reward starting from time t, where T represents the total number of time steps and γ represents the discount factor. Let ρ represent the instantaneous reward predicted at time t+k in the future, λ represent the state offset penalty coefficient, and ρ represent the instantaneous reward predicted at time t+k. t+k This represents the state distance at time t+k.

[0136] S45. Calculate the advantage function and adjust the value deviation based on the execution of terminal operation resource allocation:

[0137]

[0138] Among them, A t Represents the dominance function. V represents the predicted immediate reward. ψ Represents the value network, s t and s t+1 These represent the current state and the next state, respectively, and γ represents the discount factor. a represents the motion disturbance coefficient. t Indicates the current action. This represents the average of historical actions. V represents the L2 norm between the current action and historical actions, k' represents the category, and V ψ (s t ) and V ψ (s t+1 ) indicates that in state s t Below, the value estimation of the state is based on the parameter ψ output by the value network;

[0139] S46. Optimize the policy network, calculate the policy loss function, optimize the value network, calculate the value error of resource allocation decisions, and define the value loss function:

[0140]

[0141] Among them, L π L represents the policy loss function. V Let E represent the value loss function, E represent the expected value, and D represent the experience pool. Let A represent the logarithmic probability of the current action. t Let κ represent the advantage function, and let κ represent the entropy regularization coefficient. V represents the entropy value of the policy network output distribution. ψ (s t R represents the value estimate of the current state. t Let δ represent the reward function, δ represent the value difference adjustment coefficient, and ΔV represent the value difference adjustment coefficient. t This represents the difference between the predicted values ​​of adjacent states;

[0142] S47. Perform parameter optimization by adjusting the parameters of the policy network and value network based on gradient descent.

[0143] S48. Output the optimized terminal operation resource allocation strategy, store the optimized strategy network parameters and value network parameters, apply the optimized resource allocation strategy to the terminal operation scheduling system, and adjust the resource allocation scheme in the actual operation process.

[0144] This invention constructs a reinforcement learning agent based on an optimized world model. By setting a multi-dimensional state space and a composite reward function, and designing cumulative reward and advantage functions, it guides the policy network and value network to efficiently optimize resource allocation strategies in the potential space. It has the advantages of fast policy learning speed, high action output stability, compatibility with multi-objective tasks, and adaptability to complex scheduling scenarios.

[0145] In this embodiment, S5 specifically includes:

[0146] S51. Initialize the reinforcement learning module optimization process for the water cycle algorithm, and optimize the policy network parameters in the reinforcement learning module. Using the value network parameter ψ as a joint optimization variable, and setting the droplet population size to G, G individual droplets are randomly generated in the parameter space.

[0147] S52, Based on the current individual water droplet Perform terminal operation resource allocation tasks within the optimized world model, utilizing a policy network. With Value Network V ψ Conduct simulated interactions to obtain resource allocation strategies and generate experience trajectories D.i ={(s t ,a t ,R t ,s t+1 )};

[0148] S53. Calculate the fitness function for each water droplet based on the empirical trajectory:

[0149]

[0150] in, This represents the fitness value of the i-th water droplet. L represents the policy loss corresponding to the policy network. V (ψ i ) represents the value loss corresponding to the value network, and δ represents a positive constant to prevent division by zero overflow;

[0151] S54. Set the update rules for water droplets in the parameter space, execute the water droplet flow process, and select water droplet pairs based on relative fitness. and

[0152]

[0153] in, and Let these represent the policy network parameters and value network parameters of the i-th droplet in round t+1, respectively. and Let ᵢ represent the policy network parameters and value network parameters of the i-th droplet in round t, respectively, and let ι represent the learning rate. This represents the disturbance factor. and Let ξ1 and ξ2 represent another set of droplet parameters used for reference in round t, where ξ1 and ξ2 represent random perturbation terms under uniform distribution.

[0154] S55. After updating all water droplets, re-execute the resource allocation simulation in the world model, generate policy trajectories based on the new parameters, and recalculate the fitness function. Determine whether the convergence condition is met;

[0155] S56. If the convergence condition is not met, the evaporation-precipitation mechanism of the water cycle algorithm is executed, wherein the evaporation-precipitation mechanism randomly resets the M water droplets with the lowest fitness ranking:

[0156]

[0157] in, and Let represent the policy network parameters and value network parameters of the k-th water droplet that has been reset, and η represent the perturbation intensity coefficient in the evaporation-precipitation mechanism. and ψ rand ξ1′ and ξ2′ represent the newly generated random parameters, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism;

[0158] S57. Repeat steps S54 to S56 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination of the reinforcement learning module and replace the original parameters to optimize the dock operation resource allocation scheme.

[0159] This invention uses a water cycle algorithm to collaboratively optimize the parameters of the policy network and value network in the reinforcement learning module. It dynamically adjusts the network weights by combining water droplet flow and evaporation-precipitation mechanisms, avoiding the problem of traditional gradient descent algorithms getting trapped in local optima during training. This improves the global search capability of policy learning and the adaptability of scheduling strategies, and enhances the generalization performance of the model in uncertain environments.

[0160] Example 1:

[0161] To verify the feasibility of this invention in practice, it was applied to the operation scheduling system of a large automated container terminal on the southeast coast. The terminal has a total shoreline length exceeding 2000 meters, an annual designed throughput of 6.5 million TEUs, and is equipped with 32 quay cranes, 68 yard cranes, and approximately 300 unmanned container trucks. It handles an average of over 25 ships per day, with peak hourly container loading and unloading capacity reaching 5000 TEUs. The operations are intensive, dynamically changing, and the scheduling tasks are complex and frequent. Traditional scheduling systems suffer from significant response delays, severe local resource conflicts, and unstable scheduling results in dealing with high-frequency equipment status changes and sudden operational adjustments.

[0162] In this scenario, the dynamic allocation method for terminal operation resources based on a multi-objective optimization algorithm proposed in this invention is deployed in the terminal operation scheduling platform, replacing the original rule-driven scheduling module. In the early stage of deployment, the system collects the terminal's operation data for the past three months, including ship arrival time, planned operation tasks, equipment availability, operation progress, and environmental status. A standardized dataset is constructed and used to train the world model in the Dreamer reinforcement learning model. Through the state encoder layer, state transition layer, reward predictor layer, and termination predictor layer, the state changes of the operation environment are modeled, and the operation state sequence and resource demand change trend in the future time period are generated by using historical data.

[0163] After completing the world model training, the system constructs a reinforcement learning module agent. The state space is set to include indicators such as potential job states, equipment load distribution, and yard density changes. The action space is the set of equipment allocation decisions for different job areas. The reward function introduces multi-objective parameters such as average job completion time, equipment energy consumption, and yard load balance. To ensure that the model has good adaptability and generalization ability among different equipment groups, a water cycle algorithm is further introduced to optimize the policy network and value network parameters in the agent and iteratively search for the optimal resource allocation strategy.

[0164] In actual operation, the system collects the latest job status data in real time and adjusts resource allocation in 15-minute time windows. When the deviation between the scheduling result and the actual job exceeds the preset threshold, the evaporation-precipitation mechanism of the water cycle algorithm is triggered to quickly fine-tune the current strategy and realize the dynamic correction of job scheduling. After the system has been deployed and running continuously for 30 days, it has been compared and evaluated with the original scheduling system in terms of job efficiency, resource utilization, response speed and other dimensions.

[0165] Table 1 Comparison of Terminal Operation Resource Allocation Performance

[0166]

[0167]

[0168] In terms of work efficiency, this invention optimizes the average work completion time from 42.6 minutes in the original system to 35.2 minutes, an improvement of 17.3%. This effect is due to the fact that this invention introduces the Dreamer reinforcement learning model to accurately predict the future work status, and on this basis generates a more reasonable resource scheduling strategy, avoiding problems such as lagging scheduling rules and redundant resource configuration in traditional methods, and improving the continuity and coordination of work flow.

[0169] In terms of resource utilization, this invention improves the utilization efficiency of various key equipment. The utilization rate of quay cranes increased from 71.5% to 83.9%, yard cranes from 68.1% to 79.3%, and AGV vehicle utilization rate from 64.4% to 76.7%. These data show that this invention can achieve better resource matching based on multi-objective coordination, enabling more effective allocation of equipment in both spatial and temporal dimensions, thereby reducing resource idleness and conflicts, and optimizing the overall scheduling flow.

[0170] In terms of scheduling response performance, the average response latency of the system of this invention is significantly reduced from the original 8.3 seconds to 3.1 seconds, and the response speed is improved by 62.7%. This is due to the water cycle algorithm introduced in this method to optimize the parameters of the policy network and value network, which enables the scheduling system to have a stronger real-time adjustment capability. In addition, through dynamic deviation detection and evaporation-precipitation mechanism, the system can correct the scheduling deviation in time when it occurs, further reducing the response lag.

[0171] In terms of system robustness, the present invention achieves a strategy stability rate of 99.1% under abnormal weather conditions, which is 3.7% higher than the original system's 95.4%. At the same time, in terms of handling emergencies, the scheduling failure rate is significantly reduced from 2.4% to 0.6%, reducing scheduling errors by 75%. This fully demonstrates that the present invention effectively enhances the system's adaptability to external disturbances and has higher reliability and fault tolerance through the environmental modeling capability of the world model and the continuous learning mechanism of the agent.

[0172] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm, characterized in that, Includes the following steps: S1. Collect dock operation data, preprocess the collected data, and generate a standardized dataset; S2. Construct the Dreamer reinforcement learning model, train the world model of the Dreamer reinforcement learning model based on the standardized dataset, and predict the port operation status through the world model. The Dreamer reinforcement learning model includes a world model and a reinforcement learning module. The world model includes a state encoder layer, a state transition layer, a reconstruction decoder layer, a reward predictor layer, and a termination predictor layer. The reinforcement learning module includes a policy network and a value network. The world model predicts the terminal operation status, reward information, and task termination information based on terminal operation data. The policy network is used to optimize resource allocation schemes in the world model, and the value network is used to evaluate the merits of different resource allocation schemes. S3. Optimize the prediction process of the world model using the water cycle algorithm. Set water droplets to represent different combinations of training parameters, calculate the prediction error of the world model, adjust the parameters of the world model based on the mechanism of water droplet flow to the optimal solution, and generate optimized prediction data. S4. Based on the predicted data, set the state space, action space and reward function of the reinforcement learning module agent, train the reinforcement learning module agent, learn the terminal operation resource allocation strategy, and generate a resource allocation scheme. S5. Optimize the training process of the reinforcement learning module agent using the water cycle algorithm. Set water droplets to represent different combinations of reinforcement learning module parameters, calculate the fitness of the resource allocation scheme, and adjust the reinforcement learning module parameters based on the water droplet flow process to optimize the dock operation resource allocation scheme. S6. Collect real-time dock operation data, calculate the deviation of dock operation resource allocation, and if the resource allocation deviation exceeds the set threshold, adjust the dock operation resource allocation scheme based on the evaporation-precipitation mechanism of the water cycle algorithm. S7. Optimize the Dreamer reinforcement learning model based on historical terminal operation data, use incremental learning to update the parameters of the Dreamer reinforcement learning model, and adjust the terminal operation resource allocation scheme.

2. The method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that, The terminal operation data includes vessel arrival time, cargo loading and unloading schedule, equipment availability, operation progress, and environmental status. The preprocessing includes format conversion, outlier detection, time synchronization, and normalization.

3. The method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that, S2 specifically includes: S21. Construct the Dreamer reinforcement learning model; S22. Train a world model based on a standardized dataset, construct a state encoder layer, and process the input standardized dataset... Feature extraction is performed, and a variational autoencoder is used to map the standardized dataset into latent states. : ； in, Indicates the current potential state. and This represents the weight matrix of the state encoder layer. Represents the feature extraction vector. and This represents the bias term of the state encoder layer. Represents a non-linear activation function. This represents a noise variable that follows a standard normal distribution. This represents a standard normal distribution with a mean of 0 and a variance of 1. This represents the standardized dataset of the input; S23. Construct a state transition layer and use gated loop units to recursively model historical latent states and actions to obtain the current hidden state. And generate predicted potential states Prior distribution: ； in, Indicates the current hidden state. Indicates a gated loop unit. This indicates the hidden state of the previous time step. This represents the potential state at the previous time step. Indicates the action at the previous time step. Indicates the predicted potential state, Indicates a normal distribution. and This represents the weight matrix of the state transition layer. and This indicates the bias of the state transition layer. Represents a non-linear activation function; S24. Construct the reconstruction decoder layer to predict the latent states. Decoded into a reconstructed standardized dataset The reconstruction decoder layer uses a multilayer perceptron for decoding; S25. Construct a reward predictor layer to predict the immediate reward under the current potential state and action. The reward predictor layer uses a fully connected network; S26. Construct a termination predictor layer to determine whether the current potential state is a termination state and output the termination flag of the prediction. The termination predictor layer uses a fully connected network combined with a sigmoid activation function; S27. Jointly train all modules in the world model, construct a comprehensive loss function, and minimize the reconstruction prediction error, reward prediction error, termination prediction error, and KL divergence: ； in, Represents the comprehensive loss function. , , and This represents the weighting coefficient of each loss term. This represents the standardized input dataset. Standardized data representing reconstruction, Denotes the square norm. Indicates the predicted potential state, This indicates a real reward. This represents the predicted current potential state and the immediate reward under the given action. Indicates the termination probability of the prediction, and describes the termination marker of the prediction. Indicates the true termination probability. Indicates the Kullback-Leibler divergence. The probability distribution representing the true potential state, and the ideal distribution describing the world model. This represents the distribution of potential states predicted by the world model, describing the world model's current prediction of the environmental state. and The mean and variance represent the true potential states. and This represents the mean and variance of the world model's predictions; S28. Based on the trained world model, input the current task data to generate a sequence of potential states for future time steps. Predicting reward sequences and termination probability sequence .

4. The method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that, S3 specifically includes: S31. Based on the trained world model, set the optimization objective of the water cycle algorithm. The optimization objective is to minimize the prediction error of the world model and optimize the parameters of the state transition layer of the world model. and parameters of the state encoder layer ; S32. Initialize the water cycle algorithm and set the water droplet population size. Randomly generated within the parameter search space Group of candidate parameter combinations , where represent the parameters of the state transition layer and the state encoder layer corresponding to each water droplet, respectively. The initial fitness corresponding to each set of parameters is calculated. The fitness is used to measure the performance of the current parameter combination on the optimization objective and is represented by the prediction error. S33. For each set of parameter combinations In the world model, forward prediction is performed, the corresponding prediction error is calculated, and the error function is defined: ； in, Represents the error function. This represents the standardized input dataset. Standardized data representing reconstruction, Indicates the Kullback-Leibler divergence. The probability distribution representing the true potential state, and the ideal distribution describing the world model. This represents the distribution of potential states predicted by the world model, describing the world model's current prediction of the environmental state. and Represents the mean and variance of the true potential states. and This represents the mean and variance of the world model's predictions. and Indicates the loss weight. Represents the square norm; S34. Execute the water droplet flow process, guide the search direction among the water droplets according to the fitness value, and update the parameter combination of each group of water droplets: ； in, and They represent the first Wheel of Life The parameters of the state transition layer and state encoder layer of each water droplet and They represent the first Wheel of Life The parameters of the state transition layer and state encoder layer of each water droplet Indicates the learning rate. This represents the disturbance factor. and Represents independent random disturbance terms. and Indicates the first Another set of droplet parameters used for reference in the wheel has better fitness than the first set. A water droplet; S35. Calculate the prediction error of each updated parameter combination. If the absolute value of the difference between the prediction errors of adjacent rounds is less than the convergence threshold, then mark the droplet parameters as converged. S36. For non-converged water droplet combinations, execute the evaporation-precipitation mechanism of the water cycle algorithm, wherein the evaporation-precipitation mechanism is applied to the worst-fitting combination. Reset the group of water droplet parameters: ； in, and Indicates the number that has been reset A new combination of parameters for each water droplet and This represents a combination of water droplet parameters that are randomly generated again in the parameter search space. This represents the disturbance intensity coefficient during the evaporation-precipitation process. and This represents the independent random perturbation term introduced in the evaporation-precipitation mechanism; S37. Repeat steps S34 to S36 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination, update the world model based on the optimized parameter combination, and recalculate the predicted values ​​of the operation status for future time steps.

5. The method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that, S4 specifically includes: S41. Construct a reinforcement learning module and define a policy network for optimizing the allocation of terminal operation resources. and value network Initialize the parameters of the policy network and value network, set the input of the agent to the dock operation status, reward information and task termination information provided by the world model, and set the output of the agent to the allocation strategy of dock operation resources. S42. Construct the state space, action space, and reward function for terminal operation resource allocation, and define the state vector. ,in The world model predicts the first Potential state at all times Indicates the immediate reward for the prediction. Indicates the probability of prediction termination. This represents the change in the potential state. This represents the Euclidean distance between the current potential state and the average potential state. S43. Based on the world model, sample the resource allocation strategy for terminal operations and set the action strategy distribution: ； in, In the state vector The action obtained by downsampling Represents the policy network, This indicates a normalization operation. and This represents the weight matrix and bias vector of the policy network. Represents a non-linear activation function; S44. Calculate the cumulative reward of the terminal operation resource allocation strategy and define the reward function: ； in, Represents the reward function, describing the reward from the first... Accumulated rewards starting from a certain moment. Indicates the total number of time steps. Indicates the discount factor. Indicates the future number Instant rewards for predicting the moment. This represents the state offset penalty coefficient. Indicates the first Distance between states at any given moment; S45. Calculate the advantage function and adjust the value deviation based on the execution of terminal operation resource allocation: ； in, Represents the dominance function. Indicates the immediate reward for the prediction. Represents a value network. and These represent the current state and the next state, respectively. Indicates the discount factor. This represents the motion disturbance coefficient. Indicates the current action. This represents the average of historical actions. This represents the L2 norm between the current action and historical actions. Indicates category, and Indicates the state Below, the value network is based on parameters Value estimation of the output state; S46. Optimize the policy network, calculate the policy loss function, optimize the value network, and calculate the value error of resource allocation decisions. Define the value loss function: ； ； in, Represents the policy loss function. Represents the value loss function, Represents the mathematical expectation. Indicates the experience pool. This represents the logarithmic probability of the current action. Represents the dominance function. Represents the entropy regularization coefficient. The entropy value represents the output distribution of the policy network. This represents the value estimate of the current state. Represents the reward function, This represents the value difference adjustment coefficient. This represents the difference between the predicted values ​​of adjacent states; S47. Perform parameter optimization by adjusting the parameters of the policy network and value network based on gradient descent. S48. Output the optimized terminal operation resource allocation strategy, store the optimized strategy network parameters and value network parameters, apply the optimized resource allocation strategy to the terminal operation scheduling system, and adjust the resource allocation scheme in the actual operation process.

6. A method for dynamic allocation of terminal operation resources based on a multi-objective optimization algorithm according to claim 5, characterized in that, S5 specifically includes: S51. Initialize the reinforcement learning module optimization process for the water cycle algorithm, and optimize the policy network parameters in the reinforcement learning module. and value network parameters As a joint optimization variable, the water droplet population size is set to... Randomly generated in the parameter space Individual water droplets ; S52, Based on the current individual water droplet In the optimized world model, the task of allocating port operation resources is performed using a policy network. With value network Conduct simulated interactions to obtain resource allocation strategies and generate experience trajectories. ; S53. Calculate the fitness function for each water droplet based on the empirical trajectory: ； in, Indicates the first The fitness value of each water droplet. This represents the policy loss corresponding to the policy network. This represents the value loss corresponding to the value network. Represents a positive constant to prevent overflow when divided by zero; S54. Set the update rules for water droplets in the parameter space, execute the water droplet flow process, and select water droplet pairs based on relative fitness. and : ； in, and They represent the first Wheel of Life The policy network parameters and value network parameters of each water droplet. and They represent the first Wheel of Life The policy network parameters and value network parameters of each water droplet. Indicates the learning rate. This represents the disturbance factor. and Indicates the first Another set of droplet parameters used for reference in the wheel, and This represents the random disturbance term under a uniform distribution; S55. After updating all water droplets, re-execute the resource allocation simulation in the world model, generate policy trajectories based on the new parameters, and recalculate the fitness function. Determine whether the convergence condition is met; S56. If the convergence condition is not met, the evaporation-precipitation mechanism of the water cycle algorithm is executed, wherein the evaporation-precipitation mechanism randomly resets the M water droplets with the lowest fitness ranking: ； in, and Indicates the number that has been reset The policy network parameters and value network parameters of each water droplet. This represents the disturbance intensity coefficient in the evaporation-precipitation mechanism. and This represents the regenerated random parameters. and This represents the independent random perturbation term introduced in the evaporation-precipitation mechanism; S57. Repeat steps S54 to S56 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination of the reinforcement learning module and replace the original parameters to optimize the dock operation resource allocation scheme.

Images