Wharf operation resource dynamic allocation method based on multi-objective optimization algorithm

Abstract

The invention discloses a multi-objective optimization algorithm-based wharf operation resource dynamic allocation method, which comprises the following steps of S1, acquiring wharf operation data, and preprocessing the acquired data; s2, constructing a Dreamer reinforcement learning model, and predicting a wharf operation state through a world model of the Dreamer reinforcement learning model; s3, optimizing the prediction process of the world model by using a water circulation algorithm; s4, training a reinforcement learning module agent, and learning a wharf operation resource allocation strategy; s5, using a water circulation algorithm to optimize the training process of the reinforcement learning module agent; s6, calculating a wharf operation resource allocation deviation and a wharf operation resource allocation scheme; and S7, updating parameters of the Dreamer reinforcement learning model by adopting incremental learning, and adjusting a wharf operation resource allocation scheme. According to the method, the Dreamer reinforcement learning model and the water circulation algorithm are fused, dynamic optimization allocation of the wharf operation resources is achieved, and the method has the advantages of being fast in response, high in adaptability and high in multi-target coordination capacity.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent scheduling and resource optimization, and particularly to a dynamic allocation method for terminal operation resources based on a multi-objective optimization algorithm. Background Art

[0002] With the rapid development of global port operations and the increasing complexity of the international logistics chain, the terminal operation system is facing unprecedented pressure for efficient operation. As the core node of port logistics, the resource scheduling efficiency of the terminal directly affects the entire port operation capacity, the smoothness of the logistics channel, and the port throughput. Traditional methods for allocating terminal operation resources generally rely on empirical rules or heuristic algorithms. Although they have certain practicality in specific scenarios, in the face of a dynamically changing operation environment, complex and variable operation tasks, and multi-objective coordination requirements, their robustness, real-time performance, and adaptability often have significant limitations and are difficult to meet the high-standard management requirements of current intelligent ports and automated terminals.

[0003] Current mainstream research on terminal resource scheduling mainly focuses on subtasks such as optimizing the operation sequence of equipment such as quay cranes, yard cranes, and trailers, berth allocation, and yard scheduling. Traditional methods widely used in related research include integer programming, linear programming, genetic algorithms, ant colony algorithms, and particle swarm optimization. Although these methods have achieved certain results in static or semi-dynamic scenarios, when dealing with the highly dynamic and strongly uncertain actual working conditions in the terminal system, they often face problems such as difficult parameter adjustment, being easily trapped in local optima, and low computational efficiency, and are particularly difficult to respond to complex events such as equipment failures, ship schedule adjustments, and changes in task priorities in real time.

[0004] In recent years, with the development of artificial intelligence and reinforcement learning, some research has attempted to introduce reinforcement learning models such as Q-learning, Deep Q Network, and Actor-Critic into terminal operation scheduling to learn the optimal scheduling strategy from the environment. However, there are still several bottlenecks in the application of existing reinforcement learning methods in port operation scenarios: First, the port operation state is complex and high-dimensional, and it is difficult for the reinforcement learning model to accurately model the long-term state transition law, resulting in low sample efficiency in the policy learning process; Second, real-world terminal scheduling often involves multiple objectives (such as equipment utilization rate, energy consumption balance, operation completion time, operation fairness, etc.), while most reinforcement learning methods are more suitable for single-objective optimization and lack the ability to coordinate the conflict relationships between multiple objectives; Third, the reinforcement learning agent usually relies on a large amount of interaction data during the training process, and the actual port system does not have enough safety space for a large number of trial-and-error learning, which limits the deployability of the model in the real system.

[0005] In terms of optimization algorithms, although traditional swarm intelligence optimization methods (such as genetic algorithms and particle swarm algorithms) have been used in some scheduling optimization tasks, these algorithms generally have problems such as slow convergence speed and being easily trapped in local optima, and most of them are static structures, lacking a linkage mechanism with deep learning models. As a new type of global optimization algorithm proposed in recent years, the water cycle algorithm is inspired by the principle of the natural water cycle, introducing dynamic processes such as evaporation, precipitation, and flow, and has advantages such as strong global search ability, good convergence stability, and simple parameter control. However, there is no standard method for its joint optimization with reinforcement learning agents, especially the cooperation mechanism in the core links of deep reinforcement learning such as world model parameter optimization and policy network update needs to be further studied.

[0006] In summary, there is still much room for improvement in the existing quay operation resource allocation methods in terms of multi-objective coordination, dynamic adaptation, and intelligent optimization. In particular, there is a lack of a systematic solution that can integrate world model prediction, reinforcement learning policy generation, and swarm intelligence optimization algorithm for collaborative optimization. The existing technologies cannot effectively solve the problem of how to generate accurate, efficient, and stable resource scheduling strategies based on historical and real-time data in complex, dynamic, and multi-constrained quay operation scenarios, and it is difficult to support the high requirements of future intelligent quays for a "dynamic, real-time, and adaptive" resource scheduling system.

[0007] Therefore, there is an urgent need to propose a dynamic allocation method for quay operation resources based on a multi-objective optimization algorithm, which systematically integrates the world model prediction ability, reinforcement learning policy training ability, and water cycle optimization mechanism to achieve efficient, stable, and adaptive control of complex resource scheduling systems in multi-objective scenarios. Summary of the Invention

[0008] An object of the present invention is to propose a dynamic allocation method for quay operation resources based on a multi-objective optimization algorithm. The present invention integrates the Dreamer reinforcement learning model and the water cycle algorithm to accurately predict the quay operation state and dynamically optimize the resource scheduling strategy, achieving efficient and stable multi-objective resource allocation, with both policy self-learning ability and multi-objective collaborative optimization ability, and being able to adapt to complex and changeable quay operation environments, improving the real-time performance, intelligence, and overall operation efficiency of scheduling.

[0009] A dynamic allocation method for quay operation resources based on a multi-objective optimization algorithm according to an embodiment of the present invention includes the following steps:

[0010] S1. Collect quay operation data, preprocess the collected data, and generate a standardized data set;

[0011] S2. Build a Dreamer reinforcement learning model, train the world model of the Dreamer reinforcement learning model based on the standardized dataset, and predict the terminal operation status through the world model;

[0012] S3. Use the water cycle algorithm to optimize the prediction process of the world model, set water droplets to represent different combinations of training parameters, calculate the prediction error of the world model, adjust the parameters of the world model based on the mechanism of water droplets flowing towards the optimal solution, and generate optimized prediction data;

[0013] S4. Based on the prediction data, set the state space, action space, and reward function of the intelligent agent in the reinforcement learning module, train the intelligent agent in the reinforcement learning module, learn the terminal operation resource allocation strategy, and generate a resource allocation plan;

[0014] S5. Use the water cycle algorithm to optimize the training process of the intelligent agent in the reinforcement learning module, set water droplets to represent different combinations of parameters of the reinforcement learning module, calculate the fitness of the resource allocation plan, and adjust the parameters of the reinforcement learning module based on the water droplet flow process to optimize the terminal operation resource allocation plan;

[0015] S6. Collect real-time terminal operation data, calculate the deviation of terminal operation resource allocation. If the resource allocation deviation exceeds the set threshold, adjust the terminal operation resource allocation plan based on the evaporation-precipitation mechanism of the water cycle algorithm;

[0016] S7. Optimize the Dreamer reinforcement learning model based on historical terminal operation data, update the parameters of the Dreamer reinforcement learning model using incremental learning, and adjust the terminal operation resource allocation plan.

[0017] Optionally, the terminal operation data includes the ship arrival time, cargo handling plan, equipment availability, operation progress, and environmental status, and the preprocessing includes format conversion, outlier detection, time synchronization, and normalization processing.

[0018] Optionally, the specific steps of S2 are as follows:

[0019] S21. Build a Dreamer reinforcement learning model. The Dreamer reinforcement learning model includes a world model and a reinforcement learning module. The world model includes a state encoder layer, a state transition layer, a reconstruction decoder layer, a reward predictor layer, and a termination predictor layer. The reinforcement learning module includes a policy network and a value network. The world model predicts the terminal operation status, reward information, and task termination information based on the terminal operation data. The policy network is used to optimize the resource allocation plan in the world model, and the value network is used to evaluate the advantages and disadvantages of different resource allocation plans;

[0020] S22. Train the world model based on the standardized dataset, build a state encoder layer, and encode the input standardized dataset ot Feature extraction is performed, and a variational autoencoder is used to map the standardized dataset to the latent state z t :

[0021] z t = W μ ·φ(o t ) + b μ + softplus(W σ ·φ(o t ) + b σ )·∈, ∈~N(0,1);

[0022] Among them, z t represents the current latent state, W μ and W σ represent the weight matrices of the state encoder layer, φ(o t ) represents the feature extraction vector, b μ and b σ represent the bias terms of the state encoder layer, softplus represents the non-linear activation function, ∈ represents the noise variable subject to the standard normal distribution, N represents the standard normal distribution with a mean of 0 and a variance of 1, and o t represents the input standardized dataset;

[0023] S23. Construct a state transition layer, and use a gated recurrent unit to recursively model the historical latent state and action to obtain the current hidden state h t and generate the prior distribution of the predicted latent state :

[0024]

[0025] Among them, h t represents the current hidden state, GRU represents the gated recurrent unit, h t-1 represents the hidden state at the previous time step, z t-1 represents the latent state at the previous time step, a t-1 represents the action at the previous time step, represents the predicted latent state, N represents the normal distribution, and represent the weight matrices of the state transition layer, and represent the biases of the state transition layer, and softplus represents the non-linear activation function;

[0026] S24. Construct a reconstruction decoder layer to decode the predicted latent state into the reconstructed standardized dataset The reconstruction decoder layer uses a multi-layer perceptron for decoding;

[0027] S25. Construct a reward predictor layer to predict the immediate reward under the current latent state and action The reward predictor layer adopts a fully connected network;

[0028] S26. Construct a termination predictor layer to determine whether the current latent state is a termination state and output the predicted termination flag The termination predictor layer adopts a fully connected network combined with a sigmoid activation function;

[0029] S27. Jointly train all modules in the world model, construct a comprehensive loss function, and minimize the reconstruction prediction error, reward prediction error, termination prediction error, and KL divergence:

[0030]

[0031] where L represents the comprehensive loss function, λ1, λ2, λ3, and λ4 represent the weight coefficients of each loss term, o t represents the input standardized data set, represents the reconstructed standardized data, ||·|| 2 represents the squared norm, represents the predicted latent state, r t represents the true reward, represents the immediate reward under the predicted current latent state and action, represents the predicted termination probability, describing the predicted termination flag, d t represents the true termination probability, KL(·||·) represents the Kullback-Leibler divergence, represents the probability distribution of the true latent state, describing the ideal distribution of the world model, represents the latent state distribution predicted by the world model, describing the current prediction of the world model for the environmental state, μ t and represent the mean and variance of the true latent state, and represent the mean and variance predicted by the world model;

[0032] S28. Based on the trained world model, input the current job data to generate a sequence of latent states at future time steps Predict the reward sequence and the termination probability sequence

[0033] Optionally, the specific content of S3 includes:

[0034] S31. Based on the trained world model, set the optimization objective of the water cycle algorithm. The optimization objective is to minimize the prediction error of the world model, and optimize the parameters θ of the state transition layer and the parameters of the state encoder layer of the world model.

[0035] S32. Initialize the water cycle algorithm, set the size P of the water droplet population, and randomly generate P groups of candidate parameter combinations within the parameter search space. respectively represent the parameters of the state transition layer and the state encoder layer corresponding to each water droplet, calculate the initial fitness corresponding to each group of parameter combinations. The fitness is used to measure the performance of the current parameter combination in terms of the optimization objective, and is represented by the prediction error. Indicated;

[0036] S33. For each group of parameter combinations Perform forward prediction in the world model, calculate the corresponding prediction error, and define the error function:

[0037]

[0038] Among them, represents the error function, o t represents the input standardized data set, represents the reconstructed standardized data, KL(·||·) represents the Kullback-Leibler divergence, represents the probability distribution of the true latent state, describing the ideal distribution of the world model, represents the predicted latent state distribution of the world model, describing the current prediction of the world model for the environmental state, μ t and represent the mean and variance of the true latent state, and represent the mean and variance predicted by the world model, α1 and α2 represent the loss weights, ||·|| 2 represents the squared norm;

[0039] S34. Execute the water droplet flow process, guide the search direction among the water droplets according to the fitness size, and update the parameter combinations of each group of water droplets:

[0040]

[0041] Among them, and respectively represent the parameters of the state transition layer and the state encoder layer of the i-th water droplet in the (t + 1)-th round, and respectively represent the parameters of the state transition layer and the state encoder layer of the i-th water droplet in the t-th round, α represents the learning rate, β represents the perturbation factor, and ξ1 and ξ2 represent independent random perturbation terms. and represents another set of water droplet parameters used for reference in the t-th round, with a fitness superior to that of the i-th water droplet;

[0042] S35. Calculate the prediction error of each updated parameter combination. If the convergence condition is met then mark that the water droplet parameters have converged, where ∈ represents the convergence threshold;

[0043] S36. For the non-converged water droplet combinations, execute the evaporation-precipitation mechanism of the water cycle algorithm. The evaporation-precipitation mechanism resets the M water droplet parameters with the worst fitness:

[0044]

[0045] where and represent the new parameter combination of the k-th water droplet to be reset, and θ rand and represent the water droplet parameter combination randomly generated again in the parameter search space. γ represents the perturbation intensity coefficient in the evaporation-precipitation process, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism;

[0046] S37. Repeat steps S34 to S36 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination, update the world model based on the optimized parameter combination, and recalculate the predicted values of the job status at future time steps.

[0047] Optionally, the specific steps of S4 include:

[0048] S41. Construct a reinforcement learning module and define a policy network for optimizing the allocation of dock operation resources and a value network V ψ , initialize the parameters of the policy network and the value network, set the input of the intelligent agent to the dock operation status, reward information, and task termination information provided by the world model, and set the output of the intelligent agent to the allocation policy of dock operation resources;

[0049] S42. Construct the state space, action space, and reward function for the allocation of dock operation resources, and set the state vector where represents the potential state predicted by the world model at the t-th moment, represents the predicted immediate reward, represents the predicted termination probability, represents the change in the potential state, and ρ t represents the Euclidean distance between the current potential state and the average potential state;

[0050] S43. Sample the quay operation resource allocation strategy based on the world model and set the action policy distribution:

[0051]

[0052] where a t represents the action sampled under the state vector s t , represents the policy network, Softmax represents the normalization operation, W π and b π represent the weight matrix and bias vector of the policy network respectively, and σ represents the non-linear activation function;

[0053] S44. Calculate the cumulative reward of the quay operation resource allocation strategy and define the reward function:

[0054]

[0055] where R t represents the reward function, describing the cumulative reward starting from the t-th moment, T represents the total number of time steps, γ represents the discount factor, represents the predicted immediate reward at the future (t + k)-th moment, λ represents the state deviation penalty coefficient, and ρ t+k represents the state distance at the (t + k)-th moment;

[0056] S45. Calculate the advantage function and adjust the value deviation based on the execution of the quay operation resource allocation:

[0057]

[0058] where A t represents the advantage function, represents the predicted immediate reward, V ψ represents the value network, s t and s t+1 represent the current state and the next state respectively, γ represents the discount factor, represents the action perturbation coefficient, a t represents the current action, represents the historical action mean, represents the Euclidean norm between the current action and the historical action, k' represents the category, and V ψ (s t ) and V ψ (s t+1 ) represent the value estimates of the state output by the value network based on the parameter ψ under the state s t ;

[0059] S46. Optimize the policy network, calculate the policy loss function, optimize the value network, and calculate the value error of the resource allocation decision. Set the value loss function:

[0060]

[0061] Among them, L π represents the policy loss function, L V represents the value loss function, E represents the mathematical expectation, D represents the experience pool, represents the logarithm probability of the current action, A t represents the advantage function, κ represents the entropy regularization coefficient, represents the entropy value of the output distribution of the policy network, V ψ (s t ) represents the value estimation of the current state, R t represents the reward function, δ represents the value difference adjustment coefficient, ΔV t represents the difference between adjacent state value predictions;

[0062] S47. Perform parameter optimization, and adjust the parameters of the policy network and the value network based on the gradient descent method;

[0063] S48. Output the optimized quay operation resource allocation strategy, store the optimized policy network parameters and value network parameters, and apply the optimized resource allocation strategy to the quay operation scheduling system to adjust the resource allocation plan in the actual operation process.

[0064] Optionally, the S5 specifically includes:

[0065] S51. Initialize the optimization process of the reinforcement learning module of the water cycle algorithm, take the policy network parameters and the value network parameters ψ as the joint optimization variables, set the size of the water droplet population to G, and randomly generate G water droplet individuals in the parameter space

[0066] S52. Based on the current water droplet individual perform the quay operation resource allocation task in the optimized world model, and use the policy network to perform simulated interaction with the value network V ψ to obtain the resource allocation strategy and generate the experience trajectory D i ={(s t ,a t ,R t ,s t+1 )};

[0067] S53. Calculate the fitness function of each water droplet according to the experience trajectory:

[0068]

[0069] Among them, represents the fitness value of the \(i\)-th water droplet, represents the policy loss corresponding to the policy network, \(L\) V (\(\psi\) i ) represents the value loss corresponding to the value network, and \(\delta\) represents a positive constant to prevent division by zero overflow;

[0070] S54. Set the update rule of the water droplets in the parameter space, execute the water droplet flow process, and select water droplet pairs according to the relative fitness and :

[0071]

[0072] Among them, and respectively represent the policy network parameters and value network parameters of the \(i\)-th water droplet in the \((t + 1)\)-th round, and respectively represent the policy network parameters and value network parameters of the \(i\)-th water droplet in the \(t\)-th round, \(\iota\) represents the learning rate, represents the perturbation factor, and represent another set of water droplet parameters for reference in the \(t\)-th round, and \(\xi_1\) and \(\xi_2\) represent random perturbation terms under the uniform distribution;

[0073] S55. After updating all the water droplets, re - execute the resource allocation simulation in the world model, generate a policy trajectory based on the new parameters, and recalculate the fitness function Judge whether the convergence condition is satisfied;

[0074] S56. If the convergence condition is not satisfied, execute the evaporation - precipitation mechanism of the water cycle algorithm, and the evaporation - precipitation mechanism randomly resets the \(M\) water droplet individuals with the lowest fitness rankings:

[0075]

[0076] Among them, and represent the policy network parameters and value network parameters of the \(k\)-th water droplet to be reset, \(\eta\) represents the perturbation intensity coefficient in the evaporation - precipitation mechanism, and \(\psi\) rand represent the newly generated random parameters, and \(\xi_1'\) and \(\xi_2'\) represent the independent random perturbation terms introduced in the evaporation - precipitation mechanism;

[0077] Repeat steps S54 to S56 until all water droplets meet the convergence condition or reach the maximum number of iteration rounds. Finally, output the optimal parameter combination of the reinforcement learning module, replace the original parameters, and optimize the quay operation resource allocation plan.

[0078] The beneficial effects of the present invention are as follows:

[0079] First of all, a quay operation resource dynamic allocation method based on a multi-objective optimization algorithm provided by the present invention overcomes the problems in the prior art such as static operation scheduling mode, lagging strategy update, poor adaptability to complex operation scenarios, and lack of multi-objective coordination ability. By constructing a comprehensive intelligent scheduling framework integrating a world model, a reinforcement learning module and a water cycle optimization mechanism, the intelligent level and operation efficiency of quay operation resource allocation are improved. Based on standardized quay operation data and combined with the Dreamer reinforcement learning model, the present invention realizes the dynamic prediction of future operation states, and completes the iterative optimization of operation strategies without a large number of real system interactions, improving the utilization rate of training samples and the strategy generation speed.

[0080] Secondly, the world model of the present invention not only includes a state transition and state encoding module, but also introduces a reward predictor layer and a termination predictor layer, which can simultaneously model quay operation states, immediate benefits and task termination information, enabling the policy learning process to have stronger environmental modeling ability and target perception ability in the latent state space. At the same time, the present invention introduces the water cycle algorithm into the world model parameter optimization and reinforcement learning policy network training process. During the optimization process, by simulating the water droplet flow, evaporation and precipitation mechanisms, it explores the optimal parameter path, avoiding problems such as parameter selection relying on experience and optimization being prone to local optima in traditional methods, and improving the global search ability and convergence stability of the overall scheduling model.

[0081] Finally, the present invention combines the core idea of multi-objective optimization. In the resource scheduling process, it not only focuses on a single performance index, but also incorporates multiple scheduling objectives (such as energy consumption balance, operation fairness, yard balance rate, etc.) into the reward function system. Through multi-factor integration design in reinforcement learning, the comprehensive improvement of multi-objective scheduling performance is realized. The method also sets up a real-time resource allocation deviation monitoring mechanism. By calculating the difference between the actual operation actions and the policy output, when the deviation exceeds the set threshold, the policy adjustment process is automatically triggered, and parameter reset is carried out in combination with the perturbation mechanism in the water cycle algorithm, enabling the model to have fast adaptability and continuous optimization ability in the dynamically changing quay environment. Description of the Drawings

[0082] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0083] Figure 1 This is a flowchart of a dynamic allocation method for terminal operation resources based on a multi-objective optimization algorithm proposed by the present invention. Specific implementation manners

[0084] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0085] Reference Figure 1 , a dynamic allocation method for terminal operation resources based on a multi-objective optimization algorithm, includes the following steps:

[0086] S1. Collect terminal operation data, preprocess the collected data, and generate a standardized data set;

[0087] S2. Construct a Dreamer reinforcement learning model, train the world model of the Dreamer reinforcement learning model based on the standardized data set, and predict the terminal operation state through the world model;

[0088] S3. Use the water cycle algorithm to optimize the prediction process of the world model, set water droplets to represent different combinations of training parameters, calculate the prediction error of the world model, and adjust the parameters of the world model based on the mechanism of water droplets flowing to the optimal solution to generate optimized prediction data;

[0089] S4. Based on the prediction data, set the state space, action space, and reward function of the intelligent agent of the reinforcement learning module, train the intelligent agent of the reinforcement learning module, learn the terminal operation resource allocation strategy, and generate a resource allocation plan;

[0090] S5. Use the water cycle algorithm to optimize the training process of the intelligent agent of the reinforcement learning module, set water droplets to represent different combinations of parameters of the reinforcement learning module, calculate the fitness of the resource allocation plan, and adjust the parameters of the reinforcement learning module based on the water droplet flow process to optimize the terminal operation resource allocation plan;

[0091] S6. Collect real-time terminal operation data, calculate the deviation of terminal operation resource allocation. If the resource allocation deviation exceeds the set threshold, adjust the terminal operation resource allocation plan based on the evaporation-precipitation mechanism of the water cycle algorithm;

[0092] S7. Optimize the Dreamer reinforcement learning model based on historical terminal operation data, update the parameters of the Dreamer reinforcement learning model by incremental learning, and adjust the terminal operation resource allocation plan.

[0093] The present invention constructs a dynamic scheduling framework that integrates the Dreamer reinforcement learning model and the water cycle algorithm, achieving high-precision modeling of the quay operation status and adaptive optimization of the resource allocation strategy, enhancing the intelligence, real-time performance, and multi-objective coordination ability of resource scheduling, and meeting the actual requirements of efficient resource allocation in the complex environment of automated quays.

[0094] In this embodiment, the quay operation data includes vessel arrival time, cargo handling plan, equipment availability, operation progress, and environmental status, and the preprocessing includes format conversion, outlier detection, time synchronization, and normalization processing.

[0095] The present invention collects multi-source heterogeneous operation data such as vessel arrival time, handling plan, and equipment availability, performs format conversion, anomaly detection, time synchronization, and normalization processing on the data, constructs a standardized dataset with high quality and unified format, provides a stable and reliable data basis for the subsequent training of the world model and policy learning, and improves the accuracy and generalization ability of model training.

[0096] In this embodiment, the specific steps of S2 are as follows:

[0097] S21. Construct a Dreamer reinforcement learning model, where the Dreamer reinforcement learning model includes a world model and a reinforcement learning module. The world model includes a state encoder layer, a state transition layer, a reconstruction decoder layer, a reward predictor layer, and a termination predictor layer. The reinforcement learning module includes a policy network and a value network. The world model predicts the quay operation status, reward information, and task termination information based on the quay operation data. The policy network is used to optimize the resource allocation scheme in the world model, and the value network is used to evaluate the advantages and disadvantages of different resource allocation schemes;

[0098] S22. Train the world model based on the standardized dataset, construct a state encoder layer, extract features from the input standardized dataset o t and map the standardized dataset to a latent state z t using a variational autoencoder:

[0099] z t = W μ ·φ(o t ) + b μ + softplus(W σ ·φ(o t ) + b σ )·∈, ∈~N(0,1);

[0100] where z t represents the current latent state, W μ and W σDenote the weight matrix of the state encoder layer, φ(o t ) represents the feature extraction vector, b μ and b σ represent the bias terms of the state encoder layer, softplus represents the non-linear activation function, ∈ represents the noise variable following the standard normal distribution, N represents the standard normal distribution with mean 0 and variance 1, o t represents the input standardized data set;

[0101] S23. Construct a state transition layer, use a gated recurrent unit to recursively model the historical latent state and action, and obtain the current hidden state h t and generate the prior distribution of the predicted latent state :

[0102]

[0103] where h t represents the current hidden state, GRU represents the gated recurrent unit, h t-1 represents the hidden state at the previous time step, z t-1 represents the latent state at the previous time step, a t-1 represents the action at the previous time step, represents the predicted latent state, N represents the normal distribution, and represent the weight matrices of the state transition layer, and represent the biases of the state transition layer, softplus represents the non-linear activation function;

[0104] S24. Construct a reconstruction decoder layer to decode the predicted latent state into the reconstructed standardized data set The reconstruction decoder layer uses a multi-layer perceptron for decoding;

[0105] S25. Construct a reward predictor layer to predict the immediate reward under the current latent state and action The reward predictor layer uses a fully connected network;

[0106] S26. Construct a termination predictor layer to determine whether the current latent state is a termination state and output the predicted termination flag The termination predictor layer uses a fully connected network combined with a sigmoid activation function;

[0107] S27. Jointly train all modules in the world model, construct a comprehensive loss function, and minimize the reconstruction prediction error, reward prediction error, termination prediction error, and KL divergence:

[0108]

[0109] Among them, L represents the comprehensive loss function, λ1, λ2, λ3, and λ4 represent the weight coefficients of each loss term, and o t represents the input standardized data set, represents the reconstructed standardized data, ||·|| 2 represents the squared norm, represents the predicted latent state, r t represents the true reward, represents the immediate reward under the predicted current latent state and action, represents the predicted termination probability, describing the predicted termination flag, d t represents the true termination probability, KL(·||·) represents the Kullback-Leibler divergence, represents the probability distribution of the true latent state, describing the ideal distribution of the world model, represents the latent state distribution predicted by the world model, describing the current prediction of the world model for the environmental state, μ t and represent the mean and variance of the true latent state, and represent the mean and variance predicted by the world model;

[0110] S28. Based on the trained world model, input the current job data to generate a sequence of latent states for future time steps Predict the reward sequence and the termination probability sequence

[0111] The present invention constructs a Dreamer world model including a state encoder, a state transition layer, a reward predictor, and a termination predictor, which can model the job environment based on the latent state, predict the future job state, reward, and termination signal, enabling the reinforcement learning agent to complete policy learning without relying on a large number of real interactions, effectively improving the training efficiency and the accuracy and stability of the scheduling strategy.

[0112] In this embodiment, the S3 specifically includes:

[0113] S31. Based on the trained world model, set the optimization objective of the water cycle algorithm, and the optimization objective is to minimize the prediction error of the world model, and optimize the parameters θ of the state transition layer and the parameters of the state encoder layer

[0114] S32. Initialize the water cycle algorithm, set the water droplet population size P, and randomly generate P groups of candidate parameter combinations within the parameter search space respectively represent the parameters of the state transition layer and the state encoder layer corresponding to each water droplet, calculate the initial fitness corresponding to each set of parameter combinations, and the fitness is used to measure the performance of the current parameter combination on the optimization objective, through the prediction error is represented by;

[0115] S33. For each set of parameter combinations perform forward prediction in the world model, calculate the corresponding prediction error, and define the error function:

[0116]

[0117] where represents the error function, o t represents the input standardized data set, represents the reconstructed standardized data, KL(·||·) represents the Kullback-Leibler divergence, represents the probability distribution of the true latent state, describing the ideal distribution of the world model, represents the latent state distribution predicted by the world model, describing the current prediction of the world model on the environmental state, μ t and represent the mean and variance of the true latent state, and represent the mean and variance predicted by the world model, α1 and α2 represent the loss weights, ||·|| 2 represents the squared norm;

[0118] S34. Execute the water droplet flow process, guide the search direction among the water droplets according to the fitness, and update the parameter combinations of each group of water droplets:

[0119]

[0120] where and respectively represent the parameters of the state transition layer and the state encoder layer of the i-th water droplet in the (t + 1)-th round, and respectively represent the parameters of the state transition layer and the state encoder layer of the i-th water droplet in the t-th round, α represents the learning rate, β represents the perturbation factor, ξ1 and ξ2 represent independent random perturbation terms, and represent another set of water droplet parameters used for reference in the t-th round, and the fitness is better than that of the i-th water droplet;

[0121] S35. Calculate the prediction error of each updated set of parameter combinations. If the convergence condition is satisfied, then mark that the water droplet parameters have converged, where ∈ represents the convergence threshold;

[0122] S36. For the un - converged water - droplet combinations, execute the evaporation - precipitation mechanism of the water - cycle algorithm. The evaporation - precipitation mechanism resets the M sets of water - droplet parameters with the worst fitness:

[0123]

[0124] Among them, and represent the new parameter combination of the k - th water - droplet to be reset. θ rand and represent the water - droplet parameter combinations randomly generated again in the parameter search space. γ represents the disturbance intensity coefficient in the evaporation - precipitation process. ξ1′ and ξ2′ represent the independent random disturbance terms introduced in the evaporation - precipitation mechanism;

[0125] S37. Repeat steps S34 to S36 until all water - droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination, update the world model based on the optimized parameter combination, and recalculate the predicted values of the job status at future time steps.

[0126] The present invention globally optimizes the parameters of the state encoder layer and the state transition layer in the world model by using the water - cycle algorithm, adjusts the model parameters through the water - droplet flow and the evaporation - precipitation mechanism, effectively avoids the problems of falling into local optimality or difficult parameter convergence in traditional training, improves the prediction accuracy and robustness of the world model, and provides more reliable prediction support for subsequent policy optimization.

[0127] In this embodiment, S4 specifically includes:

[0128] S41. Construct a reinforcement - learning module, define a policy network and a value network V ψ for optimizing the dock operation resource allocation, initialize the parameters of the policy network and the value network, set the input of the agent as the dock operation status, reward information, and task termination information provided by the world model, and set the output of the agent as the allocation policy of the dock operation resources;

[0129] S42. Construct the state space, action space, and reward function for the dock operation resource allocation, set the state vector where represents the predicted latent state at the t - th moment by the world model, represents the predicted immediate reward, represents the predicted termination probability, represents the change amount of the latent state, ρ t represents the Euclidean distance between the current latent state and the average latent state;

[0130] S43. Sample the quay operation resource allocation strategy based on the world model and set the action policy distribution:

[0131]

[0132] where a t represents the action sampled under the state vector s t , represents the policy network, Softmax represents the normalization operation, W π and b π represent the weight matrix and bias vector of the policy network respectively, and σ represents the non-linear activation function;

[0133] S44. Calculate the cumulative reward of the quay operation resource allocation strategy and define the reward function:

[0134]

[0135] where R t represents the reward function, describing the cumulative reward starting from the t-th moment, T represents the total number of time steps, γ represents the discount factor, represents the predicted immediate reward at the future (t + k)-th moment, λ represents the state deviation penalty coefficient, and ρ t+k represents the state distance at the (t + k)-th moment;

[0136] S45. Calculate the advantage function and adjust the value deviation based on the execution of the quay operation resource allocation:

[0137]

[0138] where A t represents the advantage function, represents the predicted immediate reward, V ψ represents the value network, s t and s t+1 represent the current state and the next state respectively, γ represents the discount factor, represents the action perturbation coefficient, a t represents the current action, represents the historical action mean, represents the two-norm between the current action and the historical action, k' represents the category, and V ψ (s t ) and V ψ (s t+1 ) represent the value estimation of the state output by the value network based on the parameter ψ under the state s t ;

[0139] S46. Optimize the policy network, calculate the policy loss function, optimize the value network, and calculate the value error of the resource allocation decision. Set the value loss function:

[0140]

[0141] Among them, L π represents the policy loss function, L V represents the value loss function, E represents the mathematical expectation, D represents the experience pool, represents the logarithm probability of the current action, A t represents the advantage function, κ represents the entropy regularization coefficient, represents the entropy value of the output distribution of the policy network, V ψ (s t ) represents the value estimation of the current state, R t represents the reward function, δ represents the value difference adjustment coefficient, ΔV t represents the difference between the value predictions of adjacent states;

[0142] S47. Perform parameter optimization and adjust the parameters of the policy network and the value network based on the gradient descent method;

[0143] S48. Output the optimized quay operation resource allocation strategy, store the optimized policy network parameters and value network parameters, and apply the optimized resource allocation strategy to the quay operation scheduling system to adjust the resource allocation plan in the actual operation process.

[0144] Based on the optimized world model, the present invention constructs a reinforcement learning agent. By setting a multi-dimensional state space and a composite reward function, and designing an accumulated reward and an advantage function, the policy network and the value network are guided to efficiently optimize the resource allocation strategy in the latent space, having the advantages of fast policy learning speed, high action output stability, and compatibility with multi-objective tasks, and adapting to complex scheduling scenarios.

[0145] In this embodiment, the S5 specifically includes:

[0146] S51. Initialize the optimization process of the reinforcement learning module of the water cycle algorithm, take the policy network parameters and the value network parameters ψ in the reinforcement learning module as joint optimization variables, set the size of the water droplet population to G, and randomly generate G water droplet individuals in the parameter space

[0147] S52. Based on the current water droplet individual perform the quay operation resource allocation task in the optimized world model, and use the policy network and the value network V ψ to perform simulation interactions, obtain the resource allocation strategy and generate the experience trajectory Di = {(s t , a t , R t , s t+1 )};

[0148] S53. Calculate the fitness function of each water droplet according to the empirical trajectory:

[0149]

[0150] Among them, represents the fitness value of the i-th water droplet, represents the policy loss corresponding to the policy network, L V (ψ i ) represents the value loss corresponding to the value network, and δ represents a positive constant to prevent division by zero overflow;

[0151] S54. Set the update rule of the water droplets in the parameter space, execute the water droplet flow process, and select the water droplet pair according to the relative fitness and

[0152]

[0153] Among them, and respectively represent the policy network parameters and value network parameters of the i-th water droplet in the (t + 1)-th round, and respectively represent the policy network parameters and value network parameters of the i-th water droplet in the t-th round, ι represents the learning rate, represents the perturbation factor, and represent another set of water droplet parameters for reference in the t-th round, and ξ1 and ξ2 represent random perturbation terms under the uniform distribution;

[0154] S55. After updating all the water droplets, re-execute the resource allocation simulation in the world model, generate a policy trajectory based on the new parameters, and recalculate the fitness function Judge whether the convergence condition is satisfied;

[0155] S56. If the convergence condition is not satisfied, execute the evaporation-precipitation mechanism of the water cycle algorithm, and the evaporation-precipitation mechanism randomly resets the M water droplet individuals with the lowest fitness ranking:

[0156]

[0157] Among them, and represent the policy network parameters and value network parameters of the k-th water droplet to be reset, and η represents the perturbation intensity coefficient in the evaporation-precipitation mechanism, and ψ rand represent the regenerated random parameters, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism;

[0158] S57. Repeat steps S54 to S56 until all water droplets meet the convergence condition or reach the maximum number of iteration rounds, and finally output the optimal parameter combination of the reinforcement learning module, replace the original parameters, and optimize the quay operation resource allocation plan.

[0159] The present invention collaboratively optimizes the parameters of the policy network and value network of the reinforcement learning module through a water cycle algorithm, dynamically adjusts the network weights by combining the water droplet flow and evaporation-precipitation mechanism, avoids the problem that the traditional gradient descent algorithm is prone to fall into local optima during training, improves the global search ability of policy learning and the adaptability of the scheduling policy, and enhances the generalization performance of the model in an uncertain environment.

[0160] Embodiment 1:

[0161] To verify the feasibility of the present invention in implementation, the present invention is applied to the operation scheduling system of a large container automated quay in the southeast coast of China. The total quay length of this quay exceeds 2000 meters, the annual designed throughput is 6.5 million TEUs, the quay operation area is equipped with 32 quay cranes, 68 yard cranes and about 300 automated guided vehicles, the average number of vessels processed daily exceeds 25, and the container handling volume per hour can reach 5000 standard containers during the peak operation period. The operation tasks are intensive and dynamically change violently, the scheduling tasks are complex and frequent, and the traditional scheduling system has obvious problems such as response lag, serious local resource conflicts and unstable scheduling results in coping with high-frequency equipment status changes and sudden operation adjustments.

[0162] In this scenario, the quay operation resource dynamic allocation method based on a multi-objective optimization algorithm proposed by the present invention is deployed in the quay operation scheduling platform to replace the original rule-driven scheduling module. In the early stage of deployment, the system collected the operation data of the quay in the past three months, including vessel arrival time, planned operation tasks, equipment availability, operation progress and environmental status, etc., constructed a standardized data set and used it to train the world model in the Dreamer reinforcement learning model. Through the state encoder layer, state transition layer, reward predictor layer and termination predictor layer, the state changes of the operation environment are modeled, and the operation state sequence and resource demand change trend in the future time period are generated through historical data.

[0163] After completing the training of the world model, the system constructs an agent for the reinforcement learning module. The state space is set to include indicators such as potential job status, equipment load distribution, and yard density changes. The action space is a set of equipment allocation decisions for different job areas. Multiple objective parameters such as average job completion time, equipment energy consumption, and yard load balance are introduced into the reward function. To ensure that the model has good adaptability and generalization ability among different equipment groups, a water cycle algorithm is further introduced to optimize the parameters of the policy network and value network in the agent, and iteratively search for the optimal resource allocation strategy.

[0164] During the actual operation process, the system takes a 15-minute time window, collects the latest job status data in real time and adjusts the resource allocation. When it is detected that the deviation between the scheduling result and the actual operation exceeds the preset threshold, the evaporation-precipitation mechanism of the water cycle algorithm is triggered to quickly fine-tune the current policy and achieve dynamic correction of job scheduling. After the system has been deployed and continuously run for 30 days, a comparative evaluation is carried out with the original scheduling system from dimensions such as job efficiency, resource utilization rate, and response speed.

[0165] Table 1 Comparison table of quay operation resource allocation performance

[0166]

[0167]

[0168] In terms of job efficiency, the present invention optimizes the average job completion time from 42.6 minutes in the original system to 35.2 minutes, with an improvement rate of 17.3%. This effect is due to the present invention's accurate prediction of future job status by introducing the Dreamer reinforcement learning model and generating a more reasonable resource scheduling strategy on this basis, avoiding problems such as lagging scheduling rules and repeated resource allocation in traditional methods, and improving the continuity and coordination of job flow.

[0169] In terms of resource utilization rate, the present invention improves the utilization efficiency of various key equipment. The utilization rate of quay cranes is increased from 71.5% to 83.9%, the utilization rate of yard cranes is increased from 68.1% to 79.3%, and the utilization rate of AGV vehicles is increased from 64.4% to 76.7%. These data indicate that the present invention can achieve better resource matching on the basis of multi-objective coordination, enabling the equipment to be more effectively allocated in both the spatial and temporal dimensions, thereby reducing resource vacancy and conflict and optimizing the overall scheduling flow line.

[0170] In terms of scheduling response performance, the average response delay of the system of the present invention has been significantly reduced, from the original 8.3 seconds to 3.1 seconds, and the response speed has increased by 62.7%. This is due to the introduction of the water cycle algorithm in this method to optimize the parameters of the policy network and the value network, enabling the scheduling system to have a stronger real-time adjustment ability. In addition, through dynamic deviation detection and the evaporation-precipitation mechanism, the system can correct in time when scheduling deviations occur, further reducing response lag.

[0171] In terms of system robustness, the policy stability rate of the present invention reaches 99.1% under abnormal weather conditions, an increase of 3.7% compared with 95.4% of the original system. At the same time, in terms of the ability to handle emergencies, the scheduling failure rate has been significantly reduced from 2.4% to 0.6%, reducing 75% of scheduling errors. This fully demonstrates that through the environmental modeling ability of the world model and the continuous learning mechanism of the agent, the present invention effectively enhances the system's adaptability to external disturbances and has higher reliability and fault tolerance.

[0172] The above is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes should be covered within the protection scope of the present invention.

Claims

1. A dynamic allocation method for quay operation resources based on a multi-objective optimization algorithm, characterized in that It includes the following steps: S1. Collect terminal operation data, preprocess the collected data, and generate a standardized data set; S2. Build a Dreamer reinforcement learning model, train the world model of the Dreamer reinforcement learning model based on the standardized data set, and predict the terminal operation status through the world model; S3. Use the water cycle algorithm to optimize the prediction process of the world model, set water droplets to represent different combinations of training parameters, calculate the prediction error of the world model, and adjust the parameters of the world model based on the mechanism of water droplets flowing towards the optimal solution to generate optimized prediction data; S4. Based on the prediction data, set the state space, action space, and reward function of the intelligent agent in the reinforcement learning module, train the intelligent agent in the reinforcement learning module, learn the terminal operation resource allocation strategy, and generate a resource allocation plan; S5. Use the water cycle algorithm to optimize the training process of the intelligent agent in the reinforcement learning module, set water droplets to represent different combinations of reinforcement learning module parameters, calculate the fitness of the resource allocation plan, and adjust the reinforcement learning module parameters based on the water droplet flow process to optimize the terminal operation resource allocation plan; S6. Collect real-time terminal operation data, calculate the deviation of terminal operation resource allocation. If the resource allocation deviation exceeds the set threshold, adjust the terminal operation resource allocation plan based on the evaporation-precipitation mechanism of the water cycle algorithm; S7. Optimize the Dreamer reinforcement learning model based on historical terminal operation data, update the parameters of the Dreamer reinforcement learning model using incremental learning, and adjust the terminal operation resource allocation plan.

2. The dynamic allocation method of quay operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that The terminal operation data includes the ship arrival time, cargo handling plan, equipment availability, operation progress, and environmental status. The preprocessing includes format conversion, outlier detection, time synchronization, and normalization processing.

3. A dynamic allocation method of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that, The specific content of S2 includes: S21. Build a Dreamer reinforcement learning model. The Dreamer reinforcement learning model includes a world model and a reinforcement learning module. The world model includes a state encoder layer, a state transition layer, a reconstruction decoder layer, a reward predictor layer, and a termination predictor layer. The reinforcement learning module includes a policy network and a value network. The world model predicts the terminal operation status, reward information, and task termination information based on the terminal operation data. The policy network is used to optimize the resource allocation plan in the world model, and the value network is used to evaluate the advantages and disadvantages of different resource allocation plans; S22. Train a world model based on a standardized dataset, construct a state encoder layer, and perform feature extraction on the input standardized dataset o t Map the standardized dataset to a latent state z using a variational autoencoder t : z t = W μ ·φ(o t ) + b μ + softplus(W σ ·φ(o t ) + b σ )·∈, ∈~N(0,1); where z t represents the current latent state, W μ and W σ represent the weight matrices of the state encoder layer, φ(o t ) represents the feature extraction vector, b μ and b σ represent the bias terms of the state encoder layer, softplus represents the non-linear activation function, ∈ represents a noise variable following the standard normal distribution, N represents the standard normal distribution with mean 0 and variance 1, and o t represents the input standardized data set; S23. Construct a state transition layer, and use a gated recurrent unit to recursively model the historical latent state and action to obtain the current hidden state h t and generate a predicted latent state with a prior distribution of: Among them, h t represents the current hidden state, GRU represents the gated recurrent unit, h t-1 represents the hidden state of the previous time step, z t-1 represents the latent state of the previous time step, a t-1 represents the action of the previous time step, represents the predicted latent state, N represents the normal distribution, and represent the weight matrix of the state transition layer, and represent the bias of the state transition layer, softplus represents the non-linear activation function; S24. Construct a reconstruction decoder layer to decode the predicted latent state into a reconstructed normalized data set The reconstruction decoder layer uses a multi-layer perceptron for decoding; S25. Construct a reward predictor layer to predict the immediate reward under the current latent state and action The reward predictor layer adopts a fully connected network; S26. Construct a termination predictor layer to determine whether the current latent state is a termination state and output a predicted termination flag The termination predictor layer uses a fully connected network combined with a sigmoid activation function; S27. Jointly train all modules in the world model, build a comprehensive loss function, and minimize the reconstruction prediction error, reward prediction error, termination prediction error, and KL divergence; Among them, \(L\) represents the comprehensive loss function, \(\lambda_1\), \(\lambda_2\), \(\lambda_3\) and \(\lambda_4\) represent the weight coefficients of each loss term, \(o\) t represents the input standardized data set, represents the reconstructed standardized data, \(\|\cdot\|\) 2 represents the squared norm, represents the predicted latent state, \(r\) t represents the true reward, represents the immediate reward under the predicted current latent state and action, represents the predicted termination probability, describing the predicted termination marker, \(d\) t represents the true termination probability, \(KL(\cdot\|\cdot)\) represents the Kullback-Leibler divergence, represents the probability distribution of the true latent state, describing the ideal distribution of the world model, represents the latent state distribution predicted by the world model, describing the current prediction of the world model for the environmental state, \(\mu\) t and represent the mean and variance of the true latent state, and represent the mean and variance predicted by the world model; S28. Based on the trained world model, input the current job data to generate a sequence of latent states at future time steps Predict the reward sequence and the termination probability sequence 4. A dynamic allocation method of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that, The specific content of S3 includes: S31. Based on the trained world model, set the optimization objective of the water cycle algorithm. The optimization objective is to minimize the prediction error of the world model and optimize the parameters θ of the state transition layer and the parameters of the state encoder layer of the world model. S32. Initialize the water cycle algorithm, set the size P of the water droplet population, and randomly generate P groups of candidate parameter combinations within the parameter search space. They respectively represent the parameters of the state transition layer and the state encoder layer corresponding to each water droplet, calculate the initial fitness corresponding to each group of parameter combinations, and the fitness is used to measure the performance of the current parameter combination in the optimization objective, and is represented by the prediction error. Indicates. S33. For each set of parameter combinations Perform forward prediction in the world model, calculate the corresponding prediction error, and define the error function: Among them, represents the error function, o t represents the normalized input data set, represents the reconstructed normalized data, KL(·||·) represents the Kullback-Leibler divergence, represents the probability distribution of the true latent state, describing the ideal distribution of the world model, represents the distribution of the latent state predicted by the world model, describing the current prediction of the world model for the environmental state, μ t and represent the mean and variance of the true latent state, and represent the mean and variance predicted by the world model, α1 and α2 represent the loss weights, ||·|| 2 represents the squared norm; S34. Execute the water droplet flow process, guide the search direction among water droplets according to the fitness size, and update the parameter combination of each group of water droplets; Among them, and respectively represent the parameters of the state transition layer and the state encoder layer of the $i$-th water droplet in the $(t + 1)$-th round, and respectively represent the parameters of the state transition layer and the state encoder layer of the $i$-th water droplet in the $t$-th round, $\alpha$ represents the learning rate, $\beta$ represents the perturbation factor, and $\xi_1$ and $\xi_2$ represent independent random perturbation terms, and represent another set of water droplet parameters used for reference in the $t$-th round, and the fitness is better than that of the $i$-th water droplet; S35. Calculate the prediction error of each group of updated parameter combinations. If the convergence condition is satisfied then mark that the water droplet parameters have converged, where ∈ represents the convergence threshold; S36. For the water droplet combinations that have not converged, execute the evaporation-precipitation mechanism of the water cycle algorithm. The evaporation-precipitation mechanism resets the parameters of the M groups of water droplets with the worst fitness; Among them, and represent the new parameter combination of the k-th water droplet to be reset, and θ rand and represent the parameter combination of the water droplet randomly regenerated again in the parameter search space. γ represents the disturbance intensity coefficient in the evaporation-precipitation process, and ξ1′ and ξ2′ represent the independent random disturbance terms introduced in the evaporation-precipitation mechanism; S37. Repeat steps S34 to S36 until all water droplets meet the convergence condition or reach the maximum number of iterations. Finally, output the optimal parameter combination, update the world model based on the optimized parameter combination, and recalculate the predicted values of the job status for future time steps.

5. A dynamic allocation method of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, characterized in that The specific steps of S4 include: S41. Construct a reinforcement learning module, and define a policy network for optimizing the allocation of terminal operation resources and a value network V ψ , initialize the parameters of the policy network and the value network, set the input of the agent to the terminal operation status, reward information, and task termination information provided by the world model, and set the output of the agent to the allocation policy of terminal operation resources; S42. Construct the state space, action space, and reward function for quay operation resource allocation, and set the state vector where represents the potential state at the t-th moment predicted by the world model, represents the predicted immediate reward, represents the predicted termination probability, represents the change in the potential state, ρ t represents the Euclidean distance between the current potential state and the average potential state; S43. Sample the quay operation resource allocation strategy based on the world model and set the action policy distribution: Among them, a t represents the action obtained by downsampling the state vector s t The following sampling is obtained, represents the policy network, Softmax represents the normalization operation, W π and b π represent the weight matrix and bias vector of the policy network, and σ represents the non-linear activation function; S44. Calculate the cumulative reward of the quay operation resource allocation strategy and define the reward function: Among them, R t represents the reward function, describing the cumulative reward starting from the t-th moment, T represents the total number of time steps, γ represents the discount factor, represents the predicted immediate reward at the (t + k)-th moment in the future, λ represents the state deviation penalty coefficient, ρ t+k represents the state distance at the (t + k)-th moment; S45. Calculate the advantage function and adjust the value deviation based on the execution of the quay operation resource allocation: Among them, A t represents the advantage function, represents the predicted immediate reward, V ψ represents the value network, s t and s t+1 represent the current state and the next state respectively, γ represents the discount factor, represents the action perturbation coefficient, a t represents the current action, represents the historical action mean, represents the L2 norm between the current action and the historical action, k' represents the category, V ψ (s t ) and V ψ (s t+1 ) represent the value estimates of the state output by the value network based on the parameter ψ in the state s t ; S46. Optimize the policy network, calculate the policy loss function, optimize the value network, and calculate the value error of the resource allocation decision. Set the value loss function: Among them, L π represents the policy loss function, L V represents the value loss function, E represents the mathematical expectation, D represents the experience pool, represents the log probability of the current action, A t represents the advantage function, κ represents the entropy regularization coefficient, represents the entropy value of the policy network output distribution, V ψ (s t ) represents the value estimate of the current state, R t represents the reward function, δ represents the value difference adjustment coefficient, ΔV t represents the difference between adjacent state value predictions; S47. Perform parameter optimization and adjust the parameters of the policy network and the value network based on the gradient descent method; S48. Output the optimized quay operation resource allocation strategy, store the optimized policy network parameters and value network parameters, and apply the optimized resource allocation strategy to the quay operation scheduling system to adjust the resource allocation plan in the actual operation process.

6. The dynamic allocation method of terminal operation resources based on a multi-objective optimization algorithm according to claim 1, wherein The specific steps of S5 include: S51. Initialize the optimization process of the reinforcement learning module for the water cycle algorithm, and use the policy network parameters and the value network parameter ψ in the reinforcement learning module as joint optimization variables. Set the size of the water droplet population to G, and randomly generate G water droplet individuals in the parameter space S52. Based on the current water droplet individual Execute the dock operation resource allocation task in the optimized world model, and use the policy network to perform simulated interaction with the value network V ψ to obtain the resource allocation strategy and generate the experience trajectory D i ={(s t , a t , R t , s t+1 )}; S53. Calculate the fitness function of each water droplet according to the empirical trajectory: wherein, represents the fitness value of the i-th water droplet, represents the policy loss corresponding to the policy network, L V (ψ i ) represents the value loss corresponding to the value network, and δ represents a positive constant to prevent division by zero overflow; S54. Set the update rule of water droplets in the parameter space, execute the water droplet flow process, and select water droplet pairs according to the relative fitness and Among them, and respectively represent the policy network parameters and value network parameters of the i-th water droplet in the (t + 1)-th round, and respectively represent the policy network parameters and value network parameters of the i-th water droplet in the t-th round, ι represents the learning rate, represents the perturbation factor, and represent another set of water droplet parameters used for reference in the t-th round, ξ1 and ξ2 represent random perturbation terms under the uniform distribution; After updating all the water droplets, re - execute the resource allocation simulation in the world model, generate a policy trajectory based on the new parameters, and recalculate the fitness function Determine whether the convergence condition is met; S56. If the convergence condition is not met, execute the evaporation-precipitation mechanism of the water cycle algorithm. The evaporation-precipitation mechanism randomly resets the M water droplet individuals with the lowest fitness rankings: Among them, and represent the policy network parameters and value network parameters of the k-th water droplet to be reset, η represents the perturbation intensity coefficient in the evaporation-precipitation mechanism, and ψ rand represent the regenerated random parameters, and ξ1′ and ξ2′ represent the independent random perturbation terms introduced in the evaporation-precipitation mechanism; S57. Repeat steps S54 to S56 until all water droplets meet the convergence condition or reach the maximum number of iteration rounds. Finally, output the optimal parameter combination of the reinforcement learning module, replace the original parameters, and optimize the quay operation resource allocation plan.

Images