Off-line reinforcement learning harbor equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement

Abstract

The invention belongs to the field of harbor equipment resource allocation, and discloses an offline reinforcement learning harbor equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement. Comprising the steps of collecting data such as operation object operation and equipment position, operation starting and ending time and equipment type in each production scene; deleting interference data; according to ship loading and unloading tasks of each ship, time window sliding processing is carried out on each production scene data, and the number of devices in each time period is obtained; learning from historical data by using an uncertainty perception and generalization enhancement reinforcement learning method to obtain a robust offline reinforcement learning model, and meanwhile, having the capability of processing out-of-distribution data; deploying a model in a port environment, acquiring environment information by using a distributed execution method, and recommending the number of devices in real time; one reinforcement learning model can process ship loading and unloading tasks of a plurality of ships at the same time, and the requirement for real-time resource allocation in the field of unmanned wharfs is met.

Description

Technical Field

[0001] The present invention belongs to the field of dynamic resource allocation of global port equipment, and in particular to a robust resource allocation algorithm for out-of-distribution data. Background Art

[0002] Due to the dynamic and complex nature of resource allocation in unmanned ports, it is difficult to make real-time recommendations for multiple devices serving a ship. In addition, data distribution shifts due to factors such as equipment failures, weather factors, and dynamic environments, which in turn leads to a decline in the performance of offline reinforcement learning models.

[0003] At present, in order to achieve dynamic allocation of port equipment resources, the existing methods include:

[0004] (1) Physical simulation: By building a simulation platform, the port operation environment is constructed in the simulation environment setting, and then the operation process is simulated. However, this method cannot simulate various real abnormal weather conditions, and it is even more difficult to simulate a large number of equipment; due to the complexity of the operation process, the planned execution of each device is related to other devices, and this relationship is difficult to reflect in the simulation environment.

[0005] (2) Resource allocation based on traditional algorithms: mathematical modeling of the port environment and use of heuristic algorithms to recommend equipment. Due to the complexity of algorithm modeling, it is difficult to achieve global optimization in a short period of time, and the correlation between equipment makes it difficult to deploy real-time recommendations for multiple equipment.

[0006] (3) Dynamic resource allocation based on offline reinforcement learning: Due to the safety and reliability requirements of the port equipment operating environment, it is impossible to interact with the environment, resulting in insufficient exploration capabilities. In addition, the dynamic environment of the port and random equipment failures lead to insufficient generalization capabilities of the model for distributed data. Existing offline reinforcement learning cannot meet the requirements of robust real-time port equipment recommendation. Summary of the invention

[0007] In order to solve the deficiencies in the above-mentioned prior art, an offline reinforcement learning port equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement is provided. Uncertainty perception and generalization enhanced offline reinforcement learning are used to achieve out-of-distribution feature processing capabilities, and to achieve real-time dynamic recommendation of the number of port equipment, so as to achieve the purpose of robust global scheduling of equipment and improve productivity.

[0008] The technical solution adopted by the present invention is:

[0009] An offline reinforcement learning port equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement includes the following steps:

[0010] Step S1: Collecting ship operation information (operation start and end time, berth, number of containers, etc.) and equipment operation information (operation equipment location and operation area, operation time, equipment type, etc.);

[0011] Said ship operation includes loading and unloading operations;

[0012] The equipment includes: bridge crane, AGV (Automated Guided Vehicle) and rail crane;

[0013] Step S2: pre-process the data obtained in step S1, and delete data containing null values, plan cancellations, equipment failures, etc.;

[0014] Step S3: According to the loading and unloading tasks of each ship, each operation scene data of the port operation is processed by sliding time window to obtain the number of equipment in each time slice;

[0015] Among them, port operations include three operation scenarios: ship operation scenario, container operation scenario and yard operation scenario. In simplified cases, each scenario uses a type of equipment, namely bridge crane, AGV and rail crane.

[0016] Step S4: Use uncertainty-aware and generalization-enhanced reinforcement learning methods to learn from historical data and obtain a robust offline reinforcement learning model;

[0017] The present invention improves the offline reinforcement learning algorithm, constructs a twin network with Dropout for the value network in the offline reinforcement learning algorithm to generate sparse features, and uses MSE loss and KL divergence loss to perform generalization constraints, thereby improving the robustness of function estimation and the ability to process out-of-distribution data.

[0018] Step S5: Deploy the model obtained in step S4 in the port environment, use the central server to obtain environmental information as input, and output the recommended number of equipment in real time.

[0019] Furthermore, in step S1, the equipment operation information collection method: collects data such as the number Did of each operation equipment, the position P of the operation equipment, the operation object Oid, and the equipment type Dy for a period of half a year. A record can be represented as (Did, P, Oid, St, Et, Dy), where the operation equipment position P is represented by x, y coordinates in a two-dimensional coordinate system, the operation object Oid indicates the ship to which the loaded cargo belongs, the operation start time St and the operation end time Et are accurate to minutes, and the equipment type Dy includes various transport equipment at unmanned terminals.

[0020] The ship operation information collection method adopts the existing scheduling algorithm to train and schedule each operation scene separately and in real time, and collects the scheduling operation history information for half a year.

[0021] Furthermore, in the step S2, data containing null values ​​and data whose job plans are canceled are deleted and filled into the already opened state space, action space, and reward to obtain high-quality data set samples.

[0022] Furthermore, step S3 also includes the following method: recording at the start and end nodes of each ship to obtain the precise start and end time of the loading and unloading task of each ship.

[0023] Port operations include three operation scenarios: ship operation scenario, container operation scenario and yard operation scenario, which use bridge crane, AGV and rail crane respectively.

[0024] According to the loading and unloading tasks of each ship, the data of each operation scenario of the port operation is processed by sliding time window to obtain the number of equipment serving the ship Oid in each time period. The process of processing data using sliding time window includes the following steps:

[0025] Step 301: Perform statistics based on the job object, and calculate the job equipment Did, job start time St, job end time Et, and equipment type Dy of each job object Oid. The statistical results are saved in a dictionary format.

[0026] The statistical results for each job object are expressed as

[0027] {Oid: [Did 1 ,St 1 ,Et 1 ,Dy 1 ],[Did 2 ,St 2 ,Et 2 ,Dy 2 ],...,[Did M ,St M ,Et M ,Dy M ]}

[0028] Where M represents the number of operating equipment currently assigned to the vessel.

[0029] Step 302: Perform sliding time window processing on the statistical results, perform slicing every K minutes, and obtain the number of equipment serving the operation object (ship);

[0030] Using K as the window size of each shard, the data after sliding time window processing can be expressed as

[0031] {Oid:[Tw,{Dy 1 :[Dn 1 ],Dy 2 :[Dn 2 ],...,Dy M :[Dn M ]}]}

[0032] Where Dn represents the number of current object ship resource allocation equipment, Tw represents the timestamp, and K = Tw i+1 -Tw i , indicating that the time interval between two timestamps is K.

[0033] Furthermore, step S4 uses the Siamese network with Dropout to generate sparse features, and uses MSE loss and KL divergence loss for generalization constraints. These losses can be further tuned using weighted methods.

[0034] The details are as follows:

[0035] First, the state space s and action space a are generated according to the offline historical data set obtained from step 1 to step 3, as follows:

[0036] The state space s of the port is a discrete space (one sampling point per time step). Each ship has two tasks (loading and unloading). The state space can be expressed as (berth D, number of remaining loading containers, number of remaining unloading containers, number of available bridge cranes, number of available AGVs, number of available rail cranes). The berth ID represents the berth index where the ship is docked, and the loading / unloading containers are represented by the Int type.

[0037] The action space a is a discrete space. For each operation scenario, the action space can be expressed as (equipment plus one, minus one, unchanged). The information of the action is determined according to the changes in the equipment information in adjacent time steps. A reward is set for each time step data. 500 is added when loading and unloading are completed, and the loading and unloading process is rewarded with 0.

[0038] The offline data is represented as a state transition five-tuple (s, a, r, γ, s′), where s represents the state space; a represents the action space; r represents the immediate reward; γ represents the reward discount factor, the default value is 0.99; s′ represents the state at the next moment;

[0039] The reinforcement learning of the present invention uses an actor-critic algorithm. The critic model learns the value function and iterates through the Bellman operator. The actor network is optimized based on the feedback of the current state and the action truth value calculation loss.

[0040] Specifically, the learning target y of setting the value function Q(s,a) according to the Bellman operator can be expressed as

[0041] y=r+γmax a′ Q(s′,a′)

[0042] Among them, r represents the reward value of a single step, and a′ represents the action distribution at the next moment.

[0043] The Bellman loss L can be expressed as

[0044] L=(Q(s,a)-y) 2

[0045] In order to obtain the robustness of value function estimation and the ability to process out-of-distribution data, a twin model (weight θ') with Dropout (default node loss rate p = 0.5) is constructed for the value network (weight θ) in the offline reinforcement learning algorithm.

[0046] During each batch training, the twin value network loads the value network weights, and the value network returns V, feature_V = Q θ (s,a) value, its twin value network return value is V′, s_feature_V=Q θ′ (s,a). Where feature_V and s_feature_V represent the returned feature values, and V and V′ represent the scalar values ​​predicted by the Q network.

[0047] The model uncertainty is generated by using the random dropout method. The uncertainty loss can be expressed as

[0048] U(s,a)=MSE(Q θ (s,a)[0],Q θ ′(s,a)[0]) / w+KL(Q θ (s,a)[1]|Q θ ′(s,a)[1])

[0049] +KL(Q θ ′(s,a)[1]|Q θ (s,a)[1])

[0050] =MSE(V,V′) / w+KL(feature_V|s_feature_V)+KL(s_feature_V,feature_V)

[0051] Among them, w represents the weighted value of MSE loss. KL loss makes the feature distribution of the value network and the twin value network as similar as possible, so that the model has the ability to represent out-of-distribution. MSE (V, V′) is to make the model robust to out-of-distribution environments or behaviors. The MSE loss function can be expressed as:

[0052]

[0053] Where N represents BatchSize.

[0054] The weighted value w can be expressed as:

[0055]

[0056] The KL loss can be expressed as:

[0057]

[0058] In value function learning, we use uncertainty-aware loss and generalization-enhanced loss as a regularization method to re-set the value network loss

[0059] L critic =L+U(s,a)=(Q(s,a)-y) 2 +U(s,a)

[0060] Construct Actor loss for the current observation and action

[0061] L actor =λQ(s,π(s))-(π(s)-a) 2

[0062] λ represents the weighting parameter, α is set to 2.5 by default.

[0063] Furthermore, when step S5 deploys the model obtained in step S4, the collected dataset is used to fine-tune the model of the offline reinforcement learning method.

[0064] The model is deployed in the port environment, and the central server is used to collect environmental information in real time (including the current operating equipment ID, operating object Oid, equipment type Dy and other data) as input. Then, the Actor network is used to predict the deterministic action a′=π(s) at the next moment, so as to recommend the number of equipment in real time.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. The present invention provides an offline reinforcement learning port equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement, and utilizes an improved offline reinforcement learning algorithm to perform out-of-distribution learning, thereby achieving robustness in dynamic environments and enhancing the generalization performance of dynamic recommendations for port equipment.

[0067] 2. Compared with the data enhancement method based on simulation, the present invention avoids the problem of more variables in the real operating environment of the port leading to simulation difficulties and more human interference factors. Compared with the method based on the traditional resource allocation algorithm, it is difficult to achieve robust resource allocation for dynamic environments and random failures. Compared with the reinforcement learning algorithm, the present invention achieves the requirements of robust resource allocation under dynamic environments, random failures and weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of an offline reinforcement learning port equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement of the present invention;

[0069] Figure 2 A schematic diagram showing the change in the number of equipment serving a ship for loading and unloading at one time according to the present invention;

[0070] Figure 3 Schematic diagram of an offline reinforcement learning model based on uncertainty perception and generalization enhancement of the present invention;

[0071] Figure 4 It is a schematic diagram of distributed extrasensing and generalization of the present invention;

[0072] Figure 5 This is a diagram comparing the training results of the method according to the embodiment of the present invention and the traditional TD3BC method in global device scheduling;

[0073] Figure 6 Schematic diagram of online deployment and real-time recommendation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The content of the present invention is further illustrated below in conjunction with specific examples, but it should not be construed as limiting the present invention. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0075] Port offline data is the historical data of the operation of various equipment.

[0076] Offline reinforcement learning algorithms improve the value function. For out-of-distribution feature generation methods, Dropout, feature channel shuffling, or data enhancement can be used.

[0077] like Figure 1As shown, an offline reinforcement learning based on uncertainty perception and generalization enhancement of the present invention includes preprocessing the historical port equipment resource allocation data, constructing the state space and action space of the unmanned terminal and setting rewards for each time step, and then using an improved offline reinforcement learning method to perform out-of-distribution generalization perception and learning.

[0078] First, if Figure 2 As shown, the historical resource configuration data of port equipment is constructed. Specifically, each ship has two tasks, loading and unloading, and generally performs loading or unloading tasks at the same time or only.

[0079] like Figure 3 This is a schematic diagram of the offline reinforcement learning model based on uncertainty perception and generalization enhancement of the present invention. We have added a twin critic network to learn out-of-distribution features, and used constraint functions to enable the critic network to have out-of-distribution data processing capabilities. In value function learning, the actor-critic algorithm is used. Among them, the critic model performs value function learning and iterates through the Bellman operator. The actor network is optimized based on the feedback of the current state and the action truth value calculation loss.

[0080] The historical log files obtained from the offline dataset are preprocessed and saved as a state five-tuple (s, a, r, γ, s′), including the state space s, the action space a, the return reward r, the reward discount γ, and the state s′ at the next moment.

[0081] The Bellman operator sets the learning target of the value function Q(s,a) to y=r+γmax a 'Q(s',a'), where r represents the instantaneous reward value of a single step, a' represents the action distribution at the next moment, and the Bellman loss L can be expressed as L = (Q(s,a)-y) 2 .

[0082] In the uncertainty perception and generalization enhancement algorithm, the twin value network is used to perform out-of-distribution perception and generalization enhancement methods to constrain features and achieve the ability to robustly process out-of-distribution data. Specifically, the twin value network has the same architecture as the value network, and the Dropout of the value network is set to 0. The twin value network loads the weights of the value network during each batch training, and then enables the use of Dropout to generate out-of-distribution sparse representations. No gradient calculation is performed during training, and only U(s,a) is used to achieve robust processing capabilities for out-of-distribution features. The role of U(s,a) is as follows: Figure 4As shown in the figure, the Q network is used to obtain in-distribution features, and the twin Q network obtains out-of-distribution features. U(s,a) distills the knowledge of the twin Q network into the Q network by reducing the distance between the two feature distributions, so that the Q network can obtain generalized features, indicating that the Q network has the ability to process out-of-distribution features.

[0083] The present invention does not limit the method for obtaining out-of-distribution features. Preferably, it can be DropConnect, channel shuffling, or data enhancement.

[0084] The present invention does not limit the deployment method of the reinforcement learning model. It can be deployed in a single operation scenario or in a global resource allocation scenario with only slight modifications.

[0085] For the port operation equipment allocation scenario, the TD3BC (Twin Delayed Deep Deterministic Policy Gradient + Behavior Cloning) offline reinforcement method is used to compare the situation of adding the twin Q network (the method of the present invention) and not adding the twin Q network (the traditional method). Reward is used as the evaluation indicator. The results are as follows: Figure 5 As shown, it can be seen that the method of the present invention is significantly better than the TD3BC method and has better performance.

[0086] For the model trained in the offline scenario, the centralized node deployment method is used to recommend the number of devices serving a ship in real time, such as Figure 6 As shown in the figure, the state of each ship at the current time t is used as input, and the trained Actor is used to predict the number of bridge cranes and AGV cranes allocated at time t+K, so as to achieve the purpose of global equipment resource allocation.

[0087] Starting from the development needs of the dynamics and robustness of unmanned terminals, the present invention proposes an offline reinforcement learning port equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement, which realizes the dynamic allocation of port resources and has a certain out-of-distribution generalization capability. It is a more effective and scientific technical means.

[0088] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement modes and shall be included in the protection scope of the present invention.

Claims

1. An offline reinforcement learning port equipment global resource allocation algorithm based on uncertainty perception and generalization enhancement, characterized by: The following steps are involved: Step S1: Collecting ship operation information (operation start and end time, berth, number of containers, etc.) and equipment operation information (operation equipment location and operation area, operation time, equipment type, etc.); Said ship operation includes loading and unloading operations; The equipment includes: bridge crane, AGV and rail crane; Step S2: pre-process the data obtained in step S1, and delete the data containing null values, plan cancellations, and equipment failures; Step S3: According to the loading and unloading tasks of each ship, each operation scene data of the port operation is processed by sliding time window to obtain the number of equipment in each time slice; Among them, port operations include three operation scenarios: ship operation scenario, container operation scenario and yard operation scenario. In simplified cases, each scenario uses one type of equipment, namely bridge crane, AGV and rail crane; Step S4: Use uncertainty-aware and generalization-enhanced reinforcement learning methods to learn from historical data and obtain a robust offline reinforcement learning model; For the value network in the offline reinforcement learning algorithm, a twin network with Dropout is constructed to generate sparse features, and MSE loss and KL divergence loss are used for generalization constraints, thereby improving the robustness of function estimation and the ability to handle out-of-distribution data; Step S5: Deploy the model obtained in step S4 in the port environment, use the central server to obtain environmental information as input, and output the recommended number of equipment in real time.

2. The allocation algorithm according to claim 1, characterized in that In step S1, the equipment operation information collection method is as follows: data such as the number Did of each operation equipment, the position P of the operation equipment, the operation object Oid, and the equipment type Dy for a period of half a year are collected; a record can be expressed as (Did, P, Oid, St, Et, Dy), wherein the operation equipment position P is expressed using x, y coordinates in a two-dimensional coordinate system, the operation object Oid indicates the ship to which the loaded cargo belongs, the operation start time St and the operation end time Et are accurate to minutes, and the equipment type Dy includes various transport equipment at unmanned terminals; The ship operation information collection method adopts the existing scheduling algorithm to train and schedule each operation scene separately and in real time, and collects the scheduling operation history information for half a year.

3. The allocation algorithm according to claim 1, characterized in that In step S2, data containing null values ​​and data whose job plans are canceled are deleted and filled into the already opened state space, action space, and reward to obtain high-quality data set samples.

4. The allocation algorithm according to claim 1, characterized in that In step 3, the start and end nodes of each ship are recorded to obtain the precise start and end time of each ship's loading and unloading task; Port operations include three operation scenarios: ship operation scenario, container operation scenario and yard operation scenario, which use bridge crane, AGV and rail crane respectively. According to the loading and unloading tasks of each ship, the data of each operation scenario of port operations is processed by sliding time window to obtain the number of equipment serving the ship Oid in each time period. The process of processing data using sliding time window includes the following steps: Step 301: Perform statistics based on the job object, and calculate the job equipment Did, job start time St, job end time Et, and equipment type Dy of each job object Oid. The statistical results are saved in a dictionary format. The statistical results for each job object are expressed as {Oid: [Did1, St1, Et1, Dy1], [Did2, St2, Et2, Dy2], ..., [Did M ,St M ,Et M ,Dy M ]} Where M represents the number of operating equipment currently assigned to the vessel; Step 302: Perform sliding time window processing on the statistical results, perform slicing every K minutes, and obtain the number of equipment serving the operation object (ship); Using K as the window size of each shard, the data after sliding time window processing is represented as {Oid: [Tw, {Dy1: [Dn1], Dy2: [Dn2], ..., Dy M :[Dn M ]}]} Where Dn represents the number of current object ship resource allocation equipment, Tw represents the timestamp, and K = Tw i+1 -Tw i , indicating that the time interval between two timestamps is K.

5. The allocation algorithm according to claim 1, characterized in that In step S4, the state space s and action space a are generated according to the offline historical data set obtained in steps 1 to 3, as follows: The state space s of the port is a discrete space (one sampling point per time step). Each ship has two tasks (loading and unloading). The state space can be expressed as (berth D, number of remaining loading containers, number of remaining unloading containers, number of available bridge cranes, number of available AGVs, number of available rail cranes). Berth ID indicates the berth index where the ship is docked, and loading / unloading containers are represented by Int type. The action space a is a discrete space. For each operation scenario, the action space can be expressed as (equipment plus one, minus one, unchanged). The information of the action is determined according to the change of the equipment information in the adjacent time steps. A reward is set for each time step data. When loading and unloading are completed, 500 is added. The loading and unloading process is 0 reward. The offline data is represented as a state transition five-tuple (s, a, r, γ, s′), where s represents the state space; a represents the action space; r represents the immediate reward; γ represents the reward discount factor, the default value is 0.99; s′ represents the state at the next moment.

6. The allocation algorithm according to claim 1, characterized in that Step S4 uses the Siamese network with Dropout to generate sparse features, and uses MSE loss and KL divergence loss for generalization constraints, which are further tuned using a weighted method.

7. The allocation algorithm according to claim 6, characterized in that Reinforcement learning uses the actor-critic algorithm, in which the critic model learns the value function and iterates through the Bellman operator; the actor network is optimized based on the feedback of the current state and the true value of the action. Specifically, the learning target y of the value function Q(s,a) is set according to the Bellman operator, expressed as y=r+γmax a′ Q(s′,a′) Among them, r represents the reward value of a single step, and a′ represents the action distribution at the next moment; The Bellman loss L can be expressed as L=(Q(s,a)-y) 2 Construct a twin model (weight θ') with Dropout (default node loss rate p = 0.5) for the value network (weight θ) in the offline reinforcement learning algorithm; During each batch training, the twin value network loads the value network weights, and the value network returns V, feature_V = Q θ (s,a) value, its twin value network return value is V′, s_feature_V=Q θ ′(s,a); where feature_V and s_feature_V represent the returned feature values, and V and V′ represent the scalar values ​​predicted by the Q network; The model uncertainty is generated using the random dropout method, and the uncertainty loss is expressed as U(s,a)=MSE(Q θ (s,a)[0],Q θ′ (s,a)[0]) / w+KL(Q θ (s,a)[1]|Q θ′ (s,a)[1])+KL(Q θ′ (s,a)[1]|Q θ (s,a)[1]) =MSE(V,V′) / w+KL(feature_V|s_feature_V)+KL(s_feature_V,feature_V) Among them, w represents the weighted value of MSE loss, KL loss makes the feature distribution of the value network and the twin value network as similar as possible, so that the model has the ability to represent out-of-distribution, and MSE(V,V′) makes the model robust to out-of-distribution environments or behaviors; the MSE loss function can be expressed as: Where N represents BatchSize; The weighted value w is expressed as: The KL loss is expressed as: In value function learning, uncertainty-aware loss and generalization-enhanced loss are used as a regularization method to reset the value network loss. L critic =L+U(s,a)=(Q(s,a)-y) 2 +U(s,a) Construct Actor loss for the current observation and action L actor =λQ(s,π(s))-(π(s)-a) 2 λ represents the weighting parameter, α is set to 2.5 by default.

8. The allocation algorithm according to claim 1, characterized in that In step S5, when deploying the model obtained in step S4, the collected dataset is used to fine-tune the model of the offline reinforcement learning method; The model is deployed in the port environment, and the central server is used to collect environmental information in real time (including the current operating equipment ID, operating object Oid, equipment type Dy and other data) as input. Then, the Actor network is used to predict the deterministic action a′=π(s) at the next moment, so as to recommend the number of equipment in real time.

Images