Boundary perception data enhancement method for port resource allocation

Abstract

The invention relates to port resource allocation optimization, in particular to a boundary perception data enhancement method for port resource allocation, which is used for solving the problem that the performance of off-line reinforcement learning cannot be exerted due to the fact that historical interaction data of a port production operation environment is difficult to collect, and enhancing the generalization performance of port equipment resource allocation. The method comprises the following steps of: querying a nearest neighbor state for a state in each round in historical data, generating new round data by using an interpolation method, and realizing boundary perception data enhancement; and combining the generated new round data with the original historical data to obtain training data for the port resource allocation model. The obtained data can be used for offline training of a port resource allocation model, so that the model can achieve the purpose of real-time dynamic port global resource allocation, and the port productivity is improved.

Description

Technical Field

[0001] The present invention relates to the field of port resource allocation optimization, and in particular to a boundary perception data enhancement method for port resource allocation. Background Art

[0002] Due to the safety requirements and environmental randomness in the port environment, data collection is difficult, which may lead to poor performance of offline reinforcement learning models under limited data sets.

[0003] To achieve dynamic allocation of port equipment resources in a noisy and highly random environment, current methods include:

[0004] (1) Pre-training method based on massive big data: Collecting relevant historical data of the sequence decision process to pre-train the intelligent agent can improve the generalization of the intelligent agent to a certain extent. Due to the safety requirements of the port environment, historical data on actual production is relatively scarce.

[0005] (2) Simulation-based port equipment resource allocation process: By building a simulation platform, a twin port operation environment is constructed in the simulation environment to simulate the equipment resource allocation process. However, this method is difficult to model and analyze complex resource allocation processes due to the large number of intelligent agents in the port and the complexity of the equipment resource allocation process.

[0006] (3) Dynamic equipment resource configuration based on offline reinforcement learning: Due to the safety and reliability requirements of the port equipment operating environment, it is impossible to interact with the environment in real time, resulting in insufficient exploration capabilities. In addition, the amount of data in unmanned terminals is small, resulting in the offline reinforcement learning method not being able to exert its performance advantages. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, the purpose of the present invention is to provide a boundary-aware data enhancement algorithm for port resource allocation. By using global nearest neighbor search and then data expansion, the generated new data is used for training in offline reinforcement learning to achieve the purpose of real-time dynamic port global resource allocation and improve port productivity. The specific technical solution is as follows.

[0008] On the first aspect, this case proposes a boundary-aware data enhancement method for port resource allocation, which includes the following steps: for the state in each round constructed based on historical data, query its nearest neighbor state, and use the interpolation method to generate new round data, which is data based on boundary-aware enhancement; wherein, after the boundary-aware enhanced data is merged with the original historical data, it can train the port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0009] In one implementation of the above technical solution, the historical data acquisition step includes: collection and preprocessing, and deleting items containing null values ​​and canceled plans.

[0010] In one implementation of the above technical solution, the resource configuration includes configuring the bridge crane, AGV, rail crane and container area.

[0011] In one implementation of the above technical solution, for each state in the historical data, query its nearest neighbor state, use the interpolation method to generate new round data, and implement boundary-aware data enhancement. The steps include: using the Chebyshev distance to measure the current t corresponding to the state s in the round t and recent status The Chebyshev distance d between t If the Chebyshev distance is between the set maximum value and the set minimum value, then b t The value is equal to d t , otherwise b t The value of is equal to the set maximum value; based on b t Perform uniform distribution sampling to obtain the data fluctuation intensity μ t ; Based on the data fluctuation intensity μ t , generate new mixed data

[0012] in: is the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, and γ is the reward discount factor.

[0013] Secondly, this case proposes a port resource allocation method, which merges the data based on boundary perception enhancement obtained by any of the above methods with the original historical data to obtain training data for the port resource allocation model; uses the obtained training data to train the port resource allocation model built based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0014] On the third aspect, according to the above-mentioned method and technical scheme, this case proposes a boundary-aware data enhancement system for port resource allocation, the system includes a boundary-aware module, and the boundary-aware module is configured to query the nearest neighbor state in each round constructed based on historical data, and use an interpolation method to generate new round data, which is data based on boundary-aware enhancement; wherein, after the boundary-aware enhanced data is merged with the original historical data, it can train the port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in the port environment, and can perform resource allocation for ship unloading and loading tasks at each berth.

[0015] In the above system technical solution, the boundary perception module includes a distance calculation unit and a generation unit; the distance calculation unit is configured to use the Chebyshev distance to measure the state s in the current round corresponding to t t and recent status The Chebyshev distance d between t The generating unit is configured based on d t Determine b t , then based on b t Perform uniform distribution sampling to obtain the data fluctuation intensity μ t ; Then based on the data fluctuation intensity μ t , generate new mixed data

[0016]

[0017] in: is the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, γ is the reward discount factor; t Determine b t When the Chebyshev distance is between the set maximum value and the set minimum value, b t The value is equal to d t , otherwise b t The value is equal to the maximum value set.

[0018] Fourthly, this case proposes a port resource allocation system, which uses the data based on boundary perception enhancement obtained by any of the above-mentioned systems, merges it with the original historical data, and obtains training data for the port resource allocation model; uses the obtained training data to train the port resource allocation model built based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0019] In a fifth aspect, the present case proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the above methods.

[0020] Compared with the prior art, the beneficial technical effects of this case are:

[0021] 1. This scheme uses an improved data enhancement method to perform data amplification, thereby maintaining performance under low data volume and enhancing the generalization performance of port equipment resource configuration.

[0022] 2. Compared with the resource allocation algorithm based on online reinforcement learning, this solution avoids the problem of simulation difficulties caused by the large number of variables in the real operating environment of the port. In addition, online reinforcement learning requires massive data interaction and consumes a lot of resources to collect real-time data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 , one A flow chart of a boundary perception data enhancement method for port resource configuration in an implementation manner.

[0025] Figure 2 , one Schematic diagram of the comparison of hybrid data distribution generated by boundary-aware data enhancement and Mixup method for port resource allocation in one implementation method.

[0026] Figure 3 , one Schematic diagram of an offline reinforcement learning training method based on edit-aware data enhancement in one implementation.

[0027] Figure 4 . Schematic diagram of actions executed by reasoning of a trained Actor in one implementation. DETAILED DESCRIPTION

[0028] In order to overcome the simulation difficulties caused by the large number of variables in the real port operation environment and enhance the generalization performance of port equipment resource configuration, this case proposes a boundary-aware data enhancement method for port resource configuration, including preprocessing the historical port equipment scheduling data, constructing the state space and action space of the unmanned terminal and setting rewards for each time step, and then using the improved offline reinforcement learning method for adversarial training. The historical port equipment scheduling data used in the training is the historical scheduling data of the operation of each bridge crane, AGV, rail crane and container area equipment. The bridge crane, AGV, rail crane and container area are the four resources required for the ship unloading and loading tasks of the port berth. The method can predict the ratio of the four resources for the ship unloading and loading tasks of each berth for deploying the port resource configuration model in the port environment.

[0029] The following will be combined with the accompanying drawings to clearly and completely describe how to implement the boundary perception data enhancement method for port resource configuration. Obviously, the described implementation method is only a part of the implementation method of this case, not all of the implementation methods. Based on the implementation method in this case, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] like Figure 1 A boundary-aware data enhancement method for port resource configuration is shown, and the steps are as follows.

[0031] Step 1: Collect and preprocess historical data of port resource configuration, and delete items containing null values ​​and cancelled plans.

[0032] The log records of the global equipment resource allocation of the port for half a year were collected, including ship logs and equipment resource configuration logs. Data with null values ​​were filled, and the data of the cancelled operation plan was deleted to obtain a clean data set. After a demand survey on the global equipment resource allocation of Shanghai Port, the ship operation logs were analyzed, and the various attributes of the operation tasks were summarized and analyzed. Among them, 12 attributes of the ship (ship berth, ship type, ship length, ship width, number of ordinary containers remaining in the loading task, number of refrigerated containers remaining in the loading task, number of oversized containers remaining in the loading task, number of dangerous goods containers remaining in the loading task, number of ordinary containers remaining in the unloading task, number of refrigerated containers remaining in the unloading task, number of oversized containers remaining in the unloading task, number of dangerous goods containers remaining in the unloading task) were set as the state space of reinforcement learning. The equipment operation log file records the equipment scheduling container loading and unloading: including the equipment ID of the transshipment container (high level, empty weight, box status), the equipment start operation time, and the end operation time. The global resources of the port are composed of quarry cranes (QC), automated guided vehicles (AGV), yard cranes (YC) and yard blocks (YB). The purpose of data enhancement is to realize port resource configuration based on offline reinforcement learning with low data volume, so as to maximize resource utilization and minimize ship operation time.

[0033] Step 2: Build a unified action space and state space for all devices in the port, and set rewards based on minimizing container turnaround time.

[0034] First, we divide the time into 6 minutes according to the start and end time of the operation. According to the resource allocation requirements of the port, the time is divided into 6 minutes, that is, the timestamp is a 6-minute interval to obtain the number of operating equipment of various types, the number of remaining containers, the status of the ship, etc., in order to generate a standard offline reinforcement learning dataset. Construct a standard offline reinforcement learning dataset, including the following fields observations, next _ observations, actions, rewards, terminals, where observations represents the current observation state, next _ Observations represent the observed state at the next moment, actions represent the current state, rewards represent immediate rewards, and terminals represent whether the current environment state is a completed state.

[0035] The observation space S is:

[0036]

[0037] Among them, the superscript 1 represents the berth id, the berth number is i, and the st 1 Indicates the type of ship with berth id 1, sl 1 Indicates the length of the ship with berth ID 1, sw 1 Indicates the width of the ship at berth id 1. Indicates the number of common boxes for unloading tasks, represents the number of refrigerated containers for unloading tasks, Indicates the number of dangerous goods containers for unloading tasks, Indicates the number of oversized containers for unloading tasks, Indicates the number of common boxes for the shipment task, Indicates the number of refrigerated containers for the loading task, Indicates the number of dangerous goods containers for the loading task. Indicates the number of oversized containers in the loading task.

[0038] The action space A of each berth is:

[0039]

[0040] The observation space dimension of each berth is 11, and there are 7 berths in total, so the total observation space dimension is 77. Indicates the number of bridge cranes in the ship unloading task, Indicates the number of AGVs in the unloading task, Indicates the number of rail cranes in the ship unloading task, Indicates the number of container areas in the unloading task, Indicates the number of bridge cranes in the loading task, Indicates the number of AGVs in the loading task, Indicates the number of rail cranes in the loading task, Indicates the number of container areas in the loading task.

[0041] The action space dimension of each berth is 8, there are 7 berths in total, and the total action space dimension is 56. The goal of port resource allocation is to minimize the transit time of equipment and minimize the idle rate of equipment. The reward space is constructed using dense rewards.

[0042] In order to minimize the equipment transit time, each pre-processed data needs to be associated with the operating ship of the port, and all the same type of a time slice does not exceed the maximum transit time threshold, otherwise a certain weight is used for penalty. In one embodiment, the percentage of containers operated by the equipment in the current time interval (the number of containers operated / the number of containers on the current ship) is set. If the time interval for completing the unloading and loading tasks of the equipment (the end operation time and the start operation time) is less than 10 minutes, no penalty is imposed, otherwise a penalty with a weight of -0.1 is imposed each time the time interval is exceeded (for example, if the operation time of the bridge crane exceeds 10 minutes, the penalty is -0.1 / (time interval-10 minutes)).

[0043] To minimize the idle rate of the equipment The idle rate of the same type of equipment in a time slice does not exceed the maximum idle rate threshold, otherwise a certain weight is used for penalty. For example, the idle rate thresholds for bridge cranes, AGVs and rail cranes are 0.3, 0.5 and 0.4, and no penalty is imposed on the idle rate of the container area. For bridge cranes, AGVs and rail cranes, if the idle rate threshold is exceeded, a penalty of -0.08 is imposed (for example, if the idle rate of a bridge crane exceeds 0.3, the penalty is -0.08*(idle rate -0.3)). The value of terminal is (True / False), and terminal indicates whether the current ship has completed the operation. Terminal will only be True only after both tasks are completed.

[0044] Step 3: For each state in an episode, query its nearest neighbor state and use the interpolation method to generate new episode data. The episode is a tuple of reinforcement learning.

[0045] Figure 2 The comparison chart of the data obtained by using boundary-aware enhancement and mixed data obtained by Mixup is shown. Mixup uses two rounds of random data for interpolation, and the mixed data has a large distribution deviation from the original data, while the boundary-aware data enhancement method can generate mixed data that conforms to the distribution of the original data.

[0046] In one implementation, the state of each round is traversed for the obtained offline data set, the Chebyshev distance is used to query the most recent state of each round, and then the data of these two rounds are interpolated to obtain new data to expand the original data.

[0047] Specifically, offline data is represented as a state transition quintuple (s t , a t ,s t+1 , r t ,γ), where s t∈S represents the current observation state, a t ∈A means taking action based on the current observation, s t+1 represents the state at the next moment, r t represents the immediate reward, γ represents the reward discount factor; the Chebyshev distance is used to measure the state s in each round t The distance between them, and then find the closest state It can be expressed as:

[0048]

[0049] Then based on the most recent state found Calculate the Chebyshev distance d between the two closest states t , which can be expressed as:

[0050]

[0051] In order to reduce the distribution shift caused by data enhancement, a threshold β is set to further constrain d t , which can be expressed as:

[0052] b t =clip(d t , min = 0, max = β),

[0053] Among them, the clip function is used to d t Make maximum and minimum constraints so that d t The minimum value is 0 and the maximum value is β. t ∈[0,β], the clip function returns d t ; in d t ∈[0, β], the clip function returns β. Exemplarily, β is preferably set to 0.2.

[0054] Further setting b t It conforms to the uniform distribution and uses the uniform distribution sampling to obtain the data fluctuation intensity μ t , which can be expressed as:

[0055]

[0056] Finally, based on a certain data fluctuation intensity μ t , generate new mixed data It can be expressed as

[0057]

[0058] in: is the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, and γ is the reward discount factor. The reward discount factor remains unchanged.

[0059] The trained agent is tested in a simulation environment until the loading and unloading task of a ship is completed, indicating that the mission of the ship is completed.

[0060] Step 4: Merge the new data generated by boundary-aware data augmentation with the original historical data and train it using the Actor-Critical algorithm.

[0061] Merge the blended data with the original data And use the Actor-Critic algorithm for training.

[0062] First, set the value function Q according to the Bellman operator φ (s t , a t )(φ represents the parameter), the learning goal y can be expressed as:

[0063]

[0064] Among them, r represents the reward value of a single step, a t+1 Indicates the action at the next moment.

[0065] Bellman loss L Q It can be expressed as:

[0066] L Q =(Q(s t , a t )-y) 2

[0067] After training the Critic network, train the Actor network π θ (θ represents a parameter), Actor will return the prediction result π(s t ), using weighted Q predictions to guide Actor network learning, the following loss function can be constructed:

[0068] L actor =λQ(s t ,π(s t ))-(π(s t )-a t ) 2

[0069] Among them, λ represents the weighting parameter, α is set to 2.5 by default, and B represents BatchSize.

[0070] Step 5: Deploy the port resource allocation model in the port environment to predict the allocation ratio of the four resources for the ship unloading and loading tasks at each berth.

[0071] The model is deployed in the port environment, and the central server is used to collect environmental information in real time as input, and then the deterministic action π(s t ), thereby realizing the function of real-time resource configuration.

[0072] In summary, those skilled in the art can easily understand the boundary-aware data enhancement method for port resource allocation disclosed in the present invention, which includes the following steps: for the state in each round constructed based on historical data, querying its nearest neighbor state, and using an interpolation method to generate new round data, which is data based on boundary-aware enhancement; wherein, after the boundary-aware enhanced data is merged with the original historical data, it is possible to train a port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in a port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0073] At the same time, those skilled in the art may also understand a port resource configuration method disclosed in the present invention, which uses the data based on boundary perception enhancement obtained by the above method, merges the data based on boundary perception enhancement with the original historical data, and obtains training data for the port resource configuration model; uses the obtained training data to train the port resource configuration model built based on reinforcement learning, and the trained model is deployed in the port environment, so as to perform resource configuration for ship unloading and loading tasks at each berth.

[0074] Through the description of the above implementation methods, the technicians in the relevant field can clearly understand that the corresponding system can be realized according to the disclosed method. Exemplarily, a boundary perception data enhancement system for port resource configuration, the system includes a boundary perception module, the boundary perception module is configured to query the nearest neighbor state of each round constructed based on historical data, and use the interpolation method to generate new round data, which is data based on boundary perception enhancement; wherein, after the boundary perception enhanced data is merged with the original historical data, it can train the port resource configuration model constructed based on reinforcement learning, and the trained model is deployed in the port environment, and it can perform resource configuration for the ship unloading tasks and loading tasks of each berth. Exemplarily, a port resource configuration system, the system uses the boundary perception enhanced data obtained by the boundary perception data enhancement system for port resource configuration, merges it with the original historical data, and obtains training data for the port resource configuration model; uses the obtained training data to train the port resource configuration model constructed based on reinforcement learning, and the trained model is deployed in the port environment, and it can perform resource configuration for the ship unloading tasks and loading tasks of each berth.

[0075] Through the description of the above implementation modes, those skilled in the art can clearly understand that the method or system of the present disclosure can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation mode in most cases.

[0076] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A boundary perception data enhancement method for port resource allocation, characterized in that: The method comprises the following steps: For each state in the round constructed based on historical data, query its nearest neighbor state and use the interpolation method to generate new round data, which is data enhanced based on boundary perception; Among them, after the boundary perception enhanced data is merged with the original historical data, it can train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can perform resource allocation for ship unloading and loading tasks at each berth.

2. The method according to claim 1, characterized in that The historical data acquisition step includes: collection and preprocessing, and deleting items containing null values ​​and canceled plans.

3. The method according to claim 1, characterized in that The resource configuration includes configuration of bridge cranes, AGVs, rail cranes and container areas.

4. The method according to claim 1, characterized in that For each state in the historical data, query its nearest neighbor state, use the interpolation method to generate new round data, and implement boundary-aware data enhancement. The steps include: Use Chebyshev distance to measure the state s in the current round corresponding to the current t t and recent status The Chebyshev distance d between t ; If the Chebyshev distance is between the set maximum value and the set minimum value, then b t The value is equal to d t , otherwise b t The value of is equal to the set maximum value; Based on b t Perform uniform distribution sampling to obtain the data fluctuation intensity μ t ; Based on the data fluctuation intensity μ t , generate new mixed data in: is the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, and γ is the reward discount factor.

5. A port resource allocation method, characterized in that: The method uses the data based on boundary perception enhancement obtained by any of the methods described in claims 1-4, and merges it with the original historical data to obtain training data for the port resource allocation model; The obtained training data is used to train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can allocate resources for ship unloading and loading tasks at each berth.

6. A boundary perception data enhancement system for port resource allocation, characterized in that: The system includes a boundary perception module, which is configured to query the nearest neighbor state of each round state constructed based on historical data, and use an interpolation method to generate new round data, which is data enhanced based on boundary perception; Among them, after the boundary perception enhanced data is merged with the original historical data, it can train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can perform resource allocation for ship unloading and loading tasks at each berth.

7. The system according to claim 1, characterized in that: The resource configuration includes configuration of bridge cranes, AGVs, rail cranes and container areas.

8. The system according to claim 1, characterized in that: The boundary perception module includes a distance calculation unit and a generation unit; The distance calculation unit is configured to use the Chebyshev distance to measure the state s in the current round corresponding to t t and recent status The Chebyshev distance d between t ; The generating unit is configured based on d t Determine b t , then based on b t Perform uniform distribution sampling to obtain the data fluctuation intensity μ t ; Then based on the data fluctuation intensity μ t , generate new mixed data in: is the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, γ is the reward discount factor; t Determine b t When the Chebyshev distance is between the set maximum value and the set minimum value, b t The value is equal to d t , otherwise b t The value is equal to the maximum value set.

9. A port resource allocation system, characterized in that: The system uses the data based on boundary perception enhancement obtained by any system described in claims 6-8, and merges it with the original historical data to obtain training data for the port resource allocation model; The obtained training data is used to train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can allocate resources for ship unloading and loading tasks at each berth.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 4.

Images