Boundary-aware data augmentation method for port resource configuration

Abstract

The application relates to port resource configuration optimization, in particular to a boundary perception data enhancement method for port resource configuration, which is used for solving the problem that the performance of offline reinforcement learning cannot be exerted due to the difficulty in collecting historical interaction data of a port production operation environment and enhancing the generalization performance of port equipment resource configuration. The method steps comprise the following: for the state in each round of historical data, the nearest neighbor state is inquired, an interpolation method is used to generate new round data, boundary perception data enhancement is realized; and the generated new round data is merged with the original historical data to obtain training data for a port resource configuration model. The obtained data can be used for offline training of the port resource configuration model, so that the model can realize the purpose of real-time dynamic global resource configuration of a port and improve the productivity of the port.

Description

Technical Field

[0001] The present invention relates to the field of port resource allocation optimization, and in particular to a boundary perception data enhancement method for port resource allocation. Background Art

[0002] Due to the safety requirements and environmental randomness in the port environment, data collection is difficult, which may lead to poor performance of offline reinforcement learning models under limited data sets.

[0003] To achieve dynamic allocation of port equipment resources in a noisy and highly random environment, current methods include:

[0004] (1) Pre-training method based on massive big data: Collecting relevant historical data of the sequence decision process to pre-train the intelligent agent can improve the generalization of the intelligent agent to a certain extent. Due to the safety requirements of the port environment, historical data on actual production is relatively scarce.

[0005] (2) Simulation-based port equipment resource allocation process: By building a simulation platform, a twin port operating environment is constructed in the simulation environment to simulate the equipment resource allocation process. However, this method is difficult to model and analyze complex resource allocation processes due to the large number of intelligent agents in the port and the complexity of the equipment resource allocation process.

[0006] (3) Dynamic equipment resource configuration based on offline reinforcement learning: Due to the safety and reliability requirements of the port equipment operating environment, it is impossible to interact with the environment in real time, resulting in insufficient exploration capabilities. In addition, the amount of data in unmanned terminals is small, which makes the offline reinforcement learning method unable to exert its performance advantages. Summary of the Invention

[0007] To address the aforementioned issues in existing technologies, the present invention aims to provide a boundary-aware data augmentation algorithm for port resource allocation. This algorithm utilizes a global nearest neighbor search followed by data augmentation, and uses the generated new data for training in offline reinforcement learning. This algorithm achieves real-time dynamic global port resource allocation, thereby improving port productivity. The specific technical solution is as follows.

[0008] First, this case proposes a boundary-aware data enhancement method for port resource allocation, which includes the following steps: for the state in each round constructed based on historical data, query its nearest neighbor state, and use the interpolation method to generate new round data, which is data based on boundary-aware enhancement; wherein, after the boundary-aware enhanced data is merged with the original historical data, it can train the port resource allocation model constructed based on reinforcement learning. The trained model is deployed in the port environment and can perform resource allocation for ship unloading and loading tasks at each berth.

[0009] In one implementation of the above technical solution, the historical data acquisition step includes: collection and preprocessing, and deleting items containing null values ​​and canceled plans.

[0010] In one implementation of the above technical solution, the resource configuration includes configuring the bridge crane, AGV, rail crane and container area.

[0011] In one embodiment of the above technical solution, for each state in the historical data, query its nearest neighbor state, use the interpolation method to generate new round data, and implement boundary-aware data enhancement. The steps include: using the Chebyshev distance to measure the current t corresponding to the state s in the round t and recent status The Chebyshev distance d between t If the Chebyshev distance is between the set maximum value and the set minimum value, then b t The value is equal to d t Otherwise, make b t The value of is equal to the set maximum value; based on b t Perform uniform distribution sampling to obtain data fluctuation intensity μ t ; Based on the data fluctuation intensity μ t , generate new mixed data

[0012] in: For the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, and γ is the reward discount factor.

[0013] Secondly, this case proposes a port resource allocation method, which merges the data based on boundary perception enhancement obtained by any of the above methods with the original historical data to obtain training data for the port resource allocation model; uses the obtained training data to train the port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0014] On the third aspect, according to the above-mentioned method and technical solution, this case proposes a boundary perception data enhancement system for port resource allocation, which includes a boundary perception module. The boundary perception module is configured to query the nearest neighbor state of the state in each round constructed based on historical data, and use an interpolation method to generate new round data, which is data based on boundary perception enhancement; wherein, after the boundary perception enhanced data is merged with the original historical data, it can train the port resource allocation model constructed based on reinforcement learning. The trained model is deployed in the port environment, and can perform resource allocation for ship unloading and loading tasks at each berth.

[0015] In the above system technical solution, the boundary perception module includes a distance calculation unit and a generation unit; the distance calculation unit is configured to use Chebyshev distance to measure the state s in the current round t t and recent status The Chebyshev distance d between t The generating unit is configured based on d t Determine b t , then based on b t Perform uniform distribution sampling to obtain data fluctuation intensity μ t ; Then based on the data fluctuation intensity μ t , generate new mixed data

[0016]

[0017] in: For the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, γ is the reward discount factor; the d-based t Determine b t When the Chebyshev distance is between the set maximum value and the set minimum value, b t The value is equal to d t Otherwise, make b t The value is equal to the maximum value set.

[0018] Fourthly, this case proposes a port resource allocation system, which uses the boundary perception enhanced data obtained by any of the above-mentioned systems, merges it with the original historical data, and obtains training data for the port resource allocation model; uses the obtained training data to train the port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0019] In a fifth aspect, the present application provides a computer readable storage medium storing a computer program capable of being loaded and executed by a processor to perform any of the above methods.

[0020] Compared with the prior art, the present application has the beneficial technical effects of:

[0021] 1. The present application uses an improved data enhancement method for data augmentation, thereby maintaining performance at low data volume and enhancing the generalization performance of port equipment resource allocation.

[0022] 2. Compared with the method of resource allocation algorithm based on online reinforcement learning, the present application avoids the problem of difficulty in simulation caused by many variables in the real running environment of the port. Moreover, online reinforcement learning requires massive data interaction and consumes a large amount of resources for real-time data collection. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 、 One In one embodiment, the flowchart of the boundary perception data enhancement method for port resource allocation is shown.

[0025] Figure 2 、 One In one embodiment, the mixed data distribution comparison diagram generated by the boundary perception data enhancement and Mixup method for port resource allocation is shown.

[0026] Figure 3 、 One In one embodiment, the offline reinforcement learning training method based on editing perception data enhancement is shown.

[0027] Figure 4 In one embodiment, the action diagram of the inference execution of the trained Actor is shown. DETAILED DESCRIPTION

[0028] In order to overcome the simulation difficulties caused by the large number of variables in the real port operation environment and enhance the generalization performance of port equipment resource allocation, this case proposes a boundary-aware data enhancement method for port resource allocation, including preprocessing of historical port equipment scheduling data, constructing the state space and action space of the unmanned terminal and setting rewards for each time step, and then using an improved offline reinforcement learning method for adversarial training. The historical port equipment scheduling data used in training is the historical scheduling data of the operation of various bridge cranes, AGVs, rail cranes and container area equipment. Bridge cranes, AGVs, rail cranes and container areas are the four resources required for ship unloading and loading tasks at port berths. The method can predict the ratio of the four resources for ship unloading and loading tasks at each berth when deploying a port resource allocation model in a port environment.

[0029] The following, combined with the accompanying drawings, provides a clear and complete description of how to implement the boundary-aware data enhancement method for port resource allocation. Obviously, the described implementations represent only a portion of the implementations of this case, not all of them. Based on the implementations in this case, all other implementations derived by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application.

[0030] like Figure 1 A boundary-aware data enhancement method for port resource configuration is shown, and the steps are as follows.

[0031] Step 1: Collect and preprocess historical data on port resource allocation, and delete items with null values ​​and cancelled plans.

[0032] Log records of the port's global equipment resource allocation were collected for six months, including ship logs and equipment resource configuration logs. Data containing null values ​​were padded, and data with canceled operation plans was deleted to obtain a clean dataset. Following a demand survey on the global equipment resource allocation at Shanghai Port, ship operation logs were analyzed, and the various attributes of the operation tasks were summarized and analyzed. Twelve ship attributes (berth, ship type, length, ship width, number of ordinary containers remaining in the loading task, number of refrigerated containers remaining in the loading task, number of oversized containers remaining in the loading task, number of hazardous materials containers remaining in the loading task, number of ordinary containers remaining in the unloading task, number of refrigerated containers remaining in the unloading task, number of oversized containers remaining in the unloading task, and number of hazardous materials containers remaining in the unloading task) were set as the state space for reinforcement learning. The equipment operation log file records the equipment scheduling of container loading and unloading, including the equipment ID of the transshipment container (high level, empty weight, and container status), the start time of the equipment operation, and the end time of the operation. The port's global resources consist of quarry cranes (QC), automated guided vehicles (AGVs), yard cranes (YC), and yard blocks (YBs). The purpose of data augmentation is to implement port resource allocation based on offline reinforcement learning with low data volume, thereby maximizing resource utilization and minimizing ship operation time.

[0033] Step 2: Build a unified action space and state space for all equipment in the port, and set rewards based on minimizing container turnaround time.

[0034] First, we divide the time into 6 minutes according to the start and end time of the operation. According to the resource allocation requirements of the port, the time is divided into 6 minutes, that is, the timestamp is a 6-minute interval to obtain the number of running equipment of each type, the number of remaining containers, the status of the ship, etc., in order to generate a standard offline reinforcement learning dataset. Construct a standard offline reinforcement learning dataset, including the following fields: observations, next _ observations, actions, rewards, terminals, where observations represents the current observation state, next _ Observations represent the observed state at the next moment, actions represent the current state, rewards represent immediate rewards, and terminal s represents whether the current environment state is a completed state.

[0035] The observation space S is:

[0036]

[0037] where the superscript 1 represents the berth id, the number of berths is i, st 1 represents the ship type of berth id 1, sl 1 represents the ship length of berth id 1, sw 1 represents the ship width of berth id 1, represents the number of general containers of the unloading task, represents the number of refrigerated containers of the unloading task, represents the number of dangerous goods containers of the unloading task, represents the number of oversized containers of the unloading task, represents the number of general containers of the loading task, represents the number of refrigerated containers of the loading task, represents the number of dangerous goods containers of the loading task, represents the number of oversized containers of the loading task.

[0038] The action space A of each berth is:

[0039]

[0040] The observation space dimension of each berth is 11, and there are 7 berths in total, so the total observation space dimension is 77. Among them, represents the number of bridge cranes in the unloading task, represents the number of AGVs in the unloading task, represents the number of rail cranes in the unloading task, represents the number of container areas in the unloading task, represents the number of bridge cranes in the loading task, represents the number of AGVs in the loading task, represents the number of rail cranes in the loading task, represents the number of container areas in the loading task.

[0041] The action space dimension of each berth is 8, and there are 7 berths in total, so the total action space dimension is 56. The goal of port resource allocation is to minimize the transfer time of equipment and minimize the idle rate of equipment, and the construction method of reward space uses dense rewards.

[0042] To minimize equipment transit time, each piece of pre-processed data needs to be associated with an operating vessel at the port. All vessels of the same type within a time slice must not exceed the maximum transit time threshold, otherwise a penalty is applied using a certain weight. In one implementation, the percentage of containers operated by the equipment within the current time interval (number of operated containers / number of containers on the current vessel) is calculated. If the time interval between the completion of the unloading and loading tasks (end operation time and start operation time) is less than 10 minutes, no penalty is applied. Otherwise, a penalty weighted at -0.1 is applied for each time the time interval is exceeded (for example, if the gantry crane's operating time exceeds 10 minutes, the penalty is -0.1 / (time interval - 10 minutes)).

[0043] To minimize equipment idleness The idle rate of the same type of equipment in a time slice does not exceed the maximum idle rate threshold, otherwise a certain weight is used for punishment. For example, the idle rate thresholds for bridge cranes, AGVs and rail cranes are 0.3, 0.5 and 0.4, and no punishment is imposed on the idle rate of the container area. For bridge cranes, AGVs and rail cranes, if the idle rate threshold is exceeded, a penalty of -0.08 weight is imposed (for example, if the idle rate of a bridge crane exceeds 0.3, the penalty is -0.08*(idle rate -0.3)). The value of terminal is (True / False), and terminal indicates whether the current ship has completed the operation. Terminal will be True only after both tasks are completed.

[0044] Step 3: For each state in an episode, query its nearest neighbor state and use interpolation to generate new episode data. The episode is a tuple of reinforcement learning.

[0045] Figure 2 This diagram shows a comparison of data generated using boundary-aware data augmentation and mixed data generated using Mixup. Mixup uses two random rounds of data for interpolation, resulting in mixed data with a large distribution deviation from the original data. However, boundary-aware data augmentation methods can generate mixed data that conforms to the original data distribution.

[0046] In one implementation, the obtained offline data set is traversed through the states of each round, the Chebyshev distance is used to query the most recent state of each round, and then the data of these two rounds are interpolated to obtain new data to expand the original data.

[0047] Specifically, offline data is represented as a state transition quintuple (s t , a t , s t+1 , r t ,γ), where s t∈S represents the current observation state, a t ∈A represents taking an action for the current observation, s t+1 represents the state at the next time, r t represents the immediate reward, and γ represents the reward discount factor; the distance between states s t in each round is measured using the Chebyshev distance, and the nearest state is then found can be represented as:

[0048]

[0049] The Chebyshev distance d between the two nearest states is then calculated t , which can be represented as:

[0050]

[0051] To reduce the distribution shift caused by data augmentation, a threshold β is set to further constrain d t , which can be represented as:

[0052] b t = clip(d t , min = 0, max = β),

[0053] where the clip function is used to constrain the maximum and minimum values of d t , so that d t does not exceed the range with a minimum value of 0 and a maximum value of β. In one embodiment, when d t ∈ [0, β], the clip function returns d t ; when d t ∈ [0, β], the clip function returns β. Exemplarily, β is preferably set to 0.2.

[0054] Further, b t is set to conform to a uniform distribution, and the data fluctuation intensity μ t is obtained by sampling using the uniform distribution, which can be represented as:

[0055]

[0056] Finally, based on a certain data fluctuation intensity μ t , new mixed data is generated, which can be represented as

[0057]

[0058] where: is the generated state, is the generated action, Rt+1=∑t+1Tγt+1T Rt+1=∑t+1Tγt+1T

[0059] The trained agent is tested in the simulation environment until the unloading task of a ship is completed, indicating the end of the task of the ship.

[0060] Step 4: Merge the new data generated by using boundary perception-based data augmentation with the original historical data, and train using the Actor-Critic algorithm.

[0061] Merge the mixed data with the original data and train using the Actor-Critic algorithm.

[0062] First, set the learning target y of the value function Q φ (s t , a t ) according to the Bellman operator (φ represents the parameters), which can be expressed as:

[0063]

[0064] where r represents the reward value of a single step, and a t+1 represents the action at the next time.

[0065] The Bellman loss L Q can be expressed as:

[0066] L Q = (Q(s t , a t ) - y) 2

[0067] After training the Critic network, the Actor network π θ (θ represents the parameters) is trained, and the Actor returns the predicted result π(s t ) of the next step, and the weighted Q prediction is used to guide the learning of the Actor network. The loss function can be constructed as follows:

[0068] L actor = λQ(s t , π(s t )) - (π(s t ) - a t ) 2

[0069] where λ represents the weighting parameter, α is set to 2.5 by default, and B represents BatchSize.

[0070] Step 5: Deploy the port resource allocation model in the port environment and predict the allocation ratio of the four resources for the ship unloading and loading tasks at each berth.

[0071] The model is deployed in the port environment, and the central server is used to collect environmental information in real time as input, and then the deterministic action π(s t ), thereby realizing the function of real-time resource configuration.

[0072] In summary, those skilled in the art can easily understand the boundary-aware data enhancement method for port resource allocation disclosed in the present invention, which includes: querying the nearest neighbor state for the state in each round constructed based on historical data, and using an interpolation method to generate new round data, which is data based on boundary-aware enhancement; wherein, after the boundary-aware enhanced data is merged with the original historical data, it can train the port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0073] At the same time, those skilled in the art may also understand a port resource allocation method disclosed in the present invention, which uses the boundary perception enhancement-based data obtained by the above method, merges the boundary perception enhancement-based data with the original historical data, and obtains training data for the port resource allocation model; uses the obtained training data to train the port resource allocation model constructed based on reinforcement learning, and the trained model is deployed in the port environment to perform resource allocation for ship unloading and loading tasks at each berth.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that the corresponding system can be realized according to the method of the present disclosure. Exemplarily, a boundary-aware data enhancement system for port resource configuration includes a boundary-aware module configured to query the nearest neighbor state of the state in each round constructed based on historical data, generate new round data using an interpolation method, and obtain boundary-aware enhanced data; wherein the boundary-aware enhanced data can train a port resource configuration model constructed based on reinforcement learning after being combined with original historical data, and the trained model is deployed in a port environment to configure resources for ship unloading and loading tasks of each berth. Exemplarily, a port resource configuration system merges the boundary-aware enhanced data obtained by the boundary-aware data enhancement system for port resource configuration with original historical data to obtain training data for a port resource configuration model; uses the obtained training data to train a port resource configuration model constructed based on reinforcement learning, and the trained model is deployed in a port environment to configure resources for ship unloading and loading tasks of each berth.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the method or system of the present disclosure can be realized by means of software and necessary general hardware, of course, it can also be realized by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily realized by corresponding hardware, and specific hardware structures for realizing the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better embodiment.

[0076] Although the embodiments of the present application are described above in combination with the drawings, the present application is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and guiding, but not limiting. Those skilled in the art can make many forms under the guidance of the present disclosure and without departing from the scope protected by the claims of the present application, which all belong to the protection of the present application.

Claims

1. A boundary perception data enhancement method for port resource allocation, characterized in that: The method comprises the following steps: Collect and pre-process historical data on port resource allocation, deleting items with empty values ​​and cancelled plans; Build a unified action space and state space for all equipment in the port, and set rewards based on minimizing container turnaround time; For each state in the round constructed based on historical data, query its nearest neighbor state and use the interpolation method to generate new round data based on boundary-aware enhanced data. The steps include: Use Chebyshev distance to measure the state in the current round t and recent status Chebyshev distance between ; If the Chebyshev distance is between the set maximum value and the set minimum value, then The value is equal to , otherwise The value of is equal to the set maximum value; based on Perform uniform distribution sampling to obtain data fluctuation intensity ; Based on the intensity of data fluctuations , generate new mixed data ( , , ): in: For the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, is the reward discount factor; Among them, after the boundary perception enhanced data is merged with the original historical data, it can be used to train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can perform resource allocation for ship unloading and loading tasks at each berth.

2. The method according to claim 1, characterized in that The resource configuration includes the configuration of bridge cranes, AGVs, rail cranes and container areas.

3. A port resource allocation method, characterized in that: The method uses the data based on boundary perception enhancement obtained by the method of any one of claims 1-2, and merges it with the original historical data to obtain training data for the port resource allocation model; The obtained training data is used to train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can allocate resources for ship unloading and loading tasks at each berth.

4. A boundary perception data enhancement system for port resource allocation, characterized by: The system includes a boundary perception module configured to collect and preprocess historical data on port resource configurations, removing entries containing null values ​​and plan cancellations; construct a unified action space and state space for all equipment in the port, and set rewards based on minimizing container turnaround time; for each state in a round constructed based on historical data, query its nearest neighbor state and use an interpolation method to generate new round data, which is data enhanced based on boundary perception; The boundary perception module includes a distance calculation unit and a generation unit; The distance calculation unit is configured to use Chebyshev distance to measure the state in the current round t and recent status Chebyshev distance between ; The generating unit is configured based on Sure , and then based on Perform uniform distribution sampling to obtain data fluctuation intensity ; Based on the intensity of data fluctuations , generate new mixed data ( , , ): in: For the generated state, For the generated action, is the state generated at time t+1, is the reward generated at time t, is the reward discount factor; Sure When the Chebyshev distance is between the set maximum value and the set minimum value, The value is equal to , otherwise The value of is equal to the set maximum value; Among them, after the boundary perception enhanced data is merged with the original historical data, it can be used to train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can perform resource allocation for ship unloading and loading tasks at each berth.

5. The system according to claim 4, characterized in that: The resource configuration includes the configuration of bridge cranes, AGVs, rail cranes and container areas.

6. A port resource allocation system, characterized by: The system uses the data based on boundary perception enhancement obtained by the system of any one of claims 4-5, and merges it with the original historical data to obtain training data for the port resource allocation model; The obtained training data is used to train the port resource allocation model built based on reinforcement learning. The trained model is deployed in the port environment and can allocate resources for ship unloading and loading tasks at each berth.

7. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 2.

Images