Automatic guided vehicle scheduling method of automatic container terminal and related device

Abstract

The invention discloses an automatic guided vehicle scheduling method of an automatic container terminal and a related device. The method comprises the following steps: constructing a layered framework; the upper layer model adopts a Markov decision process model to model the task assignment of the AGV; the lower layer model adopts a heuristic path generation algorithm to carry out path planning on the AGV; constructing a reinforcement learning network; task assignment is carried out according to the system state through a strategy network, and a task assignment action is generated; the lower-layer model adopts a path generation algorithm to generate an optimal path and a reward value for each task assignment action; evaluating an estimated value of a value function of the current strategy according to the reward value through a value network, updating the strategy according to the estimated value of the value function of the current strategy through a strategy network, and obtaining an optimal strategy; wherein the strategy network takes maximization of accumulated rewards as an optimization target; and obtaining a task assignment result and a transportation path of the AGV through the optimal strategy. According to the invention, AGV path conflicts are reduced, and the working efficiency of the AGV is improved.

Description

Technical Field

[0001] This application relates to the field of automated guided vehicle (AGV) scheduling technology, and in particular to an AGV scheduling method and related apparatus for automated container terminals. Background Technology

[0002] Container terminals are crucial cargo transshipment hubs in the global supply chain, handling over 80% of international maritime trade cargo. Faced with increasing demand for container handling and cost pressures, global container terminals are undergoing a transformation towards automated and intelligent terminals. A key characteristic of automated container terminals is the use of Automated Guided Vehicles (AGVs) instead of traditional container trucks. Because AGVs are driverless, their operation differs significantly from that of traditional container trucks.

[0003] In most automated container terminals, AGVs are controlled by two software systems simultaneously: the Terminal Operation System (TOS) and the Equipment Control System (ECS), such as... Figure 1 As shown in (a), the TOS assigns container loading and unloading tasks to AGVs with the goal of completing ship services on time, while the ECS controls the movement of the AGVs and assigns travel paths to them to complete the assigned tasks. This sequential AGV scheduling mechanism has several key drawbacks that can lead to inefficient AGV operations. First, the TOS is unaware of the AGV's traffic environment when assigning tasks; therefore, task assignment decisions generated by the TOS may lead to frequent path conflicts and local congestion between AGVs. Second, because the TOS cannot specify the AGV's path, its estimation of task start and finish times is inaccurate, and task assignment decisions made by the TOS may lead to significant task delays. Third, because the ECS performs tasks passively, its path planning decisions are often short-sighted, potentially leading to inefficient AGV path planning and high path conflicts. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an automated guided vehicle (AGV) scheduling method and related apparatus for automated container terminals.

[0005] In view of this, the first aspect of this application provides a method for scheduling automated guided vehicles (AGVs) in an automated container terminal, comprising:

[0006] A hierarchical architecture consisting of an upper-level model and a lower-level model is constructed. The upper-level model uses a Markov decision process model to model the task assignment of the automated guided vehicle and determine the system state, action space, reward function, and objective function. The lower-level model uses a heuristic path generation algorithm to perform path planning for the automated guided vehicle.

[0007] Construct a reinforcement learning network that includes a policy network and a value network; assign tasks based on the system state through the policy network and generate task assignment actions; use a heuristic path generation algorithm in the lower-level model to generate the optimal path for each task assignment action and calculate the reward value of the optimal path.

[0008] The value network evaluates the estimated value of the current policy's value function based on the reward value, and the policy network updates the policy based on the estimated value of the current policy's value function to obtain the optimal policy; wherein, the policy network has the optimization objective of maximizing the cumulative reward.

[0009] The optimal strategy is used to obtain the task assignment results and transportation routes of the automated guided vehicles.

[0010] Optionally, the system status is... t is the current time point, Let s be the set of tasks that have not yet been served, and let s be the current state vector of the automated guided vehicle. The current state includes the current position, the current battery charge state, the start point and the end point of the current task.

[0011] In system state s, the action space is A(s), where each task in the action space is assigned an action. , The allocation decision for task i;

[0012] The reward function is ,in,

[0013]

[0014] In the formula, To take action for task i under system state s. The reward value obtained; The revenue generated from completing a transportation task; The travel time required for the automated guided vehicle to perform service task i under system state s; The delay amount for the automated guided vehicle to execute task i in system state s; These are the container transportation cost coefficient and the task delay cost coefficient, respectively. The penalty value imposed on actions that cannot be taken;

[0015] The objective function is , Indicates the state s of the system e The following is a random strategy The generated action, This is the initial system state. As the discount factor, Let e ​​be the optimal strategy and 'e' be the decision time. The timeframe for decision-making.

[0016] Optionally, a heuristic path generation algorithm is used for path planning of the automated guided vehicle, including:

[0017] S11. Under the current system state, obtain the main path and road segments of the current task based on the initial position, start point and end point of the current task of the current automated guided vehicle service.

[0018] S12. Obtain the feasible lanes of each road segment, and sort the feasible lanes of each road segment in descending order according to lane priority to obtain the set of feasible lanes of each road segment.

[0019] S13. Select the highest priority driveway from the set of driveways for each road segment to generate a path;

[0020] S14. Generate a trajectory based on the path and the initial time of the current automated guided vehicle (AGV) serving the current task, and determine the arrival time of the current AGV to the starting point of the current task and the completion time of the current task based on the trajectory.

[0021] S15. Detect conflict based on the current trajectory of the automated guided vehicle (AGV) and the trajectories of other AGVs. If a path conflict is detected between the current AGV and other AGVs at the target time, proceed to step S16; if no path conflict is detected between the current AGV and other AGVs, proceed to step S17.

[0022] S16. Obtain the target road segment where the current automated guided vehicle is located at the target time, delete the currently selected feasible lane from the feasible lane set of the target road segment, and obtain the updated feasible lane set of the target road segment; if the updated feasible lane set of the target road segment is empty, it is determined that the current task assignment action is not feasible, and the preset penalty value is returned as the reward value; if the updated feasible lane set of the target road segment is not empty, return to step S13.

[0023] S17. Determine that the current task assignment action is feasible, and calculate the execution time of the current task based on the initial time and completion time of the current task, and calculate the delay based on the arrival time and expected arrival time; return the trajectory, execution time and delay of the current task of the current automated guided vehicle service.

[0024] Optionally, tasks can be assigned based on system state via a policy network, generating task assignment actions. This process may also include:

[0025] S21. Aggregate tasks based on the container dimensions corresponding to each task in the current task set, and obtain the possible routes for each aggregated task.

[0026] S22. Calculate the distance saved by each possible route of each aggregation task relative to the task before aggregation;

[0027] S23. Sort all possible routes for the aggregation tasks in descending order based on the distance saved to obtain a route list;

[0028] S24. Select the possible route that saves the most distance from the route list as the target route, delete the target route from the route list, and delete the remaining possible routes of the aggregated task corresponding to the target route from the route list to obtain the remaining route list.

[0029] S25. Repeatedly select the possible route with the largest distance saving as the target route for the remaining route list until the distance saving of the first route in the remaining route list is 0 or negative, or the remaining route list is empty; generate the current task set based on the aggregated tasks corresponding to all selected target routes and the remaining unaggregated tasks.

[0030] Optionally, the method also includes:

[0031] Construct a two-layer grid map, which consists of an upper coarse-grained grid map and a lower fine-grained grid map;

[0032] The upper coarse-grained grid map is used for task assignment; the grid size of the upper coarse-grained grid map is determined by the width of the container yard area and the width of the horizontal transport area in the container terminal.

[0033] The lower-level fine-grained grid map is used for path planning; the grid size of the lower-level fine-grained grid map is determined according to the speed of the automated guided vehicle.

[0034] A second aspect of this application provides an automated guided vehicle (AGV) scheduling device for an automated container terminal, comprising:

[0035] The first building unit is used to construct a hierarchical architecture consisting of an upper-level model and a lower-level model. The upper-level model uses a Markov decision process model to model the task assignment of the automated guided vehicle and determine the system state, action space, reward function, and objective function. The lower-level model uses a heuristic path generation algorithm to perform path planning for the automated guided vehicle.

[0036] The second building unit is used to construct a reinforcement learning network that includes a policy network and a value network; the policy network assigns tasks according to the system state and generates task assignment actions; the lower-level model uses a heuristic path generation algorithm to generate the best path for each task assignment action and calculate the reward value of the best path.

[0037] The update unit is used to evaluate the estimated value of the current policy's value function based on the reward value through the value network, and update the policy based on the estimated value of the current policy's value function through the policy network to obtain the optimal policy; wherein, the policy network has the optimization objective of maximizing the cumulative reward;

[0038] The scheduling unit is used to obtain the task assignment results and transportation routes of the automated guided vehicles through the optimal strategy.

[0039] Optionally, the system status is... t is the current time point, Let s be the set of tasks that have not yet been served, and let s be the current state vector of the automated guided vehicle. The current state includes the current position, the current battery charge state, the start point and the end point of the current task.

[0040] In system state s, the action space is A(s), where each task in the action space is assigned an action. , The allocation decision for task i;

[0041] The reward function is ,in,

[0042]

[0043] In the formula, To take action for task i under system state s. The reward value obtained; The revenue generated from completing a transportation task; The travel time required for the automated guided vehicle to perform service task i under system state s; The delay amount for the automated guided vehicle to execute task i in system state s; These are the container transportation cost coefficient and the task delay cost coefficient, respectively. The penalty value imposed on actions that cannot be taken;

[0044] The objective function is , Indicates the state s of the system e The following is a random strategy The generated action, This is the initial system state. As the discount factor, Let e ​​be the optimal strategy and 'e' be the decision time. The timeframe for decision-making.

[0045] Optionally, the device also includes:

[0046] The third building unit is used to build a two-layer grid map, which includes an upper coarse-grained grid map and a lower fine-grained grid map.

[0047] The upper coarse-grained grid map is used for task assignment; the grid size of the upper coarse-grained grid map is determined by the width of the container yard area and the width of the horizontal transport area in the container terminal.

[0048] The lower-level fine-grained grid map is used for path planning; the grid size of the lower-level fine-grained grid map is determined according to the speed of the automated guided vehicle.

[0049] A third aspect of this application provides an electronic device, the device including a processor and a memory;

[0050] The memory is used to store program code and transmit the program code to the processor;

[0051] The processor is used to execute the automated guided vehicle scheduling method for any of the first aspects of the automated container terminal according to the instructions in the program code.

[0052] The fourth aspect of this application provides a computer-readable storage medium for storing program code, which, when executed by a processor, implements the automated guided vehicle scheduling method for an automated container terminal as described in any of the first aspects.

[0053] As can be seen from the above technical solutions, this application has the following advantages:

[0054] The automated guided vehicle (AGV) scheduling method for automated container terminals provided in this application integrates the two decision-making processes of AGV task assignment and path planning to form a hierarchical architecture. This architecture optimizes task assignment at the upper layer and path planning at the lower layer. Reinforcement learning is used to enhance the system's ability to assign AGV tasks, and the path generation algorithm is embedded into the reinforcement learning algorithm. The two-layer architecture enables the AGV path planning model to have a feedback mechanism. When there are many path conflicts, the task assignment scheme can be adjusted, fundamentally avoiding AGVs from going to areas with frequent conflicts, thereby avoiding conflicts and deadlocks, significantly reducing path conflicts, and improving AGV working efficiency.

[0055] Furthermore, this application proposes two acceleration strategies: a two-layer mesh map and a container matching method. The container matching method aggregates transportation tasks, increasing the chances of AGVs performing dual-container transport, thereby shortening the AGV's travel distance, reducing the number of transportation tasks, and further improving AGV efficiency. Constructing a two-layer mesh map allows the upper layer to use a larger-granularity mesh to aggregate the AGV's position hierarchically, reducing the state space, decreasing the complexity of the task assignment MDP model, improving the learning and generalization ability of the reinforcement learning algorithm, and thus enhancing task assignment capability. The lower layer uses a smaller-granularity mesh to simulate precise AGV positions, improving the accuracy of conflict detection between AGVs and further reducing path conflicts. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A schematic diagram of the existing AGV scheduling method and the improved AGV scheduling method proposed in this application, provided for the embodiments of this application;

[0058] Figure 2 A flowchart illustrating an automated guided vehicle (AGV) scheduling method for an automated container terminal, as provided in an embodiment of this application.

[0059] Figure 3 This is a schematic diagram of the layout of an automated container terminal provided in an embodiment of this application;

[0060] Figure 4 This is a schematic diagram of the AGV path selection process provided in the embodiments of this application;

[0061] Figure 5 This is a schematic diagram of AGV trajectory recording provided in an embodiment of this application;

[0062] Figure 6 A pseudocode diagram illustrating the path generation algorithm provided in the embodiments of this application;

[0063] Figure 7 A schematic diagram of the hierarchical architecture for dynamic AGV scheduling provided in the embodiments of this application;

[0064] Figure 8 This is a schematic diagram of a two-layer grid map of the wharf layout provided in an embodiment of this application;

[0065] Figure 9This is a schematic diagram of an automated guided vehicle (AGV) scheduling device for an automated container terminal, provided as an embodiment of this application. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0067] To address the shortcomings of current automated guided vehicle (AGV) scheduling methods and improve operational efficiency, this application proposes an integrated AGV scheduling method. This method combines task assignment with path planning, coordinating the terminal operating system and equipment control system, such as... Figure 1 As shown in (b). It should be noted that this integrated decision-making approach can be implemented without changing the existing TOS and ECS functions, simply by coordinating the two systems through the construction of middleware.

[0068] For easier understanding, please refer to Figure 2 This application provides a method for scheduling automated guided vehicles (AGVs) in an automated container terminal, including:

[0069] Step 110: Construct a layered architecture consisting of an upper-level model and a lower-level model; the upper-level model uses a Markov decision process model to model the task assignment of the automated guided vehicle, and determines the system state, action space, reward function, and objective function; the lower-level model uses a heuristic path generation algorithm to perform path planning for the automated guided vehicle.

[0070] This application employs a layered architecture to implement automated guided vehicle (AGV) scheduling in automated container terminals. The layered architecture includes an upper-layer model and a lower-layer model. The upper-layer model uses a Markov decision process model to model AGV task assignment, while the lower-layer model uses a heuristic path generation algorithm to plan paths for the AGVs. The entire architecture is built within a reinforcement learning algorithm environment, achieving coupling and unification between the two sub-models.

[0071] When a vessel is operating at an automated container terminal, containers are loaded onto or unloaded by multiple quay cranes (QCs) assigned to that vessel. For loading tasks, a container needs to be transported from the yard (YB) to a QC; for unloading tasks, a container needs to be transported from a quay crane to the yard. Container transport tasks are performed by a set of AGVs assigned to the vessel. Because tasks are dynamically issued, newly arriving tasks need to be assigned to AGVs in real time.

[0072] AGV task assignment needs to follow some practical operational requirements, as follows:

[0073] (1) Time window requirements. Each container loading or unloading request has a time window within which the container should be loaded or unloaded by the designated QC. Therefore, the AGV should arrive at the QC within the time window specified for the loading and unloading task. Failure to arrive within the time window will result in task delays and reduce the service level of the vessel.

[0074] (2) Return to Position Requirements. Once an AGV has completed its task, it should return to its parking space in the buffer zone if there are no subsequent tasks requiring its service. Remaining in the traffic lanes would obstruct traffic and is therefore prohibited. The buffer zone is located between the seaside and landside lanes and is large enough to accommodate AGVs.

[0075] (3) Battery capacity limitation. AGVs are powered by lithium batteries. When the State of Charge (SOC) of an AGV's battery drops below a preset threshold, the AGV must immediately proceed to a battery swapping station to replenish its battery after completing its current task. Typically, the battery swapping station is located at one end of the landside lane, where the AGV can replace its current battery with a fully charged one. Battery swapping requests are modeled as a special type of task for each AGV.

[0076] (4) Transport capacity limitations. Containers are divided into two types according to their size: 20-foot containers and 40-foot containers. An AGV can transport one 40-foot container, one 20-foot container, or two 20-foot containers. Therefore, if each task requires the transport of one 20-foot container, the two tasks can be assigned to one AGV.

[0077] Since tasks arrive dynamically, static mixed-integer programming models are unsuitable for real-time task assignment. This application proposes a Markov Decision Process (MDP) model to capture system dynamics and optimize task assignment decisions.

[0078] We consider a time range [0, T] containing a set of decision points. Every moment Corresponding to time point t ∈ [0, T], an event occurs at this time point and triggers a task assignment decision. Here, the event can be the arrival of a new task or the completion of a task. At each time point, the system state is defined as follows: t is the current time point, For the set of tasks (transportation tasks) that have not yet been served. Let be the current state vector of the automated guided vehicle (AGV). For each AGV k∈ The state of the automated guided vehicle can be represented as:

[0079]

[0080] in, These are the current position of the automated guided vehicle k, the current battery charge state, the remaining time before completing the current task, the start point of the current task, and the end point of the current task, respectively.

[0081] In system state s, there is a set of actions A(s) available for task assignment, each task assignment action... Defined as:

[0082]

[0083] Where, for each task i∈ , It is the allocation decision for task i. This refers to the number of AGVs. Specifically, This indicates that task i at the current moment has not been assigned to any AGV, therefore it will be considered in the next moment; while This indicates that task i is assigned to the automated guided vehicle k. It should be noted that this application allows the action space A(s) to include non-actionable actions. (Regarding...) If there is no conflict-free path when the AGV executes a task, then the task assignment action will be... This is not feasible. Selecting an inactive action for a task will result in a negative reward.

[0084] In addition to performing assigned transportation tasks, AGVs also need to perform battery replacement and return to their designated positions when necessary. Since battery replacement and return are mandatory actions, and these actions are unique to each AGV, these two special types of actions are enforced during state transitions. After the battery replacement and return actions are completed, the AGV's state is updated based on these actions. The battery replacement and return actions are described below:

[0085] Battery replacement. When the SOC of an AGV drops below a preset threshold, each AGV must travel to a battery swapping station for a battery replacement. Terminals typically use battery swapping for recharging, and this process takes a specific amount of time. After the battery is replaced, the AGV's status is updated. It is also assumed that even if the AGV's SOC reaches the battery swapping threshold, the AGV can still complete its current task. Therefore, no AGV in the system will experience operational interruptions.

[0086] AGV Return to Position. If the AGV has completed its task and no new tasks are assigned to it, it should return to its parking space. If the parking space is large enough, the AGV will return to the nearest parking space, after which it will be idle.

[0087] Take action under system state s. At that time, the reward received by the intelligent agent (i.e., the AGV) is ,in, To take action for task i under system state s. Reward value obtained:

[0088]

[0089] In the formula, The revenue generated from completing a transportation task; The travel time required for the automated guided vehicle to perform service task i under system state s; The delay amount for the automated guided vehicle to execute task i in system state s; These are the container transportation cost coefficient and the task delay cost coefficient, respectively. The penalty value imposed on inactive actions (i.e. actions that cause conflicts between AGVs).

[0090] set up Indicates the state s of the system e The following is a random strategy The generated action, This is the initial system state. As the discount factor, This determines the importance of future rewards relative to immediate rewards. The goal of the MDP model is to find the optimal policy. This strategy maximizes the decision-making timeframe. The expected total discounted reward within the period, i.e., the objective function, is:

[0091]

[0092] The above process constructs a Markov decision process model for task assignment. The path planning problem in the lower-level model will be described next.

[0093] To complete transportation tasks on time, an efficient AGV movement scheme needs to be designed. Each task has a starting point (i.e., the start of the task) and a destination point (i.e., the end of the task), and typically there is a limited set of options on the AGV's path connecting the starting point and the destination. Take a loading task as an example (unloading tasks are similar). After the AGV retrieves a container from the container yard, it needs to first enter one of the landside lanes for horizontal transport of the container; then, it approaches the dock via a parking space in the buffer zone; and finally, it enters one of the seaside lanes to approach the target QC. Let N... L N P and N SThese represent the number of landside lanes, parking spaces, and seaside lanes available for AGVs, respectively. Therefore, the number of paths connecting the starting point and destination of a loading task is determined by N. L ×N P ×N S The given information is provided. However, please note that due to additional path selection by the AGV from its initial position to the task start position, the number of paths the AGV takes to complete the loading task may be greater than N. L ×N P ×N S .

[0094] Given a set of task paths, the ideal paths are those that cause zero or minimal latency at the QC (Quality Control) point. However, ideal paths are only available if they do not conflict with the paths of other AGVs. Please refer to [reference needed]. Figure 3 In actual AGV operation, there are usually three path conflict scenarios:

[0095] (1) Head-on collision. In this case, two AGVs travel towards each other along the same lane and stop moving when they approach each other, resulting in a deadlock. Because both the landside and seaside lanes are one-way traffic lanes, all AGVs must travel in the same direction in each lane, so head-on collisions will not occur in these lanes. Therefore, head-on collisions can only occur in the buffer zone.

[0096] (2) Lateral Crossing Conflict. In this case, two AGVs arrive at an intersection of a parking space and a traffic lane at the same time and attempt to pass through. Lateral crossing conflicts can also lead to deadlock because neither AGV will continue its current task at the shared intersection to avoid a collision.

[0097] (3) Overtaking conflict. In this case, one AGV attempts to overtake another AGV that is slower in the same lane. Overtaking conflict will force the following AGV to change lanes or slow down to resolve the conflict, resulting in an extension of task execution time. In practice, overtaking conflict rarely occurs because AGVs are usually set to travel at the same speed.

[0098] When assigning tasks to AGVs, the route for transporting containers should also be determined. For example... Figure 4 As shown, the AGV has multiple path options when traveling between YB and QC. Since the travel distances of candidate paths may differ, the shortest path is the ideal choice. However, when selecting a path for an AGV, it is also necessary to consider paths already assigned to other AGVs and avoid head-on and side-to-side collisions between AGVs. Because conflict elimination is required, there are situations where a single shortest path is not feasible.

[0099] This application proposes a dynamic and conflict-free AGV path generation algorithm. The algorithm records the paths already assigned to AGVs and the time points when these paths need to be executed. To evaluate the effectiveness of AGV candidate paths, the path generation algorithm provided in this application generates the trajectories of all AGVs with tasks to be executed and checks whether the candidate paths intersect with the trajectories of other AGVs both spatially and temporally. This is accomplished by utilizing a grid map of traffic lanes and buffer zones, such as... Figure 5 As shown. Specifically, let's assume... This represents the set of grids in the grid map. The set of candidate paths P for AGV k to perform task i can be calculated. ki Assume the movement forms a trajectory. ,in It is the set of grids occupied by AGV k at time t. Therefore, whenever At time t, a conflict was detected.

[0100] To efficiently generate conflict-free paths, the AGV's main path is first divided into continuous segments. Then, specific paths are constructed by finding the optimal traffic lanes for these segments. To illustrate this process, [the text continues with a description of a process involving a specific AGV and its associated traffic lanes]. Figure 4 Taking the container unloading task shown in the image as an example, the main path serving this task has four segments: Segment 1 (the segment containing the seaside lanes) is at the seaside; Segment 2 (the buffer zone) passes through a parking space; Segment 3 (the segment containing the landside lanes) is on the land side; and Segment 4 (the segment between the boundary of Segment 3 near the container yard area and the container yard area) connects to the designated container yard area. For Segment 1, four seaside lanes (lanes 2, 3, 5, and 6) are feasible; for Segment 2, seven parking spaces (the parking spaces between two adjacent QCs) are feasible; for Segment 3, three landside lanes (lanes 1, 3, and 5) are feasible; and for Segment 4, five lanes are feasible. For each segment, priorities can be assigned to its feasible lanes. For Segment 1, lanes near the buffer zone are given higher priority to prioritize shorter travel distances. For segment 2, lanes closer to designated QCs are given higher priority to reduce conflicts with AGVs serving adjacent QCs. For segment 3, lanes closer to the buffer zone are given higher priority because AGVs occupy less land lane when turning left and entering the lane. For segment 4, lanes closer to designated QCs at horizontal distance are given higher priority to prioritize shorter travel distances.

[0101] In addition to assigning a lane to each road segment, it is also necessary to generate a trajectory. This involves using the motion characteristics of the AGV to calculate grid occupancy (see...). Figure 5 ). In the grid map In this system, the length and width of each grid are equal to the distance traveled by the AGV at each time step (the AGV's travel speed is typically 4 meters per second). The physical dimensions of the AGV are generally 3 meters × 15 meters. Therefore, during travel, the AGV occupies at least four grids at each time step, and the AGV's trajectory records the number of grids occupied by the AGV at each time step t (using...). (Represented). The entire AGV trajectory is a sequence strung together in chronological order (using...). (Representation). When the AGV moves straight, the number of grid cells occupied at each time step can be easily calculated; that is, for every grid cell the AGV advances, it moves in the direction of travel... Insert a new mesh into the head and remove it. The grid at the rear of the queue; when the AGV turns, it is affected by the turning radius, Figure 5 The turning process shown is used to depict the grid occupied by the AGV at each time step. Figure 5 The AGV's turning radius is 16 meters. For ease of explanation, we have extracted frames from the turning process and simplified it into four time steps, among which... At that time, the AGV was in the initial transverse lane; At this point, the AGV completes the turn and is in the target longitudinal lane. After constructing the AGV's trajectory, it's necessary to check if the current trajectory conflicts with existing trajectories of other AGVs. If a conflict is detected in a road segment, the current lane for that segment is canceled, and another lane is selected for that segment. This process is repeated until a complete task completion trajectory is generated. It's important to note that if a conflict cannot be resolved in a road segment, it indicates that the current task assignment is not feasible.

[0102] Figure 6 Pseudocode for the path generation algorithm is provided, in which:

[0103] This indicates that AGV k can be used for the initial position / initial time of the current task;

[0104] This represents the set of grids occupied by AGV k at time t;

[0105] Let t0 and t be the trajectory of AGV k along path p. p These are the start and end times of the path, respectively.

[0106] This represents the set of trajectories for all AGVs. = };in, This refers to all AGVs in the AGV set except for AGV k.

[0107] The first in the main path Each section of road, , The number of road segments on the main path;

[0108] Indicates road segment The set of feasible lanes, among which For road section The number of available lanes It is a section of road The Each lane, the set of feasible lanes is sorted in descending order of lane priority;

[0109] For the first Index of the selected lanes for each road segment ;

[0110] To assign the kth AGV The lanes of each section of road, ;

[0111] When AGV k follows the trajectory At time t, on the road segment where time t is located;

[0112] This represents the start / end point of the current task.

[0113] The expected destination of AGV k Time;

[0114] For AGV k to reach the location via path p Time;

[0115] Indicates the current task assignment It is not feasible; Indicates the current task assignment It is feasible.

[0116] The specific process of using a heuristic path generation algorithm for path planning of automated guided vehicles in this embodiment of the application includes:

[0117] S11. Under the current system state s, based on the initial position of the current automated guided vehicle serving the current task i. The starting point of the current task and the end point Get the main path and segments of the main path for the current task i. ;

[0118] S12, Obtain each road segment The feasible lanes are determined, and the feasible lanes of each road segment are sorted in descending order according to lane priority to obtain the feasible lanes of each road segment. Collection of feasible lanes ;

[0119] S13. Select the highest priority feasible lane from the feasible lane set of each road segment to generate a path (i.e., select lanes according to the lane order in the feasible lane set).

[0120] S14. Based on path p and the initial time of the current automated guided vehicle service for the current task. Generate trajectory According to the trajectory Determine the arrival time of the current automated guided vehicle (AGV) to the starting point of the current task. and the completion time of the current task ;

[0121] S15. Based on the current trajectory of the automated guided vehicle k and other automated guided vehicles... The trajectory is used to perform conflict detection. If a path conflict is detected between the current automated guided vehicle and other automated guided vehicles at the target time, step S16 is executed; if no path conflict is detected between the current automated guided vehicle and other automated guided vehicles, step S17 is executed.

[0122] S16. Obtain the target road segment where the current automated guided vehicle is located at the target time, delete the currently selected feasible lane from the feasible lane set of the target road segment, and obtain the updated feasible lane set of the target road segment; if the updated feasible lane set of the target road segment is empty, it is determined that the current task assignment action is not feasible, and the preset penalty value is returned as the reward value; if the updated feasible lane set of the target road segment is not empty, return to step S13.

[0123] S17. Determine that the action to be assigned to the current task is feasible, and based on the initial time of the current task... and completion time Calculate the execution time of the current task. According to arrival time Calculate the delay amount with the expected arrival time E. Returns the trajectory, execution time, and delay of the current task for the current automated guided vehicle service.

[0124] Step 120: Construct a reinforcement learning network including a policy network and a value network; assign tasks based on the system state through the policy network and generate task assignment actions; use a heuristic path generation algorithm in the lower-level model to generate the best path for each task assignment action and calculate the reward value of the best path.

[0125] Step 130: Evaluate the estimated value of the current policy's value function based on the reward value through the value network, and update the policy based on the estimated value of the current policy's value function through the policy network to obtain the optimal policy; wherein, the policy network aims to maximize the cumulative reward.

[0126] In the hierarchical architecture for joint AGV task assignment and path planning proposed in this application embodiment, the upper-layer model uses a reinforcement learning (RL) algorithm to dynamically optimize task assignment, and the reward for actions in RL is evaluated by the path generation algorithm of the lower-layer model. Figure 7 The solution architecture for AGV scheduling is demonstrated.

[0127] In a reinforcement learning environment, the agent represents a fleet of AGVs, and the arrival and completion of tasks trigger decision moments. At each decision moment, a container matching heuristic can be applied to aggregate 20-foot containers and generate an aggregated task. This embodiment uses a proximal policy optimization (PPO) algorithm with an Actor-Critic structure to train the agent. In this algorithm, the Actor Network generates actions based on the system state and learns the optimal policy to maximize cumulative reward, while the Critic Network evaluates the value of the system state and minimizes the error in the Actor Network's cumulative reward estimation. Both the Actor Network and the Critic Network employ a multilayer perceptron (MLP), a fully connected neural network. Each network consists of an input layer, two hidden layers, and an output layer. In the input layer, the system state is flattened into a one-dimensional vector with multiple features and then input to the hidden layers. The hidden layers, equipped with many neurons, are crucial for capturing the complexity and nonlinearity within the state space. Furthermore, each neuron uses an activation function to nonlinearly transform the input data to learn complex patterns. In the Actor Network, the output layer outputs the probability distribution of discrete actions. In the Critic Network, the output layer generates a state-action value estimate under the current policy. To improve the performance of the PPO algorithm, the number of parallel environments can be increased, which allows us to collect a larger and more diverse set of experience in a shorter time, thereby improving sample efficiency and generalization ability.

[0128] The underlying path generation algorithm generates the optimal path for the AGV to perform transportation tasks. Therefore, the underlying layer needs to return a transportation path and a corresponding reward value for each task assignment action generated by the Actor Network. The reward values ​​generated at the underlying layer are then input into the Critic Network to update the estimate of the state value function. The reward values ​​are also used to calculate the advantage value to update the Actor Network's policy. The calculation of the value function and advantage value is prior art, and its specific calculation process will not be elaborated here.

[0129] Furthermore, in order to improve the performance of the two-layer architecture, this application embodiment also provides a container matching method, which aims to increase the chances of AGVs performing dual container transportation in order to shorten the travel distance of AGVs.

[0130] Specifically, before assigning tasks based on system status through the policy network and generating task assignment actions, a container matching method can be used for task aggregation, including:

[0131] S21. Aggregate tasks based on the container dimensions corresponding to each task in the current task set, and obtain the possible routes for each aggregated task.

[0132] S22. Calculate the distance saved by each possible route of each aggregation task relative to the task before aggregation;

[0133] S23. Sort all possible routes for the aggregation tasks in descending order based on the distance saved to obtain a route list;

[0134] S24. Select the possible route that saves the most distance from the route list as the target route, delete the target route from the route list, and delete the remaining possible routes of the aggregated task corresponding to the target route from the route list to obtain the remaining route list.

[0135] S25. Repeatedly select the possible route with the largest distance saving as the target route for the remaining route list until the distance saving of the first route in the remaining route list is 0 or negative, or the remaining route list is empty; generate the current task set based on the aggregated tasks corresponding to all selected target routes and the remaining unaggregated tasks.

[0136] As mentioned above, containers are divided into 20-foot containers and 40-foot containers. Each AGV can carry one 40-foot container or up to two 20-foot containers. Maximizing the opportunity for double container transport can reduce AGV travel distance and achieve higher operational efficiency, i.e., encouraging each AGV to carry two containers when serving tasks requiring 20-foot containers. To this end, this application embodiment develops a container matching heuristic method to aggregate 20-foot container transport tasks so that each aggregated task includes two 20-foot containers transported simultaneously by one AGV. By matching 20-foot containers, the opportunity for double container transport is increased, thereby reducing the overall travel distance of the AGVs; the number of tasks in the system is reduced, making the RL algorithm computationally more efficient.

[0137] Consider a situation where two 20-foot containers need to be carried to complete two shipping tasks. and AGV. Let's assume... (or )and (or ) represent tasks respectively (or task) The starting and destination locations of the AGV. There are four possible routes for AGVs to perform dual-container transport:

[0138] Route 1: ;

[0139] Route 2: ;

[0140] Route 3: ;

[0141] Route 4: .

[0142] Suppose that at some point in time, there are n 20-foot containers that need to be shipped. The number of possible combinations of matching 20-foot containers is (The question is incomplete in the original text.) Since each pair of matches has four possible dual-task execution routes, and there are n possible routes to execute these tasks separately, the total number of possible routes to serve these 20-foot containers is . This application uses a heuristic method to match 20-foot containers. This method enumerates all possible routes and selects the route that maximizes distance savings. Specifically, let... This represents the distance traveled between positions i and j. Therefore, for executing the task... and The distance savings achieved by routes 1, 2, 3, and 4 are as follows:

[0143] The distance saved by Route 1: ;

[0144] Route 2 saves distance: ;

[0145] Route 3 saves distance: ;

[0146] Route 4 saves distance: .

[0147] The number of tasks at each decision point (i.e.) The number of possible routes for container matching is relatively small. Therefore, the data on possible routes is limited and can be efficiently enumerated. The heuristic sorts all routes in descending order of saving distance, resulting in a list of routes. Initially, the heuristic selects the first route in the list so that both tasks on the first route are matched. Subsequently, the selected route is removed from the list, and all routes in the remaining list containing matched tasks are also removed to prevent duplicate matching. Finally, the method continues to match tasks by selecting the first route in the remaining list. This process is repeated until the saving distance of the first route in the remaining list is non-positive, or the list becomes empty. If the saving distance of the first route is non-positive, no task is matched, and the container is transported alone. For each selected route, the two tasks on the selected route are removed, and an aggregate task is created that requires the 20-foot container to be transported along the selected route. Note that the container matching method is executed at each time step of the RL algorithm. Since the number of tasks at each time step is relatively small, the runtime of the container matching heuristic is negligible.

[0148] Furthermore, in order to improve the performance of the two-layer architecture, this application embodiment also provides a two-layer grid map method. Since the two decisions of task assignment and path planner have different requirements for AGV position accuracy, different grid sizes can be used for task assignment and path planning.

[0149] This application embodiment constructs a two-layer grid map, which includes an upper coarse-grained grid map and a lower fine-grained grid map;

[0150] The upper coarse-grained grid map is used for task assignment; the grid size of the upper coarse-grained grid map is determined by the width of the container yard area and the width of the horizontal transport area in the container terminal.

[0151] The lower-level fine-grained grid map is used for path planning; the grid size of the lower-level fine-grained grid map is determined according to the speed of the automated guided vehicle.

[0152] While path planning decisions require precise AGV location information to calculate travel distance and task delays, task assignment decisions only use location information to record the AGV's state. This means that AGV positions can be recorded at an aggregation level to reduce the state space, decrease the complexity of the task assignment MDP model, and improve the learning and generalization capabilities of the RL algorithm. Based on this, this application embodiment creates a two-layer mesh map for AGV task assignment and path planning, such as... Figure 8 As shown in the diagram, in the two-layer mesh map, the upper layer uses a larger-granularity mesh to aggregate the hierarchical simulation of AGV locations, providing relatively coarse locations for QC, YB, seaside and land lane segments, and parking space subgroups; while the lower layer uses a smaller-granularity mesh to simulate precise AGV locations. Therefore, in the solution architecture of this application, task assignment uses a coarse-grained mesh to describe the start and end locations of AGVs and transportation tasks, while path planning uses a fine-grained mesh to characterize the paths and trajectories of AGVs.

[0153] The size of the lower-level fine-grained grid is determined based on the AGV's travel speed. The width of the lower-level fine-grained grid is equal to the distance the AGV travels per second. Therefore, the lower-level fine-grained grid map can capture the specific position of the AGV every second, which is crucial for detecting conflicts between AGVs. The size of the upper-level coarse-grained grid is determined by the width of the YB and the width of the horizontal transport area (sea / land lanes and buffer zones). In the TOS system, when assigning tasks, the start and end points of a task are usually described using its corresponding YB or QC number. Therefore, we follow this principle in the reinforcement learning environment, using a coarse-grained grid to describe the positions of tasks and AGVs. Conversely, using a fine-grained grid to describe position information would result in a huge state space in the reinforcement learning environment, directly affecting the convergence and generalization performance of the algorithm. At the same time, path planning needs to clearly define the specific path of the AGV and avoid conflict locations as much as possible, thus requiring a focus on local information, which is why we use a fine-grained grid. For these two reasons, different grid sizes are used in the two-layer grid map for different purposes.

[0154] This application proposes a grid-based path model that performs conflict detection at the motion level, considering the motion characteristics of AGVs, such as AGV speed, physical dimensions, and turning radius, to more accurately capture AGV occupancy of traffic lanes and buffer zones. Compared with existing methods, the proposed method can calculate the exact AGV trajectory and perform conflict detection on each grid of the map, improving the accuracy of path planning.

[0155] Step 140: Obtain the task assignment results and transportation routes of the automated guided vehicles through the optimal strategy;

[0156] The optimal strategy can be obtained through the above process. Based on the optimal strategy, the best task assignment action can be determined, thereby determining the AGV task assignment result. According to the task assignment result of each AGV, the optimal transportation path of each AGV can be obtained by using a heuristic path planning algorithm.

[0157] This application recognizes that existing technologies oversimplify the coupling between task assignment and path planning decisions, with both decision-making processes relying on partial information and failing to achieve information sharing and overall coordination. To address this issue, this application proposes an integrated two-layer architecture that combines task assignment and path planning. This architecture optimizes task assignment at the upper layer and path planning at the lower layer, employing reinforcement learning to enhance the system's AGV task assignment capabilities. The path generation algorithm is embedded into the reinforcement learning algorithm. This two-layer architecture provides a feedback mechanism to the AGV path planning model, enabling adjustments to the task assignment scheme when path conflicts are frequent. This fundamentally prevents AGVs from venturing into conflict-prone areas, thereby avoiding conflicts and deadlocks, significantly reducing path conflicts, and improving AGV efficiency.

[0158] Furthermore, this application proposes two acceleration strategies: a two-layer mesh map and a container matching method. The container matching method aggregates transportation tasks, increasing the chances of AGVs performing dual-container transport, thereby shortening AGV travel distances, reducing the number of transportation tasks, and improving AGV efficiency. Constructing a two-layer mesh map allows the upper layer to use a larger-granularity mesh to aggregate the AGV positions hierarchically, reducing the state space, decreasing the complexity of the task assignment MDP model, and improving the learning and generalization capabilities of the RL algorithm. The lower layer uses a smaller-granularity mesh to simulate precise AGV positions, improving the accuracy of conflict detection between AGVs and reducing path conflicts.

[0159] Please refer to Figure 9 This application also provides an automated guided vehicle (AGV) scheduling device for an automated container terminal, comprising:

[0160] The first building unit 210 is used to build a hierarchical architecture consisting of an upper-level model and a lower-level model. The upper-level model uses a Markov decision process model to model the task assignment of the automated guided vehicle and determine the system state, action space, reward function, and objective function. The lower-level model uses a heuristic path generation algorithm to perform path planning for the automated guided vehicle.

[0161] The second building unit 220 is used to build a reinforcement learning network including a policy network and a value network; the policy network assigns tasks according to the system state and generates task assignment actions; the lower-level model uses a heuristic path generation algorithm to generate the best path for each task assignment action and calculate the reward value of the best path.

[0162] The update unit 230 is used to evaluate the estimated value of the current policy's value function based on the reward value through the value network, and update the policy based on the estimated value of the current policy's value function through the policy network to obtain the optimal policy; wherein, the policy network aims to maximize the cumulative reward.

[0163] The scheduling unit 240 is used to obtain the task assignment results and transportation routes of the automated guided vehicles through the optimal strategy.

[0164] Optionally, the system status is... t is the current time point, Let s be the set of tasks that have not yet been served, and let s be the current state vector of the automated guided vehicle. The current state includes the current position, the current battery charge state, the start point and the end point of the current task.

[0165] In system state s, the action space is A(s), where each task in the action space is assigned an action. , The allocation decision for task i;

[0166] The reward function is ,in,

[0167]

[0168] In the formula, To take action for task i under system state s. The reward value obtained; The revenue generated from completing a transportation task; The travel time required for the automated guided vehicle to perform service task i under system state s; The delay amount for the automated guided vehicle to execute task i in system state s; These are the container transportation cost coefficient and the task delay cost coefficient, respectively. The penalty value imposed on actions that cannot be taken;

[0169] The objective function is , Indicates the state s of the system e The following is a random strategy The generated action, This is the initial system state. As the discount factor, Let e ​​be the optimal strategy and 'e' be the decision time. The timeframe for decision-making.

[0170] As a further improvement, the device also includes:

[0171] The third building unit is used to build a two-layer grid map, which includes an upper coarse-grained grid map and a lower fine-grained grid map.

[0172] The upper coarse-grained grid map is used for task assignment; the grid size of the upper coarse-grained grid map is determined by the width of the container yard area and the width of the horizontal transport area in the container terminal.

[0173] The lower-level fine-grained grid map is used for path planning; the grid size of the lower-level fine-grained grid map is determined according to the speed of the automated guided vehicle.

[0174] This application recognizes that existing technologies oversimplify the coupling between task assignment and path planning decisions, with both decision-making processes relying on partial information and failing to achieve information sharing and overall coordination. To address this issue, this application proposes an integrated two-layer architecture that combines task assignment and path planning. This architecture optimizes task assignment at the upper layer and path planning at the lower layer, employing reinforcement learning to enhance the system's AGV task assignment capabilities. The path generation algorithm is embedded into the reinforcement learning algorithm. This two-layer architecture provides a feedback mechanism to the AGV path planning model, enabling adjustments to the task assignment scheme when path conflicts are frequent. This fundamentally prevents AGVs from venturing into conflict-prone areas, thereby avoiding conflicts and deadlocks, significantly reducing path conflicts, and improving AGV efficiency.

[0175] Furthermore, this application proposes two acceleration strategies: a two-layer mesh map and a container matching method. The container matching method aggregates transportation tasks, increasing the chances of AGVs performing dual-container transport, thereby shortening AGV travel distances, reducing the number of transportation tasks, and improving AGV efficiency. Constructing a two-layer mesh map allows the upper layer to use a larger-granularity mesh to aggregate the AGV positions hierarchically, reducing the state space, decreasing the complexity of the task assignment MDP model, and improving the learning and generalization capabilities of the RL algorithm. The lower layer uses a smaller-granularity mesh to simulate precise AGV positions, improving the accuracy of conflict detection between AGVs and reducing path conflicts.

[0176] This application also provides an electronic device, which includes a processor and a memory;

[0177] The memory is used to store program code and transfer the program code to the processor;

[0178] The processor is used to execute the automated guided vehicle scheduling method for automated container terminals in the foregoing method embodiments according to the instructions in the program code.

[0179] This application also provides a computer-readable storage medium for storing program code, which, when executed by a processor, implements the automated guided vehicle scheduling method for automated container terminals described in the foregoing method embodiments.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0181] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0182] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0187] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for scheduling automated guided vehicles (AGVs) in an automated container terminal, characterized in that, include: Construct a layered architecture consisting of upper-level models and lower-level models; The upper-level model uses a Markov decision process model to model the task assignment of the automated guided vehicle, determining the system state, action space, reward function, and objective function; the lower-level model uses a heuristic path generation algorithm to perform path planning for the automated guided vehicle. Construct a reinforcement learning network that includes a policy network and a value network; assign tasks based on the system state through the policy network and generate task assignment actions; use a heuristic path generation algorithm in the lower-level model to generate the optimal path for each task assignment action and calculate the reward value of the optimal path. The value network evaluates the estimated value of the current policy's value function based on the reward value, and the policy network updates the policy based on the estimated value of the current policy's value function to obtain the optimal policy; wherein, the policy network has the optimization objective of maximizing the cumulative reward. The optimal strategy is used to obtain the task assignment results and transportation routes of the automated guided vehicles.

2. The automated guided vehicle (AGV) scheduling method for automated container terminals according to claim 1, characterized in that, The system status is t is the current time point, Let s be the set of tasks that have not yet been served, and let s be the current state vector of the automated guided vehicle. The current state includes the current position, the current battery charge state, the start point and the end point of the current task. In system state s, the action space is A(s), where each task in the action space is assigned an action. , The allocation decision for task i; The reward function is ,in, In the formula, To take action for task i under system state s. The reward value obtained; The revenue generated from completing a transportation task; The travel time required for the automated guided vehicle to perform service task i under system state s; The delay amount for the automated guided vehicle to execute task i in system state s; These are the container transportation cost coefficient and the task delay cost coefficient, respectively. The penalty value imposed on actions that cannot be taken; The objective function is , Indicates the state s of the system e The following is a random strategy The generated action, This is the initial system state. As the discount factor, Let e ​​be the optimal strategy and 'e' be the decision time. The decision-making timeframe.

3. The automated guided vehicle (AGV) scheduling method for automated container terminals according to claim 1, characterized in that, A heuristic path generation algorithm is used for path planning of automated guided vehicles, including: S11. Under the current system state, obtain the main path and road segments of the current task based on the initial position, start point and end point of the current task of the current automated guided vehicle service. S12. Obtain the feasible lanes of each road segment, and sort the feasible lanes of each road segment in descending order according to lane priority to obtain the set of feasible lanes of each road segment. S13. Select the highest priority driveway generation path from the set of driveways in each road segment; S14. Generate a trajectory based on the path and the initial time of the current automated guided vehicle (AGV) serving the current task, and determine the arrival time of the current AGV to the starting point of the current task and the completion time of the current task based on the trajectory. S15. Detect conflict based on the current trajectory of the automated guided vehicle (AGV) and the trajectories of other AGVs. If a path conflict is detected between the current AGV and other AGVs at the target time, proceed to step S16; if no path conflict is detected between the current AGV and other AGVs, proceed to step S17. S16. Obtain the target road segment where the current automated guided vehicle is located at the target time, delete the currently selected feasible lane from the feasible lane set of the target road segment, and obtain the updated feasible lane set of the target road segment; if the updated feasible lane set of the target road segment is empty, it is determined that the current task assignment action is not feasible, and the preset penalty value is returned as the reward value; if the updated feasible lane set of the target road segment is not empty, return to step S13. S17. Determine that the current task assignment action is feasible, and calculate the execution time of the current task based on the initial time and completion time of the current task, and calculate the delay based on the arrival time and expected arrival time; return the trajectory, execution time and delay of the current task of the current automated guided vehicle service.

4. The automated guided vehicle (AGV) scheduling method for automated container terminals according to claim 1, characterized in that, Tasks are assigned based on system status using a policy network, generating task assignment actions. This process also includes: S21. Aggregate tasks based on the container dimensions corresponding to each task in the current task set, and obtain the possible routes for each aggregated task. S22. Calculate the distance saved by each possible route of each aggregation task relative to the task before aggregation; S23. Sort all possible routes for the aggregation tasks in descending order based on the distance saved to obtain a route list; S24. Select the possible route that saves the most distance from the route list as the target route, delete the target route from the route list, and delete the remaining possible routes of the aggregated task corresponding to the target route from the route list to obtain the remaining route list. S25. Repeatedly select the possible route with the largest distance saving as the target route for the remaining route list until the distance saving of the first route in the remaining route list is 0 or negative, or the remaining route list is empty; generate the current task set based on the aggregated tasks corresponding to all selected target routes and the remaining unaggregated tasks.

5. The automated guided vehicle (AGV) scheduling method for an automated container terminal according to claim 1, characterized in that, The method also includes: Construct a two-layer grid map, which consists of an upper coarse-grained grid map and a lower fine-grained grid map; The upper coarse-grained grid map is used for task assignment; the grid size of the upper coarse-grained grid map is determined by the width of the container yard area and the width of the horizontal transport area in the container terminal. The lower-level fine-grained grid map is used for path planning; the grid size of the lower-level fine-grained grid map is determined according to the speed of the automated guided vehicle.

6. An automated guided vehicle (AGV) scheduling device for an automated container terminal, characterized in that, include: The first building unit is used to construct a layered architecture consisting of upper-level models and lower-level models. The upper-level model uses a Markov decision process model to model the task assignment of the automated guided vehicle, determining the system state, action space, reward function, and objective function; the lower-level model uses a heuristic path generation algorithm to perform path planning for the automated guided vehicle. The second building unit is used to construct a reinforcement learning network that includes a policy network and a value network; the policy network assigns tasks according to the system state and generates task assignment actions; the lower-level model uses a heuristic path generation algorithm to generate the best path for each task assignment action and calculate the reward value of the best path. The update unit is used to evaluate the estimated value of the current policy's value function based on the reward value through the value network, and update the policy based on the estimated value of the current policy's value function through the policy network to obtain the optimal policy; wherein, the policy network has the optimization objective of maximizing the cumulative reward; The scheduling unit is used to obtain the task assignment results and transportation routes of the automated guided vehicles through the optimal strategy.

7. The automated guided vehicle (AGV) scheduling device for an automated container terminal according to claim 6, characterized in that, The system status is t is the current time point, Let s be the set of tasks that have not yet been served, and let s be the current state vector of the automated guided vehicle. The current state includes the current position, the current battery charge state, the start point and the end point of the current task. In system state s, the action space is A(s), where each task in the action space is assigned an action. , The allocation decision for task i; The reward function is ,in, In the formula, To take action for task i under system state s. The reward value obtained; The revenue generated from completing a transportation task; The travel time required for the automated guided vehicle to perform service task i under system state s; The delay amount for the automated guided vehicle to execute task i in system state s; These are the container transportation cost coefficient and the task delay cost coefficient, respectively. The penalty value imposed on actions that cannot be taken; The objective function is , Indicates the state s of the system e The following is a random strategy The generated action, This is the initial system state. As the discount factor, Let e ​​be the optimal strategy and 'e' be the decision time. The decision-making timeframe.

8. The automated guided vehicle (AGV) scheduling device for an automated container terminal according to claim 6, characterized in that, The device also includes: The third building unit is used to build a two-layer grid map, which includes an upper coarse-grained grid map and a lower fine-grained grid map. The upper coarse-grained grid map is used for task assignment; the grid size of the upper coarse-grained grid map is determined by the width of the container yard area and the width of the horizontal transport area in the container terminal. The lower-level fine-grained grid map is used for path planning; the grid size of the lower-level fine-grained grid map is determined according to the speed of the automated guided vehicle.

9. An electronic device, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the automated guided vehicle scheduling method for an automated container terminal as described in any one of claims 1-5, according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, implements the automated guided vehicle scheduling method for an automated container terminal as described in any one of claims 1-5.

Images