Reinforcement learning network training method, storage location scheduling method and corresponding apparatus

Abstract

This invention discloses a reinforcement learning network training method, a warehouse location scheduling method, and a corresponding device. The method includes: Step 1, acquiring multi-source business data, including: a goods specification data table, a historical shelf structure table, and a historical inbound / outbound order table; Step 2, generating a heterogeneous graph based on the goods specification data table and the historical shelf structure table; Step 3, obtaining a state representation based on the heterogeneous graph using GAT; Step 4, obtaining a first probability distribution of actions in the action set based on the state representation and a first target historical inbound / outbound order table in the historical inbound / outbound order table using a reinforcement learning network; Step 5, determining a first action based on the first probability distribution; Step 6, updating the heterogeneous graph based on the first action to generate the first heterogeneous graph; Step 7, repeating steps 3 to 6 until the iteration termination condition is met to obtain the trained reinforcement learning network. This method abandons the traditional static scheduling rules that rely on human experience and achieves autonomous evolution of scheduling strategies through reinforcement learning networks.

Description

Technical Field

[0001] This invention belongs to the field of warehousing and logistics technology, and specifically relates to a reinforcement learning network training method, a cargo location scheduling method, and a corresponding device. Background Technology

[0002] With the rapid development of e-commerce, new retail, and intelligent manufacturing, the logistics and warehousing industry is facing multiple challenges, including fragmented orders, high frequency of inbound and outbound operations, and complex inventory categories. In sectors with high turnover, such as tobacco, pharmaceuticals, and fast-moving consumer goods, automated high-bay warehouses have become core infrastructure, using stacker cranes to perform high-frequency inbound and outbound tasks across multiple aisles and multi-level storage locations. Currently, mainstream warehouse management systems generally rely on rule-based static scheduling strategies (such as proximity storage, high-level priority, and SKU-based storage) or simple heuristic algorithms for location allocation. This rigid scheduling logic makes it difficult to dynamically respond to changes in inventory status and fluctuations in task load, thus limiting the overall efficiency of the system.

[0003] To meet the demands of modern logistics services characterized by high concurrency, high timeliness, and high accuracy, warehousing systems urgently need intelligent scheduling capabilities that are adaptive, self-optimizing, and multi-objective collaborative. Specifically, the industry urgently needs a location scheduling method that can simultaneously consider task execution efficiency (e.g., minimizing stacker crane movement time), racking structure safety (e.g., avoiding heavy loads on high shelves and horizontal off-center loading), inventory health (e.g., promoting reasonable clustering of similar SKUs, accelerating the flow of high-turnover goods, and reducing inventory age), and the evolvability of strategies. This method should be able to continuously learn based on real business data and support rapid verification and iteration in a high-fidelity simulation environment, thereby achieving an upgrade from "experience-driven" to "data and intelligence-driven."

[0004] Existing high-bay automated warehouse location scheduling technologies have significant limitations: On the one hand, traditional rule-based methods lack generalization ability, cannot capture the dynamic relationships and task collaboration patterns between locations, and are prone to getting trapped in local optima; on the other hand, optimization models based on mathematical programming have high computational complexity, are difficult to deploy online, and do not adequately model equipment dynamics (such as acceleration and deceleration processes) and task stochasticity; although some studies in recent years have attempted to introduce reinforcement learning, most of them use vectorized or gridded state representations, ignoring the inherent graph structure characteristics of high-bay automated warehouses (such as aisle isolation, spatial relationships between locations, and dedicated service ranges for stacker cranes), resulting in redundant state representations and low learning efficiency; in addition, most algorithm verifications rely on idealized simulators and do not integrate real business data such as historical task flows and cargo attributes (such as weight, SKU, turnover rate, and value), making it difficult to support the effective transfer and continuous evolution of strategies in real-world scenarios. Summary of the Invention

[0005] To address the aforementioned problems, embodiments of the present invention provide a reinforcement learning network training method, a cargo location scheduling method, and a corresponding apparatus.

[0006] In a first aspect, embodiments of the present invention provide a reinforcement learning network training method for scheduling inbound and outbound storage locations in an elevated automated warehouse, comprising: Step 1: Obtain multi-source business data, which includes: a goods specification data table, a historical shelf structure table, and a historical inbound / outbound order table. The historical inbound / outbound order table is used to represent the original task requests for unassigned storage locations. The historical inbound / outbound order table includes a first target historical inbound / outbound order table. Step 2: Generate a heterogeneous graph based on the product specification data table and the historical shelf structure table. The heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes. Step 3: Based on the heterogeneous graph, obtain the state representation using a graph attention network; Step 4: Based on the state representation and the first target historical inbound / outbound order table, obtain the first probability distribution of actions in the action set using a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and availability. Step 5: Based on the first probability distribution of the actions in the action set, determine the first action, which is used to perform in the environment of entering and exiting the elevated automated warehouse, and generate an instant reward to be sent to the reinforcement learning network. Step 6: Update the heterogeneous graph according to the first action to generate a first heterogeneous graph; Step 7: Repeat steps 3 to 6 until the iteration termination condition is met to obtain the trained reinforcement learning network.

[0007] In one possible implementation, generating the heterogeneity graph based on the product specification data table and the historical shelf structure table includes: The product specification data table and the historical shelf structure table of the goods are preprocessed to generate preprocessed data. The preprocessing includes removing duplicate data, filling in missing data, normalizing numerical data, and embedding and encoding non-numerical data. A heterogeneous graph is generated based on the preprocessed data.

[0008] In one possible implementation, obtaining the state representation based on the heterogeneous graph and a graph attention network includes: Based on graph attention network, for the heterogeneous graph, the features of storage location nodes, stacker crane nodes, edge features between storage location nodes, and edge features between storage location nodes and stacker crane nodes are extracted. The features of the storage location node, the features of the stacker crane node, the edge features between storage location nodes, and the edge features between storage location nodes and stacker crane nodes are fused to obtain a state representation.

[0009] In one possible implementation, the fusion of the features of the storage location node, the features of the stacker crane node, the edge features between storage location nodes, and the edge features between storage location nodes and stacker crane nodes to obtain the state representation includes: Based on the features of the storage location node and the edge features between storage location nodes, determine the first attention weight between the storage location node and its neighboring storage location nodes; Based on the features of the storage location node, the features of the stacker crane node, and the edge features between the storage location node and the stacker crane node, determine the second attention weight between the storage location node and its neighboring stacker crane node; The state representation is determined based on the characteristics of the neighboring warehouse node and the corresponding first attention weight, and the characteristics of the neighboring stacker crane node and the corresponding second attention weight.

[0010] In one possible implementation, determining the first action based on a first probability distribution of actions in the action set includes: Select the action with the highest probability in the first probability distribution and determine it as the first action.

[0011] In one possible implementation, the function for the immediate reward is as follows: R = −ω1T + ω2S + ω3C + ω4F Where: R is the immediate reward; T is the task duration, representing the time from the start to the completion of the current task; S is the cargo dispersion, measuring the degree of clustering of SKUs of the same inventory quantity in storage locations; C is the center of gravity evaluation index, reflecting the proportion of heavy cargo at high levels; F is the turnover rate distribution index, measuring whether high-turnover cargo is concentrated in easily accessible areas; ω1, ω2, ω3, and ω4 are the weighting coefficients, respectively.

[0012] Secondly, embodiments of the present invention provide a method for scheduling storage locations in and out of an elevated automated warehouse, including: Step 1: Obtain current multi-source business data, which includes: a goods specification data table, a current shelf structure table, and a current inbound / outbound order table. The current inbound / outbound order table is used to represent current task requests for unassigned storage locations. Step 2: Generate a current heterogeneous graph based on the product specification data table and the current shelf structure table. The current heterogeneous graph includes the current storage location node, the current stacker crane node, the edges between current storage location nodes, and the edges between current storage location nodes and current stacker crane nodes. Step 3: Based on the current heterogeneous graph, obtain the current state representation using a graph attention network; Step 4: Based on the current state representation and the current inbound / outbound order table, obtain the second probability distribution of actions in the current action set using the trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability. Step 5: Determine a second action based on the second probability distribution of actions in the current action set. The second action is used to execute in the environment of entering and exiting the elevated automated warehouse. Step 6: Update the current heterogeneous graph according to the second action, and generate a second heterogeneous graph; Step 7: Repeat steps 3 to 6 until, in the case of outbound from the high-bay warehouse, all occupied storage locations for the first item have been assigned to the corresponding stacker cranes for scheduling; in the case of inbound from the high-bay warehouse, all empty storage locations that meet the conditions of the second item, weight, and availability have been assigned to the corresponding stacker cranes for scheduling.

[0013] Thirdly, embodiments of the present invention provide a reinforcement learning network training device for scheduling inbound and outbound storage locations in an elevated automated warehouse, comprising: The multi-source business data acquisition module is used to acquire multi-source business data, which includes: a goods specification data table, a historical shelf structure table, and a historical inbound and outbound order table. The historical inbound and outbound order table is used to represent the original task requests for unassigned storage locations. The historical inbound and outbound order table includes a first target historical inbound and outbound order table. The heterogeneous graph generation module is used to generate a heterogeneous graph based on the product specification data table of the goods and the historical shelf structure table. The heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes. The state representation acquisition module is used to obtain a state representation based on the heterogeneous graph and a graph attention network. The first probability distribution acquisition module is used to obtain the first probability distribution of actions in the action set based on the state representation and the first target historical inbound / outbound order table, using a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and availability. The first action determination module is used to determine a first action based on the first probability distribution of actions in the action set. The first action is used to be executed in the environment of entering and leaving the elevated automated warehouse, and an instant reward is generated and sent to the reinforcement learning network. The first update module is used to update the heterogeneous graph according to the first action and generate a first heterogeneous graph; The first processing module is used to repeat the functions of the aforementioned state representation acquisition module to the first update module until the iteration termination condition is met, thereby obtaining the trained reinforcement learning network.

[0014] Fourthly, embodiments of the present invention provide a storage location scheduling device for inbound and outbound operations in an elevated automated warehouse, comprising: The current multi-source business data acquisition module is used to acquire current multi-source business data, which includes: a goods specification data table, a current shelf structure table, and a current inbound / outbound order table. The current inbound / outbound order table is used to represent current task requests for unassigned storage locations. The current heterogeneous graph generation module is used to generate a current heterogeneous graph based on the product specification data table of the goods and the current shelf structure table. The current heterogeneous graph includes the current storage location node, the current stacker crane node, the edge between the current storage location node and the edge between the current storage location node and the current stacker crane node. The current state representation acquisition module is used to obtain the current state representation based on the current heterogeneous graph and a graph attention network. The second probability distribution acquisition module is used to obtain the second probability distribution of actions in the current action set based on the current state representation and the current inbound / outbound order table, using a trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability. The second action determination module is used to determine a second action based on the second probability distribution of actions in the current action set. The second action is used to execute in the environment of the high-rise automated warehouse entry and exit. The second update module is used to update the current heterogeneous graph according to the second action and generate a second heterogeneous graph; The second processing module is used to repeat the function of the current state representation acquisition module to the second update module until, when it is an outbound scenario of an elevated warehouse, all occupied storage locations storing the first specification have been allocated to the corresponding stacker cranes for scheduling; when it is an inbound scenario of an elevated warehouse, all empty storage locations that meet the conditions of the second specification, weight, and vacancy have been allocated to the corresponding stacker cranes for scheduling.

[0015] Fifthly, embodiments of the present invention provide an electronic device, comprising: The system includes a memory and a processor, which communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method described in the first or second aspect and various steps in various possible implementations.

[0016] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first or second aspect and various possible implementations.

[0017] In a seventh aspect, embodiments of the present invention provide a computer program product containing instructions that, when the computer program product is run on a computer, cause the steps of the method as described in the first or second aspect and in various possible implementations to be executed by the computer.

[0018] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows: 1) For the first time, the physical layout, equipment constraints (i.e., stacker cranes in each aisle can only be used in their own aisle and cannot be allocated between aisles) and task coordination relationships of the elevated automated warehouse are uniformly modeled as a heterogeneous graph structure, and this graph is used as the perception environment of the reinforcement learning network, breaking through the limitations of traditional scheduling methods that rely on gridded or vectorized state representations; the graph attention network GAT is used to calculate the first attention weight between the storage location and its neighboring storage locations, and the second attention weight between the storage location and the corresponding stacker crane, and dynamic aggregation is performed based on the first and second attention weights, so that the state representation is naturally embedded in spatial reachability. The relevance to historical tasks provides a structured, interpretable, and high-information-density input foundation for reinforcement learning, significantly improving the efficiency and generalization ability of policy learning; 2) It abandons the traditional static scheduling rules (such as proximity principle, high-level priority, etc.) that rely on human experience, and realizes the autonomous evolution and real-time decision-making of scheduling strategies through an end-to-end reinforcement learning framework; When a task arrives, it outputs the optimal location allocation action in real time based on the current heterogeneous graph state, without the need for preset logical branches, and can dynamically balance the conflicts of multiple objectives such as efficiency, safety, and inventory health. It has strong adaptability and online response capabilities, truly realizing the transformation from "human-made rules" to "intelligent decision-making", meeting the urgent needs of high-turnover warehousing scenarios for real-time and intelligent scheduling. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a reinforcement learning network training method for scheduling inbound and outbound storage locations in an elevated automated warehouse, provided as an embodiment of the present invention. Figure 2 A flowchart illustrating the storage location scheduling method for inbound and outbound operations in an elevated automated warehouse provided in an embodiment of the present invention; Figure 3 This is another flowchart illustrating a reinforcement learning network training method for scheduling inbound and outbound storage locations in an elevated automated warehouse, as provided in an embodiment of the present invention. Figure 4 A system architecture diagram provided for an embodiment of the present invention; Figure 5 A schematic block diagram of a reinforcement learning network training device for scheduling inbound and outbound storage locations in an elevated automated warehouse, provided in an embodiment of the present invention; Figure 6 A schematic block diagram of a storage location scheduling device for inbound and outbound operations in an elevated automated warehouse, provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if monitoring (the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when monitoring (the stated condition or event)," or "in response to monitoring (the stated condition or event)."

[0024] Existing high-bay automated warehouse location scheduling technologies have significant limitations: On the one hand, traditional rule-based methods lack generalization ability, cannot capture the dynamic relationships and task collaboration patterns between locations, and are prone to getting trapped in local optima; on the other hand, optimization models based on mathematical programming have high computational complexity, are difficult to deploy online, and do not adequately model equipment dynamics (such as acceleration and deceleration processes) and task stochasticity; although some studies in recent years have attempted to introduce reinforcement learning, most of them use vectorized or gridded state representations, ignoring the inherent graph structure characteristics of high-bay automated warehouses (such as aisle isolation, spatial relationships between locations, and dedicated service ranges for stacker cranes), resulting in redundant state representations and low learning efficiency; in addition, most algorithm verifications rely on idealized simulators and do not integrate real business data such as historical task flows and cargo attributes (such as weight, SKU, turnover rate, and value), making it difficult to support the effective transfer and continuous evolution of strategies in real-world scenarios.

[0025] In view of this, embodiments of the present invention provide a reinforcement learning network training method for inbound and outbound storage location scheduling in elevated automated warehouses. A flowchart of the method is shown below. Figure 1 As shown in the image. Figure 1 The method may include the following steps: Step 101: Obtain multi-source business data, which includes: product specification data table, historical shelf structure table, and historical inbound / outbound order table. The historical inbound / outbound order table is used to represent the original task requests for unassigned storage locations. The historical inbound / outbound order table includes the first target historical inbound / outbound order table.

[0026] Step 102: Generate a heterogeneous graph based on the product specification data table and the historical shelf structure table. The heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes.

[0027] Step 103: Based on the heterogeneous graph, obtain the state representation using a graph attention network.

[0028] Step 104: Based on the state representation and the historical inbound / outbound order table of the first target, obtain the first probability distribution of actions in the action set using a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and availability.

[0029] Step 105: Based on the first probability distribution of actions in the action set, determine the first action. The first action is used to execute in the environment of entering and exiting the high-rise warehouse, and generate an immediate reward to send to the reinforcement learning network.

[0030] Step 106: Update the heterogeneous graph according to the first action and generate the first heterogeneous graph.

[0031] Step 107: Repeat steps 103 to 106 until the iteration termination condition is met to obtain the trained reinforcement learning network.

[0032] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments of the present invention. It should be noted that the terms "first" and "second" involved in the embodiments of the present invention do not have limitations in terms of size, order, or quantity, but are only used to distinguish them in name. For example, "first specification" and "second specification" are used to distinguish two different specifications.

[0033] First, in conjunction with the embodiments of the present invention, the above step 101, namely "acquiring multi-source business data, the multi-source business data including: a product specification data table, a historical shelf structure table, and a historical inbound and outbound order table, the historical inbound and outbound order table being used to represent the original task request for unassigned storage locations, the historical inbound and outbound order table including a first target historical inbound and outbound order table", will be described in detail.

[0034] In this embodiment of the invention, the goods specification data table records the static attribute information of each type of goods, including indicators such as: specification code, weight, turnover rate, dimensions (length × width × height), unit value, etc., which serve as the feature input for the storage location nodes. The historical racking structure table describes the physical layout and storage location status of the entire high-bay automated warehouse, including indicators such as: storage location number, whether it is occupied, the specification code of the currently stored goods, the inbound time, the age of the warehouse, the number of rows, columns, and layers of the racking, the aisle number, the number of aisles, etc., which are used to initialize the dynamic status attributes of the storage location nodes, and may vary at different stages. The historical inbound / outbound order table is used to represent the original task requests for unassigned storage locations, including a first target historical inbound / outbound order table. This historical inbound / outbound order table includes indicators such as: task ID, specification code, issuance time, quantity, priority, etc., which are used to generate a queue of tasks to be executed. It can be understood that the historical inbound / outbound order table also includes a second target historical inbound / outbound order table. For example, 80% of the historical inbound and outbound orders in the historical inbound and outbound order table constitute the first target historical inbound and outbound order table, which is used to train the reinforcement learning network. 20% of the historical inbound and outbound orders in the historical inbound and outbound order table constitute the second target historical inbound and outbound order table, which is used to validate the trained reinforcement learning network.

[0035] The following describes in detail step 102, namely, "generating a heterogeneous graph based on the product specification data table and the historical shelf structure table, wherein the heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes," in conjunction with embodiments of the present invention.

[0036] As one possible implementation, the product specification data table and historical shelf structure table are preprocessed to generate preprocessed data. Preprocessing includes removing duplicate data, filling in missing data (e.g., using interpolation), normalizing numerical data (e.g., normalizing weight and dimensions to the [0, 1] range), and embedding encoding into non-numerical data. This ultimately forms a structured database with a unified format. Based on the preprocessed data, a heterogeneous graph is generated. In this graph, rows in the historical shelf structure table are bound to aisles (see the aisle number). For example, aisle 1 corresponds to rows 1 (left) and 2 (right). Stacker crane 1 is bound to aisle 1 and can only operate on rows 1 and 2. Therefore, this stacker crane node can only be associated with these location nodes, thus creating "edges." In other words, the stacker crane nodes in the heterogeneous graph are determined by the aisle numbers in the historical shelf structure table.

[0037] The following describes step 103, namely "obtaining a state representation based on a graph attention network according to a heterogeneous graph", in detail with reference to embodiments of the present invention.

[0038] In this embodiment of the invention, the heterogeneous graph is used to represent the spatial topology and task association relationship of the elevated automated warehouse.

[0039] As one possible implementation, based on graph attention networks, for heterogeneous graphs, features of storage location nodes, stacker crane nodes, edge features between storage location nodes, and edge features between storage location nodes and stacker crane nodes are extracted. The features of storage location nodes, stacker crane nodes, edge features between storage location nodes, and edge features between storage location nodes and stacker crane nodes are then fused to obtain a state representation.

[0040] Specifically, fusing the features of the storage location node, the features of the stacker crane node, the edge features between storage location nodes, and the edge features between the storage location node and the stacker crane node to obtain the state representation can be done as follows: First attention weights between a storage location node and its neighboring storage location nodes are determined based on the storage location node features and the edge features between storage location nodes; second attention weights between a storage location node and its neighboring stacker crane nodes are determined based on the storage location node features, the stacker crane node features, and the edge features between the storage location node and the stacker crane node; and the state representation is determined based on the neighboring storage location node features and their corresponding first attention weights, and the neighboring stacker crane node features and their corresponding second attention weights. The state representation is naturally embedded with spatial reachability (e.g., stacker crane A only serves aisle 1, and storage location B is located in aisle 2, so there is no "stacker crane A – storage location B" edge in the heterogeneous graph; while storage location C is located in aisle 1, so there is a "stacker crane A – storage location C" edge, and the edge attributes include travel_time=8s and energy_cost=12J. After GAT aggregation, the final representation of storage location C will contain the semantic information that it can be accessed quickly and with low cost by stacker crane A, while storage location B does not have this information. During scheduling, the actor network naturally tends to select storage locations with high reachability) and historical task relevance (e.g., the product specifications "Zhonghua cigarettes" and "Yuxi cigarettes" have a high co-occurrence frequency in historical orders (co_access_score=0.78), so there is a high-weight edge between storage location nodes that stores these two products). GAT Aggregation enhances the similarity of representations between products. When a new storage location needs to be allocated for "Zhonghua Tobacco," the actor network may prioritize locations closer to the "Yuxi" inventory area to reduce the number of stacker crane round trips for future multi-product orders.

[0041] Specifically, determining the first attention weights of a storage location node and its neighboring storage locations based on the node's features and the edge features between them can be achieved by: using Gaussian Interference Amplifier (GAT), concatenating the node's features, the features of its neighboring storage locations, and the edge features between them to obtain a first concatenated vector. This first concatenated vector is then input into a small, learnable neural network, which outputs a first score. The first scores of all neighboring storage locations are then normalized to obtain the first attention weights for each of the neighboring storage locations.

[0042] Based on the features of the storage location node, the features of the stacker crane node, and the edge features between the storage location node and the stacker crane node, the second attention weights for the storage location node and its corresponding stacker crane node are determined as follows: Using Gaussian Interference Aspects (GATs), the features of the storage location node, the features of its neighboring stacker crane nodes, and the edge features between the storage location node and its neighboring stacker crane nodes are concatenated to obtain a second concatenated vector. This second concatenated vector is then input into a learnable small neural network, which outputs a second score. The second scores of all neighboring stacker crane nodes are normalized to obtain the individual second attention weights for each neighboring stacker crane node.

[0043] Based on the features of neighboring warehouse nodes and their corresponding first attention weights, and the features of neighboring stacker crane nodes and their corresponding second attention weights, the state representation can be determined as follows: The features of each neighboring warehouse node and its corresponding first attention weight are summed in weight to obtain a first weighted result. The features of each neighboring stacker crane node and its corresponding second attention weight are summed in weight to obtain a second weighted result. The first and second weighted results are added together to obtain aggregated information. This aggregated information is processed by a non-linear activation function, which is the first state representation of the warehouse node in the current GAT layer. The above steps are repeated in each layer of the GAT to obtain multiple first state representations. These multiple first state representations are further merged into the state representation of the entire warehouse through global pooling (e.g., taking the maximum value or weighted average).

[0044] It should be noted that the characteristics of the storage location nodes are as follows: Each storage location is an independent node, and its feature vector is jointly generated by the product specification data table and the historical shelf structure table. Specifically, it includes: Aisle number (aisle_id): integer, used for cross-aisle task allocation decisions; Column coordinate (col): integer, used for proximity calculation and path planning; Level coordinate (level): integer, used for vertical movement cost evaluation; Storage location occupancy status (is_occupied): boolean, indicating whether there is already goods, serving as the basis for determining vacancy; Product specification identifier (sku_id): integer, supporting optimization of the same SKU; Product weight (weight): floating-point, used for center of gravity safety control (C objective); Turnover rate (turnover_rate): floating-point, the frequency of inbound and outbound operations in the past 7 days, used for turnover efficiency optimization (F objective); Stock age days (stock_age_days): floating-point, the number of days since the goods were put into storage, used for inventory aging control (F objective). Stacker crane node characteristics: Each stacker crane corresponds to one node, whose characteristics include: Fixed service aisle (aisle_id): Integer, used for equipment-cargo location binding; Current column coordinate (current_col_norm): Floating-point, normalized value of the current column coordinate (current_col / N_col), used for location awareness; Current row coordinate (current_level_norm): Floating-point, normalized value of the current level coordinate (current_level / N_level), used for vertical positioning; Running status (status): Integer, running status (0=idle, 1=moving, 2=working), used for action feasibility judgment; Task queue length (task_queue_len): Integer, number of tasks to be processed, used for load balancing optimization. Edge characteristics between storage location nodes (within the same aisle): Storage location distance: Floating-point type, the cost of moving between two storage locations, proportional to the sum of column difference and layer difference, used for GAT aggregation reference; Storage location co-access score: Floating-point type ∈ [0, 1], dynamically accumulated co-access score, initially 0, this value increases by a fixed step size (e.g., 0.01) whenever the stacker crane operates two storage locations consecutively, with an upper limit of 1.0, reflecting task continuity and the strength of association between storage locations, supporting strategy evolution. Edge characteristics between storage location nodes and stacker crane nodes (only when the storage location is within the stacker crane service aisle): Stacker crane arrival time: Floating-point type, the time required for the stacker crane to move from its current position to the target storage location, calculated based on an acceleration / deceleration model, used for task response speed optimization; Stacker crane arrival energy cost: Floating-point type, positively correlated with stacker crane arrival time (travel_time), reflecting energy consumption cost, used for energy efficiency optimization objectives.

[0045] The following describes in detail step 104, namely, "based on the state representation and the historical inbound / outbound order table of the first target, obtain the first probability distribution of actions in the action set based on a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and vacancy."

[0046] For example, the reinforcement learning network is a proximal policy optimization algorithm based on an actor-critic framework. This proximal policy optimization algorithm includes an actor-based policy network and a critic-based policy network.

[0047] In one embodiment of the invention, as a possible implementation, a policy network based on actors can be trained using state representations and a historical inbound / outbound order table of the first target as training samples. The output is a first probability distribution of actions in the action set. Actions with higher probabilities (i.e., the corresponding storage locations) are more suitable as the execution target of the current task. To ensure the legality of actions, storage locations that do not conform to business rules (such as those that are already occupied or beyond the service range of the stacker crane) will be automatically blocked, making their probability zero.

[0048] Here, the state represents the value of the scheduled state predicted by the policy network based on the critics, used to characterize the quality of the action. A positive prediction indicates a good action, and a negative prediction indicates a poor action.

[0049] The following describes in detail step 105, namely, "determining a first action based on the first probability distribution of actions in the action set, the first action being executed in the environment of entering and exiting an elevated automated warehouse, and generating an immediate reward to be sent to the reinforcement learning network," in conjunction with an embodiment of the present invention.

[0050] As one possible implementation, the action with the highest probability in the first probability distribution is selected and determined as the first action.

[0051] As one possible implementation, the instant reward function is as follows: R = −ω1T + ω2S + ω3C + ω4F Where: R is the immediate reward; T is the task duration, representing the time from the start to the completion of the current task; S is the cargo dispersion, measuring the degree of clustering of SKUs of the same inventory quantity in storage locations; C is the center of gravity evaluation index, reflecting the proportion of heavy cargo at high levels; F is the turnover rate distribution index, measuring whether high-turnover cargo is concentrated in easily accessible areas; ω1, ω2, ω3, and ω4 are the weighting coefficients, respectively.

[0052] The following describes step 106, namely "updating the heterogeneous graph and generating the first heterogeneous graph according to the first action", in detail with reference to the embodiments of the present invention.

[0053] In one embodiment of the present invention, as a possible implementation, after the first action is performed, the idle state of the storage location changes. At this time, the structured database is updated first, and then the heterogeneous graph is updated according to the data that has changed in the structured database.

[0054] The following describes step 107, namely, "repeating steps 103 to 106 until the iteration termination condition is met to obtain the trained reinforcement learning network," in conjunction with an embodiment of the present invention.

[0055] In this embodiment of the invention, the iteration termination condition can be a threshold number of iterations, such as 1000 times or 10000 times.

[0056] It should be noted that after obtaining the trained reinforcement learning network, it is necessary to validate it. Specifically, the multi-source business data obtained in step 101 also includes a historical inbound / outbound task table. This historical inbound / outbound task table represents the details of tasks that have been assigned to storage locations and completed, including indicators such as: task type (outbound / inbound), target storage location number, execution start and end time, stacker crane number, and actual time consumption. These fields are used to analyze task continuity, calculate the co-access score, and construct task flows. This historical inbound / outbound task table includes a first target historical inbound / outbound task table and a second target historical inbound / outbound task table. 80% of the historical inbound / outbound task table constitutes the first target historical inbound / outbound task table. 20% of the historical inbound / outbound task table constitutes the second target historical inbound / outbound task table. The second target historical inbound / outbound task table corresponds to the second target historical inbound / outbound order table. The validation process is as follows: Step 1: Based on the state representation and the historical inbound / outbound order table of the second target, and using the trained reinforcement learning network, the third probability distribution of actions in the action set can be obtained, and the action corresponding to the highest probability is determined as the third action.

[0057] Step 2: Update the heterogeneous graph according to the third action to obtain the third heterogeneous graph.

[0058] Step 3: Based on the third heterogeneous graph, obtain the target state representation using a graph attention network.

[0059] Step 4: Repeat steps 1 to 3 until all orders in the second objective's historical inbound / outbound order table are executed, resulting in a third action. If the calculated immediate reward after the third action is completed is better than the immediate reward for completing the tasks in the second objective's historical inbound / outbound task table, then the scheduling strategy of the trained reinforcement learning network is determined to be superior to the traditional scheduling strategy. Otherwise, the scheduling strategy of the trained reinforcement learning network is determined to be inferior to the traditional scheduling strategy.

[0060] It should also be noted that a simulation environment can be set up in this embodiment of the invention. Steps 104 and 105 can be performed in this simulation environment, thus saving the training cost of the reinforcement learning network. Similarly, the process of verifying the trained reinforcement learning network can also be performed in this simulation environment.

[0061] The beneficial effects of the technical solution provided by this invention are as follows: 1) For the first time, the physical layout, equipment constraints (i.e., stacker cranes in each aisle can only be used in their own aisle and cannot be allocated between aisles) and task collaboration relationships of the elevated automated warehouse are uniformly modeled as a heterogeneous graph structure, and this graph is used as the perception environment of the reinforcement learning network, breaking through the limitations of traditional scheduling methods that rely on gridded or vectorized state representations; the graph attention network (GAT) is used to calculate the first attention weight between a storage location and its neighboring storage locations, and the second attention weight between a storage location and its corresponding stacker crane, and dynamically aggregates the first and second attention weights, so that the state representation is naturally embedded with spatial reachability and historical task relevance, providing a structured, interpretable, and high-information-density input foundation for reinforcement learning, significantly improving the efficiency and generalization ability of policy learning; 2) It abandons the traditional static scheduling rules (such as proximity principle, high-level priority, etc.) that rely on manual experience, and realizes the autonomous evolution and real-time decision-making of scheduling strategies through an end-to-end reinforcement learning framework; When a task arrives, the optimal location allocation action is output instantly based on the current heterogeneous graph state, without the need for preset logic branches. It can dynamically balance multiple conflicting objectives such as efficiency, safety, and inventory health, and has strong adaptability and online response capabilities. It truly realizes the transformation from "human-made rules" to "intelligent decision-making," meeting the urgent needs of high-turnover warehousing scenarios for real-time and intelligent scheduling. 3) A data-driven simulation test environment based on real historical task flows and cargo attributes has been constructed, which can reproduce the operation logic of high-bay automated warehouses, the dynamic behavior of stacker cranes, and the randomness of tasks with high fidelity. It supports offline training, online verification, and continuous iteration of reinforcement learning strategies before deployment. This simulation environment not only significantly reduces the cost of on-site trial and error, but also ensures the accuracy and comparability of algorithm performance evaluation. Combining heterogeneous graph structure modeling, multi-objective reinforcement learning training, and high-fidelity simulation closed loop, this invention forms a complete technical chain of "data input - graph modeling - intelligent decision-making - simulation verification - strategy deployment." The overall system has high integration, portability, and engineering implementation value, providing a replicable and scalable new paradigm for intelligent warehouse scheduling.

[0062] This invention also provides a method for scheduling storage locations in and out of an elevated automated warehouse. A flowchart of this method is shown below. Figure 2 As shown. Figure 2 The method is shown to include the following steps: Step 201: Obtain the current multi-source business data, which includes: the product specification data table, the current shelf structure table, and the current inbound / outbound order table. The current inbound / outbound order table is used to represent the current task requests for unassigned storage locations.

[0063] Step 202: Generate the current heterogeneous graph based on the product specification data table and the current shelf structure table. The current heterogeneous graph includes the current storage location node, the current stacker crane node, the edges between the current storage location nodes, and the edges between the current storage location node and the current stacker crane node.

[0064] Step 203: Based on the current heterogeneous graph, obtain the current state representation using a graph attention network.

[0065] Step 204: Based on the current state representation and the current inbound / outbound order table, obtain the second probability distribution of actions in the current action set based on the trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability.

[0066] Step 205: Determine the second action based on the second probability distribution of the actions in the current action set. The second action is to be executed in the environment of entering and leaving the high-rise warehouse.

[0067] Step 206: Based on the second action, update the current heterogeneous graph and generate the second heterogeneous graph.

[0068] Step 207: Repeat steps 203 to 206 until, in the case of an outbound scenario from an automated warehouse, all occupied storage locations for the first product specification have been assigned to the corresponding stacker cranes for scheduling; in the case of an inbound scenario from an automated warehouse, all empty storage locations that meet the conditions of the second product specification, weight, and availability have been assigned to the corresponding stacker cranes for scheduling.

[0069] First, in conjunction with the embodiments of the present invention, the above step 201, namely "obtaining current multi-source business data, the current multi-source business data including: goods specification data table, current shelf structure table, current inbound and outbound order table, the current inbound and outbound order table is used to represent the current task request for unassigned storage location", will be described in detail.

[0070] In this embodiment of the invention, the current shelf structure table and the historical shelf structure table include the same indicators. The current inbound / outbound order table and the historical inbound / outbound order table include the same indicators.

[0071] The following describes in detail step 202, namely, "generating a current heterogeneous graph based on the product specification data table and the current shelf structure table, wherein the current heterogeneous graph includes the current storage location node, the current stacker crane node, the edge between the current storage location node and the edge between the current storage location node and the current stacker crane node," in conjunction with an embodiment of the present invention.

[0072] In one embodiment of the invention, as a possible implementation, the product specification data table and the current shelf structure table are preprocessed to generate preprocessed current data. Preprocessing includes removing duplicate data, filling in missing data (e.g., using interpolation methods), normalizing numerical data (e.g., normalizing weight and dimensions to the [0, 1] range), and embedding encoding into non-numerical data. Based on the preprocessed current data, a current heterogeneous graph is generated.

[0073] The following describes step 203, namely "obtaining the current state representation based on the current heterogeneous graph and a graph attention network", in detail with reference to embodiments of the present invention.

[0074] As one possible implementation, based on graph attention networks, for the current heterogeneous graph, the features of the current storage location node, the current stacker crane node, the edge features between the current storage location nodes, and the edge features between the current storage location node and the current stacker crane node are extracted. These features are then fused to obtain the current state representation.

[0075] As one possible implementation, fusing the features of the current storage location node, the current stacker crane node, the edge features between the current storage location nodes, and the edge features between the current storage location node and the current stacker crane node to obtain the current state representation can be specifically as follows: Based on the features of the current storage location node and the edge features between the current storage location nodes, determine the first current attention weight between the current storage location node and its neighboring current storage location nodes; based on the features of the current storage location node, the current stacker crane node, and the edge features between the current storage location node and the current stacker crane node, determine the second current attention weight between the current storage location node and its neighboring current stacker crane node; based on the features of the neighboring current storage location nodes and their corresponding first current attention weights, and the features of the neighboring current stacker crane nodes and their corresponding second current attention weights, determine the current state representation.

[0076] Specifically, determining the first current attention weights of the current storage location node and its neighboring current storage locations based on the current storage location node's features and the edge features between the current storage location node and its neighbors can be achieved by: using Gaussian Interference Amplifier (GAT), concatenating the current storage location node's features, the features of its neighboring current storage locations, and the edge features between the current storage location node and its neighbors to obtain a first current concatenation vector. This first current concatenation vector is then input into a learnable small neural network, which outputs a first current score. The first current scores of all neighboring current storage locations are then normalized to obtain the first current attention weights for each of the neighboring current storage locations.

[0077] Based on the features of the current storage location node, the features of the current stacker crane node, and the edge features between the current storage location node and the current stacker crane node, the second current attention weights for the current storage location node and its neighboring current stacker crane nodes can be determined as follows: Using Gaussian Interference Aspects (GATs), the features of the current storage location node, the features of its neighboring current stacker crane nodes, and the edge features between them are concatenated to obtain a second current concatenated vector. This second current concatenated vector is then input into a learnable small neural network, which outputs a second current score. The second current scores of all neighboring current stacker crane nodes are normalized to obtain the second current attention weights for each of the neighboring current stacker crane nodes.

[0078] Based on the features of the current storage location nodes and their corresponding first current attention weights, and the features of the current stacker crane nodes and their corresponding second current attention weights, the current state representation is determined as follows: The features of each current storage location node and its corresponding first current attention weight are summed in weight to obtain a first current weighted result. The features of each current stacker crane node and its corresponding second current attention weight are summed in weight to obtain a second current weighted result. The first and second current weighted results are added together to obtain the current aggregated information. This current aggregated information is processed by a non-linear activation function, which becomes the second state representation of the current storage location node at the current GAT layer. The above steps are repeated at each layer of the GAT to obtain multiple second state representations. These multiple second state representations are further merged into the current state representation of the entire warehouse through global pooling (e.g., taking the maximum value or weighted average).

[0079] The following describes in detail step 204, namely, "based on the current state representation and the current inbound / outbound order table, obtain the second probability distribution of actions in the current action set based on the trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated automated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated automated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability."

[0080] In this embodiment of the invention, the current state representation and the current inbound / outbound order table are input into the trained reinforcement learning network to obtain the second probability distribution of actions in the current action set.

[0081] The following describes step 205, namely, "determining a second action based on the second probability distribution of actions in the current action set, the second action being executed in the environment of entering and exiting the elevated automated warehouse," in conjunction with an embodiment of the present invention.

[0082] In one embodiment of the invention, as a possible implementation, the action in the current action set corresponding to the highest probability in the second probability distribution is determined as the second action.

[0083] The following describes step 206, namely "updating the current heterogeneous graph and generating a second heterogeneous graph according to the second action", in conjunction with the embodiments of the present invention.

[0084] In one possible implementation of this invention, after the second action is performed, the current vacancy status of the cargo location changes, and a second heterogeneous graph is generated based on the changed cargo location.

[0085] The following describes in detail step 207, namely, "repeating steps 203 to 206 until, in the case of an outbound scenario from an automated warehouse, all occupied storage locations storing the first product specification have been allocated to the corresponding stacker cranes for scheduling; and in the case of an inbound scenario from an automated warehouse, all empty storage locations that meet the conditions of the second product specification, weight, and availability have been allocated to the corresponding stacker cranes for scheduling."

[0086] In this embodiment of the invention, steps 203 to 206 are repeated until, in the case of an outbound scenario from an automated warehouse, all occupied storage locations for the first product specification have been allocated to the corresponding stacker cranes for scheduling; in the case of an inbound scenario from an automated warehouse, all empty storage locations that meet the conditions of the second product specification, weight, and availability have been allocated to the corresponding stacker cranes for scheduling.

[0087] The beneficial effects of the technical solution provided by the embodiments of the present invention are: it realizes the autonomous evolution of the scheduling strategy (the training process of the reinforcement learning network) and real-time decision-making (using the training reinforcement learning network to make scheduling decisions for real tasks).

[0088] It should be noted that, according to another embodiment, a reinforcement learning network training method for inbound and outbound storage location scheduling in an elevated automated warehouse is also provided. A flowchart of this method is shown below. Figure 3 As shown. This method is based on, Figure 4 The system architecture implementation is shown below. Figure 3 As shown, the method includes the following steps: Step 301: Collect and process product, inventory, and equipment data.

[0089] Step 302: Structural design of elevated automated warehouse - stacker crane diagram.

[0090] Step 303: Set up the simulation environment.

[0091] Step 304: Reinforcement learning network design and training.

[0092] Step 305: Perform simulation testing on the trained reinforcement learning network.

[0093] First, the above step 301, namely "collecting and processing commodity, inventory, and equipment data", will be described in detail with reference to the embodiments of the present invention.

[0094] In this embodiment of the invention, by Figure 4 The product, inventory, and equipment data layer in the system architecture shown is responsible for collecting and processing product, inventory, and equipment data, i.e., multi-source business data, i.e., step 101.

[0095] The following describes step 302, namely "high-rise automated warehouse - stacker crane structural design", in detail with reference to the embodiments of the present invention.

[0096] In this embodiment of the invention, by Figure 4 The elevated automated warehouse-stacking crane diagram structure layer in the system architecture shown is responsible for the structural design of the elevated automated warehouse-stacking crane diagram, i.e., step 102.

[0097] The following describes step 303, namely "building a simulation environment", in detail with reference to the embodiments of the present invention.

[0098] In this embodiment of the invention, by Figure 4 The simulation environment layer in the system architecture shown is responsible for building the simulation environment.

[0099] The following describes step 304, namely "reinforcement learning network design and training", in detail with reference to embodiments of the present invention.

[0100] In this embodiment of the invention, by Figure 4 The reinforcement learning network layer in the system architecture shown is responsible for the design and training of the reinforcement learning network, namely steps 104 and 105.

[0101] It should be noted that step 304 can be performed in the simulation environment built in step 303.

[0102] The following describes step 305, namely "simulation testing of the trained reinforcement learning network", in detail with reference to the embodiments of the present invention.

[0103] In this embodiment of the invention, simulation testing of the trained reinforcement learning network is described above.

[0104] According to another embodiment, a reinforcement learning network training device for scheduling inbound and outbound storage locations in an elevated automated warehouse is provided. Figure 5 A schematic block diagram of a reinforcement learning network training apparatus for inbound and outbound storage location scheduling in an automated warehouse, according to one embodiment, is shown. Figure 5 As shown, the device 500 may include: a multi-source service data acquisition module 501, a heterogeneous graph generation module 502, a state representation acquisition module 503, a first probability distribution acquisition module 504, a first action determination module 505, a first update module 506, and a first processing module 507. The main functions of each component module are as follows: The multi-source business data acquisition module 501 is used to acquire multi-source business data, which includes: a goods specification data table, a historical shelf structure table, and a historical inbound and outbound order table. The historical inbound and outbound order table is used to represent the original task request for unassigned storage locations. The historical inbound and outbound order table includes a first target historical inbound and outbound order table. The heterogeneous graph generation module 502 is used to generate a heterogeneous graph based on the product specification data table of the goods and the historical shelf structure table. The heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes. The state representation acquisition module 503 is used to obtain a state representation based on the heterogeneous graph and a graph attention network. The first probability distribution acquisition module 504 is used to obtain the first probability distribution of actions in the action set based on the state representation and the first target historical inbound / outbound order table, using a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and availability; The first action determination module 505 is used to determine a first action based on the first probability distribution of actions in the action set. The first action is used to be executed in the environment of entering and leaving the high-rise warehouse and to generate an instant reward and send it to the reinforcement learning network. The first update module 506 is used to update the heterogeneous graph according to the first action and generate a first heterogeneous graph; The first processing module 507 is used to repeat the functions of the aforementioned state representation acquisition module to the first update module until the iteration termination condition is met, thereby obtaining the trained reinforcement learning network.

[0105] In one possible implementation, the heterogeneous graph generation module 502 is specifically used to preprocess the product specification data table and the historical shelf structure table of the goods to generate preprocessed data. The preprocessing includes removing duplicate data, filling in missing data, normalizing numerical data, and embedding encoding of non-numerical data. Based on the preprocessed data, a heterogeneous graph is generated.

[0106] In one possible implementation, the state representation acquisition module 503 is specifically used to extract features of storage location nodes, stacker crane nodes, edge features between storage location nodes, and edge features between storage location nodes and stacker crane nodes based on a graph attention network for the heterogeneous graph; and to fuse the features of storage location nodes, stacker crane nodes, edge features between storage location nodes, and edge features between storage location nodes and stacker crane nodes to obtain a state representation.

[0107] In one possible implementation, the state representation acquisition module 503 includes a first attention weight determination module, a second attention weight determination module, and a state representation determination module; The first attention weight determination module is used to determine the first attention weight between a storage location node and its neighboring storage location nodes based on the characteristics of the storage location node and the edge characteristics between the storage location nodes. The second attention weight determination module is used to determine the second attention weight between the storage location node and its neighboring stacker crane node based on the characteristics of the storage location node, the characteristics of the stacker crane node, and the edge characteristics between the storage location node and the stacker crane node. The state representation determination module is used to determine the state representation based on the characteristics of neighboring storage location nodes and the corresponding first attention weight, and the characteristics of neighboring stacker crane nodes and the corresponding second attention weight.

[0108] In one possible implementation, the first action determination module 505 is specifically used to select the action with the highest current probability from the first probability distribution and determine it as the first action.

[0109] In one possible implementation, the function for the immediate reward is as follows: R = −ω1T + ω2S + ω3C + ω4F Where: R is the immediate reward; T is the task duration, representing the time from the start to the completion of the current task; S is the cargo dispersion, measuring the degree of clustering of SKUs of the same inventory quantity in storage locations; C is the center of gravity evaluation index, reflecting the proportion of heavy cargo at high levels; F is the turnover rate distribution index, measuring whether high-turnover cargo is concentrated in easily accessible areas; ω1, ω2, ω3, and ω4 are the weighting coefficients, respectively.

[0110] According to another embodiment, a storage location scheduling device for inbound and outbound operations of an elevated automated warehouse is provided. Figure 6 A schematic block diagram of a storage location scheduling device for inbound and outbound operations in an automated warehouse according to one embodiment is shown. Figure 6 As shown, the device 600 may include: a current multi-source service data acquisition module 601, a current heterogeneous graph generation module 602, a current state representation acquisition module 603, a second probability distribution acquisition module 604, a second action determination module 605, a second update module 606, and a second processing module 607. The main functions of each component module are as follows: The current multi-source business data acquisition module 601 is used to acquire current multi-source business data, which includes: a goods specification data table, a current shelf structure table, and a current inbound / outbound order table. The current inbound / outbound order table is used to represent current task requests for unassigned storage locations. The current heterogeneous graph generation module 602 is used to generate a current heterogeneous graph based on the product specification data table of the goods and the current shelf structure table. The current heterogeneous graph includes the current storage location node, the current stacker crane node, the edge between the current storage location node and the edge between the current storage location node and the current stacker crane node. The current state representation acquisition module 603 is used to obtain the current state representation based on the current heterogeneous graph and a graph attention network. The second probability distribution acquisition module 604 is used to obtain the second probability distribution of actions in the current action set based on the current state representation and the current inbound / outbound order table, and on the basis of the trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability; The second action determination module 605 is used to determine a second action based on the second probability distribution of actions in the current action set. The second action is used to be executed in the environment of the high-rise automated warehouse entry and exit. The second update module 606 is used to update the current heterogeneous graph according to the second action and generate a second heterogeneous graph; The second processing module 607 is used to repeat the function of the current state representation acquisition module to the second update module until, when it is an outbound scenario of an elevated warehouse, all occupied storage locations storing the first item have been allocated to the corresponding stacker cranes to complete the scheduling; when it is an inbound scenario of an elevated warehouse, all empty storage locations that meet the conditions of the second item, weight, and vacancy have been allocated to the corresponding stacker cranes to complete the scheduling.

[0111] In one possible implementation, the current heterogeneous graph generation module 602 is specifically used to preprocess the product specification data table of the goods and the current shelf structure table to generate preprocessed current data. The preprocessing includes removing duplicate data, filling in missing data, normalizing numerical data, and embedding encoding for non-numerical data. Based on the preprocessed current data, the current heterogeneous graph is generated.

[0112] In one possible implementation, the current state representation acquisition module 603 is used to extract features of the current storage location node, the current stacker crane node, the edge features between the current storage location nodes, and the edge features between the current storage location node and the current stacker crane node based on a graph attention network, for the current heterogeneous graph. The current storage location node features, the current stacker crane node features, the edge features between the current storage location nodes, and the edge features between the current storage location node and the current stacker crane node are then fused to obtain the current state representation.

[0113] In one possible implementation, the current state representation acquisition module 603 includes a first current attention weight determination module, a second current attention weight determination module, and a current state representation determination module.

[0114] The first current attention weight determination module is used to determine the first current attention weight between the current storage location node and its neighboring current storage location nodes based on the characteristics of the current storage location node and the edge characteristics between the current storage location nodes. The second current attention weight determination module is used to determine the second current attention weight between the current storage location node and its neighboring current storage location node based on the characteristics of the current storage location node, the characteristics of the current stacker crane node, and the edge characteristics between the current storage location node and the current stacker crane node. The current state representation determination module is used to determine the current state representation based on the characteristics of the neighbor's current storage location node and the corresponding first current attention weight, and the characteristics of the neighbor's current stacker crane node and the corresponding second current attention weight.

[0115] In one possible implementation, the second action determination module 605 is specifically used to select the action with the highest current probability in the second probability distribution and determine it as the second action.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0117] In addition, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0118] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0119] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0120] in, Figure 7 An exemplary architecture of an electronic device is shown, which may specifically include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720 can communicate with each other via a communication bus 730.

[0121] The processor 710 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0122] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store the operating system 721 for controlling the operation of the electronic device 700, and the basic input / output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. Additionally, it can store a web browser 723, a data storage management system 724, and a reinforcement learning network training device or a storage location scheduling device 725 for inbound and outbound storage in an automated warehouse, etc. The aforementioned reinforcement learning network training device or storage location scheduling device 725 for inbound and outbound storage in an automated warehouse can be the application program that specifically implements the aforementioned steps in this embodiment of the invention. In summary, when implementing the technical solution provided in this embodiment of the invention through software or firmware, the relevant program code is stored in the memory 420 and executed by the processor 710.

[0123] Input / output interface 713 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0124] Network interface 714 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0125] Bus 730 includes a pathway for transmitting information between various components of the device, such as processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720.

[0126] It should be noted that although the above-described device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning network training method for inbound and outbound storage location scheduling in an elevated automated warehouse, characterized in that, include: Step 1: Obtain multi-source business data, which includes: a goods specification data table, a historical shelf structure table, and a historical inbound / outbound order table. The historical inbound / outbound order table is used to represent the original task requests for unassigned storage locations. The historical inbound / outbound order table includes a first target historical inbound / outbound order table. Step 2: Generate a heterogeneous graph based on the product specification data table and the historical shelf structure table. The heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes. Step 3: Based on the heterogeneous graph, obtain the state representation using a graph attention network; Step 4: Based on the state representation and the first target historical inbound / outbound order table, obtain the first probability distribution of actions in the action set using a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and availability. Step 5: Based on the first probability distribution of the actions in the action set, determine the first action, which is used to perform in the environment of entering and exiting the elevated automated warehouse, and generate an instant reward to be sent to the reinforcement learning network. Step 6: Update the heterogeneous graph according to the first action to generate a first heterogeneous graph; Step 7: Repeat steps 3 to 6 until the iteration termination condition is met to obtain the trained reinforcement learning network.

2. The method according to claim 1, characterized in that, The step of generating a heterogeneity diagram based on the product specification data table and the historical shelf structure table includes: The product specification data table and the historical shelf structure table of the goods are preprocessed to generate preprocessed data. The preprocessing includes removing duplicate data, filling in missing data, normalizing numerical data, and embedding and encoding non-numerical data. A heterogeneous graph is generated based on the preprocessed data.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the state representation based on the heterogeneous graph and a graph attention network includes: Based on graph attention network, for the heterogeneous graph, the features of storage location nodes, stacker crane nodes, edge features between storage location nodes, and edge features between storage location nodes and stacker crane nodes are extracted. The features of the storage location node, the features of the stacker crane node, the edge features between storage location nodes, and the edge features between storage location nodes and stacker crane nodes are fused to obtain a state representation.

4. The method according to claim 3, characterized in that, The process of fusing the features of the storage location node, the features of the stacker crane node, the edge features between storage location nodes, and the edge features between storage location nodes and stacker crane nodes to obtain a state representation includes: Based on the characteristics of the storage location node and the edge characteristics between storage location nodes, determine the first attention weight between the storage location node and its neighboring storage location nodes; Based on the features of the storage location node, the features of the stacker crane node, and the edge features between the storage location node and the stacker crane node, determine the second attention weight between the storage location node and its neighboring stacker crane node; The state representation is determined based on the characteristics of the neighboring warehouse node and the corresponding first attention weight, and the characteristics of the neighboring stacker crane node and the corresponding second attention weight.

5. The method according to claim 1, characterized in that, Determining the first action based on the first probability distribution of actions in the action set includes: Select the action with the highest probability in the first probability distribution and determine it as the first action.

6. The method according to claim 1, characterized in that, The function for the instant reward is as follows: R = −ω1T + ω2S + ω3C + ω4F Where: R is the immediate reward; T is the task duration, representing the time from the start to the completion of the current task; S is the cargo dispersion, measuring the degree of clustering of SKUs of the same inventory quantity in storage locations; C is the center of gravity evaluation index, reflecting the proportion of heavy cargo at high levels; F is the turnover rate distribution index, measuring whether high-turnover cargo is concentrated in easily accessible areas; ω1, ω2, ω3, and ω4 are the weighting coefficients, respectively.

7. A reinforcement learning network training device for scheduling inbound and outbound storage locations in an elevated automated warehouse, characterized in that, include: The multi-source business data acquisition module is used to acquire multi-source business data, which includes: a goods specification data table, a historical shelf structure table, and a historical inbound and outbound order table. The historical inbound and outbound order table is used to represent the original task requests for unassigned storage locations. The historical inbound and outbound order table includes a first target historical inbound and outbound order table. The heterogeneous graph generation module is used to generate a heterogeneous graph based on the product specification data table of the goods and the historical shelf structure table. The heterogeneous graph includes storage location nodes, stacker crane nodes, edges between storage location nodes, and edges between storage location nodes and stacker crane nodes. The state representation acquisition module is used to obtain a state representation based on the heterogeneous graph and a graph attention network. The first probability distribution acquisition module is used to obtain the first probability distribution of actions in the action set based on the state representation and the first target historical inbound / outbound order table, using a reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the action set is all occupied storage locations storing the first target product specification; when it is an inbound scenario of an elevated warehouse, the action set is all empty storage locations that meet the conditions of the second target product specification, weight, and availability. The first action determination module is used to determine a first action based on the first probability distribution of actions in the action set. The first action is used to be executed in the environment of entering and leaving the elevated automated warehouse, and an instant reward is generated and sent to the reinforcement learning network. The first update module is used to update the heterogeneous graph according to the first action and generate a first heterogeneous graph; The first processing module is used to repeat the functions of the aforementioned state representation acquisition module to the first update module until the iteration termination condition is met, thereby obtaining the trained reinforcement learning network.

8. A method for scheduling warehouse locations for inbound and outbound operations in an elevated automated warehouse, characterized in that, include: Step 1: Obtain current multi-source business data, which includes: a goods specification data table, a current shelf structure table, and a current inbound / outbound order table. The current inbound / outbound order table is used to represent current task requests for unassigned storage locations. Step 2: Generate a current heterogeneous graph based on the product specification data table and the current shelf structure table. The current heterogeneous graph includes the current storage location node, the current stacker crane node, the edges between current storage location nodes, and the edges between current storage location nodes and current stacker crane nodes. Step 3: Based on the current heterogeneous graph, obtain the current state representation using a graph attention network; Step 4: Based on the current state representation and the current inbound / outbound order table, obtain the second probability distribution of actions in the current action set using the trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability. Step 5: Determine a second action based on the second probability distribution of actions in the current action set. The second action is to be executed in the environment of entering and leaving the elevated automated warehouse. Step 6: Update the current heterogeneous graph according to the second action to generate a second heterogeneous graph; Step 7: Repeat steps 3 to 6 until, in the case of outbound from the high-bay warehouse, all occupied storage locations for the first item have been assigned to the corresponding stacker cranes for scheduling; in the case of inbound from the high-bay warehouse, all empty storage locations that meet the conditions of the second item, weight, and availability have been assigned to the corresponding stacker cranes for scheduling.

9. A storage location scheduling device for inbound and outbound operations in an elevated automated warehouse, characterized in that, include: The current multi-source business data acquisition module is used to acquire current multi-source business data, which includes: a goods specification data table, a current shelf structure table, and a current inbound / outbound order table. The current inbound / outbound order table is used to represent current task requests for unassigned storage locations. The current heterogeneous graph generation module is used to generate a current heterogeneous graph based on the product specification data table of the goods and the current shelf structure table. The current heterogeneous graph includes the current storage location node, the current stacker crane node, the edge between the current storage location node and the edge between the current storage location node and the current stacker crane node. The current state representation acquisition module is used to obtain the current state representation based on the current heterogeneous graph and a graph attention network. The second probability distribution acquisition module is used to obtain the second probability distribution of actions in the current action set based on the current state representation and the current inbound / outbound order table, using a trained reinforcement learning network; wherein, when it is an outbound scenario of an elevated warehouse, the current action set is all occupied storage locations storing the first item; when it is an inbound scenario of an elevated warehouse, the current action set is all empty storage locations that meet the conditions of the second item, weight, and availability. The second action determination module is used to determine a second action based on the second probability distribution of actions in the current action set. The second action is used to execute in the environment of the high-rise automated warehouse entry and exit. The second update module is used to update the current heterogeneous graph according to the second action and generate a second heterogeneous graph; The second processing module is used to repeat the function of the current state representation acquisition module to the second update module until, when it is an outbound scenario of an elevated warehouse, all occupied storage locations storing the first specification have been allocated to the corresponding stacker cranes for scheduling; when it is an inbound scenario of an elevated warehouse, all empty storage locations that meet the conditions of the second specification, weight, and vacancy have been allocated to the corresponding stacker cranes for scheduling.

10. An electronic device, characterized in that, include: The memory and the processor communicate with each other via a bus; The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method as described in any one of claims 1 to 6 or claim 8.

Images