An intelligent warehouse resource dynamic allocation method based on deep reinforcement learning

Abstract

This invention discloses an intelligent warehousing resource dynamic allocation method based on deep reinforcement learning, belonging to the field of warehousing allocation technology. The method includes: based on operational status representation vectors and dynamic weight vectors, using a high-level task allocation network and a low-level resource assignment network of an initial strategy model to allocate resource types and screen executors, respectively, to obtain a candidate scheduling instruction set; using a digital twin model, a multi-objective reward function, and dynamic weight vectors to perform forward-looking inference and multi-objective comprehensive evaluation of the candidate scheduling instruction set; selecting the scheduling instruction with the optimal evaluation result and passing safety rule verification; and issuing the selected optimal scheduling instruction to the warehousing operation entity for execution. This invention, by generating optimal scheduling instructions, not only improves resource scheduling efficiency but also reduces equipment collision conflict rate, reduces empty mileage, reduces average waiting latency, and lowers equipment energy consumption during task execution.

Description

Technical Field

[0001] This invention relates to the field of warehouse allocation technology, and in particular to an intelligent method for dynamic allocation of warehouse resources based on deep reinforcement learning. Background Technology

[0002] As global manufacturing accelerates its evolution towards intelligence and flexibility, intelligent warehousing, as the core hub of the modern supply chain system, directly determines the response speed and cost control capabilities of the entire logistics network through its operational efficiency and reliability. In recent years, automated equipment, represented by automated storage and retrieval systems (AS / RS), automated guided vehicles (AGVs), and intelligent sorting systems, has been widely used in warehousing scenarios, driving the transformation of warehousing operations from the previous manual mode to a highly integrated and digital intelligent mode. At the same time, combined with advanced algorithm architectures (such as attention-based policy networks, dual-network update mechanisms, and priority experience replay), a solid technical foundation has been provided for building truly adaptive intelligent warehouse scheduling.

[0003] Nevertheless, there is still room for improvement in existing warehouse scheduling. First, although some studies have attempted to introduce digital twin technology to build virtual simulation environments for strategy verification, they generally lack a two-way closed-loop linkage mechanism with the real physical environment, resulting in a discrepancy between simulation results and actual execution effects. Second, most of the current mainstream reinforcement learning frameworks rely on online interactive training, and the exploration process carries uncontrollable risks, making it difficult to meet the stringent safety and stability requirements of industrial sites. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning to solve the problems of discrepancies between simulation results and actual execution effects, and the difficulty in meeting the stringent safety and stability requirements of industrial sites.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning, comprising: Collect operational data of the warehouse operation entities, and perform different feature engineering processes on categorical and numerical data in the operational data to obtain operational status representation vectors; After collecting the design data of the warehouse operation entity and establishing a digital twin model, historical operation data is used for simulation to obtain interaction experience. Based on the interaction experience, an offline reinforcement learning algorithm is used to pre-train the deep reinforcement learning policy network to obtain the initial policy model. The desired optimization allocation target and multi-objective reward function are preset; Based on the operational data of the warehouse operation entity, the task backlog coefficient, path congestion coefficient, and remaining power constraint coefficient are calculated to obtain the operational status index, and the operational status index is mapped to a dynamic weight vector in the multi-objective reward function. Based on the operational status representation vector and dynamic weight vector, the high-level task allocation network and low-level resource assignment network of the initial strategy model are used to allocate resource types and screen execution entities to obtain a candidate scheduling instruction set. Using digital twin models, multi-objective reward functions, and dynamic weight vectors, a forward-looking simulation and multi-objective comprehensive evaluation of the candidate scheduling instruction set are conducted, and the scheduling instruction with the best evaluation results and that passes the safety rule verification is selected. The selected optimal scheduling instruction is issued to the warehouse operation entity for execution.

[0007] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the operation data of the warehousing operation entity includes equipment status data, task data, path occupancy data, and equipment remaining power data.

[0008] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, wherein: obtaining the operational status representation vector specifically involves: One-hot encoding is performed on the categorical data in the running data to obtain the first numerical feature; The numerical data in the operational data is normalized to obtain the second numerical feature; The first numerical feature is concatenated with the second numerical feature to generate an operational status representation vector.

[0009] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the step of establishing a digital twin model based on the design data of the warehousing operation entity specifically includes: Based on the design data of the warehouse operation entity, construct a static scene map; Dynamic modeling and behavior rule extraction are performed on movable equipment entities in static scene maps to form a dynamic behavior simulator; Establish a two-way real-time data channel between the dynamic behavior simulator and the warehouse operation entity for equipment status, order information and environmental parameters; A virtual simulation environment integrating static scene maps, dynamic behavior simulators, and two-way real-time data channels will be deployed as a digital twin model of the warehouse operation entity.

[0010] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the step of obtaining the initial strategy model specifically involves: Collect historical operational data of the warehouse operation entities, input the historical operational data into the digital twin model, and the digital twin model drives the internal dynamic behavior simulator to simulate the historical motion trajectory and operation process of the movable equipment entities in the static scene map to obtain interactive experience; Based on interactive experience, an offline reinforcement learning algorithm is used to pre-train the deep reinforcement learning policy network to obtain an initial policy model.

[0011] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the preset allocation target and multi-objective reward function are specifically as follows: Preset the desired optimization allocation target; The desired optimization allocation objectives include sub-objectives for task completion efficiency, operational safety constraints, and equipment energy consumption control. The multi-objective reward function is preset to a linear weighted sum structure.

[0012] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, wherein mapping the operational status indicators to a dynamic weight vector in a multi-objective reward function specifically involves: Calculate the task backlog coefficient based on task data, calculate the path congestion coefficient based on path occupancy data, and calculate the remaining power constraint coefficient based on device remaining power data. The task backlog coefficient, path congestion coefficient, and remaining power constraint coefficient are mapped to the weight coefficients of the multi-objective reward function, and then concatenated after normalization to form a dynamic weight vector.

[0013] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the step of obtaining the candidate scheduling instruction set specifically includes: The operational status representation vector and the dynamic weight vector are combined into a composite input vector; The composite input vector is input into the initial policy model, and the high-level task allocation network in the initial policy model calculates and generates resource type allocation suggestions based on the composite input vector. The low-level resource assignment network in the initial strategy model calculates attention scores among all available resources based on resource type allocation suggestions and operational status representation vectors, and selects the N resource individuals with the highest scores as candidate executors through an attention mechanism. The resource type allocation suggestions and the selected candidate executors are combined to form a candidate scheduling instruction set.

[0014] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the step of selecting the scheduling instruction with the optimal evaluation result and that passes the safety rule verification specifically includes: Each scheduling instruction in the candidate scheduling instruction set is sequentially input into the digital twin model for state simulation to obtain simulation data; Based on the preset set of security rules, a security check is performed on the deduction process of each scheduling instruction in the digital twin model, and scheduling instructions that pass the security check are selected. For scheduling instructions that pass the security check, the expected multi-objective reward value is calculated by calling the simulation data, dynamic weight vector and multi-objective reward function, and the scheduling instruction with the highest expected multi-objective reward value is taken as the optimal scheduling instruction.

[0015] As a preferred embodiment of the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning described in this invention, the step of issuing the selected optimal scheduling instruction to the warehousing operation entity for execution specifically includes: Convert the optimal scheduling instructions into a sequence of low-level control instructions that the device can recognize; The underlying control command sequence is sent to the corresponding device controller. After receiving and parsing, the device controller drives the physical devices in the warehouse operation entity to execute.

[0016] The beneficial effects of this invention are as follows: It constructs a digital twin model that integrates a static scene graph and a dynamic behavior simulator, enabling the state reproduction and deduction of the operational process of warehouse entities in a virtual environment. This provides a repeatable interactive environment for the pre-training of deep reinforcement learning strategy networks. By generating dynamic weight vectors through task backlog coefficients, path congestion coefficients, and remaining power constraint coefficients, it can dynamically adjust the weight coefficients corresponding to different sub-objectives in the multi-objective reward function, thereby improving the adaptability of the multi-objective reward function to changes in the current operating state. Simultaneously, it performs deduction, safety checks, and multi-objective comprehensive evaluation of candidate scheduling instructions within the digital twin model. This allows for the identification of abnormal situations such as potential path conflicts, insufficient remaining power, and faulty equipment participating in execution before the scheduling instructions are issued. This improves the executability of candidate scheduling instructions, reduces equipment collision conflict rates, reduces empty mileage, and lowers average waiting latency and equipment energy consumption during task execution. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1The flowchart shows a method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning. Figure 2 A flowchart for driving physical devices to perform operations; Figure 3 A flowchart for generating operational status representation vectors; Figure 4 A flowchart for forming a dynamic weight vector. Detailed Implementation

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0022] Reference Figures 1-4 This is one embodiment of the present invention, which provides a method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning, including the following steps: S1. Collect operational data of the warehouse operation entities. For categorical and numerical data in the operational data, perform different feature engineering processes to obtain operational status representation vectors.

[0023] S1.1 should be explained as follows: Read the data source connection configuration corresponding to the warehouse operation entity; according to the data source connection configuration, access the status record interface corresponding to the device, the task log file corresponding to the task data, the path occupancy record file corresponding to the path occupancy data, and the device power record file corresponding to the device remaining power data, respectively; extract the device identifier, current location coordinates, current speed, current orientation, load status, and fault status from the status records corresponding to each device to form device status data; scan the task records in the task log file one by one in chronological order, extract the task identifier, task type, task priority, pickup location, delivery location, task generation time, task issuance time, and task completion status from each task record to form task data; read the segment records in the path occupancy record file segment by segment according to the path segment identifier, extract the path segment identifier, segment occupancy status, occupancy start time, occupancy end time, occupant device identifier, and path travel direction from each segment record to form path occupancy data; read the power records in the device power record file one by one according to the device identifier, extract the device identifier, collection time, current remaining power value, charging status, and low power alarm status from each power record to form device remaining power data.

[0024] S1.2. It should be noted that categorical data with discrete enumeration values ​​are filtered from the operational data, including load status and fault status in device status data, task type, task priority, and task completion status in task data, path segment occupancy status and path direction in path occupancy data, and charging status and low battery alarm status in device remaining power data. For each categorical data field, all category values ​​for the corresponding field are counted, and a binary vector with a length equal to the total number of categories is created for each category value. In the binary vector, all positions are 0 except for the corresponding category index position, which is 1. Each record in the operational data is traversed, and the specific value of the categorical data field in the record is replaced with the corresponding binary vector representation. All binary vectors are concatenated, and the resulting merged vector is the first numerical feature. Numerical data fields in the operational data are identified, including current position coordinates, current speed, and current orientation in device status data; task generation time, task issuance time, and task waiting time in task data; occupancy start time, occupancy end time, and path segment occupancy duration in path occupancy data; and current remaining power value in device remaining power data. The raw values ​​of the numerical data fields are read, and the raw values ​​are normalized to their minimum and maximum values. All normalized values ​​are then concatenated to form a merged vector, which is the second numerical feature. The first and second numerical features are then concatenated end-to-end according to the field order to generate the operational status representation vector.

[0025] S2. After collecting the design data of the warehouse operation entity and establishing a digital twin model, historical operation data is used for simulation to obtain interaction experience. Based on the interaction experience, an offline reinforcement learning algorithm is used to pre-train the deep reinforcement learning policy network to obtain the initial policy model.

[0026] S2.1 It should be noted that the design data of the warehouse operation entity is collected, including warehouse layout drawings, racking layout drawings, equipment installation point diagrams, and network topology diagrams. Key entity elements and their spatial and logical relationships are extracted. Key entity elements include racks, aisles, workstations, charging piles, and automated guided vehicle (AGV) parking points. A node is created for each key entity element, and attributes are assigned to the node, including node type, unique node identifier, and two-dimensional or three-dimensional spatial coordinates. Based on the connection, adjacency, and containment relationships described in the network topology diagram, edges are established between corresponding nodes, and relationship type attributes are assigned to the edges, including "adjacent to," "connected to," and "located within." The networked data structure composed of all nodes and edges, reflecting the fixed physical structure and logical connection relationships of the warehouse operation entity, constitutes the constructed static scene graph.

[0027] S2.2 It should be noted that nodes belonging to movable equipment entities are identified in the static scene map. Movable equipment entities include automated guided vehicles, stacker cranes, and shuttle vehicles. Based on the technical specifications of the movable equipment entities, a dynamic model is established for each movable equipment entity node. Specifically, the technical specifications include the physical dimensions, mass, maximum travel speed, maximum acceleration, maximum deceleration, minimum turning radius, and maximum load of the movable equipment entity. Based on Newton's laws of motion, the differential equations of motion for the movable equipment entity are defined using technical specifications. This abstracts the motion of the movable equipment entity as the motion of a point mass or rigid body on a two-dimensional plane. Using Newton's second law, the relationship between the net external force and acceleration of the movable equipment entity is established, where the net external force equals the difference between the driving force, braking force, and rolling friction applied to the movable equipment entity at the current moment. Acceleration is defined as the first derivative of velocity with respect to time, and velocity is defined as the first derivative of position with respect to time. Thus, a differential equation of motion is obtained, with time as the independent variable and the position, velocity, acceleration, and control force (driving force / braking force) of the movable equipment entity as the state and control variables. Initial conditions and boundary constraints are set for the differential equation of motion. The initial conditions include the initial position and velocity of the movable equipment entity, and the boundary constraints include the upper and lower limits of velocity and acceleration, as well as the physical space boundaries. The mathematical descriptions of the differential equation of motion, the initial conditions, and the boundary constraints constitute the dynamic model of the movable equipment entity's nodes.

[0028] Historical control instruction sets and operation logs are extracted and analyzed from the controller of the movable equipment entity. Standard behavior patterns and operating rules for the movable equipment entity are extracted from these logs. These standard behavior patterns and operating rules include "straight-line acceleration," "curving and turning," "precise docking at the platform," "performing grabbing / storing actions," "entering the charging position," and "responding to emergency stop signals." The dynamic model is encapsulated and integrated with the standard behavior patterns and operating rules to create a dynamic behavior simulator capable of receiving scheduling instructions, simulating the physical movement of the movable equipment entity, and executing preset behavioral logic.

[0029] S2.3. It should be noted that a data publishing interface and a command receiving interface are deployed on the dynamic behavior simulator side, and a status acquisition interface and a control command execution interface are deployed on the warehouse operation entity side. A unified data exchange protocol and message format (such as MQTT protocol and JSON format) are defined, covering device status, sensor readings, control commands, and timestamps. Based on the industrial IoT communication middleware, an uplink data link is constructed connecting the dynamic behavior simulator's data publishing interface and the warehouse operation entity's status acquisition interface. This link is used to continuously and in real-time transmit the warehouse operation entity's operational data from the warehouse operation entity to the dynamic behavior simulator. Simultaneously, a downlink command link is constructed connecting the dynamic behavior simulator's command receiving interface and the warehouse operation entity's control command execution interface. This link is used to reliably and in real-time send control commands generated by the dynamic behavior simulator to the warehouse operation entity. The uplink data link and the downlink command link together constitute a bidirectional real-time data channel between the dynamic behavior simulator and the warehouse operation entity.

[0030] Configure the data interface between the static scene map and the dynamic behavior simulator, enabling the movable device entity nodes in the dynamic behavior simulator to be bound and synchronized with the corresponding spatial location nodes in the static scene map. Configure the interface between the bidirectional real-time data channel and the dynamic behavior simulator, allowing the real-time status of the warehouse operation entity transmitted through the uplink data link to drive the update of the corresponding entity status in the dynamic behavior simulator, and enabling the control commands generated by the dynamic behavior simulator to be output through the downlink command link. Encapsulate and instantiate the integrated whole (static scene map, dynamic behavior simulator, and bidirectional real-time data channel) to form an independently runnable software service with complete input and output interfaces, namely, a digital twin model of the warehouse operation entity.

[0031] S2.4. It should be noted that historical operational data is collected from the log files of the warehouse operation entity. This historical operational data includes historical equipment status data, historical task data, historical path occupancy data, and historical equipment remaining power data. The historical operational data of the warehouse operation entity is input into the digital twin model in chronological order. The digital twin model drives an internal dynamic behavior simulator to simulate the movement trajectory and operation process of the movable equipment entity within a static scene map. Simultaneously, the digital twin model receives historical control command sequences as input, simulates the execution of commands, and generates corresponding status responses. During this simulation reproduction process, the digital twin model continuously records the simulation status, received simulation commands, and generated simulation result data and feedback to obtain interactive experience. The simulation result data includes task completion time, task waiting time, number of safety violation events, and estimated energy.

[0032] S2.5. It should be noted that the interactive experience is input into the deep reinforcement learning policy network for forward propagation calculation. The forward propagation calculation is performed layer by layer according to the connection order of each network layer in the deep reinforcement learning policy network to obtain the output result corresponding to the current parameters of the deep reinforcement learning policy network. Based on the objective function of the offline reinforcement learning algorithm, combined with the state, action, reward and next state in the interactive experience, the error of the output result is calculated to obtain the loss value. The loss value includes at least the error term corresponding to the task completion reward, the error term corresponding to the device energy consumption penalty and the error term corresponding to the safety violation penalty. When the offline reinforcement learning algorithm imposes constraints on out-of-distribution actions, the loss value also includes the corresponding constraint term. The parameters of the deep reinforcement learning policy network are iteratively updated according to the loss value. The process of output result calculation, loss value calculation and parameter update is repeated until the preset training rounds (50 times) are reached. The trained deep reinforcement learning policy network is used as the initial policy model.

[0033] Furthermore, setting the preset training rounds to 50 times is beneficial for the deep reinforcement learning policy network to complete multiple rounds of parameter updates based on interactive experience, so that the loss value gradually decreases and tends to stabilize, while taking into account both training time overhead and parameter convergence effect.

[0034] The deep reinforcement learning policy network employs a hierarchical structure, comprising a high-level task allocation network and a low-level resource assignment network. The high-level task allocation network consists of multiple perceptrons, with parameters for each layer including the weight matrix and bias vector of the first fully connected layer and the weight matrix and bias vector of the second fully connected layer, used to generate resource type allocation suggestions. The low-level resource assignment network consists of attention layers responsible for calculating attention scores and selecting resources from the candidate resource pool; its parameters include query projection weight matrices, key projection weight matrices, and value projection weight matrices. The parameters of each layer are iteratively updated based on the loss value during offline reinforcement learning training.

[0035] S3, preset the expected optimization allocation target and multi-objective reward function.

[0036] S3.1 It should be noted that the preset optimization targets for allocation include task completion efficiency sub-targets, operational safety constraint sub-targets, and equipment energy consumption control sub-targets; a corresponding reward value is calculated for each allocation target, wherein the reward value for the task completion efficiency sub-target is calculated using the following formula: ; in, This represents the reward value for the sub-goal of task completion efficiency. This indicates the task completion time in the simulation results data. This indicates the task waiting time in the simulation results data.

[0037] The formula for calculating the reward value of the job safety constraint sub-objective is as follows: ; in, This represents the reward value for the sub-objective of the work safety constraint. This indicates the number of security violations in the simulation results data.

[0038] The reward value for the sub-objective of calculating equipment energy consumption control is calculated using the following formula: ; in, This represents the reward value for the sub-objective of equipment energy consumption control. This represents the estimated energy consumed by the device to complete the task, as shown in the simulation results data.

[0039] The reward values ​​for the task completion efficiency sub-objective, the operational safety constraint sub-objective, and the equipment energy consumption control sub-objective are respectively mapped to components in the multi-objective reward function. The multi-objective reward function is preset to a linear weighted sum structure.

[0040] S4. Calculate the task backlog coefficient, path congestion coefficient, and remaining power constraint coefficient based on the operational data of the warehouse operation entity to obtain the operational status index, and map the operational status index into a dynamic weight vector in the multi-objective reward function.

[0041] It should be noted that the number of tasks in an incomplete state is read from the task data, and the average number of tasks processed within the statistical period is read. The number of incomplete tasks and the average number of tasks processed are then compared to obtain the task backlog coefficient.

[0042] The number of path segments in a occupied state and the total number of path segments are read from the path occupancy data. The quotient of the number of path segments in a occupied state and the total number of path segments is calculated to obtain the path congestion coefficient.

[0043] The number of devices with power levels below a preset power threshold is read from the remaining power data. At the same time, the total number of devices is read. The remaining power constraint coefficient is obtained by quotienting the number of devices with power levels below the power threshold with the total number of devices.

[0044] The task backlog coefficient, path congestion coefficient, and remaining power constraint coefficient are respectively used as the initial weight coefficients for the task completion efficiency sub-objective reward value, the operation safety constraint sub-objective reward value, and the equipment energy consumption control sub-objective reward value in the multi-objective reward function. The ratios of the three initial weight coefficients are normalized and then concatenated to obtain the dynamic weight vector.

[0045] S5. Based on the operational status representation vector and dynamic weight vector, resource type allocation and execution body screening are performed using the high-level task allocation network and low-level resource assignment network of the initial strategy model to obtain a candidate scheduling instruction set.

[0046] S5.1 It should be noted that the operational status representation vector and the dynamic weight vector are concatenated to form a composite input vector. This composite input vector is input into the initial policy model. The composite input vector is then multiplied by the weight matrix of the first fully connected layer of the high-level task allocation network, and a bias vector is superimposed to obtain the first intermediate feature. The first intermediate feature is then subjected to element-wise nonlinear mapping using a rectified linear unit activation function to obtain the first activation feature. This first activation feature is input into the second fully connected layer of the high-level task allocation network, and a linear transformation is performed again to obtain the original task-resource type association score. This original task-resource type association score is then input into a Softmax function and normalized row-wise (task dimension) to obtain a probability distribution vector for each task to be processed across different resource types. Each element in the probability distribution vector corresponds to the allocation probability of a resource type, and the sum of all elements is 1. For each task to be processed, the probability values ​​of each resource type in the probability distribution vector are read, and the resource type with the highest probability value is selected as the resource type allocation suggestion for the corresponding task.

[0047] S5.2 It should be noted that the low-level resource assignment network receives resource type allocation suggestions and operational status representation vectors. The low-level resource assignment network first extracts the state features of all available resource entities from the operational status representation vector to form a resource feature matrix. The resource type allocation suggestions are one-hot encoded to obtain a resource type encoding vector. The resource type encoding vector and the operational status representation vector are concatenated, and then linearly transformed with the query projection weight matrix to generate the query vector for the attention mechanism. The resource feature matrix is ​​linearly transformed with the key projection weight matrix and the value projection weight matrix respectively to generate the key matrix and value matrix for the attention mechanism. The dot product of the query vector and each key vector in the key matrix is ​​calculated to obtain the original attention score for each resource entity. Based on the type specified in the resource type allocation suggestion, a subset of resources belonging to the corresponding type is selected from all resource entities. For the resource subset, the original attention scores are sorted, and the top N (e.g., N equals 5) resource entities with the highest original attention scores are selected to form candidate executors. For each candidate executor, a corresponding candidate scheduling instruction is generated by combining the task information of the corresponding task to be processed, the current position and running status of the candidate executor; all candidate scheduling instructions together form a candidate scheduling instruction set; the candidate scheduling instruction includes the target task identifier, starting position, target position, execution device identifier, allowed execution time window, load parameters, path constraint parameters and safety interlock parameters.

[0048] Furthermore, N is set to limit the number of candidate executors. In one embodiment, N is set to 5 to achieve a balance between the coverage of candidate schemes and the subsequent calculation overhead, avoiding an increase in scheduling latency due to an excessive number of candidates.

[0049] S6. Using a digital twin model, multi-objective reward function, and dynamic weight vector, perform forward-looking simulation and multi-objective comprehensive evaluation of the candidate scheduling instruction set, and select the scheduling instruction with the best evaluation result and that passes the safety rule verification.

[0050] S6.1 It should be noted that each scheduling instruction in the candidate scheduling instruction set is sequentially input into the digital twin model; the digital twin model parses and loads the specific action specified by the scheduling instruction (e.g., controlling an AGV to go to a specific coordinate); the digital twin model calls the internal dynamic behavior simulator and static scene map to simulate from the execution of the scheduling instruction to the completion of the instruction action. During this simulation, the digital twin model of the warehouse operation entity continuously records and outputs the time-series simulation data, which includes task completion time, task waiting time, safety violation events, estimated energy, equipment collision warning, and the final virtual environment state at the end of the simulation.

[0051] The recorded states and event sequences in the simulation data are compared with each rule in the safety rule set. The safety rule set contains specific constraints such as "prohibiting any two devices from occupying the same spatial coordinates at the same time," "prohibiting devices from executing new tasks when the battery level is below a threshold," and "prohibiting the assignment of tasks to devices already marked as faulty." If no rule in the safety rule set is triggered during the entire simulation process of a scheduling instruction, the corresponding scheduling instruction is deemed to have passed the safety check.

[0052] S6.2 It should be noted that for scheduling instructions that pass the safety check, the corresponding task completion time, task waiting time, number of safety violations, and estimated energy are read from the simulation data. Simultaneously, based on the reward value calculation formula in step S3.1, the reward values ​​for the task completion efficiency sub-objective, the operational safety constraint sub-objective, and the equipment energy consumption control sub-objective are calculated respectively. These sub-objectives are then substituted into the multi-objective reward function, and combined with the corresponding weighting coefficients (obtained in step S4), the expected multi-objective reward value is calculated. All scheduling instructions that pass the safety check are iterated through, and the scheduling instruction with the highest expected multi-objective reward value is selected as the optimal scheduling instruction.

[0053] S7. Issue the selected optimal scheduling instruction to the warehouse operation entity for execution.

[0054] It should be noted that the target device identifier, target action type (such as move, pick, and place), and action parameters (such as target coordinates and target speed) are extracted from the optimal scheduling instruction. Based on the control protocol specifications of the corresponding equipment model in the warehouse operation entity, the target action type and action parameters are encoded into low-level control instructions that can be directly parsed by the corresponding equipment controller and conform to a specific message format. If the optimal scheduling instruction involves multiple devices or continuous actions, the encoded low-level control instructions are arranged into an ordered sequence, i.e., the low-level control instruction sequence, according to the logical order and time dependency of the action execution.

[0055] The underlying control command sequence is sent to the equipment controller specified in the optimal scheduling instruction through the industrial control network inside the warehouse operation entity. The equipment controller receives the underlying control command sequence according to the corresponding communication interface, decodes the underlying control command sequence, and restores the specific control commands and parameters. The decoded control commands and parameters are input to the motion control logic of the equipment controller. The motion control logic generates the corresponding motor drive signals, servo control signals or robotic arm joint trajectory commands, and drives the corresponding equipment to perform operations.

[0056] This embodiment also provides a computer device applicable to the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the intelligent warehousing resource dynamic allocation method based on deep reinforcement learning proposed in the above embodiment.

[0057] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0058] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0059] In summary, this invention constructs a digital twin model that integrates a static scene graph and a dynamic behavior simulator. This model enables the state reproduction and deduction of the operational process of warehouse entities in a virtual environment, providing a repeatable interactive environment for the pre-training of deep reinforcement learning policy networks. By generating dynamic weight vectors through task backlog coefficients, path congestion coefficients, and remaining power constraint coefficients, the weight coefficients corresponding to different sub-objectives in the multi-objective reward function can be dynamically adjusted, thereby improving the adaptability of the multi-objective reward function to changes in the current operating state. Simultaneously, the digital twin model performs deduction, safety checks, and multi-objective comprehensive evaluation of candidate scheduling instructions. This allows for the identification of abnormal situations such as potential path conflicts, insufficient remaining power, and faulty equipment participating in execution before the scheduling instructions are issued. This improves the executability of candidate scheduling instructions, reduces equipment collision conflict rate, reduces empty mileage, and reduces average waiting latency and equipment energy consumption during task execution.

[0060] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent warehouse resource dynamic allocation based on deep reinforcement learning, characterized in that: include, Collect operational data of the warehouse operation entities, and perform different feature engineering processes on categorical and numerical data in the operational data to obtain operational status representation vectors. After collecting the design data of the warehouse operation entity and establishing a digital twin model, historical operation data is used for simulation to obtain interaction experience. Based on the interaction experience, an offline reinforcement learning algorithm is used to pre-train the deep reinforcement learning policy network to obtain the initial policy model. The desired optimization allocation target and multi-objective reward function are preset; Based on the operational data of the warehouse operation entity, the task backlog coefficient, path congestion coefficient, and remaining power constraint coefficient are calculated to obtain the operational status index, and the operational status index is mapped to a dynamic weight vector in the multi-objective reward function. Based on the operational status representation vector and dynamic weight vector, the high-level task allocation network and low-level resource assignment network of the initial strategy model are used to allocate resource types and screen execution entities to obtain a candidate scheduling instruction set. Using digital twin models, multi-objective reward functions, and dynamic weight vectors, a forward-looking simulation and multi-objective comprehensive evaluation of the candidate scheduling instruction set are conducted, and the scheduling instruction with the best evaluation results and that passes the safety rule verification is selected. The selected optimal scheduling instruction is issued to the warehouse operation entity for execution.

2. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 1, characterized in that: The operational data of the warehouse operation entity includes equipment status data, task data, path occupancy data, and equipment remaining power data.

3. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 2, characterized in that: The acquisition of the operational status representation vector specifically involves: One-hot encoding is performed on the categorical data in the running data to obtain the first numerical feature; The numerical data in the operational data is normalized to obtain the second numerical feature; The first numerical feature is concatenated with the second numerical feature to generate an operational status representation vector.

4. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 3, characterized in that: The process of establishing a digital twin model based on the design data of the warehouse operation entity specifically involves: Based on the design data of the warehouse operation entity, construct a static scene map; Dynamic behavior simulators are formed by performing dynamic modeling and behavior rule extraction on movable equipment entities in static scene maps. Establish a two-way real-time data channel between the dynamic behavior simulator and the warehouse operation entity for equipment status, order information and environmental parameters; The virtual simulation environment, which integrates static scene maps, dynamic behavior simulators, and two-way real-time data channels, will be deployed as a digital twin model of the warehouse operation entity.

5. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 4, characterized in that: The acquisition of the initial strategy model specifically involves: Collect historical operational data of the warehouse operation entities, input the historical operational data into the digital twin model, and the digital twin model drives the internal dynamic behavior simulator to simulate the historical motion trajectory and operation process of the movable equipment entities in the static scene map to obtain interactive experience; Based on interactive experience, an offline reinforcement learning algorithm is used to pre-train the deep reinforcement learning policy network to obtain an initial policy model.

6. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 5, characterized in that: The preset expected optimization allocation target and multi-objective reward function are specifically as follows: Preset the desired optimization allocation target; The desired optimization allocation objectives include sub-objectives for task completion efficiency, operational safety constraints, and equipment energy consumption control. The multi-objective reward function is preset to a linear weighted sum structure.

7. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 6, characterized in that: The mapping of operational status indicators to dynamic weight vectors in a multi-objective reward function is specifically as follows: Calculate the task backlog coefficient based on task data, calculate the path congestion coefficient based on path occupancy data, and calculate the remaining power constraint coefficient based on device remaining power data. The task backlog coefficient, path congestion coefficient, and remaining power constraint coefficient are mapped to the weight coefficients of the multi-objective reward function, and then concatenated after normalization to form a dynamic weight vector.

8. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 7, characterized in that: The acquisition of the candidate scheduling instruction set specifically includes: The operational status representation vector and the dynamic weight vector are combined into a composite input vector; The composite input vector is input into the initial policy model, and the high-level task allocation network in the initial policy model calculates and generates resource type allocation suggestions based on the composite input vector. The low-level resource assignment network in the initial strategy model calculates attention scores among all available resources based on resource type allocation suggestions and operational status representation vectors, and selects the N resource individuals with the highest scores as candidate executors through an attention mechanism. The resource type allocation suggestions and the selected candidate executors are combined to form a candidate scheduling instruction set.

9. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 8, characterized in that: The selection of the scheduling instruction with the optimal evaluation result and that passes the security rule verification is specifically as follows: Each scheduling instruction in the candidate scheduling instruction set is sequentially input into the digital twin model for state simulation to obtain simulation data; Based on the preset set of security rules, a security check is performed on the deduction process of each scheduling instruction in the digital twin model, and scheduling instructions that pass the security check are selected. For scheduling instructions that pass the security check, the expected multi-objective reward value is calculated by calling the simulation data, dynamic weight vector and multi-objective reward function, and the scheduling instruction with the highest expected multi-objective reward value is taken as the optimal scheduling instruction.

10. The method for dynamic allocation of intelligent warehousing resources based on deep reinforcement learning according to claim 9, characterized in that: The step of issuing the selected optimal scheduling instruction to the warehouse operation entity for execution is specifically as follows: Convert the optimal scheduling instructions into a sequence of low-level control instructions that the device can recognize; The underlying control command sequence is sent to the corresponding device controller. After receiving and parsing, the device controller drives the physical devices in the warehouse operation entity to execute.

Images