A digital warehouse goods outbound path planning method, device and medium

Abstract

The application discloses a warehouse goods delivery path planning method, equipment and medium, the method comprises the following steps: determining the delivery information according to the warehouse delivery application, and determining the environment information of the warehouse; inputting the delivery information and the environment information into a first path planning model set in advance to determine the delivery path through the first path planning model; generating a delivery task according to the delivery path, determining a pre-set automatic driving vehicle warehouse, and determining the running state of the automatic driving vehicle through the automatic driving vehicle warehouse; inputting the running state and the delivery task into a second path planning model set in advance to distribute the delivery task through the second path planning model, and planning the path of the automatic driving vehicle allocated to the task to determine the task path of the automatic driving vehicle, wherein the task path comprises a goods taking path and a delivery path, and the automatic driving vehicle allocated to the task takes goods in the warehouse and delivers them out of the warehouse.

Description

Technical Field

[0001] This application relates to the field of path planning technology, and in particular to a method, device and medium for outbound path planning of goods in a digital warehouse. Background Technology

[0002] In the current wave of digitalization, various industries are accelerating their digital transformation, and the logistics and warehousing sector is growing exponentially. Digital warehousing parks, as key nodes in the logistics industry, are expanding rapidly, often encompassing hundreds of thousands of square meters of storage space. They store a wide variety of goods in diverse specifications, ranging from everyday consumer goods to industrial raw materials, from precision electronics to large machinery. In recent years, with the rapid development of e-commerce and logistics, warehouses have become an indispensable part of logistics operations. However, with the expansion of warehouse scale, traditional warehouses relying primarily on manual labor can no longer meet the needs of enterprises. Especially in large warehouses, the diverse types of goods and high mobility make the handling of goods more complex, bringing many challenges to logistics and warehouse management. The drawbacks of the traditional manual outbound goods model are becoming increasingly apparent. Staff members have to run around the huge park, manually searching for goods and verifying information. This is not only extremely inefficient and makes it difficult to complete the outbound tasks of a large number of orders in a short period of time, but also prone to human error in the high-intensity and highly repetitive work, resulting in misdelivery or omission of goods, causing economic losses and reputational damage to enterprises. Obviously, it can no longer adapt to and meet the current growing logistics needs. Summary of the Invention

[0003] To address the aforementioned issues, this application proposes a method for outbound path planning of goods in a digital warehouse, comprising: receiving a warehouse transfer request; determining outbound information based on the warehouse transfer request, the outbound information including goods information, starting location, and target location; and determining warehouse environmental information, including static obstacles and dynamic obstacles; inputting the outbound information and the environmental information into a pre-set first path planning model to determine the outbound path through the first path planning model, wherein the first path planning model includes a reward function to minimize the number of turns in the outbound path; and determining the outbound path based on the outbound information. The system generates a relocation task by determining a pre-set autonomous vehicle warehouse. The operating status of the autonomous vehicles, including their task status and location, is then determined from the warehouse. The operating status and the relocation task are input into a pre-set second path planning model to allocate the relocation task and perform path planning for the assigned autonomous vehicles. The task path includes a pickup path and an outbound path. The assigned autonomous vehicles then pick up the goods from the warehouse and transport them out of the warehouse.

[0004] In one example, the outbound information and the environmental information are input into a pre-set first path planning model to determine the outbound path. Specifically, this includes: determining the first path planning model based on a deep reinforcement learning network, wherein the first path planning model includes a PPO algorithm and a GRU algorithm, the PPO algorithm serving as the basic framework, and the GRU algorithm being used to learn from sequence sample data; obtaining historical state information through the GRU algorithm, and improving the model decision quality in the current state based on the memory function of the GRU algorithm.

[0005] In one example, the first path planning model includes the PPO algorithm and the GRU algorithm. The GRU algorithm acquires historical state information and improves the model's decision-making quality in the current state based on its memory function. Specifically, this includes: introducing a pruning objective function through the PPO algorithm to directly generate action policies; embedding a GRU module in the policy network of the PPO algorithm to enable the policy network to process input sequence state data, including the outbound information and environmental information; inputting the sequence data into the GRU module for feature extraction to output a hidden state sequence; determining an objective function based on the hidden state sequence to update the policy network; and determining the historical location information of obstacles through the GRU module to predict obstacle movement trends, thereby planning a safe path based on the historical obstacle location information.

[0006] In one example, the first path planning model includes a reward function to minimize the number of turns in the outbound path. Specifically, the reward function includes an external reward function and an internal reward function. The external reward function employs a non-sparse reward mechanism, combining state transition rewards, step penalties, and turn penalties to guide the autonomous vehicle to gradually approach the target location and reduce turning actions. The internal reward function uses a pre-set curiosity module to enable the autonomous vehicle to explore the warehouse environment. The state transition reward determines the change in distance between the autonomous vehicle's current position and the target point; a positive reward is given when the distance decreases, and a negative reward is given when the distance increases. The step penalty provides a fixed negative reward at pre-set time intervals to encourage the autonomous vehicle to reach the target location faster. The turn penalty provides a negative reward when the autonomous vehicle performs a turn, reducing unnecessary turns in path planning.

[0007] In one example, the step of inputting the operating state and the relocation task into a pre-set second path planning model to allocate the relocation task through the second path planning model specifically includes: determining the second path planning model based on a multi-agent reinforcement learning network, wherein the second path planning model includes the APF-MAPPO algorithm, constructing an environmental potential field map based on the APF-MAPPO algorithm combined with a pre-set artificial potential field method, observing the state of the autonomous vehicle based on the environmental potential field map, and improving the value network structure of the second path planning model through a self-attention mechanism.

[0008] In one example, an environmental potential field map is constructed based on the APF-MAPPO algorithm combined with a pre-set artificial potential field method. This map is then used to observe the state of the autonomous vehicle. Specifically, this includes: constructing a local environmental potential field map based on the target point and obstacle information of each autonomous vehicle; calculating the total potential energy value of each location in the warehouse by superimposing gravitational and repulsive fields to generate the environmental potential field map; the gravitational field is affected by the distance between the autonomous vehicle's location and the target point location, with a larger distance resulting in a larger gravitational potential energy value; the repulsive field is affected by the distance between the autonomous vehicle's location and the obstacle location, with a decreasing repulsive potential energy value as the distance increases when the autonomous vehicle enters the obstacle's influence range; determining a pre-set local gravitational field to ensure the lowest potential energy value near the target point; observing multiple grid potential energy values ​​based on the environmental potential field map, and combining gravitational and repulsive field information to perceive environmental information, thereby minimizing the target point's potential energy and achieving state observation and path planning.

[0009] In one example, the value network structure of the second path planning model is improved by using a self-attention mechanism. Specifically, this includes: weighting the potential energy value of each location in the environmental potential field map using the self-attention mechanism to determine the state information of each location, and assigning a corresponding weight to each location to determine key information in the warehouse environment; using the state information processed by the self-attention mechanism as the input to the value network, outputting the state value of each autonomous vehicle through the value network, and performing path planning for the autonomous vehicles based on the state value combined with global and individual rewards to achieve collaborative operation of multiple autonomous vehicles.

[0010] In one example, after determining the environmental information of the warehouse, the method further includes: real-time monitoring of changes in the warehouse environment, including the addition or removal of goods, the movement of obstacles, and the addition of new tasks; when changes in the warehouse environment are detected, re-planning the path according to the first path planning model and the second path planning model, and performing online model learning to optimize the path planning strategy through continuous interaction with the environment, thereby adapting to new environmental conditions.

[0011] On the other hand, this application also proposes an outbound route planning device for digital warehouse goods, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the digital warehouse goods outbound route planning device to perform: the method described in any of the examples above.

[0012] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to be the method described in any of the examples above.

[0013] This application combines deep reinforcement learning networks, utilizing the PPO and GRU algorithms to construct a first path planning model. This effectively processes sequential sample data, improving the model's decision-making quality, particularly excelling in dynamic obstacle prediction and safe path planning. The reward function design is ingenious, employing a non-sparse reward mechanism, state transition rewards, step penalties, and steering penalties to guide the autonomous vehicle to efficiently approach the target location while reducing unnecessary steering actions, thus improving the efficiency and rationality of path planning. This application uses the APF-MAPPO algorithm, combined with an artificial potential field method to construct an environmental potential field map. By improving the value network structure through a self-attention mechanism, the autonomous vehicle can accurately perceive environmental information, enabling multi-vehicle collaborative operations and improving the overall efficiency and safety of warehouse operations. This application possesses dynamic environmental adaptability, capable of real-time monitoring of warehouse environmental changes and re-planning the path and learning the model online, ensuring that the path planning strategy always adapts to the latest environmental conditions, enhancing the system's robustness and flexibility. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating an outbound route planning method for goods in a digital warehouse, as described in an embodiment of this application. Figure 2 This is a schematic diagram of an outbound path planning device for goods in a digital warehouse, as described in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0017] like Figure 1 As shown, in order to solve the above problems, this application provides a method for outbound route planning of goods in a digital warehouse, the method including: S101. Receive a warehouse transfer request, determine outbound information based on the warehouse transfer request, the outbound information includes product information, starting location, and target location, and determine the warehouse environment information, the environment information includes static obstacles and dynamic obstacles.

[0018] Upon receiving a goods transfer request, the warehouse management system analyzes the request in detail to determine the outbound information. This outbound information includes specific details about the goods, such as name, quantity, and specifications, as well as the starting and destination locations. The starting location is the shelf or storage area where the goods are currently stored, while the destination location is the designated location where the goods need to be moved, such as a workbench or another shelf. Furthermore, the system comprehensively collects environmental information about the warehouse, including the distribution of static obstacles, such as fixed shelves and walls, and the real-time status of dynamic obstacles, such as moving autonomous vehicles and personnel. By integrating this information, the warehouse management system can build a detailed warehouse environment model, providing accurate data support for subsequent route planning and ensuring that autonomous vehicles can safely and efficiently complete goods handling tasks.

[0019] S102. Input the outbound information and the environmental information into a pre-set first path planning model to determine the outbound path through the first path planning model. The first path planning model includes a reward function to modify the number of turns in the outbound path to a minimum value through the reward function.

[0020] After receiving outbound and environmental information, the system inputs this key data into the pre-set first path planning model—the PPO-GRU algorithm model. This model not only considers the starting and destination positions of the goods, but also comprehensively integrates the distribution of static obstacles and the real-time status of dynamic obstacles within the warehouse to ensure the accuracy and safety of path planning.

[0021] The PPO-GRU algorithm incorporates a steering penalty mechanism in its reward function. The core of this mechanism is to incentivize the autonomous vehicle to minimize the number of turns when planning its exit route by assigning a negative reward value for each turn. This design not only helps shorten path length and reduce the energy consumption of the autonomous vehicle, but also improves overall handling efficiency.

[0022] During algorithm execution, the autonomous vehicle continuously adjusts its action strategy based on current state information and feedback from the reward function, gradually learning how to plan the outbound path with the fewest turns in a complex warehouse environment. Ultimately, the PPO-GRU algorithm outputs a safe and efficient outbound path, guiding the autonomous vehicle to complete the goods handling task accurately and quickly.

[0023] S103. Generate a relocation task based on the outbound route, determine a pre-set autonomous vehicle warehouse, and determine the operating status of the autonomous vehicles through the autonomous vehicle warehouse. The operating status includes the task being carried and the vehicle location.

[0024] When generating a relocation task based on the outbound route, the system first determines the specific sequence and route for goods handling based on the route planning results. Subsequently, the system accesses a pre-set autonomous vehicle warehouse, which stores information on all available autonomous vehicles, including vehicle model, carrying capacity, current location, and operating status.

[0025] By querying the autonomous vehicle warehouse, the system can monitor the real-time task status of each vehicle, including whether it is currently performing other tasks and whether it has been assigned but not yet started. Simultaneously, the system can obtain the vehicle's specific location information, which is crucial for dispatching vehicles to the starting point to load goods.

[0026] After determining the operating status of the autonomous vehicles, the system intelligently selects the most suitable vehicle to perform the current relocation task by comprehensively considering factors such as task priority, the relative distance between the vehicle's location and the starting point of the goods, and the vehicle's carrying capacity. Once a vehicle is selected, the system immediately updates its operating status, marking it as performing a new task, and sends detailed route navigation instructions to guide the vehicle along the planned outbound route to the target location to complete the goods handling. The entire process is highly efficient and automated, significantly improving the warehouse's logistics operation efficiency.

[0027] S104. Input the operating status and the transfer task into a pre-set second path planning model to allocate the transfer task through the second path planning model, and perform path planning for the autonomous driving vehicle assigned to the task to determine the task path of the autonomous driving vehicle. The task path includes a pickup path and an outbound path. The autonomous driving vehicle assigned to the task picks up the goods in the warehouse and transports them out of the warehouse.

[0028] After inputting the operating status of the autonomous vehicles and the delivery tasks into the pre-set second path planning model—the APF-MAPPO algorithm model—the system will activate a multi-autonomous vehicle cooperative path planning mechanism. This model first uses a genetic algorithm to globally optimize and allocate the delivery tasks, ensuring a balanced workload for each autonomous vehicle while minimizing the overall travel distance. During task allocation, the model comprehensively considers multi-dimensional information such as the vehicle's current location, the tasks it carries, the starting and ending points of the goods, etc., to achieve the optimal match between tasks and vehicles.

[0029] After task allocation, the APF-MAPPO algorithm performs personalized path planning for each autonomous vehicle assigned a task. The algorithm combines an environmental potential field map constructed using the artificial potential field method, providing vehicles with rich environmental state information, including the gravitational potential field of static obstacles and the repulsive potential field of dynamic obstacles. By superimposing the gravitational and repulsive fields, the algorithm accurately calculates the total potential energy value at each location in the warehouse, generating a detailed potential field map.

[0030] During path planning, the APF-MAPPO algorithm prioritizes the continuity and efficiency of the task path, seamlessly connecting the pickup and outbound paths. The algorithm utilizes a self-attention mechanism to weight global state information, enabling the vehicle to focus on critical path nodes and dynamically adjust its driving strategy to cope with real-time changes in the warehouse. Ultimately, the algorithm outputs an optimal task path for each autonomous vehicle, including a pickup path from the current location to the goods' starting point and an outbound path from the goods' starting point to the target location. The vehicle follows this path to complete the entire process of pickup and outbound transportation, achieving efficient warehouse logistics operations.

[0031] In one embodiment, the first path planning model is built upon a deep reinforcement learning network, combining the advantages of the PPO and GRU algorithms. The PPO algorithm, serving as the model's foundational framework, is responsible for generating and optimizing action policies. Its core lies in balancing exploration and exploitation by introducing a pruning objective function, ensuring that the policy effectively explores the environment while stably converging to the optimal solution during training. The GRU algorithm, as a highly efficient variant of a recurrent neural network, is specifically designed to process sequential sample data and capture temporal dependencies within the data. In its implementation, the GRU algorithm is first used to acquire historical state information, including the past pose, velocity, and acceleration of the autonomous vehicle, as well as the historical positions of obstacles in the warehouse. Through its unique gating mechanisms, such as update and reset gates, the GRU algorithm dynamically adjusts the flow of information, retaining historical information useful for the current decision while forgetting irrelevant parts. This memory function allows the model to consider a broader historical context when making decisions, significantly improving the quality of decisions in the current state. Next, the policy network of the PPO algorithm is extended to embed the GRU module. This innovative design enables the policy network to directly process sequential state data containing temporal information. These sequence state data not only contain static information such as the target location of the outbound task and the layout structure of the warehouse, but also incorporate real-time environmental information such as the dynamic position changes of obstacles. The GRU module receives this sequence data as input, performs feature extraction and pattern recognition, and outputs a series of hidden state sequences reflecting the influence of historical states. These hidden state sequences not only capture the temporal characteristics of the data but also imply a deep understanding of past states, providing the policy network with richer and more accurate decision-making basis. The policy network uses these hidden state sequences to determine the objective function, which comprehensively considers multiple optimization objectives such as path length, obstacle avoidance requirements, and energy efficiency. By continuously optimizing this objective function, the policy network can gradually update its parameters, improving the performance and effectiveness of path planning. During the optimization process, the pruning objective function of the PPO algorithm plays a crucial role, ensuring the stability and efficiency of policy updates and preventing performance fluctuations caused by excessive policy changes. Furthermore, the GRU module also predicts the possible movement trends of obstacles by analyzing their historical position information. This capability is particularly important in dynamic obstacle avoidance scenarios, as obstacles in warehouse environments are often dynamic, such as pedestrians, other autonomous vehicles, or moving goods. By combining the historical trajectories of obstacles with their current locations, the GRU module can anticipate potential collision risks and guide the policy network to plan a safe and efficient outbound route. This proactive obstacle avoidance strategy significantly enhances the autonomous navigation capabilities of autonomous vehicles in complex warehouse environments.

[0032] In one embodiment, the design of the reward function is a core component of the first path planning model. Its purpose is to guide the autonomous vehicle to plan the optimal exit route through a refined reward mechanism, particularly minimizing the number of turns within the route to improve path smoothness and execution efficiency. This reward function cleverly integrates both external and internal reward modules, which work together in the autonomous vehicle's decision-making process.

[0033] The external reward function employs a non-sparse reward mechanism. This mechanism provides clear and detailed action guidance for autonomous vehicles by comprehensively considering three key factors: state transition rewards, step penalties, and steering penalties. Specifically, the state transition reward provides feedback based on the change in distance between the autonomous vehicle's current position and the target point: when the autonomous vehicle successfully shortens the distance to the target point, the system provides a positive reward to reinforce this beneficial behavior; conversely, if the autonomous vehicle deviates from the target point, a negative reward is given, prompting the autonomous vehicle to adjust its strategy and move closer to the target point again. This design ensures that the autonomous vehicle maintains a clear pursuit of the target point throughout the path planning process.

[0034] Step-by-step penalties work by providing the autonomous vehicle with a fixed negative reward at pre-set time intervals. This mechanism effectively increases the autonomous vehicle's motivation to quickly reach the target location, as any unnecessary stops or slow movements result in a reduction in the accumulated reward. In this way, step-by-step penalties incentivize the autonomous vehicle to increase its speed as much as possible while ensuring safety, thereby shortening the task completion time.

[0035] Steering penalty is another important design element in the external reward function, directly providing a negative reward for steering actions in path planning. Whenever an autonomous vehicle performs a steering action, the system awards a negative reward to reduce unnecessary steering in the path. This design helps autonomous vehicles plan smoother, more direct paths, avoiding inefficiencies and increased energy consumption caused by frequent steering. Simultaneously, steering penalty also prompts autonomous vehicles to more carefully consider the necessity of steering in complex environments, thus making more rational decisions.

[0036] In addition to the external reward function, the internal reward function, through a pre-set curiosity module, stimulates the autonomous vehicle's desire to explore the warehouse environment. The curiosity module encourages the autonomous vehicle to try new paths and strategies, even if these attempts may not yield direct rewards in the short term. However, in the long run, this exploratory behavior helps the autonomous vehicle better understand the warehouse environment, discover potential optimized paths, and thus improve the overall path planning performance. The organic combination of internal and external rewards enables the autonomous vehicle to pursue immediate rewards while also possessing the ability for long-term planning and continuous optimization.

[0037] In practical applications, the reward function is continuously adjusted and optimized based on the behavior of autonomous vehicles and environmental feedback, ensuring that autonomous vehicles can gradually learn the optimal path planning strategy. As training progresses, autonomous vehicles will be able to more accurately determine when to turn, accelerate, and decelerate to minimize the number of turns, shorten path length, reduce energy consumption, and ultimately achieve efficient and safe outbound tasks. This reward function design not only improves the accuracy and efficiency of path planning but also provides strong support for the automated management of smart warehouses.

[0038] In one embodiment, a second path planning model is constructed based on a multi-agent reinforcement learning network, with the core algorithm being the APF-MAPPO algorithm. This algorithm combines a pre-set Artificial Potential Field (APF) method to construct an environmental potential field map, which serves as the basis for observing the state of autonomous vehicles. Specifically, for each autonomous vehicle in the warehouse environment, gravitational and repulsive fields are calculated based on its target point and obstacle information. The gravitational field is calculated based on the distance between the autonomous vehicle's position and the target point; the greater the distance, the greater the gravitational potential energy, ensuring the vehicle tends to move towards the target point. The repulsive field is calculated based on the distance between the autonomous vehicle's position and the obstacle's position; when the vehicle enters the obstacle's influence range, the repulsive potential energy decreases with increasing distance to avoid collisions.

[0039] To further optimize the potential field map and ensure the lowest potential energy value near the target point, a pre-set local gravitational field is introduced. This local gravitational field operates within a small area near the target point, applying a relatively strong gravitational force to ensure the target point becomes the point of lowest potential energy, thereby guiding the autonomous vehicle to accurately reach the target point. When generating the complete environmental potential field map, the total potential energy value at each location in the warehouse is calculated by superimposing the gravitational and repulsive fields. In this process, the total potential energy value at each location reflects the combined influences experienced by the vehicle at that location, including the gravitational force moving towards the target point and the repulsive force avoiding obstacles.

[0040] After constructing the environmental potential field map, the autonomous vehicle perceives its surrounding environment by observing the potential energy values ​​of multiple pre-divided grids and combining this with gravitational and repulsive field information. During this process, the target point is designed as the point of lowest potential energy to ensure the vehicle can clearly define its direction of movement, achieving effective state observation and path planning. To further improve the accuracy and efficiency of path planning, a self-attention mechanism is introduced to weight the potential energy value at each location in the environmental potential field map. By assigning different weights to different location information, the self-attention mechanism allows the model to focus on key information in the environment while ignoring unimportant details.

[0041] In its implementation, the self-attention mechanism first maps the potential value of each location in the environmental potential field graph to three spaces: query, key, and value, resulting in corresponding vector representations. Then, by calculating the similarity between the query vector and the key vector, the weight of each location relative to other locations is obtained. These weights are used to perform a weighted summation of the value vectors, generating a new state representation that includes information about key locations in the environment. In this way, the self-attention mechanism can effectively extract key information from the environmental potential field graph, providing strong support for subsequent path planning.

[0042] Next, the state information processed by the self-attention mechanism is used as input to the value network. Based on this state information, the value network evaluates the state value of the autonomous vehicle in the current environment. During the evaluation process, the value network considers not only the vehicle's own state but also the states of other vehicles and obstacle information in the environment to achieve globally optimal path planning. Simultaneously, to incentivize autonomous vehicles to better complete their tasks, global and individual reward mechanisms are designed. The global reward reflects the task completion status of the entire multi-vehicle system, such as whether all vehicles successfully reach the target point and whether the total path length is the shortest; individual rewards are given to each autonomous vehicle individually based on its task completion status, path length, obstacle avoidance performance, and other factors.

[0043] Finally, based on the state value output by the value network, and combining global and individual rewards, path planning is performed for the autonomous vehicles. During the planning process, each autonomous vehicle selects the optimal action strategy based on its own state value and reward information to achieve collaborative operation among multiple autonomous vehicles. Through continuous iteration and optimization, the APF-MAPPO algorithm can learn the optimal path planning strategy, enabling autonomous vehicles to complete tasks efficiently and safely in complex and ever-changing warehouse environments.

[0044] In one embodiment, after determining the initial environmental information of the warehouse, this embodiment further introduces a dynamic environmental adaptation mechanism to ensure that the intelligent warehouse system can respond to and adapt to changes in the warehouse environment in real time. Specifically, the system continuously monitors changes in the warehouse environment in real time. These changes include, but are not limited to, the addition or removal of goods, the movement of obstacles, and the addition of new tasks. The addition or removal of goods may be caused by operations such as inbound, outbound, or goods rearrangement; the movement of obstacles may involve changes in the position of equipment, personnel, or other dynamic obstacles within the warehouse; and the addition of new tasks may include new task requirements such as goods handling, sorting, or storage.

[0045] Once a change in the warehouse environment is detected, the system immediately triggers a path planning recalculation process. This process first involves re-invoking both the first and second path planning models to re-plan the path based on the latest environmental information. The first path planning model may be for single-robot path optimization, while the second path planning model may involve collaborative path planning for multiple robots, such as a model based on the APF-MAPPO algorithm. During the path replanning process, the system not only considers the latest environmental state but also incorporates previously learned experience to more efficiently generate paths adapted to the new environment.

[0046] Simultaneously, the system activates an online learning mechanism for the model, continuously optimizing its path planning strategy through interaction with the environment. During online learning, the agent adjusts its behavioral strategies based on real-time reward signals to maximize long-term cumulative rewards. This learning mechanism enables the path planning model to gradually adapt to the dynamic changes in the warehouse environment, improving the accuracy and robustness of path planning. For example, when new obstacles are added to the warehouse, the agent can quickly learn the optimal path to avoid them; when new tasks are introduced, the agent can rationally adjust the task execution order and path to ensure all tasks are completed efficiently. Through this dynamic adaptation and online learning mechanism, the intelligent warehouse system can maintain efficient and stable operation in complex and ever-changing environments.

[0047] like Figure 2 As shown in the illustration, this application also provides an outbound route planning device for goods in a digital warehouse, comprising: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable a digital warehouse goods outbound route planning device to perform the method as described in any of the embodiments above.

[0048] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which are configured as described in any of the above embodiments.

[0049] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0050] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0051] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0052] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0053] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0054] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0060] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0061] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0062] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for outbound route planning of goods in a digital warehouse, characterized in that, include: Receive a warehouse transfer request, determine outbound information based on the warehouse transfer request, the outbound information includes product information, starting location, and target location, and determine the warehouse environmental information, the environmental information includes static obstacles and dynamic obstacles; The outbound information and the environmental information are input into a pre-set first path planning model to determine the outbound path. The first path planning model includes a reward function to modify the number of turns in the outbound path to a minimum. Based on the outbound route, a relocation task is generated, a pre-set autonomous vehicle warehouse is determined, and the operating status of the autonomous vehicles is determined through the autonomous vehicle warehouse. The operating status includes the task being carried and the vehicle location. The operating status and the transfer task are input into a pre-set second path planning model to allocate the transfer task and perform path planning for the autonomous vehicle assigned to the task to determine the task path of the autonomous vehicle. The task path includes a pickup path and an outbound path. The autonomous vehicle assigned to the task picks up the goods in the warehouse and transports them out of the warehouse.

2. The method according to claim 1, characterized in that, The outbound information and the environmental information are input into a pre-set first path planning model to determine the outbound path, specifically including: The first path planning model is determined based on a deep reinforcement learning network, wherein the first path planning model includes the PPO algorithm and the GRU algorithm, the PPO algorithm serves as the basic framework, and the GRU algorithm is used to learn from sequence sample data; Historical state information is obtained through the GRU algorithm, and the model decision quality in the current state is improved based on the memory function of the GRU algorithm.

3. The method according to claim 2, characterized in that, The first path planning model includes the PPO algorithm and the GRU algorithm. The GRU algorithm acquires historical state information and improves the model's decision-making quality in the current state based on its memory function. Specifically, this includes: The PPO algorithm introduces a pruning objective function to directly generate action policies. A GRU module is embedded in the policy network of the PPO algorithm so that the policy network can process the input sequence state data, which includes the outbound information and the environmental information. The sequence data is input into the GRU module for feature extraction to output a hidden state sequence. The objective function is determined based on the hidden state sequence, and the policy network is updated based on the objective function. The GRU module determines the historical location information of obstacles to predict their movement trends, thereby planning a safe path based on the historical location information of obstacles.

4. The method according to claim 1, characterized in that, The first path planning model includes a reward function to minimize the number of turns in the outbound path, specifically including: The reward function includes an external reward function and an internal reward function. The external reward function is designed with a non-sparse reward mechanism, which combines state transition rewards, step penalties and steering penalties to guide the autonomous vehicle to gradually approach the target position and reduce steering actions. The internal reward function enables the autonomous vehicle to explore the warehouse environment through a pre-set curiosity module. The change in distance between the current position of the autonomous vehicle and the target point is determined based on the state transition reward. When the distance between the autonomous vehicle and the target point decreases, a positive reward is given; when the distance between the autonomous vehicle and the target point increases, a negative reward is given. Based on the step penalty, a fixed negative reward is given at a pre-set time period point to encourage the autonomous vehicle to reach the target location faster; The steering penalty provides a negative reward when the autonomous vehicle makes a steering maneuver, in order to reduce unnecessary steering in path planning.

5. The method according to claim 1, characterized in that, The step of inputting the operating status and the transfer task into a pre-set second path planning model, so as to allocate the transfer task through the second path planning model, specifically includes: The second path planning model is determined based on a multi-agent reinforcement learning network. The second path planning model includes the APF-MAPPO algorithm. An environmental potential field map is constructed based on the APF-MAPPO algorithm and a pre-set artificial potential field method. The state of the autonomous vehicle is observed based on the environmental potential field map. The value network structure of the second path planning model is improved through a self-attention mechanism.

6. The method according to claim 5, characterized in that, An environmental potential field map is constructed based on the APF-MAPPO algorithm combined with a pre-set artificial potential field method. The state of the autonomous vehicle is then observed based on this environmental potential field map. Specifically, this includes: A local environmental potential field map is constructed based on the target point and obstacle information of each autonomous vehicle. The total potential energy value of each location in the warehouse is calculated by superimposing gravitational and repulsive fields to generate the environmental potential field map. The gravitational field is affected by the distance between the position of the autonomous vehicle and the target point; the greater the distance, the greater the gravitational potential energy. The repulsive field is affected by the distance between the position of the autonomous vehicle and the position of the obstacle. When the autonomous vehicle enters the range of influence of the obstacle, the repulsive potential energy value decreases as the distance increases. A pre-set local gravitational field is determined to ensure that the potential energy value near the target point is minimized based on the local gravitational field; Based on the observation of multiple grid potential energy values ​​from the environmental potential field map, and combined with the information of gravitational and repulsive fields, environmental information is perceived to minimize the potential energy of the target point, thereby realizing state observation and path planning.

7. The method according to claim 6, characterized in that, The value network structure of the second path planning model is improved through a self-attention mechanism, specifically including: The self-attention mechanism is used to weight the potential energy value of each location in the environmental potential field map to determine the state information of each location and assign a corresponding weight to each location in order to determine the key information in the warehouse environment. The state information processed by the self-attention mechanism is used as the input to the value network. The value network outputs the state value of each autonomous vehicle. Based on the state value and the global reward and individual reward, the autonomous vehicle is used for path planning to achieve collaborative operation of multiple autonomous vehicles.

8. The method according to claim 1, characterized in that, After determining the environmental information of the warehouse, the method further includes: Real-time monitoring of warehouse environmental changes, including the addition or removal of goods, the movement of obstacles, and the addition of new tasks; When a change in the warehouse environment is detected, path planning is re-performed based on the first path planning model and the second path planning model, and the models are learned online to continuously optimize the path planning strategy through interaction with the environment, thereby adapting to the new environmental conditions.

9. A device for planning the outbound route of goods in a digital warehouse, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the outbound route planning device for digital warehouse goods to perform the method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to be the method as described in any one of claims 1-8.

Images