Automatic guided vehicle non-stop sorting control model training and control method and device

Abstract

The invention provides a training and control method and device for a non-stop sorting control model of an automated guided vehicle. State parameters are collected in real time through a preset sensor to construct a state space, and an action space is constructed based on sorting delivery control elements. According to whether the parcel is thrown out or not, dynamically adjusting a reward value, and when the parcel is not thrown out, introducing a parcel drop point prediction reward, an AGV position offset reward, a conveyor belt acceleration reward and a moving mechanism acceleration reward; and when the parcel is thrown, calculating a reward value based on the actual drop point. And through a reinforcement learning algorithm, calculating the return and advantage value of each time step by using the reward value, limiting the change range of the probability ratio of new and old policies in combination with the advantage value to optimize the policy network, and meanwhile, optimizing the value network by minimizing the value prediction error. A super network is introduced, state parameters recorded in an experience pool serve as input, an auxiliary neural network is trained, initial parameters of a main network are preferentially generated in a new state, and the sorting efficiency and accuracy are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic guided vehicles, and in particular to a training and control method and device for a non-stop sorting control model of an automatic guided vehicle. Background Art

[0002] With the rapid development of the logistics industry in recent years, sorting efficiency has become a key factor influencing the overall effectiveness of logistics and transportation. Logistics business volume has seen explosive growth, and sorting scenarios have become increasingly diverse, small-batch, and fast. This has placed higher demands on the flexibility, efficiency, and cost control of sorting systems. Currently, mainstream automated sorting systems fall into two categories: high-speed sorting machine systems and intelligent sorting robot systems based on automated guided vehicles (AGVs).

[0003] High-speed sorting systems include push-block, slider, flip-plate, and cross-belt systems. These systems offer rapid sorting speeds, with single-layer systems capable of reaching a maximum sorting speed of 10,800 pieces per hour and a maximum linear speed of 2.25 meters per second. They can handle items weighing no more than 30 kilograms. However, these systems are generally expensive and require a large footprint. Furthermore, their fixed size and installation location prevent them from adapting to changes in the scale of the business, leading to high energy consumption. In contrast, automated guided vehicle (AGV)-based sorting robot systems, due to their high adaptability, low cost, and scalability, have become a research hotspot in the industry. However, existing AGV sorting robot systems still face significant bottlenecks in parcel delivery. In conventional processes, the AGV must move along a planned path to the sorting slot, decelerate and stop to place the parcel, and then reaccelerate to its operating speed. This deceleration-stop-acceleration process not only prolongs the sorting cycle but also increases energy consumption. In recent years, while some studies have attempted to achieve non-stop sorting through physical models, using the principle of horizontal projection motion to deliver packages while driving, such solutions suffer from the following drawbacks: First, they have poor adaptability to individual differences. The mechanical structures of different AGVs (such as motor performance and conveyor belt friction coefficient) vary, making fixed physical models difficult to generalize and applying, and resulting in significant fluctuations in sorting efficiency. Second, they lack dynamic environmental responsiveness. Real-time factors such as mechanical wear and environmental changes can alter the dynamic relationship between the package and the conveyor belt. Traditional models are unable to dynamically adjust parameters, which can easily lead to sorting errors or package slippage. Furthermore, model maintenance costs are high, requiring frequent calibration or remodeling to adapt to equipment aging or changing operating conditions, resulting in increased operational complexity and downtime. These issues severely restrict the large-scale application and long-term stability of AGV sorting systems. Therefore, a new control method that can adapt to individual differences and dynamic environments, reduce operational costs, and improve sorting efficiency is urgently needed to overcome existing technical bottlenecks and promote the further development of logistics automation technology. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and device for training and controlling an automatic guided vehicle (AGV) non-stop sorting control model to eliminate or improve one or more defects existing in the prior art, and solve the problems of insufficient adaptability of the physical model and high operation and maintenance costs caused by individual mechanical differences and dynamic environment changes during non-stop sorting of AGVs.

[0005] One aspect of the present invention provides a method for training an AGV non-stop sorting control model, which is used to train a control model to control an AGV to deliver packages to a preset sorting port during driving. The method includes the following steps:

[0006] In a preset scenario, the AGV performs sorting and delivery, and based on state parameters collected in real time by preset sensors, a state space is constructed, and an action space is constructed based on sorting and delivery control elements;

[0007] An initial policy network and a value network are obtained to construct a main neural network with an actor-critic network structure. The policy network is used to take the current state parameters in the state space as input and output actions in the action space; the value network takes the current state parameters in the state space as input and outputs the value of the current state. In a sorting round, when the package is not thrown, a package landing prediction reward, an AGV position offset reward, a conveyor belt acceleration reward, and a moving mechanism acceleration reward are introduced to construct a reward value; when the package is thrown, the reward value is established based on the package landing point. The policy network and the value network are trained based on reinforcement learning. In each sorting round, the return and advantage value of each time step are calculated using the reward value. The policy network is updated by restricting the change range of the probability ratio of the new and old policies in combination with the advantage value; the value network is updated by minimizing the value prediction error in combination with the return. After the policy network and the value network are stable, a hypernetwork is introduced to train and output the main network parameters with the state parameters of multiple time steps recorded in the experience pool as input, and a secondary neural network is obtained to preferentially generate the initial parameters of the main neural network in a new state; the policy network obtained through multiple rounds of iteration is used as the target sorting control model.

[0008] In some embodiments, the state parameters include the distances between the AGV and the target sorting port along the x-axis and y-axis, the deviations between the current position and the target trajectory point at the current moment along the x-axis and y-axis, the angle deviation between the current traveling direction and the target trajectory direction, the angle between the current traveling direction and the positive x-axis, the distances between the package to be delivered and the target sorting port along the x-axis and y-axis, and the moving direction speed of the package to be delivered; the action space includes the conveyor belt speed and the moving mechanism speed of the AGV.

[0009] In some embodiments, the policy network adopts a multi-layer perceptron, a temporal network or a proximal policy optimization algorithm-specific policy network; the value network adopts a multi-layer perceptron; and the super network is constructed based on a recurrent neural network or a Transformer network.

[0010] In some embodiments, in a sorting round, when a package is not thrown, the calculation formula for the package landing prediction reward is:

[0011]

[0012] Among them, k1 represents the accuracy bonus coefficient for the predicted landing point of the package in the sorting grid when it is not thrown; k2 represents the accuracy bonus coefficient for the predicted landing point of the package outside the sorting grid when it is not thrown; Indicates the distance between the predicted parcel landing point and the preset sorting port in the x-axis direction, represents the distance between the predicted parcel landing point and the preset sorting port in the y-axis direction; b represents the denominator offset;

[0013] The calculation formula for the automatic guided vehicle position offset reward is:

[0014] r route =k3·(Δd x 2 +Δd y 2 )+k4·Δθ;

[0015] Wherein, k3 represents the penalty coefficient of the distance between the AGV and the target trajectory point; Δd x represents the distance between the center of the AGV and the target trajectory point along the x-axis, Δd y represents the distance between the center of the automated guided vehicle and the target trajectory point along the y-axis; k4 represents the angle deviation penalty coefficient; Δθ represents the angle deviation between the current travel direction and the target trajectory direction;

[0016] The calculation formula for the conveyor belt acceleration reward is:

[0017]

[0018] Among them, k5 represents the penalty coefficient of the current conveyor belt speed mutation; v ct represents the conveyor belt speed at time t, a cmax Indicates that the conveyor belt limits the maximum acceleration;

[0019] The mobile mechanism acceleration bonus includes the left wheel acceleration bonus and the right wheel acceleration bonus. The calculation formula of the left wheel acceleration bonus is:

[0020]

[0021] Among them, k6 represents the current left wheel speed mutation penalty coefficient; v lt represents the left wheel speed at time t, a lmax represents the maximum acceleration limit of the left wheel;

[0022] The calculation formula for the acceleration reward of the right wheel is:

[0023]

[0024] Among them, k7 represents the current right wheel speed mutation penalty coefficient; v rt represents the right wheel speed at time t, a rmax represents the maximum acceleration limit of the right wheel;

[0025] Then the calculation formula for the reward value when the package is not thrown is:

[0026] r t = r end + r route + r conStable + r leftStable + r rightStabel ;

[0027] When the package is thrown, the predicted package landing point reward is changed to the actual package landing point reward, and the calculation formula is:

[0028]

[0029] Among them, k8 represents the accuracy reward coefficient of the predicted package landing point in the sorting grid when not thrown; k9 represents the accuracy reward coefficient of the predicted package landing point outside the sorting grid when not thrown, and b represents the denominator bias.

[0030] In some embodiments, the calculation formula for the predicted package landing point is:

[0031]

[0032] Among them, d xp represents the distance along the x-axis between the package to be delivered and the target sorting port, d yp represents the distance along the y-axis between the package to be delivered and the target sorting port; v p represents the moving direction speed of the package to be delivered; v r represents the right wheel speed, v l represents the left wheel speed; h represents the height difference between the target sorting port and the package to be delivered, g represents the acceleration due to gravity; α represents the angle between the current traveling direction and the positive x-axis.

[0033] In some embodiments, in each sorting round, the reward value is used to calculate the return and advantage value of each time step, and the calculation formula of the return is:

[0034]

[0035] The calculation formula of the advantage value is:

[0036]

[0037] δ t+l =r t +γV(S t+1 )-V(S t );

[0038] Among them, δ t+l is the time domain differential error; r k represents the reward value at time step k; λ is the parameter of the generalized advantage estimate; l is the summation variable, which traverses from 0 to the remaining time steps Tt and represents the offset of multiple future steps; γ is the discount factor, which controls the importance of future rewards in reinforcement learning.

[0039] In some embodiments, in updating the policy network by limiting the range of variation of the probability ratio of the new and old policies in combination with the advantage value, the loss function is calculated as:

[0040]

[0041] in, is the importance sampling ratio, is the strategy θ a The loss function under , ε is the cropping rate;

[0042] Then the strategy network parameter update formula is:

[0043]

[0044] The value network is updated by minimizing the value prediction error in combination with the reward. The loss function is calculated as:

[0045]

[0046] Then the value network parameter update formula is:

[0047]

[0048] The hypernetwork is introduced to train the main network parameters using the state parameters of multiple time steps recorded in the experience pool as input, and the loss function is calculated as follows:

[0049]

[0050] Among them, w main represents the parameter set of the currently described main neural network; w assist represents the parameter set of the main neural network predicted by the hypernetwork;

[0051] Update the parameters of the hypernetwork through gradient descent, and the updated parameters are:

[0052]

[0053] Among them, α1, α2, and α3 are learning rates.

[0054] On the other hand, the present invention also provides a non-stop sorting control method for an automated guided vehicle, which is used to control the automated guided vehicle to deliver packages to a preset sorting port during driving. The method includes the following steps:

[0055] Receive a sorting task, and the sorting task records the information of the package to be sorted, the initial position of the package, and the target position of the package;

[0056] Based on a preset path planning algorithm, plan a target trajectory according to the current position of the automated guided vehicle, the initial position of the package, and the target position of the package;

[0057] Control the preset automated guided vehicle to pick up the package to be sorted at the initial position of the package according to the target trajectory;

[0058] Control the preset automated guided vehicle to drive towards the target position of the package according to the target trajectory, and execute the delivery task based on the target sorting control model obtained by the above non-stop sorting control model training method for the automated guided vehicle after reaching the sorting area.

[0059] On the other hand, the present invention also provides a non-stop sorting control device for an automated guided vehicle, including a processor, a memory, and a computer program / instructions stored on the memory, and the processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the above method.

[0060] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program / instructions are stored, and when the computer program / instructions are executed by a processor, the steps of the above method are implemented.

[0061] On the other hand, the present invention also provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above method are implemented.

[0062] The beneficial effects of the present invention are at least:

[0063] The training, control method and device for the non-stop sorting control model of the automatic guided vehicle (AGV) described in the present invention construct a state space by collecting the state parameters of the AGV in real time through preset sensors, and construct an action space based on the sorting and delivery control elements. Then, an actor-critic network structure is constructed using a policy network and a value network. The policy network takes the current state parameters as input and outputs actions, and the value network takes the current state parameters as input and outputs the state value. During the sorting process, the reward value is dynamically adjusted according to whether the package is thrown. When the package is not thrown, a package landing point prediction reward, an AGV position offset reward, a conveyor belt acceleration reward, and a moving mechanism acceleration reward are introduced. When the package is thrown, the reward value is calculated based on the actual landing point. Through the reinforcement learning algorithm, the return and advantage value of each time step are calculated using the reward value, and the policy network is optimized by combining the advantage value to limit the change range of the probability ratio of the new and old policies. At the same time, the value network is optimized by minimizing the value prediction error. After the policy network and the value network are stable, a super network is introduced, and the sub-neural network is trained with the state parameters recorded in the experience pool as input, so that it preferentially generates the initial parameters of the main network in the new state. The finally trained target sorting control model can achieve efficient sorting without stopping, significantly improving the sorting efficiency and accuracy.

[0064] Additional advantages, objects, and features of the present invention will be partly described below and will partly become apparent to those of ordinary skill in the art after studying the following. Or they can be learned through the practice of the present invention. The objects and other advantages of the present invention can be achieved and obtained by the structure specifically pointed out in the specification and the drawings.

[0065] Those skilled in the art will understand that the objects and advantages that can be achieved by the present invention are not limited to the above specifically described, and the above and other objects that the present invention can achieve will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention. In the drawings:

[0067] Figure 1 It is a logical schematic diagram of the training method of the non-stop sorting control model of the automatic guided vehicle according to an embodiment of the present invention.

[0068] Figure 2 It is a top view of the automatic guided vehicle delivering a package in another embodiment of the present invention.

[0069] Figure 3 It is a front view of the automatic guided vehicle delivering a package in another embodiment of the present invention DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0071] Herein, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution according to the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0072] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps, or components, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0073] In the sorting and delivery scenario, due to individual differences and environmental changes, the automated guided vehicle needs to have a strong adaptability to face the changes of various elements in the task scenario in order to meet the requirements of precise control for delivery and complete sorting.

[0074] The present invention provides a method for training a non-stop sorting control model of an automated guided vehicle. The method is used to train a control model to control the automated guided vehicle to deliver packages to a preset sorting port during driving. Referring to Figure 1 , the method includes the following steps S101 to S102:

[0075] Step S101: In a preset scenario, the automated guided vehicle performs sorting and delivery, and based on the state parameters collected in real time by a preset sensor, a state space is constructed, and an action space is constructed based on the sorting and delivery control elements.

[0076] Step S102: Obtain the initial policy network and value network to construct the main neural network of the actor-critic network structure. The policy network takes the current state parameters in the state space as input and outputs actions in the action space; the value network takes the current state parameters in the state space as input and outputs the value of the current state. In a sorting round, when the package is not thrown, introduce the package landing point prediction reward, the automatic guided vehicle position offset reward, the conveyor belt acceleration reward, and the moving mechanism acceleration reward to construct the reward value; when the package is thrown, establish the reward value based on the package landing point. Based on reinforcement learning, train the policy network and the value network. In each sorting round, use the reward value to calculate the return and advantage value at each time step, and update the policy network by restricting the change range of the probability ratio of the new and old policies in combination with the advantage value; update the value network by minimizing the value prediction error in combination with the return. After the policy network and the value network are stable, introduce a hypernetwork to train and output the main network parameters with the state parameters of multiple time steps recorded in the experience pool as input, and obtain a secondary neural network to preferentially generate the initial parameters of the main neural network in the new state; use the policy network obtained from multiple rounds of iteration as the target sorting control model.

[0077] In step S101, in the preset sorting scenario, the AGV executes the sorting and delivery task. To achieve precise control of the AGV's behavior, it is first necessary to collect the state parameters (such as position, speed, acceleration, etc.) of the AGV in real time through preset sensors, and construct a state space based on these parameters. The state space is the set of all possible states of the AGV during sorting, and is used to describe the operating state of the AGV at a certain moment.

[0078] At the same time, based on the control requirements of sorting and delivery (such as adjusting the position, speed, acceleration, etc. of the AGV), construct an action space. The action space is the set of all possible control actions that the AGV can take during sorting, and is used to guide the AGV on how to adjust its own behavior to complete the sorting task.

[0079] Specifically, as Figure 2 and 3 shown, the state parameters include the distances d xc and d yc along the x-axis and y-axis between the automatic guided vehicle and the target sorting port, the deviations Δd x and Δd y along the x-axis and y-axis between the current position and the target trajectory point at the current moment, the angle deviation Δθ between the current traveling direction and the target trajectory direction, the angle α between the current traveling direction and the positive x-axis direction, the distances d xp and d yp along the x-axis and y-axis between the package to be delivered and the target sorting port, and the moving direction speed v p of the package to be delivered, with counterclockwise being positive. The action space includes the conveyor belt speed and the moving mechanism speed of the automatic guided vehicle.

[0080] In step S102, the Actor-Critic Networks is a deep reinforcement learning algorithm that combines the policy gradient method and the value function method to achieve efficient learning through two collaborating components, including the Actor and the Critic. The Actor is responsible for generating policies, that is, selecting actions in a given state; the Critic is responsible for evaluating the value of the current policy and providing a feedback signal to the Actor to help it improve the policy. A closed-loop feedback mechanism is formed between the Actor and the Critic. The Critic evaluates the performance of the current policy based on the observed experience and provides the policy gradient to guide the Actor to update; the Actor interacts with the environment under the guidance of the new policy, generating new experience data for the Critic to further learn. This collaborative working mode effectively combines the stability of the value-based reinforcement learning method and the flexibility of the policy-based reinforcement learning method, enabling the algorithm to quickly adjust the policy parameters in a complex environment and achieve efficient learning.

[0081] The policy network, as the Actor, takes the current state parameters in the state space as input and outputs specific actions in the action space. The role of the policy network is to guide the AGV to take appropriate actions in a certain state. The value network, as the Critic, takes the current state parameters in the state space as input and outputs the value of the current state. The role of the value network is to evaluate the quality of a certain state and provide a reference for the optimization of the policy network. After the policy network and the value network are stable, a hypernetwork is introduced. The hypernetwork takes the state parameters of multiple time steps recorded in the experience pool as input and trains to output the parameters of the main network. In this way, a secondary neural network is obtained, whose role is to preferentially generate the initial parameters of the main network in a new state, thereby improving the generalization ability and training efficiency of the model.

[0082] In some embodiments, the policy network adopts a multi-layer perceptron, a temporal network, or a dedicated policy network of the proximal policy optimization algorithm; the value network adopts a multi-layer perceptron; the hypernetwork is constructed based on a recurrent neural network or a Transformer network.

[0083] During the reinforcement learning process, a reward mechanism is designed based on this delivery scenario, divided into different stages during the sorting process, and the reward value is dynamically adjusted according to whether the package is thrown:

[0084] (1) When the package is not thrown, introduce the package landing point prediction reward, the automatic guided vehicle position offset reward, the conveyor belt acceleration reward, and the moving mechanism acceleration reward.

[0085] The package drop prediction reward is given according to the matching degree between the predicted package drop point and the target sorting port. The AGV position deviation reward is given according to the deviation degree between the current position of the AGV and the ideal path. The conveyor belt acceleration reward is given according to whether the acceleration of the conveyor belt is reasonable. The moving mechanism acceleration reward is given according to whether the acceleration of the AGV moving mechanism is reasonable.

[0086] Specifically, the calculation formula for the package drop prediction reward is:

[0087]

[0088] Among them, k1 represents the precision reward coefficient of the predicted package drop point in the sorting grid when the package is not thrown; k2 represents the precision reward coefficient of the predicted package drop point outside the sorting grid when the package is not thrown; represents the distance between the predicted package drop point and the preset sorting port in the x-axis direction, represents the distance between the predicted package drop point and the preset sorting port in the y-axis direction; b represents the denominator offset.

[0089] Among them, the calculation formula for the predicted package drop point is:

[0090]

[0091] Among them, d xp represents the distance between the package to be delivered and the target sorting port along the x-axis, d yp represents the distance between the package to be delivered and the target sorting port along the y-axis; v p represents the moving direction speed of the package to be delivered; v r represents the right wheel speed, v l represents the left wheel speed; h represents the height difference between the target sorting port and the package to be delivered, g represents the acceleration due to gravity; α represents the angle between the current traveling direction and the positive x-axis direction.

[0092] The calculation formula for the AGV position deviation reward is:

[0093] r route = k3·(Δd x 2 + Δd y 2 ) + k4·Δθ;

[0094] Among them, k3 represents the distance penalty coefficient between the AGV and the target trajectory point; Δd x represents the distance between the center of the AGV and the target trajectory point along the x-axis direction, Δd y represents the distance between the center of the AGV and the target trajectory point along the y-axis direction; k4 represents the angle deviation penalty coefficient; Δθ represents the angle deviation between the current traveling direction and the target trajectory direction.

[0095] The calculation formula for the conveyor belt acceleration reward is:

[0096]

[0097] Among them, k5 represents the current conveyor belt speed mutation penalty coefficient; v ct represents the conveyor belt speed at time t, a cmax represents the maximum acceleration limit of the conveyor belt.

[0098] The moving mechanism acceleration reward includes the left wheel acceleration reward and the right wheel acceleration reward. The calculation formula for the left wheel acceleration reward is:

[0099]

[0100] Among them, k6 represents the current left wheel speed mutation penalty coefficient; v lt represents the left wheel speed at time t, a lmax represents the maximum acceleration limit of the left wheel.

[0101] The calculation formula for the right wheel acceleration reward is:

[0102]

[0103] Among them, k7 represents the current right wheel speed mutation penalty coefficient; v rt represents the right wheel speed at time t, a rmax represents the maximum acceleration limit of the right wheel.

[0104] Then the calculation formula for the reward value when the package is not thrown is:

[0105] r t = r end + r route + r conStable + r leftStable + r rightStabel .

[0106] (2) When the package is thrown, change the predicted package landing point reward to the actual package landing point reward. The calculation formula is:

[0107]

[0108] Among them, k8 represents the accuracy reward coefficient of the predicted package landing point in the sorting grid when the package is not thrown; k9 represents the accuracy reward coefficient of the predicted package landing point outside the sorting grid when the package is not thrown, and b represents the denominator offset.

[0109] Furthermore, during the training process in each sorting round, the return (i.e., cumulative reward) and the advantage value (i.e., the goodness or badness of the current action relative to the average policy) at each time step are calculated using the reward value.

[0110] When updating the policy network, the advantage value is combined, and the policy network is optimized by restricting the range of change of the probability ratio between the new and old policies (e.g., using the PPO algorithm) to ensure the stability and convergence of the policy.

[0111] When updating the value network, the value network is optimized by minimizing the value prediction error (i.e., the difference between the predicted value and the actual return) to improve the accuracy of the state value.

[0112] In reinforcement learning, the PPO (Proximal Policy Optimization) algorithm optimizes the parameters of the policy network through a clipping-based policy update method. The core idea of PPO is to ensure that each update does not cause the policy to change too much by restricting the magnitude of the policy update, thereby improving the stability and convergence of training. Specifically, PPO uses the importance sampling ratio i.e., the ratio of the action probabilities of the current policy and the old policy in a certain state, to measure the magnitude of the policy change. To avoid excessive policy updates, PPO clips this ratio, restricting its range of change to [1 - ε, 1 + ε], where ε is a small positive number (such as 0.1 or 0.2). In this way, PPO ensures that the policy update is within a reasonable range, and at the same time uses the advantage value A i as a guiding signal to help the policy improve in a better direction. Finally, PPO updates the parameters of the policy network by minimizing the clipped loss function, enabling the policy to gradually improve its performance while maintaining stability. This clipping-based update method not only improves the robustness of training but also makes PPO show good convergence and sample efficiency in complex environments.

[0113] In some embodiments, when calculating the return and the advantage value at each time step using the reward value in each sorting round, the calculation formula for the return is:

[0114]

[0115] The calculation formula for the said advantage value is:

[0116]

[0117] δ t+l =r t +γV(s t+1 )-V(S t );

[0118] where, δ t+lis the time-domain differential error; r k represents the reward value at time step k; λ is the parameter of the Generalized Advantage Estimation (GAE); l is the summation variable that traverses from 0 to the remaining time steps T - t, representing the offset of multiple future steps; γ is the discount factor that controls the importance of future rewards in reinforcement learning.

[0119] In some embodiments, when updating the policy network by combining the advantage value through restricting the variation range of the probability ratio between the new and old policies, the loss function calculation formula is:

[0120]

[0121] where is the importance sampling ratio, is the loss function under policy θ a , and ε is the clipping rate.

[0122] Then the policy network parameter update formula is:

[0123]

[0124] The value network is updated by minimizing the value prediction error in combination with the said return, and the loss function calculation formula is:

[0125]

[0126] Then the value network parameter update formula is:

[0127]

[0128] A hypernetwork is introduced to train and output the main network parameters with the state parameters of multiple time steps recorded in the experience pool as the input. The loss function calculation formula is:

[0129]

[0130] where w main represents the parameter set of the current main neural network; w assist represents the parameter set of the main neural network predicted by the hypernetwork.

[0131] The parameters of the hypernetwork are updated through gradient descent, and the updated parameters are:

[0132]

[0133] where α\(_1\), α\(_2\) and α\(_3\) are the learning rates.

[0134] On the other hand, the present invention also provides a non-stop sorting control method for an automated guided vehicle, which is used to control the automated guided vehicle to deliver packages to a preset sorting port during driving. The method includes the following steps S201 to S204:

[0135] Step S201: receiving a sorting task, which records information of the package to be sorted, the initial location of the package, and the target location of the package.

[0136] Step S202: Based on a preset path planning algorithm, a target trajectory is planned according to the current position of the AGV, the initial position of the package, and the target position of the package.

[0137] Step S203: Control the preset automatic guided transport vehicle to arrive at the initial location of the package according to the target trajectory to pick up the package to be sorted.

[0138] Step S204: Control the preset automatic guided vehicle to travel to the target location of the package according to the target trajectory, and after arriving at the sorting area, perform the delivery task based on the target sorting control model obtained by the above-mentioned automatic guided vehicle non-stop sorting control model training method.

[0139] This method involves four steps: first, receiving the sorting task and recording the initial and target locations of the package to be sorted; then, based on a preset path planning algorithm, planning the optimal driving route; then controlling the transport vehicle along the planned route to reach the initial location of the package for pickup; and finally, when the transport vehicle reaches the sorting area, utilizing the trained target sorting control model to complete the package delivery task. This non-stop sorting method can effectively reduce the downtime of the transport vehicle during the sorting process, improving overall logistics efficiency. Furthermore, through precise path planning and control models, it ensures that the package arrives at the designated location accurately and without error, reducing the sorting error rate. This method is applicable to various logistics scenarios, especially large-scale warehouses and distribution centers that require efficient sorting.

[0140] On the other hand, the present invention also provides a non-stop sorting control device for an automated guided vehicle, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0141] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when executed by a processor.

[0142] On the other hand, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0143] The present invention will be described below in conjunction with a specific embodiment:

[0144] In view of the deficiency that the AGV sorting robot relies solely on physical models and parameters for the horizontal throwing sorting of packages during non-stop sorting, that is, the physical characteristics of different AGVs are different, and the parameters of the physical model of the horizontal throwing motion need to be adjusted one by one to make the model work properly, and the parameters need to be corrected later if mechanical wear occurs, a non-stop sorting control scheme for AGVs is provided. First, the parameters of the main neural network are initially determined through offline simulation, and the physical model of the horizontal throwing motion is incorporated into the reward plasticity stage of the intermediate reward of reinforcement learning, which speeds up the training of reinforcement learning in this scenario. After completing the training of offline reinforcement learning, the trained neural network is deployed to the actual operation scenario, and the parameters of the main neural network are fine-tuned in real time using the same logic. In addition, a hypernetwork model is added in offline simulation and actual training. The input network is the state, and the output is the parameters of the main neural network. By optimizing the parameters of the secondary neural network through the gradient of the state of the main neural network and the best parameter samples, a secondary neural network that can predict the parameters of the main neural network under unknown states can be obtained, which can accelerate the convergence speed of the main neural network.

[0145] In this embodiment, the PPO algorithm in reinforcement learning is used to control the non-stop sorting of AGVs. The AGV cart carries the package and approaches the package sorting grid. Before passing through the sorting grid, the conveyor belt is started to give the package a horizontal speed, so that the AGV cart can throw the package into the sorting grid in the form of a horizontal throwing motion without stopping. In the traditional non-stop sorting process, the starting time of the conveyor belt and the speed of the target package need to be found according to the physical model of the horizontal throwing motion. This embodiment uses reinforcement learning to solve the non-stop sorting and delivery of AGVs without a physical model, including the following steps:

[0146] Modeling is carried out according to the actual non-stop sorting scenario of AGVs to establish a Markov decision process.

[0147] 1. Design of state space, action space, reward mechanism, and neural network structure:

[0148] ① First, construct the state space S: {d xc [ , d yc [ , Δd x [ , Δd y [ , Δθ, d xp [ , d yp [ , v p [ , α}, where the state space variables are the distances d xc [ and d yc [ of the automatic guided vehicle and the target sorting port along the x-axis and y-axis, respectively, the deviations Δd x [ of the current position and the target trajectory point along the x-axis and y-axis at the current moment x, Δd y , the angle deviation Δθ between the current traveling direction and the target trajectory direction, the angle α between the current traveling direction and the positive x-axis, and the distances d of the package to be delivered from the target sorting port along the x-axis and y-axis xp 、d yp , the moving direction speed v of the package to be delivered p , counterclockwise is positive. In this model, because of the addition of speed mutation limit, the conveyor belt speed will not change significantly, and it is considered that the speed of the package is the same as the conveyor belt speed. This state space can also support the scenario of dynamic movement of the sorting grid, and v p can be converted to relative speed.

[0149] ② Establish the action space A: {v c , v l , v r}, the action space includes the speed of the conveyor belt on the AGV and the speed of the AGV moving mechanism. Taking the two-wheel differential AGV as an example, it is the speed of the left and right wheels.

[0150] ③ Establish the reward mechanism. According to the model of projectile motion, calculate and design the reward based on the distance between the AGV and the sorting grid after the future package is thrown, the distance and angle difference of the vehicle deviating from the original route trajectory point during the movement under the current state space and action space. The reward calculation formula for the vehicle when the package is not thrown is as follows:

[0151] When the package is not thrown, introduce the package landing point prediction reward, the AGV position deviation reward, the conveyor belt acceleration reward, and the moving mechanism acceleration reward.

[0152] The package landing point prediction reward is given according to the matching degree between the predicted package landing point and the target sorting port. The AGV position deviation reward is given according to the deviation degree of the AGV's current position from the ideal path. The conveyor belt acceleration reward is given according to whether the acceleration of the conveyor belt is reasonable. The moving mechanism acceleration reward is given according to whether the acceleration of the AGV moving mechanism is reasonable.

[0153] Specifically, the calculation formula for the package landing point prediction reward is:

[0154]

[0155] Among them, k1 represents the precision reward coefficient of the predicted package landing point in the sorting grid when the package is not thrown; k2 represents the precision reward coefficient of the predicted package landing point outside the sorting grid when the package is not thrown; represents the distance between the predicted package landing point and the preset sorting port in the x-axis direction, represents the distance between the predicted package landing point and the preset sorting port in the y-axis direction; b represents the denominator offset.

[0156] Among them, the calculation formula for predicting the landing point of the package is as follows:

[0157]

[0158] Among them, d xp represents the distance between the package to be delivered and the target sorting port along the x-axis, and d yp represents the distance between the package to be delivered and the target sorting port along the y-axis; v p represents the moving direction speed of the package to be delivered; v r represents the speed of the right wheel, and v l represents the speed of the left wheel; h represents the height difference between the target sorting port and the package to be delivered, g represents the acceleration due to gravity; α represents the angle between the current traveling direction and the positive x-axis direction.

[0159] The calculation formula for the position offset reward of the automatic guided vehicle is as follows:

[0160] r route = k3·(Δd x 2 +Δd y 2 ) + k4·Δθ;

[0161] Among them, k3 represents the distance penalty coefficient between the automatic guided vehicle and the target trajectory point; Δd x represents the distance between the center of the automatic guided vehicle and the target trajectory point along the x-axis direction, and Δd y represents the distance between the center of the automatic guided vehicle and the target trajectory point along the y-axis direction; k4 represents the angle deviation penalty coefficient; Δθ represents the angle deviation between the current traveling direction and the target trajectory direction.

[0162] The calculation formula for the conveyor belt acceleration reward is as follows:

[0163]

[0164] Among them, k5 represents the current conveyor belt speed mutation penalty coefficient; v ct represents the conveyor belt speed at time t, and a cmax represents the maximum acceleration limit of the conveyor belt.

[0165] The moving mechanism acceleration reward includes the left wheel acceleration reward and the right wheel acceleration reward. The calculation formula for the left wheel acceleration reward is as follows:

[0166]

[0167] \nAmong them, k6 represents the current left wheel speed mutation penalty coefficient; v lt represents the left wheel speed at time t, and a lmax represents the maximum acceleration limit of the left wheel.

[0168] The calculation formula for the right wheel acceleration bonus is:

[0169]

[0170] Among them, k7 represents the penalty coefficient of the current right wheel speed mutation; v rt represents the right wheel speed at time t, a rmax Indicates that the right wheel limits the maximum acceleration.

[0171] The calculation formula for the reward value when the package is not thrown is:

[0172] r t =r end +r route +r conStable +r leftStable +r rightStabel .

[0173] The reward of the car changes after the package is thrown. The landing point of the package is the real landing point. Change the r in the above reward. end , the reward coefficient changes, and the calculation formula is:

[0174]

[0175] Among them, k8 represents the accuracy bonus coefficient for the predicted landing point of the package in the sorting grid when it is not thrown; k9 represents the accuracy bonus coefficient for the predicted landing point of the package outside the sorting grid when it is not thrown, and b represents the denominator bias.

[0176] r end is the accuracy reward for the package landing point, but the actual landing point of the package only appears once in a round, so this scenario belongs to the problem of sparse reward, and there is a reward only after the round ends. Therefore, an intermediate reward with a smaller reward coefficient r is added. end The calculation of the horizontal projection motion is used to predict the landing point and reward plasticity. The trajectory stability reward r is added in the middle process. route , keep the original track during the sorting process; conveyor belt, left and right wheel stability reward r conStable , r leftStable , r rightStable , when the speed mutates, a negative reward value is given to reduce the speed mutation.

[0177] ④ Structure Design of Neural Networks: We set up two neural networks, namely Critic and Actor. The Critic network is responsible for evaluating the rewards of the current state and outputting them, while the Actor network is responsible for calculating what actions should be output in the current state. The inputs of both networks are the state space. The input layers of the two neural networks are the same, but the hidden layers and output layers are different. The output of the Critic is the expected reward of the current state, and the final output of the Actor policy network is the speeds of the conveyor belt and the left and right wheels in the action space. The speeds of the three are continuously adjustable within a certain range. The Actor can generate three parameters of the normal distribution, namely the means μ c , μ l , μ r , and the standard deviations σ c , σ l , σ r . The speeds in the action space all conform to v ∼ N(μ, σ 2 ). After the Actor generates the parameters, it randomly samples the normal distribution of the speeds to obtain the speeds of the conveyor belt, the left wheel, and the right wheel. The AGV executes the obtained speeds as the action a t at the current time step.

[0178] ⑤ Structure Design of the Hypernetwork: Define a neural network with the state space as the input and the parameters of the Actor and Critic networks as the input. Its purpose is to predict the parameters of the main neural network through the neural network of the hypernetwork according to the input state, and use the predicted parameters as the initial parameters of the main neural network, thereby reducing the speed of gradient update during training and accelerating the convergence of the main neural network.

[0179] 2. Execution Process of a Single Sorting Round

[0180] Environment Initialization: First, when the AGV enters the sorting straight section, the reinforcement learning process is started, the environment is initialized, the initial state s0 is obtained, and the initial v p of the package at the start state is 0.

[0181] Policy Network Outputs Actions, Value Network Evaluates State Values: The Actor network generates the speed of the conveyor belt according to the current state, and this action is a t ∼ π(a|s t ). The Critic network generates the current predicted state value V(s t ) according to the current state.

[0182] Execute Actions: The AGV executes the action a t output by the policy network.

[0183] Calculate Rewards: When the package has not been thrown off the conveyor belt, since this scenario is a turn-based task, the reward value r end = 0 during the intermediate process. Therefore, the reward r during the intermediate processt For r stable .

[0184] Save data: save the data (s t ,a t ,r t ,logπ old (a t |s t ),V(s t )) is added to the cache, where logπ old (a t |s t ) Select a for the current strategy t The probability of , in this scenario, the current strategy randomly selects a from the generated normal distribution t The probability of is used to calculate the importance probability ratio in the PPO algorithm.

[0185] After Δt time, the next state s is obtained through environmental monitoring t+1 , and so on until the package leaves the conveyor belt. The time step when the package leaves the conveyor belt can be used to record the final reward r of this round. t =r end +r stable Finally, the data of all time steps stored are:

[0186] {(s0,a0,t0,logπ old (a0|s0),(s1,a1,r1,logπ old (a1|s1),...,(s T-1 ,a T-1 ,r T-1 ,logπ old (a T-1 |s T-1 )}

[0187] 3. Calculation of the return and generalized advantage estimate for each time step of a single round

[0188] After each round, the reward G of each time step can be calculated by reverse calculation of all rewards t :

[0189]

[0190] Then calculate the advantage value A at each time step t :

[0191]

[0192] δ t+l =r t +γV(S t+1)-V(S t );

[0193] where δ t+l is the time-domain differential error; r k represents the reward value at time step k; λ is the parameter of generalized advantage estimation; l is the summation variable, traversing from 0 to the remaining time steps T - t, representing the offset of multiple future steps; γ is the discount factor, which controls the importance of future rewards in reinforcement learning.

[0194] This scenario is a turn-based reinforcement learning, and the PPO algorithm is used to update the Actor network and the Critic network. To reduce the sample variance, after sampling multiple sets of episode data, they are randomly grouped for network gradient update. For example, after collecting and processing data for multiple episodes to obtain M time-step data, shuffling the data order and dividing it into m1 batches for policy update, with each batch containing the order of B time steps, and randomly selecting B data from M for updating the Actor network and the Critic network in each batch.

[0195] In updating the policy network by combining the advantage value through restricting the variation range of the probability ratio of the old and new policies, the loss function calculation formula is:

[0196]

[0197] where is the importance sampling ratio, is the loss function under policy θ a , and ε is the clipping rate.

[0198] Then the policy network parameter update formula is:

[0199]

[0200] In updating the value network by minimizing the value prediction error by combining the return, the loss function calculation formula is:

[0201]

[0202] Then the value network parameter update formula is:

[0203]

[0204] Repeat the above steps 2 - 4 until the variance of the reward value r t at the end of k consecutive episodes is less than the set value, then it is considered that the reinforcement learning result of AGV non-stop sorting has converged, the neural network parameters have reached the optimal, and the reinforcement learning ends.

[0205] 5. Update of the neural network parameters of the hypernetwork:

[0206] After the main neural networks, i.e., the Actor and Critic networks, are stable, the learning process of the neural network of the hypernetwork begins. In the experience pool, a correspondence table of the states at multiple time steps after the main neural networks are stable and the parameters of the main neural networks is recorded and added to the training of the hypernetwork for calculation. The data stored in a stable episode is: (s0, s1,..., s T-1 ), and the corresponding two sets of main neural network parameters are: (w 00 , w 01 ,..., w 2(n-1)(k-1) ). The set of parameters of the current main neural network is denoted as w main , where n is the number of hidden layers and k is the number of network nodes in each layer. The initial policy of the auxiliary neural network is denoted as θassist, and the set of predicted neural network parameters generated according to its policy is w assist . The loss function is calculated by the difference between the predicted value and the actual value, and the loss function is calculated as follows:

[0207]

[0208] The parameters of the auxiliary neural network are updated by gradient descent, and the updated policy is:

[0209]

[0210] Each time a state occurs, the neural network of the hypernetwork is updated once, and a stable episode is updated T times. Repeat the above process until the variance is less than for m2 consecutive times, then it is considered that the auxiliary neural network has converged, and the parameters of the main neural network in the new state can be predicted. The predicted parameter values can be directly used to initialize the main neural network in the new state to accelerate the gradient update of the parameters of the main neural network, i.e., the Actor and Critic networks.

[0211] Correspondingly, the present invention also provides an apparatus / system. The apparatus / system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the apparatus / system implements the steps of the method described above.

[0212] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the foregoing edge computing server deployment method are implemented. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium well known in the technical field.

[0213] In summary, for the automatic guided vehicle non-stop sorting control model training, control method and device, the state space is constructed by collecting the state parameters of the AGV in real time through preset sensors, and the action space is constructed based on the sorting and delivery control elements. Then, an actor-critic network structure is constructed using a policy network and a value network. The policy network takes the current state parameters as input and outputs actions, and the value network takes the current state parameters as input and outputs the state value. During the sorting process, the reward value is dynamically adjusted according to whether the package is thrown. When the package is not thrown, a package landing point prediction reward, an AGV position offset reward, a conveyor belt acceleration reward, and a moving mechanism acceleration reward are introduced. When the package is thrown, the reward value is calculated based on the actual landing point. Through the reinforcement learning algorithm, the return and advantage value of each time step are calculated using the reward value, the change range of the probability ratio of the new and old policies is optimized by combining the advantage value to limit the change range, and the value network is optimized by minimizing the value prediction error. After the policy network and the value network are stable, a hypernetwork is introduced, and the sub-neural network is trained with the state parameters recorded in the experience pool as input, so that it preferentially generates the initial parameters of the main network in the new state. The finally trained target sorting control model can achieve efficient sorting without stopping, significantly improving the sorting efficiency and accuracy.

[0214] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave on a transmission medium or a communication link.

[0215] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0216] In the present invention, features described and / or illustrated for one embodiment can be used in the same or similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0217] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for training an automatic guided vehicle non-stop sorting control model, characterized in that The method is used to train a control model to control an automated guided vehicle to deliver a package to a preset sorting outlet during driving. The method includes the following steps: In a preset scenario, the automated guided vehicle performs sorting and delivery, and based on state parameters collected in real time by preset sensors, a state space is constructed, and an action space is constructed based on sorting and delivery control elements; An initial policy network and a value network are obtained to construct a main neural network with an actor-critic network structure. The policy network is used to take the current state parameters in the state space as input and output an action in the action space; the value network takes the current state parameters in the state space as input and outputs the value of the current state. In a sorting round, when the package is not thrown out, a package landing point prediction reward, an automated guided vehicle position offset reward, a conveyor belt acceleration reward, and a moving mechanism acceleration reward are introduced to construct a reward value; when the package is thrown out, the reward value is established based on the package landing point. The policy network and the value network are trained based on reinforcement learning. In each sorting round, the reward value is used to calculate the return and advantage value of each time step. The policy network is updated by restricting the change range of the probability ratio of the new and old policies in combination with the advantage value; the value network is updated by minimizing the value prediction error in combination with the return. After the policy network and the value network are stable, a hypernetwork is introduced to train and output the main network parameters with the state parameters of multiple time steps recorded in the experience pool as input, and a secondary neural network is obtained to preferentially generate the initial parameters of the main neural network in a new state; the policy network obtained through multiple rounds of iteration is used as the target sorting control model.

2. The training method of the non-stop sorting control model for the automatic guided vehicle according to claim 1, characterized in that The state parameters include the distances between the automated guided vehicle and the target sorting outlet along the x-axis and y-axis, the deviations between the current position and the target trajectory points at the current moment along the x-axis and y-axis, the angle deviation between the current traveling direction and the target trajectory direction, the included angle between the current traveling direction and the positive x-axis, the distances between the package to be delivered and the target sorting outlet along the x-axis and y-axis, and the moving direction speed of the package to be delivered; the action space includes the conveyor belt speed and the moving mechanism speed of the automated guided vehicle.

3. The training method for the non-stop sorting control model of the automatic guided vehicle according to claim 2, characterized in that, The policy network adopts a multi-layer perceptron, a temporal network, or a dedicated policy network of the proximal policy optimization algorithm; the value network adopts a multi-layer perceptron; the hypernetwork is constructed based on a recurrent neural network or a Transformer network.

4. The method for training the non-stop sorting control model of the automatic guided vehicle according to claim 3, characterized in that, In a sorting round, when the package is not thrown out, the calculation formula for the package landing point prediction reward is: Among them, k1 represents the accuracy reward coefficient when the predicted landing point of the package is in the sorting grid before being thrown; k2 represents the accuracy reward coefficient when the predicted landing point of the package is outside the sorting grid before being thrown; represents the distance between the predicted landing point of the package and the preset sorting port in the x-axis direction, represents the distance between the predicted landing point of the package and the preset sorting port in the y-axis direction; b represents the denominator bias; The calculation formula for the automated guided vehicle position offset reward is: r route = k3·(Δd x 2 + Δd y 2 ) + k4·Δθ; Among them, k3 represents the distance penalty coefficient between the automatic guided vehicle and the target trajectory point; Δd x represents the distance between the center of the automatic guided vehicle and the target trajectory point in the x-axis direction, and Δd y represents the distance between the center of the automatic guided vehicle and the target trajectory point in the y-axis direction; k4 represents the angle deviation penalty coefficient; Δθ represents the angle deviation between the current traveling direction and the target trajectory direction; The calculation formula for the conveyor belt acceleration reward is: Among them, k5 represents the current conveyor belt speed mutation penalty coefficient; v ct represents the conveyor belt speed at time t, a cmax represents the maximum acceleration limit of the conveyor belt; The moving mechanism acceleration reward includes a left wheel acceleration reward and a right wheel acceleration reward. The calculation formula for the left wheel acceleration reward is: Among them, k6 represents the current mutation penalty coefficient of the left wheel speed; v lt represents the left wheel speed at time t, a lmax represents the maximum acceleration limit of the left wheel; The calculation formula for the right wheel acceleration reward is: Among them, k7 represents the current mutation penalty coefficient of the right wheel speed; v rt represents the speed of the right wheel at time t, a rmax represents the maximum acceleration limit of the right wheel; Then the calculation formula for the reward value when the package is not thrown out is: r t ＝r end +r route +r conStable +r leftStable +r rightStabel ； When the package is thrown out, the package landing point prediction reward is changed to a package actual landing point reward, and the calculation formula is: Among them, k8 represents the precision reward coefficient when the predicted landing point of the package is within the sorting bin before being thrown; k9 represents the precision reward coefficient when the predicted landing point of the package is outside the sorting bin before being thrown, and b represents the denominator bias.

5. The training method for the non-stop sorting control model of the automatic guided vehicle according to claim 4, wherein, The calculation formula for the predicted landing point of the package is: where d xp represents the distance along the x-axis between the package to be delivered and the target sorting opening, and d yp represents the distance along the y-axis between the package to be delivered and the target sorting opening; v p represents the velocity of the moving direction of the package to be delivered; v r represents the velocity of the right wheel, and v l represents the velocity of the left wheel; h represents the height difference between the target sorting opening and the package to be delivered, g represents the acceleration due to gravity; α represents the angle between the current traveling direction and the positive x-axis direction.

6. The training method of the non-stop sorting control model for the automatic guided vehicle according to claim 5, characterized in that, When calculating the return and advantage value of each time step using the reward value in each sorting round, the calculation formula for the return is The calculation formula for the advantage value is: δ t+l = r t + γV(S t+1 ) - V(S t ); where δ t+l is the time-domain differential error; r k represents the reward value at time step k; λ is the parameter of generalized advantage estimation; l is the summation variable, traversing from 0 to the remaining time steps T - t, representing the offset of multiple future steps; γ is the discount factor, which controls the importance of future rewards in reinforcement learning.

7. The method for training the non-stop sorting control model of the automatic guided vehicle according to claim 6, wherein When updating the policy network by combining the advantage value to limit the change range of the probability ratio of the old and new policies, the calculation formula for the loss function is: Among them, is the importance sampling ratio, is the loss function under the policy θ a , and ε is the clipping rate; Then the policy network parameter update formula is: When updating the value network by minimizing the value prediction error in combination with the return, the calculation formula for the loss function is: Then the value network parameter update formula is: Introduce a hypernetwork, use the state parameters of multiple time steps recorded in the experience pool as input to train and output the main network parameters, and its loss function calculation formula is: where, w main represents the parameter set of the currently described main neural network; w assist represents the parameter set of the main neural network predicted by the hypernetwork; Update the parameters of the hypernetwork through gradient descent, and the updated parameters are: Among them, α1, α2, and α3 are learning rates.

8. An automatic guided vehicle non-stop sorting control method, characterized in that The method is used to control an automated guided vehicle to deliver a package to a preset sorting port during driving, and the method includes the following steps: Receive a sorting task, and the sorting task records the information of the package to be sorted, the initial position of the package, and the target position of the package; Based on a preset path planning algorithm, plan a target trajectory according to the current position of the automated guided vehicle, the initial position of the package, and the target position of the package; Control the preset automated guided vehicle to reach the initial position of the package along the target trajectory to pick up the package to be sorted; Control the preset automated guided vehicle to drive along the target trajectory towards the target position of the package, and execute the delivery task based on the target sorting control model obtained by the automated guided vehicle non-stop sorting control model training method according to any one of claims 1 to 7 after reaching the sorting area.

9. An automatic guided vehicle non-stop sorting control device, comprising a processor, a memory, and computer programs / instructions stored on the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having computer programs / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images