Non-stop sorting control model training, control method and device for automated guided vehicle

Abstract

The application provides a kind of training, control method and device of automatic guided vehicle sorting control model without stopping.The state space is constructed by preset sensor real-time acquisition state parameter, and the action space is constructed based on sorting delivery control elements.According to whether the package is thrown out, the reward value is dynamically adjusted.When the package is not thrown out, the package drop point prediction reward, AGV position offset reward, conveyor acceleration reward and moving mechanism acceleration reward are introduced;When the package is thrown out, the reward value is calculated based on the actual drop point.Through reinforcement learning algorithm, the return and advantage value of each time step are calculated using the reward value, the policy network is optimized by limiting the change range of the probability ratio of new and old strategies combined with the advantage value, and the value network is optimized by minimizing the value prediction error.A super network is introduced, and the state parameters recorded in the experience pool are used as input to train the secondary neural network, so that the initial parameters of the primary neural network are generated in the new state, which significantly improves the sorting efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic guided vehicles, and particularly relates to a non-stop sorting control model training method and device for an automatic guided vehicle, and a control method and device. BACKGROUND

[0002] In recent years, with the rapid development of the logistics industry, sorting efficiency has become a key factor affecting the overall efficiency of logistics transportation. The logistics business volume is showing explosive growth, and the sorting scene is becoming increasingly diversified, small-batch and rapid, which puts higher requirements on the flexibility, efficiency and cost control of the sorting system. The current mainstream automated sorting system mainly falls into two categories: one is a high-speed sorting machine system, and the other is an intelligent sorting robot system based on an automatic guided vehicle (AGV).

[0003] The high-speed sorting machine system includes a push block type sorting machine system, a sliding block type sorting machine system, a flip plate type sorting machine system and a cross belt type sorting machine system. This type of sorting machine system has a relatively fast sorting speed, and the maximum sorting speed of a single-layer sorting machine system can reach 10800 pieces / hour, the highest linear speed can reach 2.25 m / s, and the weight of the processed objects does not exceed 30 kg. However, this type of sorting system generally has a high price, occupies a large area, and has a fixed equipment scale and installation location, cannot adaptively adjust the use scale with the scale change of the processing business, and has a large energy consumption. In comparison, the AGV-based sorting robot system has gradually become a research hotspot in the industry due to its high adaptability, low cost and scalability. However, the existing AGV sorting robot system still has significant bottlenecks when performing package delivery. In the conventional process, the AGV needs to move to the sorting grid according to the planned path, slow down and stop to complete the package placement when delivering, and then accelerate to the running speed again. This deceleration-stopping-acceleration process not only prolongs the sorting cycle, but also increases the energy consumption. In recent years, although some research attempts to achieve non-stop sorting through a physical model, i.e., using the principle of projectile motion to deliver packages while driving, such a scheme has the following defects: first, the individual difference adaptability is poor, and the mechanical structures (such as motor performance and conveyor belt friction coefficient) of different AGVs are different, which makes it difficult for a fixed physical model to be applied universally, and the sorting efficiency fluctuates significantly. Second, the dynamic environment response is insufficient, and real-time factors such as mechanical wear and tear and environmental changes will change the dynamics relationship between the package and the conveyor belt, and the traditional model cannot dynamically adjust the parameters, which easily causes sorting errors or package sliding. Third, the model maintenance cost is high, and frequent calibration or re-modeling is needed to adapt to equipment aging or working condition changes, which leads to an increase in operation and maintenance complexity and downtime. These problems seriously restrict the large-scale application and long-term stability of the AGV sorting system. Therefore, there is an urgent need for a new control method that can adapt to individual differences and dynamic environments, reduce operation and maintenance costs, and improve sorting efficiency, so as to break through the technical bottlenecks of the existing technology and promote the further development of logistics automation technology. SUMMARY

[0004] In view of this, the embodiment of the present application provides an AGV non-stop sorting control model training and control method and device to eliminate or improve one or more defects in the prior art, solve the problem of insufficient physical model adaptability and high operation and maintenance cost caused by individual mechanical differences and dynamic environment changes when AGV sorts without stopping.

[0005] One aspect of the present application provides an AGV non-stop sorting control model training method, which is used to train a control model to control an AGV to deliver a package to a preset sorting port during driving. The method comprises the following steps:

[0006] In a preset scenario, the AGV performs sorting delivery, and a state space is constructed based on real-time acquisition of state parameters by a preset sensor, and an action space is constructed based on sorting delivery control elements;

[0007] An initial policy network and a value network are obtained to construct an actor-critic network structure of a main neural network, the policy network is used to input the current state parameters of the state space and output actions in the action space; the value network is used to input the current state parameters of the state space and output the value of the current state; in one sorting round, when the package is not thrown out, a package drop point prediction reward, an AGV position offset reward, a conveyor belt acceleration reward and a moving mechanism acceleration reward are introduced to construct a reward value; when the package is thrown out, the reward value is established based on the package drop point; the policy network and the value network are trained based on reinforcement learning, the return and advantage value of each time step are calculated using the reward value in each sorting round, the policy network is updated by limiting the change range of the probability ratio of new and old policies in combination with the advantage value; the value network is updated by minimizing the value prediction error in combination with the return; after the policy network and the value network are stable, a super network is introduced to train the main neural network parameters using the state parameters of multiple time steps recorded in the experience pool as input, and a secondary neural network is obtained to preferentially generate the initial parameters of the main neural network in a new state; the policy network obtained through multiple iterations is used as a target sorting control model.

[0008] In some embodiments, the state parameters include the distance of the AGV and the target sorting port along the x-axis and the y-axis, the deviation of the current position and the target trajectory point along the x-axis and the y-axis at the current time, the angle deviation of the current travel direction and the target trajectory direction, the included angle between the current travel direction and the x-axis positive direction, the distance of the delivered package and the target sorting port along the x-axis and the y-axis, and the moving direction and speed of the delivered package; the action space includes the conveyor belt speed and the moving mechanism speed of the AGV.

[0009] In some embodiments, the policy network adopts a multi-layer perception, a time sequence network or a policy network dedicated to a proximal policy optimization algorithm; the value network adopts a multi-layer perception; and the super network is constructed based on a recurrent neural network or a Transformer network.

[0010] In some embodiments, in one sorting round, when a parcel is not thrown out, the calculation formula of the parcel drop point prediction reward is:

[0011] ;

[0012] wherein k1 represents an accuracy reward coefficient of the predicted drop point of the parcel in the sorting pocket when the parcel is not thrown out; k2 represents an accuracy reward coefficient of the predicted drop point of the parcel outside the sorting pocket when the parcel is not thrown out; represents the distance between the predicted drop point of the parcel and the preset sorting pocket in the x-axis direction, represents the distance between the predicted drop point of the parcel and the preset sorting pocket in the y-axis direction; and b represents a denominator bias.

[0013] The calculation formula of the automated guided vehicle position offset reward is:

[0014] ;

[0015] wherein k3 represents a distance penalty coefficient of the automated guided vehicle and the target trajectory point; Δd x represents the distance between the center of the automated guided vehicle and the target trajectory point in the x-axis direction, Δd y represents the distance between the center of the automated guided vehicle and the target trajectory point in the y-axis direction; k4 represents an angle deviation penalty coefficient; and Δθ represents the angle deviation between the current travel direction and the target trajectory direction.

[0016] The calculation formula of the conveyor belt acceleration reward is:

[0017] ;

[0018] wherein k5 represents a current conveyor belt speed mutation penalty coefficient; represents the speed of the conveyor belt at time t, a cmax represents the maximum acceleration limit of the conveyor belt;

[0019] The moving mechanism acceleration reward includes a left wheel acceleration reward and a right wheel acceleration reward, and the calculation formula of the left wheel acceleration reward is:

[0020] ;

[0021] wherein k6 represents a current left wheel speed mutation penalty coefficient; represents the speed of the left wheel at time t, almax Indicates that the maximum acceleration of the left wheel is limited;

[0022] The calculation formula for the right wheel acceleration reward is:

[0023] ;

[0024] Among them, k7 represents the penalty coefficient for the current right wheel speed mutation; represents the right wheel speed at time t, a rmax Indicates that the right wheel limits the maximum acceleration;

[0025] The calculation formula for the reward value when the package is not thrown is:

[0026] ;

[0027] When the package is thrown, the predicted landing point reward is changed to the actual landing point reward, calculated as follows:

[0028] ;

[0029] Among them, k8 represents the accuracy bonus coefficient for the predicted landing point of the package in the sorting grid when it is not thrown; k9 represents the accuracy bonus coefficient for the predicted landing point of the package outside the sorting grid when it is not thrown, and b represents the denominator bias.

[0030] In some embodiments, the formula for calculating the predicted landing point of the package is:

[0031] ;

[0032] in, Indicates the distance between the delivered package and the target sorting port along the x-axis, Indicates the distance between the delivered package and the target sorting port along the y-axis; Indicates the moving direction speed of the delivered package; represents the right wheel speed, represents the left wheel speed; h represents the height difference between the target sorting port and the delivered package; g represents the acceleration due to gravity; Indicates the angle between the current direction of travel and the positive x-axis.

[0033] In some embodiments, in each sorting round, the reward value is used to calculate the return and advantage value of each time step, and the calculation formula of the return is:

[0034] ;

[0035] The calculation formula of the advantage value is:

[0036] ;

[0037] ;

[0038] wherein, is the time-domain difference error; represents the reward value at time step k; is the parameter of generalized advantage estimation; l is a summation variable, which traverses from 0 to the remaining time step T-t, indicating the offset of future multi-steps; γ is a discount factor, which controls the importance of future rewards in reinforcement learning.

[0039] In some embodiments, the policy network is updated in combination with the advantage value by limiting the fluctuation range of the probability ratio of new and old policies, and the loss function calculation formula is:

[0040] ;

[0041] wherein, is the importance sampling ratio, is the loss function under the policy , and is the clipping rate;

[0042] The policy network parameter update formula is:

[0043] ;

[0044] The value network is updated in combination with the return by minimizing the value prediction error, and the loss function calculation formula is:

[0045] ;

[0046] The value network parameter update formula is:

[0047] ;

[0048] The hypernetwork is introduced to train the main neural network parameters with the state parameters of multiple time steps recorded in the experience pool as input, and the loss function calculation formula is:

[0049] ;

[0050] wherein, represents the parameter set of the current main neural network; represents the main neural network parameter set predicted by the hypernetwork;

[0051] The parameters of the hypernetwork are updated by gradient descent, and the updated parameters are:

[0052] ;

[0053] wherein, 、 and is a learning rate.

[0054] In another aspect, the present application also provides a non-stop sorting control method of an automated guided vehicle, which is used for controlling the automated guided vehicle to deliver a package to a preset sorting port during driving, and the method comprises the following steps:

[0055] receiving a sorting task, which records package information to be sorted, an initial position of the package, and a target position of the package;

[0056] planning a target trajectory based on a preset path planning algorithm according to a current position of the automated guided vehicle, the initial position of the package, and the target position of the package;

[0057] controlling the automated guided vehicle to reach the initial position of the package according to the target trajectory to pick up the package to be sorted;

[0058] controlling the automated guided vehicle to drive to the target position of the package according to the target trajectory, and performing a delivery task based on a target sorting control model obtained by the above-mentioned non-stop sorting control model training method after reaching a sorting area.

[0059] In another aspect, the present application also provides a non-stop sorting control device of an automated guided vehicle, which comprises a processor, a memory, and computer programs / instructions stored in the memory, and the processor is used for executing the computer programs / instructions, and the device realizes the steps of the above-mentioned method when the computer programs / instructions are executed.

[0060] In another aspect, the present application also provides a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions realize the steps of the above-mentioned method when executed by a processor.

[0061] In another aspect, the present application also provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions realize the steps of the above-mentioned method when executed by a processor.

[0062] The present application has at least the following beneficial effects:

[0063] The automatic guided vehicle non-stop sorting control model training, control method and device provided by the application constructs a state space by collecting state parameters of the AGV in real time through a preset sensor, and constructs an action space based on sorting delivery control elements. Then, an actor-critic network structure is constructed by using a policy network and a value network, wherein the policy network outputs an action with the current state parameter as input, and the value network outputs a state value with the current state parameter as input. In the sorting process, the reward value is dynamically adjusted according to whether the package is thrown out, the package drop point prediction reward, the AGV position offset reward, the conveyor acceleration reward and the moving mechanism acceleration reward are introduced when the package is not thrown out, and the reward value is calculated based on the actual drop point when the package is thrown out. Through the reinforcement learning algorithm, the reward value is used to calculate the return and advantage value of each time step, the change range of the probability ratio of the new and old policies is limited by combining the advantage value to optimize the policy network, and the value network is optimized by minimizing the value prediction error. After the policy network and the value network are stable, a super network is introduced, the state parameters recorded in the experience pool are input to train the auxiliary neural network, so that the auxiliary neural network can generate the initial parameters of the main neural network in the new state. Finally, the target sorting control model obtained by training can realize efficient sorting without stopping, and the sorting efficiency and accuracy are significantly improved.

[0064] Additional advantages, objects, and features of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0065] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0066] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0067] Figure 1 The automatic guided vehicle non-stop sorting control model training method logic diagram of an embodiment of the application.

[0068] Figure 2 The automatic guided vehicle non-stop sorting control model training method logic diagram of an embodiment of the application.

[0069] Figure 3 The automatic guided vehicle non-stop sorting control model training method logic diagram of an embodiment of the application. DETAILED DESCRIPTION

[0070] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description of the present application will be given below in conjunction with the embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but not as a limitation of the present application.

[0071] It should be further noted that, in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0072] It should be emphasized that the term "comprising" as used herein means the presence of the stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.

[0073] The automated guided vehicle needs strong adaptability to face the changes of various elements in the task scene in the sorting and delivery scenario due to individual differences and environmental changes, so as to achieve the requirement of precise control of sorting and delivery.

[0074] The present application provides a non-stop sorting control model training method for an automated guided vehicle, which is used to train a control model to control the automated guided vehicle to deliver a package to a preset sorting port during driving. Figure 1 The method comprises the following steps S101-S102:

[0075] Step S101: performing sorting and delivery by the automated guided vehicle in a preset scene, and constructing a state space based on real-time acquisition of state parameters by a preset sensor, and constructing an action space based on sorting and delivery control elements.

[0076] Step S102: Obtain the main neural network for constructing the actor-critic network structure of the initial policy network and the value network, the policy network is used for inputting the current state parameter of the state space and outputting the action in the action space; the value network is used for inputting the current state parameter of the state space and outputting the value of the current state; in one sorting round, when the package is not thrown, the package drop point prediction reward, the automated guided vehicle position offset reward, the conveyor belt acceleration reward and the moving mechanism acceleration reward are introduced to construct the reward value; when the package is thrown, the reward value is established based on the package drop point; the policy network and the value network are trained based on reinforcement learning, the return and the advantage value of each time step are calculated by using the reward value in each sorting round, the policy network is updated by limiting the change range of the probability ratio of the new and old policies in combination with the advantage value; the value network is updated by minimizing the value prediction error in combination with the return; after the policy network and the value network are stable, the super network is introduced to train the output of the main neural network parameter by using the state parameters of multiple time steps recorded in the experience pool as the input, and the secondary neural network is obtained to preferentially generate the initial parameters of the main neural network in a new state; the policy network obtained through multiple iterations is used as a target sorting control model.

[0077] In step S101, the AGV performs a sorting and delivery task in a preset sorting scenario. In order to realize accurate control of the AGV behavior, first, the state parameters (such as position, speed, acceleration, etc.) of the AGV need to be collected in real time through a preset sensor, and a state space is constructed based on these parameters. The state space is a collection of all possible states of the AGV in the sorting process, which is used to describe the running state of the AGV at a certain time.

[0078] At the same time, based on the control requirements of sorting and delivery (such as adjusting the position, speed, acceleration, etc. of the AGV), an action space is constructed. The action space is a collection of all possible control actions that the AGV can take in the sorting process, which is used to guide the AGV how to adjust its behavior to complete the sorting task.

[0079] Specifically, as shown in Figure 2 and 3 , the state parameters include the distance between the automated guided vehicle and the target sorting port along the x-axis and the y-axis, the deviation of the current position and the target trajectory point along the x-axis and the y-axis at the current time, the angle deviation of the current travel direction and the target trajectory direction, the included angle between the current travel direction and the x-axis positive direction, the distance between the delivered package and the target sorting port along the x-axis and the y-axis, the moving direction speed of the delivered package,

[0080] counterclockwise is positive. The action space includes the conveyor belt speed and the moving mechanism speed of the automated guided vehicle.In step S102, the actor-critic network is a deep reinforcement learning algorithm that combines policy gradient methods and value function methods to achieve efficient learning through two interdependent components: the actor and the critic. The actor is responsible for generating a policy, i.e., selecting an action given a state; the critic evaluates the value of the current policy, providing feedback signals to the actor to help improve the policy. A closed-loop feedback mechanism is formed between the actor and the critic, with the critic evaluating the performance of the current policy based on observed experiences and providing policy gradient guidance for the actor to update; the actor interacts with the environment under the guidance of the new policy, generating new experience data for the critic to further learn. This collaborative working mode effectively combines the stability of value-based reinforcement learning methods and the flexibility of policy-based reinforcement learning methods, enabling the algorithm to quickly adjust policy parameters in complex environments and achieve efficient learning.

[0081] The policy network acts as an actor, taking the current state parameters in the state space as input and outputting specific actions in the action space. The role of the policy network is to guide the AGV to take certain actions in a certain state. The value network acts as a critic, taking the current state parameters in the state space as input and outputting the value of the current state. The role of the value network is to evaluate the goodness of a certain state and provide a reference for the optimization of the policy network. After the policy network and the value network are stable, a hypernetwork is introduced. The hypernetwork takes the state parameters recorded in the experience pool at multiple time steps as input and trains the output parameters of the main neural network. In this way, a secondary neural network is obtained, which is used to generate the initial parameters of the main neural network in a new state, thereby improving the generalization ability and training efficiency of the model.

[0082] In some embodiments, the policy network uses a multi-layer perceptron, a temporal network, or a policy network dedicated to the proximal policy optimization algorithm; the value network uses a multi-layer perceptron; and the hypernetwork is based on a recurrent neural network or a Transformer network.

[0083] In the reinforcement learning process, a reward mechanism is designed based on the delivery scenario, different stages in the sorting process are divided, and the reward value is dynamically adjusted according to whether the package is thrown out:

[0084] (1) When the package is not thrown out, introduce a package landing point prediction reward, an automated guided vehicle position offset reward, a conveyor belt acceleration reward, and a moving mechanism acceleration reward.

[0085] The parcel drop point prediction reward gives a reward according to the matching degree of the predicted parcel drop point and the target sorting port. The AGV position offset reward gives a reward according to the offset degree of the current AGV position and the ideal path. The conveyor belt acceleration reward gives a reward according to whether the acceleration of the conveyor belt is reasonable. The moving mechanism acceleration reward gives a reward according to whether the acceleration of the AGV moving mechanism is reasonable.

[0086] Specifically, the calculation formula of the parcel drop point prediction reward is:

[0087] ;

[0088] wherein k1 represents an accuracy reward coefficient of the predicted parcel drop point in the sorting bin when not thrown out; k2 represents an accuracy reward coefficient of the predicted parcel drop point outside the sorting bin when not thrown out; represents the distance between the predicted parcel drop point and the preset sorting port in the x-axis direction, represents the distance between the predicted parcel drop point and the preset sorting port in the y-axis direction; b represents a denominator offset.

[0089] wherein the calculation formula of the predicted parcel drop point is:

[0090] ;

[0091] wherein, represents the distance between the delivered parcel and the target sorting port along the x-axis, represents the distance between the delivered parcel and the target sorting port along the y-axis; represents the moving direction speed of the delivered parcel; represents the right wheel speed, represents the left wheel speed; h represents the height difference between the target sorting port and the delivered parcel, and g represents the gravitational acceleration; represents the included angle between the current advancing direction and the x-axis positive direction.

[0092] The calculation formula of the AGV position offset reward is:

[0093] ;

[0094] wherein k3 represents an AGV distance penalty coefficient; Δd x represents the distance between the AGV center and the target trajectory point along the x-axis direction, Δd y represents the distance between the AGV center and the target trajectory point along the y-axis direction; k4 represents an angle deviation penalty coefficient; and Δθ represents the angle deviation between the current advancing direction and the target trajectory direction.

[0095] The calculation formula of the conveyor belt acceleration reward is:

[0096] ;

[0097] wherein k5 represents a current conveyor speed mutation penalty coefficient; represents the conveyor speed at time t, a cmax represents the conveyor limit maximum acceleration.

[0098] The moving mechanism acceleration reward includes a left wheel acceleration reward and a right wheel acceleration reward, and the calculation formula of the left wheel acceleration reward is:

[0099] ;

[0100] wherein k6 represents a current left wheel speed mutation penalty coefficient; represents the left wheel speed at time t, a lmax represents the left wheel limit maximum acceleration.

[0101] The calculation formula of the right wheel acceleration reward is:

[0102] ;

[0103] wherein k7 represents a current right wheel speed mutation penalty coefficient; represents the right wheel speed at time t, a rmax represents the right wheel limit maximum acceleration.

[0104] The calculation formula of the reward value when the parcel is not thrown out is:

[0105] .

[0106] (2) When the parcel is thrown out, the parcel landing point prediction reward is changed into a parcel actual landing point reward, and the calculation formula is:

[0107] ;

[0108] wherein k8 represents a precision reward coefficient of the parcel prediction landing point in the sorting pocket when not thrown out; k9 represents a precision reward coefficient of the parcel prediction landing point outside the sorting pocket when not thrown out, and b represents a denominator bias.

[0109] Further, in each sorting round of the training process, the reward value is used to calculate the return (i.e. cumulative reward) and advantage value (i.e. the degree of advantage of the current action relative to the average strategy) of each time step.

[0110] When the policy network is updated, the advantage value is combined to optimize the policy network by limiting the change range of the probability ratio of the new and old policies (for example, using the PPO algorithm), so as to ensure the stability and convergence of the policy.

[0111] The value network is updated by minimizing the value prediction error (i.e., the difference between the predicted value and the actual return) to improve the accuracy of the state value.

[0112] In reinforcement learning, the PPO (Proximal Policy Optimization) algorithm optimizes the parameters of the policy network through a clipping-based policy update method. The core idea of PPO is to limit the magnitude of policy updates to ensure that each update does not cause the policy to change too much, thereby improving the stability and convergence of training. Specifically, PPO uses the importance sampling ratio , which is the ratio of the action probabilities of the current policy and the old policy in a certain state, to measure the magnitude of policy changes. To avoid excessive policy updates, PPO clips this ratio to limit its range to [1−ε,1+ε], where ε is a small positive number (such as 0.1 or 0.2). In this way, PPO ensures that policy updates are within a reasonable range, while using the advantage value A i as a guidance signal to help the policy improve in a better direction. Ultimately, PPO updates the parameters of the policy network by minimizing the clipped loss function, allowing the policy to gradually improve its performance while maintaining stability. This clipping-based update method not only improves the robustness of training, but also enables PPO to exhibit good convergence and sample efficiency in complex environments.

[0113] In some embodiments, the reward value is used to calculate the return and advantage value at each time step in each sorting round, and the calculation formula of the return is:

[0114] ;

[0115] The calculation formula of the advantage value is:

[0116] ;

[0117] ;

[0118] wherein, is the temporal difference error; represents the reward value at time step k; is the parameter of the generalized advantage estimation GAE; l is the summation variable, which traverses from 0 to the remaining time steps T-t, representing the offset of future multiple steps; γ is the discount factor, which controls the importance of future rewards in reinforcement learning.

[0119] In some embodiments, the policy network is updated by limiting the range of changes in the probability ratio of the new and old policies in combination with the advantage value, and the loss function calculation formula is:

[0120] ;

[0121] wherein, is the importance sampling ratio, is the policy under the loss function, is the clipping rate.

[0122] The policy network parameter update formula is:

[0123] .

[0124] The value network is updated by combining the return by minimizing the value prediction error, and the loss function calculation formula is:

[0125] ;

[0126] The value network parameter update formula is:

[0127] .

[0128] The main neural network parameters are trained by introducing a hypernetwork with the state parameters recorded in the experience pool at multiple time steps as input, and the loss function calculation formula is:

[0129] ;

[0130] wherein, represents the parameter set of the current main neural network; represents the main neural network parameter set predicted by the hypernetwork.

[0131] The parameters of the hypernetwork are updated by gradient descent, and the updated parameters are:

[0132] ;

[0133] wherein, , and are learning rates.

[0134] In another aspect, the application also provides an automatic guided vehicle non-stop sorting control method, which is used for controlling the automatic guided vehicle to deliver packages to a preset sorting port during driving, and the method comprises the following steps S201-S204:

[0135] Step S201: receiving a sorting task, the sorting task recording package information to be sorted, initial position of the package, and target position of the package.

[0136] Step S202: planning a target trajectory based on a preset path planning algorithm according to the current position of the automatic guided vehicle, the initial position of the package, and the target position of the package.

[0137] Step S203: Control the preset automatic guided transport vehicle to arrive at the initial location of the package according to the target trajectory to pick up the package to be sorted.

[0138] Step S204: Control the preset automatic guided vehicle to travel to the target location of the package according to the target trajectory, and after arriving at the sorting area, perform the delivery task based on the target sorting control model obtained by the above-mentioned automatic guided vehicle non-stop sorting control model training method.

[0139] This method involves four steps: first, receiving the sorting task and recording the initial and target locations of the package to be sorted; then, based on a preset path planning algorithm, planning the optimal driving route; then controlling the transport vehicle along the planned route to reach the initial location of the package for pickup; and finally, when the transport vehicle reaches the sorting area, utilizing the trained target sorting control model to complete the package delivery task. This non-stop sorting method can effectively reduce the downtime of the transport vehicle during the sorting process, improving overall logistics efficiency. Furthermore, through precise path planning and control models, it ensures that the package arrives at the designated location accurately and without error, reducing the sorting error rate. This method is applicable to various logistics scenarios, especially large-scale warehouses and distribution centers that require efficient sorting.

[0140] On the other hand, the present invention also provides a non-stop sorting control device for an automated guided vehicle, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0141] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when executed by a processor.

[0142] On the other hand, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0143] The present invention will be described below in conjunction with a specific embodiment:

[0144] In view of the deficiency that the AGV sorting robot only relies on the physical model and parameters for the horizontal throwing sorting of the package when not stopping, that is, the physical characteristics of different AGVs are different, the parameters of the physical model of the horizontal throwing motion need to be adjusted for each AGV to make the model work normally, and if mechanical wear occurs later, the parameters need to be corrected, a non-stop sorting control scheme for AGV is provided, the parameters of the main neural network are preliminarily determined through offline simulation, the physical model of the horizontal throwing motion is combined into the reward plasticity stage of the intermediate reward of reinforcement learning, and the training of reinforcement learning in this scene is accelerated. After completing the offline reinforcement learning training, the trained neural network is deployed in the actual running scene, and the same logic is used to fine-tune the real-time main neural network parameters. In addition, the model of the super network is added in offline simulation and actual training, the input network is the state, and the output is the parameter of the main neural network. The parameter of the auxiliary neural network is optimized by the state and the best parameter sample gradient of the main neural network, so that the auxiliary neural network can predict the main neural network parameter in the unknown state, and the convergence speed of the main neural network can be accelerated.

[0145] The embodiment uses the PPO algorithm in reinforcement learning to control the AGV non-stop sorting, the AGV vehicle carries the package to approach the package sorting grid, before passing through the sorting grid, a horizontal speed is started to the package, so that the AGV vehicle can throw the package into the sorting grid in the way of horizontal throwing motion without stopping. In the traditional non-stop sorting process, the starting time of the conveyor belt and the speed of the target package are found according to the physical model of the horizontal throwing motion. The embodiment uses reinforcement learning to solve the AGV non-stop sorting delivery without physical model, including the following steps:

[0146] According to the actual AGV non-stop sorting scene, a Markov decision process is established.

[0147] 1. State space, action space, reward mechanism, neural network structure design:

[0148] ①Firstly, the state space is constructed , wherein the state space variables are the distances of the automatic guided vehicle and the target sorting port along the x-axis and the y-axis , the deviations of the current position and the target trajectory point along the x-axis and the y-axis at the current time , the angle deviation of the current advancing direction and the target trajectory direction , the included angle between the current advancing direction and the x-axis positive direction , the distances of the delivered package and the target sorting port along the x-axis and the y-axis , the moving direction speed of the delivered package , counterclockwise is positive, in this model, because of the addition of speed mutation limit, the speed of the conveyor belt will not change greatly, and it is considered that the speed of the package is consistent with the speed of the conveyor belt. This state space can also support the scenario of dynamic movement of the sorting pocket, The relative speed can be converted.

[0149] ②Establish the action space The action space includes the speed of the AGV on the moving belt and the speed of the AGV moving mechanism. Taking a two-wheel differential AGV as an example, it is the speed of the left and right wheels.

[0150] ③Establish a reward mechanism. According to the model of projectile motion, the distance between the AGV and the sorting pocket after the package is thrown, the distance and angle difference of the car from the original route trajectory point during movement are rewarded. The reward calculation formula of the car when the package is not thrown is as follows:

[0151] When the package is not thrown, the package drop point prediction reward, the AGV position offset reward, the conveyor belt acceleration reward and the moving mechanism acceleration reward are introduced.

[0152] The package drop point prediction reward gives a reward according to the matching degree of the predicted package drop point and the target sorting pocket. The AGV position offset reward gives a reward according to the offset degree of the AGV current position and the ideal path. The conveyor belt acceleration reward gives a reward according to whether the acceleration of the conveyor belt is reasonable. The moving mechanism acceleration reward gives a reward according to whether the acceleration of the AGV moving mechanism is reasonable.

[0153] Specifically, the calculation formula of the package drop point prediction reward is:

[0154] ;

[0155] Wherein, k1 represents the accuracy reward coefficient of the package prediction drop point in the sorting pocket when not thrown; k2 represents the accuracy reward coefficient of the package prediction drop point outside the sorting pocket when not thrown; The distance between the predicted package drop point and the preset sorting pocket in the x-axis direction is represented by d1, and the distance between the predicted package drop point and the preset sorting pocket in the y-axis direction is represented by d2. The distance between the predicted package drop point and the preset sorting pocket in the x-axis direction is represented by d1, and the distance between the predicted package drop point and the preset sorting pocket in the y-axis direction is represented by d2.

[0156] Wherein, the calculation formula of the package prediction drop point is:

[0157] ;

[0158] Wherein, The distance between the delivered package and the target sorting pocket along the x-axis is represented by d1, and the distance between the delivered package and the target sorting pocket along the y-axis is represented by d2. The distance between the delivered package and the target sorting pocket along the x-axis is represented by d1, and the distance between the delivered package and the target sorting pocket along the y-axis is represented by d2. Vx represents the moving direction speed of the delivered package; Vr represents the right wheel speed, Vl represents the left wheel speed; h represents the height difference between the target sorting port and the delivered package, and g represents the gravity acceleration; θ represents the included angle between the current moving direction and the x-axis positive direction.

[0159] The calculation formula of the AGV position offset reward is as follows:

[0160] ;

[0161] wherein k3 represents the AGV distance punishment coefficient from the target trajectory point; Δd x represents the distance between the AGV center and the target trajectory point along the x-axis direction, Δd y represents the distance between the AGV center and the target trajectory point along the y-axis direction; k4 represents the angle deviation punishment coefficient; and Δθ represents the angle deviation between the current moving direction and the target trajectory direction.

[0162] The calculation formula of the conveyor belt acceleration reward is as follows:

[0163] ;

[0164] wherein k5 represents the current conveyor belt speed mutation punishment coefficient; represents the conveyor belt speed at t, and a cmax represents the maximum acceleration limit of the conveyor belt.

[0165] The moving mechanism acceleration reward includes the left wheel acceleration reward and the right wheel acceleration reward, and the calculation formula of the left wheel acceleration reward is as follows:

[0166] ;

[0167] wherein k6 represents the current left wheel speed mutation punishment coefficient; represents the left wheel speed at t, and a lmax represents the maximum acceleration limit of the left wheel.

[0168] The calculation formula of the right wheel acceleration reward is as follows:

[0169] ;

[0170] wherein k7 represents the current right wheel speed mutation punishment coefficient; represents the right wheel speed at t, and a rmax represents the maximum acceleration limit of the right wheel.

[0171] The calculation formula of the reward value when the package is not thrown out is as follows:

[0172] .

[0173] The reward of the dolly changes after the package is thrown out, and the obtained package landing point is the real landing point. The in the above reward changes, and the calculation formula is:

[0174] ;

[0175] wherein k8 represents the accuracy reward coefficient of the predicted landing point of the package in the sorting pocket when not thrown out; k9 represents the accuracy reward coefficient of the predicted landing point of the package outside the sorting pocket when not thrown out, and b represents the denominator bias.

[0176] The accuracy reward of the package landing point is changed, but the actual landing point of the package appears only once in a round, so this scenario belongs to the problem of sparse reward, and only the reward is given after the round is over. Therefore, the intermediate reward with a small reward coefficient is added to calculate the position of the predicted landing point according to the projectile motion, and the reward plasticity is performed. The trajectory stability reward is added in the middle process to keep the original trajectory in the sorting process; the stability reward of the conveyor belt and the left and right wheels , , When the speed of any of them changes suddenly, a negative reward value is given to reduce the speed mutation.

[0177] ④ Structure design of neural network: we set two neural networks, Critic and Actor, respectively. The Critic network is responsible for evaluating the reward of the current state and outputting, and the Actor network is responsible for calculating what action should be output under the current state. The input of the two networks is the state space. The input layers of the two neural networks are the same, the hidden layers and the output layers are different, the output of the Critic is the expected reward of the current state, and the output of the Actor strategy network is the action space, which is the speed of the conveyor belt and the left and right wheels. The speed of the three is in the range of continuous adjustment, and the Actor can generate three normal distribution parameters, respectively , standard deviation , the speed of the action space conforms to , the Actor generates parameters, and the normal distribution of the speed is obtained by random sampling, and the speed of the conveyor belt, the left wheel and the right wheel is obtained. The speed obtained by AGV execution is the action of the current time step .

[0178] ⑤Hypernetwork structure design: define a neural network, input is state space, input is the parameter of Actor and Critic network, its purpose is to predict the parameter of main neural network according to the input state through the neural network of Hypernetwork, and take the predicted parameter as the initial parameter of the main neural network, so as to reduce the speed of gradient update during training and accelerate the convergence of the main neural network.

[0179] 2. Single sorting round execution flow

[0180] Environment initialization: first, when the AGV enters the sorting straight line segment, the reinforcement learning process is started, the environment is initialized, and the initial state is obtained , the start state of the package .

[0181] Policy network outputs action, value network evaluates state value: the Actor network generates the speed of the conveyor belt according to the current state, and the action is , the Critic network generates the current predicted state value according to the current state.

[0182] Execute action: AGV executes the action output by the policy network .

[0183] Calculate reward: when the package has not been thrown out of the conveyor belt, because this scene is a round task, the reward value in the middle process , so the reward in the middle process is .

[0184] Save data: add the data of this time to the cache, where is the probability of the current policy selection , in this scene, the probability of the current policy is randomly selected from the generated normal distribution , which is used for importance probability ratio calculation in PPO algorithm.

[0185] Through time, and through environmental monitoring, the next state is obtained, and this is repeated until the package leaves the conveyor belt. The time step when the package leaves the conveyor belt is recorded as the final reward of this round . Finally, the data of all time steps stored are:

[0186]

[0187] 3. Return and generalized advantage estimation calculation of each time step in a single round After each round, the return of each time step can be calculated in reverse through all rewards :

[0188] ;

[0189] Then the advantage value of each time step is calculated :

[0190] ;

[0191] ;

[0192] wherein, is the time domain difference error; represents the reward value of time step k; is the parameter of generalized advantage estimation; l is the summation variable, which is from 0 to the remaining time step T-t, indicating the offset of future multiple steps; γ is the discount factor, which controls the importance of future rewards in reinforcement learning.

[0193] This scenario is a round-based reinforcement learning, and the PPO algorithm is used for Actor network and Critic network update. In order to reduce the sample variance, after sampling multiple round data, the network gradient update is randomly grouped, for example, multiple rounds of data collection and processing are performed to obtain M time step data, the data order is shuffled, and m1 batches are divided for policy update, each batch contains time step order, and each batch randomly takes B data from M for Actor network and Critic network update.

[0194] The policy network is updated by limiting the change range of the probability ratio of new and old policies combined with the advantage value, and the loss function calculation formula is:

[0195] ;

[0196] wherein, is the importance sampling ratio, is the loss function under the policy , and is the clipping rate.

[0197] Then the policy network parameter update formula is:

[0198] .

[0199] The value network is updated by minimizing the value prediction error combined with the return, and the loss function calculation formula is:

[0200] ;

[0201] Then the value network parameter update formula is:

[0202] .

[0203] Repeat steps 2-4 above until the variance of the reward value at the end of the k consecutive rounds is less than a set value The reinforcement learning result of the AGV not stopping for sorting is considered to have converged, and the neural network parameters have reached the optimal value, and the reinforcement learning is ended.

[0204] 5. The neural network parameter update of the super network:

[0205] When the main neural network, i.e. the Actor and Critic network, is stable, the learning process of the neural network of the super network is started, the corresponding table of the state and the parameter of the main neural network at multiple time steps after the main neural network is stable is recorded in the experience pool, and it is added to the training of the super network for calculation. The data stored in a stable round is: , and the corresponding two main neural network parameters are: , the parameter set of the current main neural network is denoted as , n is the number of hidden layers, and k is the number of network nodes of each layer. The initial strategy of the auxiliary neural network is denoted as , and the predicted neural network parameter set generated according to the strategy is The loss function is calculated by the difference between the predicted value and the actual value, and the loss function is calculated as follows:

[0206] ;

[0207] The parameters of the auxiliary neural network are updated by gradient descent, and the updated strategy is:

[0208] .

[0209] The neural network of the super network is updated once for each state, and the neural network is updated T times in a stable round. Repeat the above process until the variance of the m2 consecutive times is less than The auxiliary neural network is considered to have converged, and the predicted parameter value of the main neural network can be used to initialize the main neural network in the new state, and the gradient update of the parameters of the main neural network, i.e. the Actor and Critic network, can be accelerated.

[0210] Corresponding to the above method, the application also provides a device / system, which comprises a computer device including a processor and a memory, the memory storing computer instructions, and the processor is configured to execute the computer instructions stored in the memory, and the device / system implements the steps of the method as described above when the computer instructions are executed by the processor.

[0211] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the aforementioned edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0212] To sum up, the AGV non-stop sorting control model training, control method and device, by presetting the sensor to collect the state parameters of the AGV in real time to construct the state space, and based on the sorting delivery control elements to construct the action space. Then, the actor-critic network structure is constructed by using the policy network and the value network, wherein the policy network takes the current state parameter as the input and outputs the action, and the value network takes the current state parameter as the input and outputs the state value. In the sorting process, the reward value is dynamically adjusted according to whether the package is thrown out, when the package is not thrown out, the package drop point prediction reward, the AGV position offset reward, the conveyor belt acceleration reward and the moving mechanism acceleration reward are introduced, and when the package is thrown out, the reward value is calculated based on the actual drop point. Through the reinforcement learning algorithm, the return and the advantage value of each time step are calculated by using the reward value, the policy network is optimized by combining the advantage value to limit the change range of the probability ratio of the new and old policies, and the value network is optimized by minimizing the value prediction error. After the policy network and the value network are stable, the super network is introduced, the state parameters recorded in the experience pool are taken as the input to train the auxiliary neural network, so that the initial parameters of the main neural network are generated in the new state. The target sorting control model finally trained can realize efficient sorting without stopping, and significantly improve the sorting efficiency and accuracy.

[0213] Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or both. The particular implementation details are not regarded as limiting, for the specific implementation depends on the particular application and design constraints imposed on the overall design. Skilled artisans appreciate that each particular application can employ different methods to implement the described functions, although such application should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, the present application can be implemented as, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, and the like. When implemented in software, the elements of the present application are program or code segments to perform the required tasks. The programs or code segments can be stored in a machine readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or communication link.

[0214] It is to be expressly understood that the invention is not limited to the specific configurations and process described above and illustrated in the accompanying drawings. For the sake of clarity, detailed descriptions of known methods are omitted. In the above-described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and various changes, modifications and additions can be made thereto by one of ordinary skill in the art without departing from the spirit of the present invention, and the order of the steps can be changed.

[0215] In the present invention, features described and / or illustrated with respect to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0216] The above description is merely illustrative of the application, and is not intended to limit the scope of the application. Various modifications and changes can be made by one of ordinary skill in the art without departing from the spirit and scope of the application. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the application should be included in the scope of the application.

Claims

1. A training method for a non-stop sorting control model of an automated guided vehicle, characterized in that: The method is used to train a control model to control an automated guided vehicle to deliver packages to a preset sorting port during driving, and the method includes the following steps: In a preset scenario, the automated guided vehicle performs sorting and delivery, and constructs a state space based on real-time state parameters collected by preset sensors, and constructs an action space based on sorting and delivery control elements; An initial policy network and a value network are obtained to construct a main neural network of an actor-critic network structure, wherein the policy network is configured to take the current state parameters of the state space as input and output actions in the action space; the value network takes the current state parameters of the state space as input and outputs the value of the current state; in a sorting round, when a package is not thrown, a reward value is constructed by introducing a package landing point prediction reward, an automated guided vehicle position offset reward, a conveyor belt acceleration reward, and a mobile mechanism acceleration reward; when a package is thrown, the reward value is established based on the package landing point; the policy network and the value network are trained based on reinforcement learning, and in each sorting round, the reward value is used to calculate the reward and advantage value of each time step, and the policy network is updated by limiting the range of variation of the probability ratio of the new and old strategies in combination with the advantage value; the value network is updated by minimizing the value prediction error in combination with the reward; after the policy network and the value network are stable, a super network is introduced to take the state parameters of multiple time steps recorded in the experience pool as input to train and output the parameters of the main neural network, thereby obtaining a secondary neural network to preferentially generate the initial parameters of the main neural network under the new state; and the policy network obtained through multiple rounds of iteration is used as the target sorting control model.

2. The training method for the automatic guided vehicle non-stop sorting control model according to claim 1 is characterized in that: The state parameters include the distance between the automated guided transport vehicle and the target sorting port along the x-axis and y-axis, the deviation between the current position and the target trajectory point at the current moment along the x-axis and y-axis, the angular deviation between the current travel direction and the target trajectory direction, the angle between the current travel direction and the positive direction of the x-axis, the distance between the delivered package and the target sorting port along the x-axis and y-axis, and the moving direction speed of the delivered package; the action space includes the conveyor belt speed and the moving mechanism speed of the automated guided transport vehicle.

3. The training method for the automatic guided vehicle non-stop sorting control model according to claim 2 is characterized in that: The policy network adopts a multi-layer perceptron, a temporal network or a proximal policy optimization algorithm-specific policy network; the value network adopts a multi-layer perceptron; and the super network is constructed based on a recurrent neural network or a Transformer network.

4. The training method for the automatic guided vehicle non-stop sorting control model according to claim 3 is characterized in that: In a sorting round, when the package is not thrown, the calculation formula for the package landing point prediction reward is: ； Among them, k1 represents the accuracy bonus coefficient for the predicted landing point of the package in the sorting grid when it is not thrown; k2 represents the accuracy bonus coefficient for the predicted landing point of the package outside the sorting grid when it is not thrown; Indicates the distance between the predicted parcel landing point and the preset sorting port in the x-axis direction, represents the distance between the predicted parcel landing point and the preset sorting port in the y-axis direction; b represents the denominator offset; The calculation formula for the automatic guided vehicle position offset reward is: ； Wherein, k3 represents the penalty coefficient of the distance between the AGV and the target trajectory point; Δd x represents the distance between the center of the AGV and the target trajectory point along the x-axis, Δd y represents the distance between the center of the automated guided vehicle and the target trajectory point along the y-axis; k4 represents the angle deviation penalty coefficient; Δθ represents the angle deviation between the current travel direction and the target trajectory direction; The calculation formula for the conveyor belt acceleration reward is: ； Among them, k5 represents the penalty coefficient for the current conveyor belt speed mutation; represents the conveyor belt speed at time t, a cmax Indicates that the conveyor belt limits the maximum acceleration; The mobile mechanism acceleration bonus includes the left wheel acceleration bonus and the right wheel acceleration bonus. The calculation formula of the left wheel acceleration bonus is: ； Among them, k6 represents the penalty coefficient for the current left wheel speed mutation; represents the left wheel speed at time t, a lmax Indicates that the maximum acceleration of the left wheel is limited; The calculation formula for the right wheel acceleration reward is: ； Among them, k7 represents the penalty coefficient for the current right wheel speed mutation; represents the right wheel speed at time t, a rmax Indicates that the right wheel limits the maximum acceleration; The calculation formula for the reward value when the package is not thrown is: ； When the package is thrown, the predicted landing point reward is changed to the actual landing point reward, calculated as follows: ； Among them, k8 represents the accuracy bonus coefficient for the predicted landing point of the package in the sorting grid when it is not thrown; k9 represents the accuracy bonus coefficient for the predicted landing point of the package outside the sorting grid when it is not thrown, and b represents the denominator bias.

5. The training method for the automatic guided vehicle non-stop sorting control model according to claim 4 is characterized in that: The calculation formula for the predicted landing point of the package is: ； in, Indicates the distance between the delivered package and the target sorting port along the x-axis, Indicates the distance between the delivered package and the target sorting port along the y-axis; Indicates the moving direction speed of the delivered package; represents the right wheel speed, represents the left wheel speed; h represents the height difference between the target sorting port and the delivered package; g represents the acceleration due to gravity; Indicates the angle between the current direction of travel and the positive x-axis.

6. The training method for the automatic guided vehicle non-stop sorting control model according to claim 5 is characterized in that: In each sorting round, the reward value is used to calculate the return and advantage value of each time step. The calculation formula of the return is: ； The calculation formula of the advantage value is: ； ； in, is the time domain differential error; represents the reward value at time step k; is the parameter of the generalized advantage estimate; l is the summation variable, which traverses from 0 to the remaining time step Tt and represents the offset of multiple steps in the future; γ is the discount factor, which controls the importance of future rewards in reinforcement learning.

7. The training method for the automatic guided vehicle non-stop sorting control model according to claim 6 is characterized in that: In combination with the advantage value, the policy network is updated by limiting the range of variation of the probability ratio of the new and old policies. The loss function is calculated as follows: ； in, is the importance sampling ratio, For strategy The loss function under is the cropping rate; A i is the advantage value, clip represents the clipping function, and the limit The range of variation is between [1−ε,1+ε]; Then the strategy network parameter update formula is: ； The value network is updated by minimizing the value prediction error in combination with the reward. The loss function is calculated as: ； Then the value network parameter update formula is: ； The hypernetwork is introduced to train and output the main neural network parameters using the state parameters of multiple time steps recorded in the experience pool as input. The loss function is calculated as follows: ； in, Represents the parameter set of the current main neural network; represents the set of main neural network parameters predicted by the hypernetwork; The parameters of the hypernetwork are updated by gradient descent, and the updated parameters are: ； in, 、 and is the learning rate.

8. A non-stop sorting control method for an automated guided vehicle, characterized in that: The method is used to control an automated guided vehicle to deliver packages to a preset sorting port during driving, and the method includes the following steps: Receive a sorting task, wherein the sorting task records information of a package to be sorted, an initial location of the package, and a target location of the package; Based on a preset path planning algorithm, a target trajectory is planned according to the current position of the automated guided transport vehicle, the initial position of the package, and the target position of the package; Controlling the preset automatic guided transport vehicle to arrive at the initial location of the package along the target trajectory to pick up the package to be sorted; The automatic guided transport vehicle is controlled to travel toward the package target location according to the target trajectory, and after arriving at the sorting area, the delivery task is performed based on the target sorting control model obtained by the automatic guided transport vehicle non-stop sorting control model training method according to any one of claims 1 to 7.

9. A non-stop sorting control device for an automated guided vehicle, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images