A local distribution area logistics vehicle distributed cooperative motion planning method and device

Abstract

This application discloses a distributed cooperative motion planning method and device for logistics vehicles in a local distribution area, relating to the field of intelligent logistics technology. The method includes constructing a kinematic model of the logistics vehicles, modeling the multi-vehicle cooperative motion planning problem as a distributed partially observable Markov decision process, and constructing a MATD3 network model; inputting the local observation information at the current moment into the corresponding independent policy network model and outputting the original action vector; calculating the safety situation assessment index of the logistics vehicles in real time, allocating the passage priority based on the first-come-first-served rule, inversely solving the safety action boundary, and using a heuristic action masking mechanism to trim the safety boundary of the original action vector to obtain the final control action vector; generating control action commands and sending them to the underlying controller. This application can realize the autonomous cooperative passage of multiple logistics vehicles under the condition of no central node scheduling, and improve the robustness and reliability of the system in the scenario of communication interruption or node failure.

Description

Technical Field

[0001] This application relates to the field of intelligent logistics technology, and in particular to a method and apparatus for planning distributed collaborative motion of logistics vehicles in local distribution areas. Background Technology

[0002] With the rapid development of smart warehousing, logistics sorting, and hub connections, the collaborative motion planning of multiple logistics vehicles within local distribution areas has become a key element in achieving efficient, safe, and conflict-free operations. Local distribution areas are typically characterized by narrow lanes, frequent intersections, and high traffic density, placing high demands on the real-time performance, safety, and robustness of multi-vehicle collaborative decision-making.

[0003] Currently, multi-vehicle cooperative motion planning mainly employs rule-based scheduling methods, centralized optimization methods, and methods based on traditional multi-agent reinforcement learning. However, existing technologies have the following drawbacks: (i) Rule-based scheduling methods rely solely on simple rules to complete scheduling, which can easily lead to resource contention, mutual waiting, deadlock, and inefficiency when vehicles are densely intersecting; (ii) The computational complexity of centralized optimization methods increases sharply with the number of vehicles, making it difficult to meet real-time decision-making requirements. Furthermore, it is highly dependent on central nodes and global communication, and is easily affected by communication delays, network packet loss, and equipment failures in local clusters. Once an anomaly occurs, the overall scheduling will fail, and the robustness and reliability of the system are difficult to guarantee; (iii) Methods based on traditional multi-agent reinforcement learning typically employ a single experience replay pool and uniform sampling strategy, mixing high-value and low-value samples for storage and sampling. During training, the effective learning signal is diluted by a large number of low-value samples, resulting in slow model convergence. At the same time, most of the above methods do not directly embed safety constraints into the decision-making process, making it difficult to avoid collision and deadlock risks from the source in complex intersection scenarios. Furthermore, there is a disconnect between high-level collaborative decision-making and low-level motion control, which may lead to deviations in the execution of decision-making commands at the vehicle level, affecting operational stability. Summary of the Invention

[0004] The purpose of this application is to provide a distributed collaborative motion planning method and device for logistics vehicles in local distribution areas, which can realize the autonomous collaborative passage of multiple logistics vehicles under the condition of no central node scheduling, and improve the robustness and reliability of the system in the scenario of communication interruption or node failure.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a distributed cooperative motion planning method for logistics vehicles in a local distribution area, including: Acquire local distribution area scene information and local observation information of multiple logistics vehicles.

[0006] Based on the local distribution area scene information, a kinematic model of logistics vehicles under the local distribution area operation scenario is constructed.

[0007] Based on the local observation information, a joint state space is constructed, and the multi-vehicle cooperative motion planning problem is modeled as a distributed partially observable Markov decision process, thus constructing the MATD3 network model. The MATD3 network model includes an independent policy network model corresponding to each logistics vehicle and a dual critic network model for centralized evaluation. The independent policy network model is obtained through centralized training of the dual critic network model for value evaluation.

[0008] The local observation information of multiple logistics vehicles at the current moment is input into the corresponding independent policy network model, and the corresponding original action vector is output.

[0009] The system calculates the safety status assessment index of each logistics vehicle and its neighboring logistics vehicles in real time, allocates passage priority based on the first-come-first-served rule, calculates the safety action boundary in reverse by combining the kinematic model of the logistics vehicle, and uses a heuristic rule action masking mechanism to trim the safety boundary of the original action vector to obtain the final control action vector.

[0010] Based on the final control motion vector, control motion commands are generated and sent to the underlying controller to perform longitudinal speed and lateral trajectory tracking control on each logistics vehicle, thereby realizing distributed collaborative motion planning for multiple logistics vehicles.

[0011] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the distributed cooperative motion planning method for local distribution area logistics vehicles described above.

[0012] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the distributed cooperative motion planning method for logistics vehicles in local distribution areas described above.

[0013] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the distributed cooperative motion planning method for logistics vehicles in local distribution areas described above.

[0014] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a distributed cooperative motion planning method and device for logistics vehicles in a localized distribution area. First, a kinematic model of the logistics vehicles in a localized distribution area operation scenario is constructed, and the multi-vehicle cooperative motion planning problem is modeled as a distributed partially observable Markov decision process (MATD3) to obtain the MATD3 network model. This ensures that the decision-making and motion control of each logistics vehicle are based on a unified physical model and mathematical framework, providing a foundation for subsequent distributed cooperative planning. Second, the MATD3 network model includes an independent policy network model corresponding to each logistics vehicle and a dual-commentator network model for centralized evaluation. The local observation information of multiple logistics vehicles at the current moment is input into the corresponding independent policy network model, and the corresponding original action vectors are output. This solves the problem of existing centralized methods being highly dependent on a central node and global communication, enabling autonomous decision-making by each logistics vehicle under conditions without a central node, avoiding the risk of single-point failure, and improving system performance. The robustness and reliability of the system are improved. Next, by real-time calculation of safety situation assessment indicators for each logistics vehicle and neighboring logistics vehicles, passage priority is allocated based on the first-come, first-served rule. The safety action boundary is then calculated inversely using the kinematic model, and a heuristic rule-based action masking mechanism is employed to trim the original action vector to obtain the final control action vector. This solves the problems of missing safety constraints and inability to avoid collisions and deadlocks at the source in existing methods, achieving a deep integration of safety constraints and collaborative decision-making. In multi-vehicle intersection scenarios, collision risks are effectively avoided, improving the safety of collaborative passage. Finally, control action commands are generated based on the final control action vector and sent to the underlying controller for longitudinal speed and lateral trajectory tracking control of each logistics vehicle. This solves the problem of disconnect between high-level decision-making and low-level motion control in existing methods, ensuring the executability of decision commands under vehicle physical constraints and improving the smoothness and operational stability of collaborative logistics vehicle passage. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an application environment diagram of a distributed cooperative motion planning method for logistics vehicles in a local distribution area according to an embodiment of this application; Figure 2 A flowchart illustrating a distributed cooperative motion planning method for logistics vehicles in a local distribution area, provided as an embodiment of this application; Figure 3 A flowchart illustrating a distributed cooperative motion planning method for logistics vehicles in a local distribution area, provided in one embodiment of this application; Figure 4 This is a schematic diagram of the kinematic model of a logistics vehicle provided in one embodiment of this application; Figure 5 This is a diagram of a multi-agent MATD3 network architecture provided in an embodiment of this application; Figure 6 This is a flowchart illustrating the execution of a heuristic action masking algorithm provided in one embodiment of this application. Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The distributed cooperative motion planning method for logistics vehicles in local distribution areas provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can send local distribution area scene information and local observation information of multiple logistics vehicles to server 104. After receiving the local distribution area scene information and the local observation information of multiple logistics vehicles, server 104 constructs a kinematic model of the logistics vehicles under the local distribution area operation scenario based on the local distribution area scene information; it constructs a joint state space based on the local observation information, models the multi-vehicle cooperative motion planning problem as a distributed partially observable Markov decision process, and constructs a MATD3 network model; the MATD3 network model includes an independent policy network model corresponding to each logistics vehicle and a dual-critic network model for centralized evaluation. The independent policy network model is connected via the dual-critic network model. The value assessment is obtained through centralized training of the critic network model; the local observation information of multiple logistics vehicles at the current moment is input into the corresponding independent policy network model, and the corresponding original action vectors are output; the safety situation assessment index of each logistics vehicle and its neighboring logistics vehicles is calculated in real time, the passage priority is allocated based on the first-come-first-served rule, the safety action boundary is calculated inversely based on the kinematic model of the logistics vehicle, and the safety boundary is pruned by the action masking mechanism of heuristic rules to obtain the final control action vector; control action commands are generated based on the final control action vector and sent to the underlying controller to perform longitudinal speed and lateral trajectory tracking control of each logistics vehicle, realizing distributed cooperative motion planning of multiple logistics vehicles. The server 104 can feed back the obtained control action commands to the terminal 102. Furthermore, in some embodiments, the distributed cooperative motion planning method for logistics vehicles in local distribution areas can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform distributed cooperative motion planning processing for logistics vehicles in local distribution areas based on the local distribution area scene information and the local observation information of multiple logistics vehicles. Alternatively, the server 104 can obtain the local distribution area scene information and the local observation information of multiple logistics vehicles from the data storage system, and perform distributed cooperative motion planning processing for logistics vehicles in local distribution areas based on the local distribution area scene information and the local observation information of multiple logistics vehicles.

[0020] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0021] In one exemplary embodiment, such as Figure 2 and Figure 3 As shown, a distributed cooperative motion planning method for logistics vehicles in a local distribution area is provided. This method mainly comprises five core modules: a distributed state observation module, a MATD3 intelligent agent decision-making module, a heuristic action masking module, a hierarchical weighted sampling module, and a low-level motion control module. This method is executed by a computer device, specifically by a terminal or server alone, or by both. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps A1 to A6. Wherein: Step A1: Obtain scene information of the local distribution area and local observation information of multiple logistics vehicles.

[0022] As an optional implementation, the local observation information includes the position, speed, and heading of each logistics vehicle, as well as the relative position, relative speed, and relative heading of neighboring logistics vehicles within the sensing range.

[0023] Step A2: Based on the local distribution area scene information, construct a kinematic model of logistics vehicles under the local distribution area operation scenario.

[0024] Step A3: Based on the local observation information, a joint state space is constructed, and the multi-vehicle cooperative motion planning problem is modeled as a distributed partially observable Markov decision process, thus constructing the MATD3 network model. The MATD3 network model includes an independent policy network model corresponding to each logistics vehicle and a dual critic network model for centralized evaluation. The independent policy network model is obtained through centralized training of the dual critic network model for value evaluation.

[0025] Step A4: Input the local observation information of the multiple logistics vehicles at the current moment into the corresponding independent policy network model, and output the corresponding original action vector.

[0026] Step A5: Calculate the safety status assessment index of each logistics vehicle and its neighboring logistics vehicles in real time, allocate passage priority based on the first-come-first-served rule, reverse-calculate the safety action boundary using the kinematic model of the logistics vehicle, and use a heuristic rule-based action masking mechanism to trim the safety boundary of the original action vector to obtain the final control action vector.

[0027] As an optional implementation, the safety situation assessment indicators include headway and collision time.

[0028] Step A6: Based on the final control action vector, control action commands are generated and sent to the underlying controller to perform longitudinal speed and lateral trajectory tracking control on each logistics vehicle, thereby realizing distributed collaborative motion planning for multiple logistics vehicles.

[0029] As an optional implementation, the underlying controller includes a longitudinal controller and a lateral controller; the longitudinal controller uses a PID controller to track and control the target speed, and the lateral controller uses a PD controller to track and control the target heading angle and the target lateral position.

[0030] Implementing steps A1 to A6 above, firstly, the embodiments of this application possess excellent scene adaptability and robustness, directly adapting to local gathering and dispersing areas and unsignalized intersections with different traffic flow densities and vehicle numbers. No reconstruction of the core network structure is required; simply adjusting the safety threshold parameters and priority rules allows for rapid adaptation to changing traffic conditions, overcoming the technical shortcomings of traditional fixed control strategies' limited adaptability to a single scenario and the instability of conventional reinforcement learning algorithms in complex dynamic environments. Secondly, by constructing the MATD3 network model and employing a dual-critic network structure for global value estimation, the traditional D... The Q-value overestimation problem commonly found in DPG and single-commentator multi-agent algorithms can be effectively avoided during training to prevent oscillations and divergences, thus improving the stability and reliability of multi-agent collaborative decision-making. Finally, this application introduces a heuristic action masking mechanism, which dynamically maintains the agent access priority list in conjunction with a first-come, first-served rule, and performs real-time risk verification through a safety threshold. This solves the problem of the passive protection mode of existing technologies that rely on post-event collision avoidance. It directly filters and constrains dangerous actions during the decision generation stage, eliminating collision and deadlock risks from the source of the algorithm, and ensuring access safety while taking into account overall access efficiency.

[0031] In one exemplary embodiment, this application first constructs a scenario model consistent with the real operating environment based on the actual road structure, lane distribution, intersection area range, and vehicle driving constraints of the local distribution area. This scenario uses an unsignalized intersection as the core conflict interaction area, where multiple unmanned logistics vehicles complete dynamic interaction behaviors such as merging, yielding, passing, and turning. During the scenario construction process, the geometric constraints, speed constraints, steering constraints, and safe distance constraints of vehicle driving are preserved to ensure that the simulation environment maintains consistency with the real physical environment in terms of dynamic characteristics.

[0032] Secondly, to ensure that the actions output by the strategy conform to the actual motion laws of vehicles, the embodiments of this application use a kinematic model of a logistics vehicle for dynamic description, such as... Figure 4 As shown, its expression is: .

[0033] in, , Representing the vehicle's global x-coordinate Global vertical coordinates of the vehicle The first derivative with respect to time Indicates the longitudinal speed of the vehicle. Indicates the vehicle's heading angle. Indicates the longitudinal acceleration of the vehicle. This indicates the rate of change of the vehicle's heading angle. For the longitudinal acceleration of the vehicle, This refers to the front wheel steering angle. This refers to the vehicle's wheelbase. It is the sideslip angle. With front wheel steering angle The relationship between them is as follows: .

[0034] in, This indicates the distance from the vehicle's center of gravity to the front axle. This indicates the distance from the center of mass to the rear axle.

[0035] At the vehicle's lowest control layer, the controller is used to track the target speed and lateral angle.

[0036] For longitudinal control, a proportional controller is used to track the vehicle's longitudinal acceleration. .

[0037] in, Represents the target speed of the vehicle. These are the acceleration proportional control parameters.

[0038] The lateral controller uses a proportional-derivative controller, as shown in the following equation: ; .

[0039] in, Indicates the lateral speed of the vehicle. Indicates the lateral position of the vehicle target. This indicates the lateral velocity proportional control parameter. This indicates the change in heading angle when the lateral speed command is applied. Simultaneously, the vehicle's lateral angle control is shown in the following formula: ; ; .

[0040] in, Indicates the target heading angle, used for the heading and position of the lane being followed; This represents the vehicle's desired heading angle determined based on the centerline of the target lane. Indicates the target heading angle The first derivative with respect to time, i.e., the expected yaw rate. The proportional parameter represents the yaw rate. Through the complete kinematics and underlying control model described above, it can be ensured that the control commands output by the policy network are executable on the actual logistics vehicle, and that no invalid actions exceed the physical boundaries.

[0041] In another exemplary embodiment of this application, each unmanned logistics vehicle acts as a distributed control intelligent agent, not relying on a global central node, but acquiring local environmental information solely through onboard sensors and vehicle-road cooperative roadside units. At the decision-making moment... , No. The logistics vehicle constructs a local observation vector based on its own sensing information. : .

[0042] in, Indicates the first The two-dimensional spatiotemporal center coordinates of the logistics vehicle. Indicates the first The current longitudinal speed of the logistics vehicle in Taiwan Indicates the first The current heading angle of the logistics vehicle in Taiwan This indicates the proximity of unmanned logistics vehicles within the local sensing range. The set of relative states is expressed as: .

[0043] in, ,and Indicates the relationship with the first There is a cluster of neighboring logistics vehicles in Taiwan that may have potential spatial and temporal trajectory conflicts.

[0044] In this embodiment, the local observation vectors of each distributed agent are integrated to form the joint state of the multi-agent decision-making system at the current moment. , is represented as: .

[0045] in, The joint state of the multi-agent decision-making system at the current moment is represented by the local observation vectors of all logistics vehicles. , This indicates the number of intelligent agents in the logistics vehicle.

[0046] Joint state It can fully characterize the current global environment and provide a basis for value assessment in the centralized training phase, while not affecting the independent decision-making of each logistics vehicle based on its own local observations in the distributed execution phase.

[0047] This application's embodiments model the multi-vehicle cooperative motion planning problem as a distributed partially observable Markov decision process, using standard tuple form. This expression ensures the entire decision-making process conforms to the reinforcement learning theoretical framework, providing a unified and standardized model foundation for subsequent policy iterations and network updates. This represents a collection of intelligent entities for logistics vehicles. Represents the global state set. This represents the local observation space of each logistics vehicle. Represents the action space, This represents the state transition probability function. This represents the global reward function. Represents the observation probability function, Indicates the discount factor. .

[0048] Specifically, the set of intelligent agents is represented as ,in This represents the number of intelligent agents on the logistics vehicle. Local observation vector. It is a local observation space One specific observation example, namely Joint state It is the global state space A specific state instance, namely At each decision-making moment, the first... The logistics vehicle is based on its local observation vector Select and execute an action ,in The actions of all logistics vehicles together constitute a coordinated action. : .

[0049] The system is based on the state transition probability function. From the current joint state Transition to the next joint state .in, , Indicates the current state of union Execute joint actions Then, the system transitions to the joint state at the next time step. The probability of each logistics vehicle being observed via the probability function. Obtain local observations corresponding to the global state and share the global reward function. The optimization objective of this application's embodiments is to maximize the cumulative expected reward through the adopted joint strategy, namely: .

[0050] in, , , , They represent the first The joint state, joint actions, global reward obtained, and corresponding discount factor at each decision moment; It represents the mathematical expectation.

[0051] In practical implementation, MATD3 is adopted as the core multi-agent algorithm. The overall network architecture consists of a distributed policy network and a centralized dual-commenter network, such as... Figure 5 As shown, it can effectively suppress the problem of value overestimation and is more suitable for continuous motion control scenarios.

[0052] Each unmanned logistics vehicle is equipped with an independent strategy network. The network takes the vehicle's own local observation vector as input and directly outputs the corresponding continuous control actions, enabling each agent to achieve fully distributed autonomous decision-making during the execution phase without relying on global communication.

[0053] During the training phase, the system sets up two centralized critic networks with identical structures but independent parameters. and Using the global joint state and joint actions as input, the system jointly evaluates the overall decision-making effect of the multi-agent system, and improves the evaluation accuracy through the complementary use of a dual-critic network model.

[0054] In practical implementation, considering the actual control requirements of logistics vehicles, this application's embodiment defines a continuous motion space. The motion commands output by the strategy network include longitudinal acceleration and lateral steering angular velocity. The original motion vector can be represented as follows: : .

[0055] Continuous motion output can avoid the jitter and abrupt changes caused by discrete motion, making the vehicle drive more smoothly. Subsequently, the original motion vector will be subjected to safety constraints in conjunction with the heuristic motion pruning mechanism described later, so as to obtain the final control motion that meets the safety constraint conditions.

[0056] In another exemplary embodiment of this application, in order to perform real-time security situation assessment and boundary constraints based on passage priority on the original action vector output by the policy network, thereby avoiding collision and deadlock risks at the decision source, the above step A5 is replaced by the following steps B1 to B5: Step B1: Based on the current motion state of each logistics vehicle and the relative motion state of neighboring logistics vehicles within the sensing range, calculate in real time the headway and collision time between each logistics vehicle and the neighboring logistics vehicle.

[0057] Step B2: Based on the time order in which each logistics vehicle enters the conflict area, dynamically maintain the passage priority list according to the first-come-first-served rule, and determine the passage priority order of each logistics vehicle; wherein, the logistics vehicle that enters the conflict area first has a high priority, and the logistics vehicle that enters the conflict area later has a low priority.

[0058] Step B3: When the headway of the low-priority logistics vehicle is less than a preset safe headway threshold or the collision time is less than a preset safe collision time threshold, it is determined that the logistics vehicle has a potential collision risk.

[0059] Step B4: Based on the logistics vehicle with potential collision risk and its low priority, and combined with the kinematic model of the logistics vehicle, the safe action boundary allowed at the current time step is calculated in reverse according to the relative distance, relative speed and heading angle deviation between the logistics vehicle and the logistics vehicle with high priority.

[0060] Step B5: Based on the action masking mechanism of the safety action boundary and heuristic rules, the original action vector is mapped and pruned using a differentiable truncation function to obtain the final control action vector.

[0061] When unmanned logistics vehicles Upon entering the conflict control area, real-time safety and priority checks are performed, and the vehicle dynamically calculates potential conflicts with other logistics vehicles on the same intersecting path. Headway between It is used to determine the risk and safe following distance of a vehicle, and the calculation formula is: .

[0062] In the formula The relative distance between the two vehicles. This represents the vehicle's current speed.

[0063] Meanwhile, the embodiments of this application calculate and predict collision time in real time. This is used to further determine the degree of collision risk between vehicles, and the expression is: .

[0064] in, The heading angle deviation between the two vehicles reflects the degree of convergence of their driving directions.

[0065] This application employs a first-come, first-served rule to allocate vehicle passage priorities. A priority list is dynamically maintained based on the order in which vehicles enter the conflict zone, and the status of vehicles entering, being in, and leaving the interaction zone is updated in real time to determine vehicle passage permissions and avoidance relationships. When an unmanned logistics vehicle enters the conflict zone, it is added to the priority list; when it leaves the interaction zone, it is removed from the priority list. Vehicles with higher priority have priority passage rights, while vehicles with lower priority execute hold, deceleration, or stopping and waiting strategies based on subsequent safety status assessments.

[0066] The system introduces a preset safe time interval threshold. Collision time threshold with preset safety The original action vector output by the policy network is subjected to safety state verification. When the vehicle meets the safety constraints, the current action output by the policy network is maintained; when the headway or collision time is lower than a preset safety threshold, the safety status is verified. or Initiate heuristic action clipping constraints, such as Figure 6 As shown, safety restrictions are applied to the original actions output by the policy network, and the vehicle adopts a deceleration strategy. In this embodiment of the application, the following is taken: , .

[0067] To ensure that the action constraints are differentiable without disrupting the policy network training, this application employs a differentiable truncation function to perform boundary clipping on the original actions for unmanned logistics vehicles. The original longitudinal control action output by the Actor policy network and lateral control movements After mapping and pruning, the legal continuous motion that finally meets the physical safety red line is obtained. The mapping truncation rules for longitudinal acceleration and lateral steering angular velocity are as follows: ; .

[0068] in, and These are the minimum longitudinal acceleration and minimum lateral steering angular velocity allowed under the physical limits of the entire vehicle, respectively. Indicates the final longitudinal acceleration; This indicates the final lateral steering angular velocity; This represents the upper limit of the allowable longitudinal safe acceleration at the current time step, obtained by inverse calculation based on safety constraints, and is used to constrain the final longitudinal acceleration. Its expression is: in, The preset safety redundancy acceleration correction coefficient can eliminate dangerous acceleration actions from the source through the above-mentioned pruning, and restrict or correct the original actions that do not meet the safety constraints without changing the main structure of the Actor policy network, so as to avoid collisions and deadlocks.

[0069] The joint action obtained after heuristic action clipping can be represented as: ;in, , indicating the first The final control vector output by the strategy network for each logistics vehicle.

[0070] In another exemplary embodiment of this application, addressing the technical shortcomings of existing single-experience-pool uniform sampling, which leads to high-value samples being diluted by a large number of low-value, low-reward invalid samples, resulting in slow model convergence and unstable decision-making in critical conflict scenarios, this embodiment improves the experience management mechanism during the MATD3 centralized training phase. It constructs a dual-experience-pool hierarchical storage and hierarchical weighted sampling strategy based on round-cumulative rewards, and designs a weighted loss function and a delayed policy gradient update mechanism to improve the utilization efficiency of high-value samples, accelerate model convergence, and enhance decision stability. Therefore, the specific process of the above step "centralized training of the independent policy network model for value evaluation via a dual-critic network model" can be replaced by the following steps C1 to C3: Step C1: Construct a global composite reward function; the global composite reward function is obtained by weighted combination of traffic efficiency reward, safe distance penalty, and motion comfort penalty.

[0071] Step C2: Calculate the round cumulative reward based on the global composite reward function, and store the complete round interaction data into the first experience pool or the second experience pool respectively according to the comparison result of the round cumulative reward and the dynamic reward threshold; the dynamic reward threshold is the product of the maximum cumulative reward upper limit and the preset classification judgment coefficient, and the maximum cumulative reward upper limit is adaptively updated with the round cumulative reward.

[0072] Specifically, the complete round interaction data is stored in either the first experience pool or the second experience pool based on the comparison result between the round's cumulative reward and the dynamic reward threshold. Step C21: If the cumulative reward of the current round is greater than the dynamic reward threshold, then the current round is determined to be a high-value round, the interaction data of the current round is stored in the first experience pool, and the maximum cumulative reward upper limit is updated to the cumulative reward of the current round.

[0073] Step C22: If the cumulative reward of the current round is less than or equal to the dynamic reward threshold, then the current round is determined to be a low-value round, and the interaction data of the current round is stored in the second experience pool.

[0074] Step C3 involves performing hierarchical joint sampling based on the first and second experience pools, calculating the target value using the global composite reward function and the target critic network model corresponding to the dual critic network model, constructing a weighted loss function to update the dual critic network model, and updating the independent policy network model using a delayed policy gradient. Specifically, this includes the following steps: Step C31: Extract a first number of samples from the first experience pool and simultaneously extract a second number of samples from the second experience pool to form a training batch.

[0075] Step C32: For each sample in the training batch, calculate the current reward using the global composite reward function, and calculate the target value of the sample by combining the value estimate of the next joint state and the next joint action by the target critic network model; wherein the target value is the minimum value output by the target critic network model.

[0076] Step C33: Based on the target value and the value estimate of the current joint state and joint action by the dual critic network model, calculate the temporal difference error corresponding to each of the dual critic network models.

[0077] Step C34: For samples from the first experience pool, the time-series difference error is weighted using a first weighting coefficient. At the same time, for samples from the second experience pool, the time-series difference error is weighted using a second weighting coefficient to obtain the weighted time-series difference error.

[0078] Step C35: Construct a weighted loss function based on the weighted temporal difference error, and use the weighted loss function to update the model parameters of the dual critic network model.

[0079] Step C36: Based on the delayed update mechanism, the model parameters of the policy network model are updated using the policy gradient theorem at preset intervals.

[0080] In this application embodiment, a global composite reward function is constructed to guide multiple agents to simultaneously consider traffic efficiency, driving safety, and smoothness of motion. Global composite reward function It is obtained by weighted combination of three terms: .

[0081] in, This is a traffic efficiency incentive item; The safety distance penalty is set when the geometric distance between any two vehicles is... At that time, a very large negative penalty is imposed. Otherwise, it is 0. To minimize the safe distance, in the embodiments of this application... ; This is a penalty item for motion comfort. , and These are the weighted coefficients corresponding to traffic efficiency rewards, safe distance penalties, and movement comfort penalties, respectively.

[0082] To constrain the smoothness of motion output, the motion comfort penalty term is expressed as: .

[0083] By constraining the global composite reward function, the policy learning direction can be aligned with the actual operational objectives. In practice, when a logistics vehicle completes its current collaborative passage task, the global composite reward function adds a task completion reward. Traffic efficiency incentive items Action comfort penalty item Collision penalty value Weighting coefficients of each item , and All are set to 1.

[0084] In the offline or cloud-based centralized training phase, two centralized critic networks are employed to suppress overestimation of value assessments in the continuous action space. and Value updates are performed while the network structure remains unchanged; only parameter updates are executed. The combined state of all logistics vehicles is received. and the combined actions obtained after heuristic action clipping As input; when updating the value network, its target value The calculation strictly adopts the minimum value of the target dual-critic network output to suppress value overestimation. The expression is: .

[0085] in, , These represent the target critic network corresponding to the critic network. To add smooth exploration noise to the target critic network output, This indicates that the mean is 0 and the variance is 0. Gaussian distribution, Indicates the standard deviation of noise. This represents the noise clipping threshold. This represents the clipping function.

[0086] The temporal difference error for each of the two critic networks is calculated based on the target value, and this is used to measure the accuracy of the current value assessment. ; .

[0087] in, and They represent the critics' network. and In the The timing difference error corresponding to time step 1. and They represent the critics' network. and Regarding the current joint state With joint actions Value estimate.

[0088] The time-step data generated by the agent's interaction with the environment is first stored in the round's temporary cache. At the end of the current driving round and when all logistics vehicles have completely passed through the distribution conflict zone or a collision has occurred, the mean of the dual-commenter network average temporal difference error of all transition samples within that complete round is calculated. : .

[0089] in, This represents the total number of time steps in this round. This parameter is used to monitor the level of bias in the critic network's estimation of the value of the current batch of samples during training, serving as a reference metric for evaluating the network's convergence status.

[0090] Simultaneously calculate the cumulative reward for that round. This application embodiment sets a dynamically adaptively updated maximum cumulative reward cap. (Initially set to a minimum value), and set the classification decision coefficient. .

[0091] In practice, based on the cumulative reward of a round and the maximum cumulative reward limit for adaptive updates, samples of a complete round of interaction data are archived into the first experience pool (i.e., the high-reward experience pool). ) and the second experience pool (i.e., the basic experience pool) This allows for the classification and storage of high-value samples and ordinary samples. If the cumulative reward of the current round is greater than the dynamic reward threshold, then the condition is met. If the sample in that round is determined to be a high-value sample, all time step data in that round will be archived and stored in the high-reward experience pool. This triggers an adaptive iteration process for the maximum cumulative reward cap, i.e. If this condition is not met, i.e. Then archive it and store it in the basic experience pool. .

[0092] When extracting batches of data to train network parameters, a fixed high-value sampling ratio coefficient is used. Perform hierarchical joint sampling, that is, from the high-reward experience pool The number of samples drawn is The samples from the basic experience pool The number of samples drawn is The samples were merged to form a total size of Training in small batches.

[0093] This application's embodiment employs a joint weighted loss function for updating the parameters of a dual-commentator network. Perform backpropagation to update parameters , This allows high-reward samples to have higher weights during training, and the loss function is: in, , These represent the number of samples drawn from the high-reward experience pool and the basic experience pool, respectively. Indicates the first The corresponding sample of the th sample The system employs a delayed update mechanism, updating at preset intervals. (Note: The original text contains several typos and incomplete sentences, making a precise translation impossible. It appears to be discussing network timing errors and time-series differences.) The policy network parameters of each distributed agent are updated using the policy gradient theorem. This enables efficient convergence of continuous motion planning strategies.

[0094] In another exemplary embodiment of this application, the final continuous control action obtained after heuristic action trimming is sent to the vehicle's underlying controller, the control action including the final longitudinal acceleration. and final lateral steering angular velocity , and serve as target inputs for the vehicle's longitudinal controller and lateral controller, respectively. , , , These represent the minimum and maximum longitudinal accelerations allowed by the vehicle's physical properties, respectively. , These represent the minimum and maximum lateral steering angular velocities allowed by the vehicle's steering system, respectively. The upper and lower limits of these actions are preset based on the dynamic performance of the logistics vehicle and the physical parameters of the actuators to ensure that the control actions sent to the lower-level controller are all within the vehicle's executable range.

[0095] The lower-level controller receives the target control input from the upper-level decision-making process and uses a pre-established vehicle kinematics model and control law to complete the tracking control. Longitudinal control uses a proportional gain of... The PID controller, when the speed tracking error At this point, it is assumed that speed tracking has reached a steady state; the lateral control uses a proportional coefficient of... and The PD controller, when the heading tracking error And lateral position error At that time, the trajectory tracking is considered to have met the requirements. , , These represent the allowable thresholds for speed tracking error, heading angle tracking error, and lateral position tracking error, respectively. These error thresholds are predetermined based on the kinematic model of the logistics vehicle established above, the response performance of the vehicle actuators, and the experimental calibration results of the control system.

[0096] All unmanned logistics vehicles independently complete the entire process of perception, decision-making, and control, without the need for a central node for unified scheduling. Each vehicle operates only within communication range. Within the communication range, the vehicle exchanges status information of nearby vehicles through vehicle-to-infrastructure (V2I) communication; outside the communication range, it relies entirely on onboard sensors for autonomous observation. When the relative distance between two vehicles is less than... At that time, the system automatically triggers a dynamic priority adjustment mechanism to determine access permissions and avoidance relationships based on the first-come, first-served rule.

[0097] The system adopts a distributed closed-loop control architecture, with a decision cycle of [missing information]. The control cycle is Control the number of updates within a cycle. ,in, This ensures the real-time and continuous nature of control. Each vehicle operates at a frequency... The system updates its own motion status and local observation information in real time, and the decision-making unit dynamically adjusts the action constraints based on the latest information, forming a complete closed loop of "perception-decision-control-feedback".

[0098] When a vehicle meets safety constraints, it executes a decision-making action. When a potential conflict risk is detected, the heuristic action masking module constrains or modifies the original action and feeds the execution result back to the decision-making module. Each vehicle can complete autonomous perception, collaborative decision-making, and motion control without the need for unified scheduling by a central node, enabling distributed collaborative passage of multiple logistics vehicles within a local distribution area.

[0099] This application also provides an application scenario in which the aforementioned distributed collaborative motion planning method for logistics vehicles in local distribution areas is applied. Specifically, the distributed collaborative motion planning method for logistics vehicles in local distribution areas provided in this application can be applied to intelligent logistics park distribution scenarios. This scenario includes cargo loading and unloading, park road transportation, and hub connection and distribution. Unmanned logistics vehicles enter the park road transportation from the cargo loading and unloading stage, and through multi-vehicle collaborative passage and dynamic avoidance, complete interactive behaviors such as merging, turning, and passing, before entering the downstream hub connection and distribution stage. The distributed collaborative motion planning method for logistics vehicles in local distribution areas provided in this application belongs to the multi-vehicle collaborative decision-making and control stage in the park road transportation stage. Specifically, when unmanned logistics vehicles pass through unsignalized intersections or narrow intersection areas, motion planning for logistics vehicles can be performed based on a collaborative approach of distributed perception and decision-making, heuristic action mask safety constraints, and underlying motion control. That is, multiple logistics vehicles are assigned passage timing and speed trajectories to achieve autonomous collaborative passage without central node scheduling.

[0100] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores local distribution area scene information and local observation information of multiple logistics vehicles. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a distributed cooperative motion planning method for logistics vehicles in a local distribution area.

[0101] Figure 7 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0102] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0103] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0105] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0106] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A distributed cooperative motion planning method for logistics vehicles in a localized distribution area, characterized in that, The method includes: Acquire local distribution area scene information and local observation information of multiple logistics vehicles; Based on the local distribution area scene information, a kinematic model of logistics vehicles under the local distribution area operation scenario is constructed; Based on the local observation information, a joint state space is constructed, and the multi-vehicle cooperative motion planning problem is modeled as a distributed partially observable Markov decision process, thus constructing the MATD3 network model. The MATD3 network model includes an independent policy network model corresponding to each logistics vehicle and a dual critic network model for centralized evaluation. The independent policy network model is obtained through centralized training of the dual critic network model for value evaluation. The local observation information of multiple logistics vehicles at the current moment is input into the corresponding independent policy network model, and the corresponding original action vector is output. The system calculates the safety status assessment index of each logistics vehicle and its neighboring logistics vehicles in real time, allocates passage priority based on the first-come-first-served rule, reverse-calculates the safety action boundary using the kinematic model of the logistics vehicle, and uses a heuristic rule action masking mechanism to trim the safety boundary of the original action vector to obtain the final control action vector. Based on the final control motion vector, control motion commands are generated and sent to the underlying controller to perform longitudinal speed and lateral trajectory tracking control on each logistics vehicle, thereby realizing distributed collaborative motion planning for multiple logistics vehicles.

2. The distributed cooperative motion planning method for logistics vehicles in local distribution areas according to claim 1, characterized in that, The local observation information includes the position, speed, and heading of each logistics vehicle, as well as the relative position, relative speed, and relative heading of neighboring logistics vehicles within the sensing range.

3. The distributed cooperative motion planning method for logistics vehicles in local distribution areas according to claim 1, characterized in that, The independent policy network model is obtained through centralized training of the value assessment of the dual critic network model, specifically including: A global composite reward function is constructed; the global composite reward function is obtained by a weighted combination of a traffic efficiency reward term, a safe distance penalty term, and a motion comfort penalty term. The round cumulative reward is calculated based on the global composite reward function. The complete round interaction data is then stored in the first experience pool or the second experience pool according to the comparison result between the round cumulative reward and the dynamic reward threshold. Hierarchical joint sampling is performed based on the first experience pool and the second experience pool. The target value is calculated using the global composite reward function and the target critic network model corresponding to the dual critic network model. A weighted loss function is constructed to update the dual critic network model, and the independent policy network model is updated using a delayed policy gradient.

4. The distributed cooperative motion planning method for logistics vehicles in local distribution areas according to claim 3, characterized in that, The dynamic reward threshold is the product of the maximum cumulative reward limit and the preset classification judgment coefficient, and the maximum cumulative reward limit is adaptively updated according to the cumulative reward of the round; The complete round interaction data is stored in either the first experience pool or the second experience pool based on the comparison result between the accumulated round reward and the dynamic reward threshold, specifically including: If the cumulative reward of the current round is greater than the dynamic reward threshold, the current round is determined to be a high-value round, the interaction data of the current round is stored in the first experience pool, and the maximum cumulative reward upper limit is updated to the cumulative reward of the current round. If the cumulative reward of the current round is less than or equal to the dynamic reward threshold, the current round is determined to be a low-value round, and the interaction data of the current round is stored in the second experience pool.

5. The distributed cooperative motion planning method for logistics vehicles in a local distribution area according to claim 3, characterized in that, Based on the first and second experience pools, hierarchical joint sampling is performed. The target value is calculated using the global composite reward function and the target critic network model corresponding to the dual critic network model. A weighted loss function is constructed to update the dual critic network model, and a delayed policy gradient is used to update the independent policy network model. Specifically, this includes: A first number of samples are drawn from the first experience pool, and a second number of samples are drawn from the second experience pool to form a training batch. For each sample in the training batch, the current reward is calculated using the global composite reward function, and the target value of the sample is calculated by combining the value estimate of the next joint state and the next joint action by the target critic network model; wherein, the target value is the minimum value output by the target critic network model; Based on the target value and the value estimate of the current joint state and joint action by the dual critic network model, calculate the temporal difference error corresponding to each of the dual critic network models; For samples from the first experience pool, the temporal difference error is weighted using a first weighting coefficient, and for samples from the second experience pool, the temporal difference error is weighted using a second weighting coefficient to obtain the weighted temporal difference error. A weighted loss function is constructed based on the weighted temporal difference error, and the model parameters of the dual critic network model are updated using the weighted loss function. Based on the delayed update mechanism, the model parameters of the policy network model are updated using the policy gradient theorem at preset time intervals.

6. The distributed cooperative motion planning method for logistics vehicles in local distribution areas according to claim 1, characterized in that, The safety situation assessment indicators include headway and collision time; The system calculates safety situation assessment indicators for each logistics vehicle and its neighboring vehicles in real time, allocates passage priorities based on a first-come, first-served rule, inversely calculates safety action boundaries using the kinematic model of the logistics vehicles, and employs a heuristic action masking mechanism to trim the original action vectors to obtain the final control action vectors, specifically including: Based on the current motion state of each logistics vehicle and the relative motion state of neighboring logistics vehicles within the sensing range, the headway and collision time between each logistics vehicle and the neighboring logistics vehicles are calculated in real time. Based on the time sequence of each logistics vehicle entering the conflict area, a passage priority list is dynamically maintained based on the first-come, first-served rule to determine the passage priority order of each logistics vehicle; wherein, the logistics vehicle that enters the conflict area first has a high priority, and the logistics vehicle that enters the conflict area later has a low priority. When the headway of the low-priority logistics vehicle is less than a preset safe headway threshold or the collision time is less than a preset safe collision time threshold, the logistics vehicle is determined to have a potential collision risk. Based on the logistics vehicle with potential collision risk and its low priority, and combined with the kinematic model of the logistics vehicle, the safe action boundary allowed at the current time step is calculated in reverse according to the relative distance, relative speed and heading angle deviation between the logistics vehicle and the logistics vehicle with high priority. Based on the action masking mechanism of the aforementioned safety action boundaries and heuristic rules, the original action vector is mapped and pruned using a differentiable truncation function to obtain the final control action vector.

7. The distributed cooperative motion planning method for logistics vehicles in local distribution areas according to claim 1, characterized in that, The underlying controller includes a longitudinal controller and a lateral controller; the longitudinal controller uses a PID controller to track and control the target speed, and the lateral controller uses a PD controller to track and control the target heading angle and the target lateral position.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the distributed cooperative motion planning method for local distribution area logistics vehicles as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the distributed cooperative motion planning method for logistics vehicles in local distribution areas as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the distributed cooperative motion planning method for logistics vehicles in local distribution areas as described in any one of claims 1-7.

Images