AGV global and local fusion navigation method and system based on LT-TD3 reinforcement learning

Abstract

The invention discloses an AGV global and local fusion navigation method and system based on LT-TD3 reinforcement learning, and the method comprises the steps: extracting environment features based on laser radar data through obstacle edge detection, channel recognition and open space detection, dynamically generating a navigation target point set, and selecting an optimal navigation point through a distance weighted evaluation strategy; a multi-modal input processing mechanism is adopted to fuse laser radar data, target position information and a historical action sequence, and a state vector is formed after processing of a convolutional neural network and a gating circulation unit; a state value network, a double-Q network and a strategy network are adopted for cooperative work, and in combination with Expectile regression, a mixed target updating mechanism and composite strategy target function optimization, continuous linear speed and angular speed control instructions are output; and collision detection and intelligent control right switching are combined to realize AGV intelligent safety navigation, so that the problem that traditional reinforcement learning is easy to fall into local optimum is solved, and the phenomenon of navigation instability caused by distribution offset is effectively relieved.

Description

Technical Field

[0001] This invention belongs to the field of automated guided vehicle (AGV) navigation technology, specifically relating to a global-local fusion navigation method and system for AGVs based on LT-TD3 reinforcement learning. Background Technology

[0002] Traditional AGV navigation technology primarily relies on pre-built, accurate environmental maps and employs graph-based path planning algorithms such as A*, Dijkstra's algorithm, and fast random tree algorithm. While map-based navigation methods can guarantee path optimization, their adaptability to environmental changes is extremely limited, requiring frequent map updates and maintenance, and they perform poorly in complex industrial environments with frequent dynamic obstacles. Reactive navigation methods, such as artificial potential field methods and dynamic window methods, offer good real-time performance and can achieve dynamic obstacle avoidance, but they generally suffer from local optima traps, especially when facing complex terrains such as U-shaped obstacles and narrow passages, where they are prone to oscillating behavior or failing to find an effective navigation path.

[0003] Deep reinforcement learning technology has shown application potential in the field of AGV navigation, and mainstream algorithms include deep learning. Q Network, near-end policy optimization, and dual-delay deep deterministic policy gradient, etc. However, existing reinforcement learning algorithms still have technical shortcomings in AGV navigation applications. The TD3 algorithm only uses... Q The function estimates state-action value but lacks explicit modeling of the state value function, leading to problems in areas outside the data distribution. Q The height estimation is inaccurate, and it tends to overestimate unseen state-action pairs. This distribution offset problem is particularly prominent in AGV navigation scenarios, where navigation performance often drops sharply when the robot faces a new environmental configuration it has not encountered during training.

[0004] Existing AGV navigation systems face core technological challenges in practical industrial applications. Balancing global optimality with local real-time performance is difficult; the hierarchical navigation architecture lacks an effective coordination mechanism between the global layer's long-term planning based on static maps and the local layer's dynamic obstacle avoidance based on real-time sensor data. Environmental adaptability and robustness are insufficient; retraining is required to maintain good performance when faced with dynamic environmental factors such as personnel movement and changes in cargo placement. Absolutely inadequate anomaly handling mechanisms lack effective diagnostic and recovery mechanisms for sensor malfunctions, accidental collisions, and other anomalies, severely impacting system reliability.

[0005] Chinese patent CN114326739A discloses a path planning method for mobile robots based on deep reinforcement learning. It employs a deep deterministic policy gradient algorithm to achieve path planning in a continuous action space and improves learning efficiency through reward function design and network structure optimization. However, this method only focuses on improving a single reinforcement learning algorithm and does not address the underlying issues. Q It suffers from overestimation and distribution offset problems, and lacks an effective mechanism for combining global path planning and local obstacle avoidance.

[0006] Therefore, it is urgent to improve traditional reinforcement learning algorithms, develop reinforcement learning methods that can effectively solve the distribution offset problem, construct a global-local hybrid navigation architecture to achieve this, and establish a sound recovery mechanism to adapt to the ever-changing industrial environment and the ever-increasing reliability requirements. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a global-local fusion navigation method and system for AGVs based on LT-TD3 reinforcement learning. This method not only solves the problem of traditional reinforcement learning easily getting trapped in local optima, but also effectively alleviates the navigation instability caused by distribution offset. It has high environmental adaptability and can significantly improve the navigation success rate and path efficiency of AGVs in complex environments without clear maps. The intelligence level and practicality of autonomous navigation vehicles are significantly improved.

[0008] To achieve the above objectives, in a first aspect, the present invention provides a global-local fusion navigation method for AGVs based on LT-TD3 reinforcement learning, comprising the following steps: Based on LiDAR data, environmental features are extracted through obstacle edge detection, channel recognition, and open space detection. A set of navigation target points is dynamically generated, and the optimal navigation point is selected by a distance-weighted evaluation strategy. It receives global optimal navigation point information, uses a multimodal input processing mechanism to fuse lidar data, target position information and historical action sequences, and forms a state vector after processing through a convolutional neural network and a gated recurrent unit; Using the state vector as input, a state value network and a dual-state value network are employed. Q The network and policy network work together, combining Expectile regression, a hybrid objective update mechanism, and composite policy objective function optimization to output continuous linear and angular velocity control commands; The system monitors the AGV's operating status in parallel and combines collision detection and intelligent control switching to achieve intelligent and safe navigation of the AGV.

[0009] Furthermore, based on LiDAR data, environmental features are extracted through obstacle edge detection, channel recognition, and open space detection to dynamically generate a set of navigation target points, including: The obstacle edge detection algorithm identifies the boundary position of obstacles in the environment by analyzing the distance jump characteristics of adjacent measurement points in the 180-degree scan data in front of the LiDAR. When the distance difference between two adjacent scan points meets the judgment condition... When the angle direction and radial distance are set in the AGV polar coordinate system, the edge obstacle navigation node is automatically generated. The channel detection algorithm identifies passageways in the environment by recognizing the spatial distribution pattern of laser scan data at distances from the AGV's set value. When a laser scan point is less than β meters away and the angular distance n between the next laser scan point also less than β meters away and the current point is greater than γ degrees, the algorithm detects the passageway. and The midpoint of the corresponding angle and the radial distance are Create a channel navigation node; The open space detection algorithm identifies free space in the environment by detecting a scanning area where the distance between multiple consecutive laser measurement points exceeds the far distance threshold. When the measurement distance of M consecutive laser scanning points is greater than θ meters, the system establishes a free space navigation node in the middle direction of that scanning angle segment, at a distance of η meters from the current position of the AGV.

[0010] Furthermore, the selection of the optimal navigation point using a distance-weighted evaluation strategy includes: The dynamic node management mechanism maintains a valid set of navigation points by monitoring and updating the status of navigation nodes in real time. After every N control steps, it checks the AGV's position change and marks navigation points that are within the arrival threshold range as reached nodes and removes them from the candidate list. Simultaneously, it continuously monitors the safe distance between all navigation points and the latest perceived obstacle, moving navigation points that do not meet the obstacle avoidance threshold safety requirements to an error node list. Navigation points that the AGV cannot reach within a certain number of consecutive control steps are marked as invalid nodes and removed. A weighted distance evaluation formula is used to determine the final destination. Calculate the comprehensive score of each candidate navigation point, where D1 is the distance from the navigation point to the current position of the AGV, D2 is the distance from the navigation point to the final target position, and a and b are weighting coefficients. Select the navigation point with the smallest evaluation score as the next intermediate navigation target of the AGV.

[0011] Furthermore, a three-layer convolutional neural network structure is used to extract spatial features from the 180-degree laser scanning data in front, mapping the original environmental obstacle information into a compact spatial feature representation; The target relative distance and angle information output by the global navigation module are linearly transformed. The feature vectors are encoded into 10-dimensional features and then normalized. The linear velocity and angular velocity control commands executed in the previous moment are linearly transformed. Encoded as a 10-dimensional action feature vector; The three types of heterogeneous features are simply concatenated to form a multidimensional integrated state vector that includes environmental perception, goal orientation, and action memory. A gated recurrent unit is used to perform sequence modeling on the integrated state vector. Through a gating mechanism, dynamic changes in the environment and long-term dependencies are captured, and a multidimensional state representation is output. .

[0012] Furthermore, the state value network adopts the same front-end multimodal information processing architecture as the actor network, sharing the network structure of lidar data processing, target position encoding, historical action embedding, feature fusion and temporal memory processing. It outputs multidimensional state representation through gated recurrent units (GRUs), which are then mapped to a single scalar state value output V(s) through a fully connected layer. The State Value Network uses the Expectile Regression training method for parameter optimization and employs an asymmetric weighted squared error loss function. ,right Q Statistical learning of the value distribution characteristics, where τ∈(0,1) is the Expectile parameter controlling regression bias, and ψ is the trainable parameter of the state-value network. Q (s, a) represents the target Q The value of network state actions The predicted output of the value network for the current state is given by E, which is the expected computation of samples from the experience replay buffer, expressed through an indicator function. Implement a weighting mechanism.

[0013] Furthermore, the hybrid target update mechanism includes: A dynamic balance between conservative estimation and aggressive exploration is achieved by balancing the Bellman optimal equation and the Bellman expectation equation, using a hybrid objective value fusion calculation formula. By combining the immediate reward r with the discount factor γ, the state value network is estimated using a mixed weight parameter λ. and double Q The network minimum estimate is weighted and combined, where, Represents the j-th target Q network, ε represents the action output of the target policy network in the next state, and ε is the added target policy smoothing regularization noise. The dynamic parameter adjustment strategy uses an exponential decay method to adjust the mixed parameters. ,in, The parameter values ​​in the initial training stage are used to ensure learning stability, λ end The parameter values ​​are used to promote policy optimization in the later stages of training, where α is the decay rate and t is the current training step.

[0014] Furthermore, the objective function of the composite strategy adopts a two-component architecture. Policy loss function ,Include Q The two complementary optimization components are the value maximization term and the advantage-weighted behavior cloning term. The first term maximizes the current policy. In double Q Network Minimum Function The expected cumulative return under the condition drives continuous improvement in policy performance. The second term applies regularization constraints to the policy output through an advantage-weighted behavior cloning mechanism. Among them, the adaptive coefficient Output actions based on the current strategy Q The absolute expected value is dynamically standardized and adjusted, where ν is a hyperparameter controlling the strength of behavioral clonal regularization. The L2 norm distance between the policy output and the actions of the data samples; the dominance weight calculation formula. Based on the advantage function The numerical values ​​are exponentially transformed, β is a temperature parameter used to control the stiffness of the weight distribution, and an upper limit for the weights is set. Truncate the calculation results.

[0015] Furthermore, a hierarchical collision recovery strategy is adopted to achieve intelligent and safe navigation of AGVs. This strategy employs a three-level recovery decision system based on increasing priority. When collision risks are detected simultaneously in both the left and right recognition areas, the first-level recovery strategy is executed, outputting a straight-line forward movement command. When a collision risk is detected in the front recognition area, the second-level recovery strategy is executed. By comparing the minimum distance values ​​of the LiDAR in the left and right half-areas, the system automatically selects the direction with the larger distance value and outputs a maneuver command with a larger turning angle. When the system detects that the AGV remains stationary for a set period of time within a preset time threshold, the third-level recovery strategy is executed. This involves detecting the presence of obstacles in the front area. If an obstacle is detected, a sharp turning escape strategy is adopted; otherwise, a forward movement command is output.

[0016] Furthermore, collision detection and intelligent control switching enable intelligent and safe navigation of AGVs, including: By dividing the 180-degree frontal LiDAR sensing space around the AGV into three independent monitoring areas—the front recognition area, the left recognition area, and the right recognition area—each area employs an independent collision risk threshold judgment and status monitoring mechanism. When any area detects a collision risk or the AGV becomes stationary, the control authority of the reinforcement learning network is automatically suspended, and the AGV motion control is transferred to the collision recovery subsystem. Based on the current collision state type, the corresponding hierarchical recovery strategy is executed until the AGV successfully escapes the collision threat state. By continuously monitoring the safety status and AGV motion state changes of each recognition area, when no collision risk is detected in all monitoring areas within a preset number of steps and the AGV returns to normal motion state, the control is automatically transferred back to the reinforcement learning navigation and the normal global-local hybrid navigation mode is restored.

[0017] Secondly, the present invention provides an AGV global-local fusion navigation system based on LT-TD3 reinforcement learning, including an optimal navigation point acquisition module, a feature fusion module, an optimization module, and a navigation module; The optimal navigation point acquisition module extracts environmental features based on LiDAR data through obstacle edge detection, channel recognition, and open space detection, dynamically generates a set of navigation target points, and selects the optimal navigation point using a distance-weighted evaluation strategy. The feature fusion module receives global optimal navigation point information, uses a multimodal input processing mechanism to fuse lidar data, target position information and historical action sequences, and forms a state vector after processing by a convolutional neural network and a gated recurrent unit. The optimization module takes the state vector as input and uses a state-value network and a dual-state optimization method. Q The network and policy network work together, combining Expectile regression, a hybrid objective update mechanism, and composite policy objective function optimization to output continuous linear and angular velocity control commands; The navigation module is used to monitor the AGV's operating status in parallel, and combines collision detection and intelligent control switching to achieve intelligent and safe navigation of the AGV.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: The method described in this invention uses a global navigation technique that combines three algorithms: obstacle edge detection, channel recognition, and open space detection. By dynamically generating a set of navigation target points and using a distance-weighted evaluation strategy to select the optimal path point, it achieves the effect of fully covering the feasible navigation space in complex environments. Compared with traditional single path planning methods, it significantly improves the accuracy of navigation target point generation and selection, and effectively avoids blind spots and dead zones in path planning. Based on the method described in this invention, multimodal input processing and deep feature fusion techniques are used to comprehensively process LiDAR data, target location information and historical action sequences through convolutional neural networks and gated recurrent units, thereby realizing the construction of a high-dimensional comprehensive state representation. Compared with the traditional single sensor input method, this comprehensively improves the system's perception and understanding capabilities and environmental adaptability to complex dynamic scenes. Based on the method described in this invention, the LT-TD3 algorithm, which combines a state value network with Expectile regression and a hybrid objective update mechanism, achieves a dynamic balance between conservative estimation and aggressive exploration through a composite policy objective function, systematically solving the problem. Q Compared to existing reinforcement learning methods, this approach significantly improves navigation decision stability and algorithm convergence speed by addressing overestimation and distribution shift problems.

[0019] Based on the method described in this invention, an active safety protection technology using spatial partition monitoring and hierarchical recovery strategies achieves seamless integration of intelligent navigation and safety assurance through real-time collision detection and intelligent control switching mechanisms. Compared with traditional passive safety mechanisms, this significantly reduces the collision risk of AGVs in complex environments and significantly improves the reliability and robustness of system operation. Attached Figure Description

[0020] Figure 1 This is a flowchart of an implementable method of the present invention; Figure 2 This is the training environment for reinforcement learning of the AGV described in this invention; Figure 3 This describes the navigation process of the AGV global-local fusion navigation system described in this invention. Figure 4 The network structure of the Actor network described in this invention; Figure 5 This invention relates to the network structure of Critic networks. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The flowchart of the AGV global-local fusion navigation method based on LT-TD3 reinforcement learning proposed in this invention is as follows: Figure 1 As shown, the details are as follows: Step 1: The global navigation subsystem, acting as the path planning layer, extracts environmental features based on LiDAR data through obstacle edge detection, channel recognition, and open space detection algorithms. It dynamically generates a set of navigation target points and selects the optimal navigation point using a distance-weighted evaluation strategy, providing clear navigation guidance for the local navigation subsystem. Furthermore, the navigation point generation and management mechanism of the global navigation subsystem described in Step 1 includes four algorithms—obstacle edge detection, channel detection, open space detection, and dynamic node management—working together to construct the foundation for environmental cognition.

[0023] 1) Obstacle edge detection algorithm Obstacle edge detection algorithms identify the boundary positions of obstacles in the environment by analyzing the distance jump characteristics of adjacent measurement points in the 180-degree scan data in front of the LiDAR. The algorithm determines the boundary position when the distance difference between two adjacent scan points meets the judgment criteria. hour, The unit is meters. i and i -1 indicates adjacent measurement points. The system is located in the AGV polar coordinate system with an angle of θ degrees and a radial distance of... The algorithm automatically generates edge obstacle navigation nodes based on the spatial location of the obstacle. It can effectively identify the boundary between the obstacle and the free space, and provide navigation path points for the AGV to bypass the obstacle.

[0024] 2) Channel detection algorithm The channel detection algorithm identifies passable channel areas in the environment by recognizing the spatial distribution patterns of areas close to the AGV in the laser scanning data. It records laser scanning points where the current distance is less than a set value, and when the angular distance *n* between the current laser scanning point and the next laser scanning point with a distance also less than the set value is greater than a set angle, a navigation node is created at the midpoint of the angle between the two laser scanning points and at a set radial distance. For example, when the system records a laser scanning point with a distance less than *β* meters... i And the next laser scanning point with the same distance less than β meters ( i -1) When the angular distance n from the current point is greater than γ degrees, and The midpoint of the corresponding angle and the radial distance are A channel navigation node is created at the location. This algorithm is specifically designed to identify gaps between narrow passages or obstacles, ensuring that the AGV can safely pass through confined spaces.

[0025] 3) Open Space Detection Algorithm The open space detection algorithm identifies free space in the environment by detecting scanning areas where the distance between multiple consecutive laser measurement points exceeds a long distance threshold. Whenever the measurement distance of M consecutive laser scanning points is greater than θ meters (M can be 20), a free space navigation node is established at the middle direction of the angle segment of the scanning area, η meters away from the current position of the AGV. This algorithm provides the AGV with a long-distance navigation target point in the open environment, avoiding invalid exploration behavior in open areas.

[0026] 4) Dynamic node management mechanism The dynamic node management mechanism maintains a valid set of navigation points by monitoring and updating the status of navigation nodes in real time. As the unified management and decision-making center for the three detection algorithms mentioned above, it checks the AGV's position change after every N control steps (N can be 30). Navigation points within the arrival threshold range of the AGV are marked as arrived nodes and removed from the candidate list. Simultaneously, it continuously monitors the safe distance between all navigation points and the latest perceived obstacle, moving navigation points that do not meet the obstacle avoidance threshold to the error node list. Navigation points that the AGV cannot reach within 700 consecutive control steps are marked as invalid nodes and cleared. The system uses a weighted distance evaluation formula... Calculate the comprehensive score of each candidate navigation point, and select the navigation point with the lowest evaluation score as the next intermediate navigation target for the AGV. Here, D1 is the distance from the navigation point to the current position of the AGV, D2 is the distance from the navigation point to the final target position, and a and b are weighting coefficients, achieving an organic combination of global path planning and local obstacle avoidance. Figure 2 As shown, navigation nodes are generated for the AGV in a mapless environment. Green nodes represent the final target point to be reached, orange nodes represent the nodes that have been reached, and blue nodes represent the current navigation point.

[0027] Step 2: The local navigation subsystem, as the information fusion layer, receives the global navigation target point set, and uses a multimodal input processing mechanism to fuse LiDAR data, target position information and historical action sequences. After processing by convolutional neural networks and gated recurrent units, a 36-dimensional state vector is formed and passed to the reinforcement learning decision layer. Figure 3 The training environment for reinforcement learning in this embodiment of the invention includes obstacles with static, unchanging positions and obstacles with static, randomly distributed positions (small yellow boxes in the view). The statically randomly distributed obstacles will change their positions during each training session.

[0028] Furthermore, the multimodal information processing module in the local navigation subsystem described in step 2 receives the target navigation point information provided by the global navigation subsystem, including a lidar data processing unit, a target position encoding unit, a historical action embedding unit, a feature fusion unit, and a temporal memory processing unit.

[0029] The lidar data processing unit employs a three-layer convolutional neural network structure (Conv(1,4), Conv(4,8), Conv(8,4)) to extract spatial features from the 180-degree forward lidar scan data. This maps the original environmental obstacle information into a compact spatial feature representation. While using the same sensor data source as the global navigation system, it focuses on extracting fine-grained spatial features required for local navigation. The target position encoding unit linearly transforms the target relative distance and angle information output by the global navigation module. The encoding is performed as a 10-dimensional feature vector and then normalized; this provides the network with clear navigation target guidance. The historical action embedding unit linearly transforms the linear velocity and angular velocity control commands executed in the previous moment. Encode it as a 10-dimensional action feature vector to ensure the continuity and smoothness of action execution; The feature fusion unit performs a simple concatenation operation on the three types of heterogeneous features to form a 36-dimensional comprehensive state vector. The temporal memory processing unit uses a gated recurrent unit to perform sequence modeling on the fused features, capturing dynamic changes in the environment and long-term dependencies through a gating mechanism, and outputting a multi-dimensional state representation. This serves as a shared input foundation for subsequent actor and critic networks, thereby enabling an effective mapping from multimodal perceptual information to intelligent decision generation.

[0030] refer to Figure 4 and Figure 5 The neural network architecture of the local navigation subsystem in step 2 mainly includes: 1) an Actor network multimodal fusion structure; 2) a Critic network dual-mode fusion structure. Q Value estimation structure; 1) Actor Network Multimodal Fusion Structure The Actor network achieves intelligent mapping from multimodal perception to continuous action decision-making through hierarchical feature processing. The 180-degree scanning data from the LiDAR is processed through a three-layer one-dimensional convolutional neural network. Spatial pattern recognition and environmental geometric feature extraction are performed, transforming the raw distance information into a compact spatial feature representation. Two-dimensional target relative position information and two-dimensional historical action information are respectively processed through linear embedding layers. Dimensional expansion and feature encoding are performed to provide navigation target guidance and action continuity assurance for the network. Feature vectors from the three processing branches are concatenated to form a 36-dimensional unified state representation vector, which is then processed through a gated recurrent unit. Temporal modeling is performed to capture the dynamic characteristics of environmental changes and historical dependencies, ultimately through a funnel-shaped fully connected network. To achieve a nonlinear mapping from state to action, the Tanh activation function will output the normalized linear velocity and angular velocity control commands of the AGV, which are constrained within the range of [-1, 1]. 2) Critic network dual Q Value estimation structure The Critic network uses two identical but parameter-independent structures. Q Value estimation networks reduce overestimation bias and improve learning stability; the state feature encoding layer uses mathematical transformations. Shared 36-dimensional state vector Expanded to 400-dimensional state feature representation ,in It is a 36×400 weight matrix. Given the input state vector from the Actor network, The state-action interaction fusion layer calculates a 400-dimensional bias vector. Generate a 300-dimensional joint feature vector This enables deep fusion modeling of state and action information, where... It is a 400×300 state weight matrix. The action weight matrix is ​​2×300, and 'a' is the input action vector. Q Value regression output layer through linear transformation Map joint features to scalars Q Value, of which Output a 1×300 weight vector, double Q Network final output As a conservative value estimate, it reduces the optimism bias of the value function and improves the convergence stability of reinforcement learning algorithms.

[0031] Step 3: The LT-TD3 (Generalized Temporal Difference) reinforcement learning algorithm subsystem acts as the intelligent decision-making layer, receiving the state vector input from the information fusion layer and employing a state-value network and dual... Q The network and policy network work together to solve problems through Expectile regression. Q The problem of estimation is addressed by a hybrid objective update mechanism that achieves a dynamic balance between conservative estimation and aggressive exploration. The composite policy objective function ensures stable optimization of the policy network, ultimately outputting continuous linear and angular velocity control commands.

[0032] Furthermore, the state-value network of the LT-TD3 reinforcement learning algorithm subsystem in step 3 is based on the state representation provided by the local navigation subsystem. For intelligent decision-making, the Expectile regression training method is used for parameter optimization, and an asymmetric weighted squared error loss function is employed. Achieve Q Statistical learning of value distribution characteristics, when τ When = 0.5, it is equivalent to mean squared error regression. τ When the value is greater than 0.5, positive errors are given greater weight to achieve a conservative estimate. ψ represents the trainable parameters of the state-value network. Q (s, a) represents the target Q The value of network state actions The predicted output of the current state value network is given by E, which represents the expected computation on samples from the experience replay buffer, through an indicator function. Implementing a weighting mechanism enables the network to... Q Differentiated modeling is performed for different regions of the value distribution to provide a basis for the dominance function. The calculation provides an accurate baseline estimate.

[0033] Furthermore, the hybrid objective update mechanism described in step 3 works in conjunction with the state value network to achieve a dynamic balance between conservative estimation and aggressive exploration by balancing the Bellman optimal equation and the Bellman expectation equation, and employs a hybrid objective value fusion calculation formula. By combining the immediate reward r with the discount factor γ, the state value network is estimated using a mixed weight parameter λ. and double Q The network minimum estimate is weighted and combined, where Represents the j-th target Q network, Let ε represent the action output of the target policy network in the next state, and ε be the added target policy smoothing regularization noise. The dynamic parameter adjustment strategy uses an exponential decay method to adjust the hybrid parameters. ,in λend represents the parameter values ​​used in the early stages of training to ensure learning stability, λend represents the parameter values ​​used in the later stages of training to promote policy optimization, α represents the decay rate, which is usually set to 0.99, and t represents the current training step.

[0034] Furthermore, the composite strategy objective function described in step 3 integrates the results of the aforementioned value estimation and objective update mechanisms, and adopts a dual-component architecture to achieve a dynamic balance between exploration and utilization. Policy loss function Include Q The two complementary optimization components are the value maximization term and the advantage-weighted behavior cloning term. The first term maximizes the current policy. In double Q Network Minimum Function The first term uses the expected cumulative return to drive continuous improvement in policy performance. The second term uses a right-weighted behavior cloning mechanism to regularize the policy output, preventing the policy from deviating too much from the distribution of the training data and causing distribution shift.

[0035] Where the adaptive coefficient Output actions based on the current strategy Q The absolute expected value is dynamically standardized and adjusted, where ν is a hyperparameter controlling the strength of behavioral clonal regularization. The L2 norm distance between the policy output and the actions of the data samples; the dominance weight calculation formula. Based on the advantage function The numerical values ​​are exponentially transformed, where β is a temperature parameter used to control the stiffness of the weight distribution. A larger β value makes the weight distribution flatter, thus avoiding extreme samples dominating the learning process, while a smaller β value strengthens the learning weights of high-dominance actions. At the same time, a weight upper limit is set. The computation results are truncated to ensure the numerical stability of the training process. This mechanism enables the network to identify and prioritize learning high-quality state-action pairs, effectively controlling the degree of deviation from the original data distribution while maintaining the policy improvement capability.

[0036] Stage 4: The collision detection and recovery subsystem serves as a safety assurance layer, monitoring the AGV's operating status in parallel. It divides the surrounding space into three monitoring areas: front, left, and right. When a collision risk is detected, it immediately takes over control and executes corresponding obstacle avoidance actions using a hierarchical recovery strategy. After escaping danger, it transfers control back to the reinforcement learning decision layer, achieving seamless integration of intelligent navigation and safety assurance.

[0037] Furthermore, the hierarchical collision recovery strategy mechanism described in section 4 is designed to address the inherent limitations of reinforcement learning neural networks in handling collision input states. Serving as a safety assurance layer for the entire system, it operates in parallel with the reinforcement learning control system. It employs a three-level recovery decision system based on increasing priority. When collision risk is detected simultaneously in both the left and right recognition areas, the first-level recovery strategy is executed, and the system outputs a straight-line forward movement command. When a collision risk is detected in the forward recognition area, the second-level recovery strategy is executed. The system automatically selects the direction with the larger distance value and outputs a maneuver command with a larger turning angle by comparing the minimum distance values ​​of the LiDAR in the left and right half-areas. When the system detects that the AGV has remained stationary for an extended period within a preset time threshold, the third-level recovery strategy is executed. First, obstacle presence detection is performed in the forward area. If an obstacle is detected, a sharp turn is adopted to escape the obstacle; if there is no obstacle in the forward area, a forward movement command is output. This mechanism, through progressive priority design, ensures that the AGV can achieve effective autonomous escape and navigation recovery in various complex collision scenarios.

[0038] Furthermore, the collision detection and reinforcement learning system integrated control method described in Section 4 divides the 180-degree sensing space around the AGV using a LiDAR into three independent monitoring areas: a front recognition area, a left recognition area, and a right recognition area. Each area employs an independent collision risk threshold judgment and status monitoring mechanism. When any area detects a collision risk or the AGV becomes stationary, the system automatically suspends the control authority of the reinforcement learning network and transfers the AGV's motion control to the collision recovery subsystem. The collision recovery subsystem executes a corresponding hierarchical recovery strategy based on the current collision state type until the AGV successfully escapes the collision threat state. When no collision risk is detected in any of the monitored areas within a preset number of steps and the AGV returns to normal motion, control is automatically transferred back to the reinforcement learning navigation system, and the normal global-local hybrid navigation mode is restored. This integrated method effectively solves the problem of insufficient processing capability of reinforcement learning algorithms under unexpected collision states, and realizes the organic integration of a rule-based safety assurance mechanism and a learning-based intelligent navigation system.

[0039] Based on the above technical concept, the present invention can also provide an AGV global-local fusion navigation system based on LT-TD3 reinforcement learning, including an optimal navigation point acquisition module, a feature fusion module, an optimization module, and a navigation module; The optimal navigation point acquisition module extracts environmental features based on LiDAR data through obstacle edge detection, channel recognition, and open space detection, dynamically generates a set of navigation target points, and selects the optimal navigation point using a distance-weighted evaluation strategy. The feature fusion module receives global optimal navigation point information, uses a multimodal input processing mechanism to fuse lidar data, target position information and historical action sequences, and forms a state vector after processing by a convolutional neural network and a gated recurrent unit. The optimization module takes the state vector as input and uses a state-value network and a dual-state optimization method. Q The network and policy network work together, combining Expectile regression, a hybrid objective update mechanism, and composite policy objective function optimization to output continuous linear and angular velocity control commands; The navigation module is used to monitor the AGV's operating status in parallel, and combines collision detection and intelligent control switching to achieve intelligent and safe navigation of the AGV.

[0040] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A global-local fusion navigation method for AGVs based on LT-TD3 reinforcement learning, characterized in that, Includes the following steps: Based on LiDAR data, environmental features are extracted through obstacle edge detection, channel recognition, and open space detection. A set of navigation target points is dynamically generated, and the optimal navigation point is selected by a distance-weighted evaluation strategy. It receives global optimal navigation point information, uses a multimodal input processing mechanism to fuse lidar data, target position information and historical action sequences, and forms a state vector after processing through a convolutional neural network and a gated recurrent unit; Using the state vector as input, a state value network and a dual-state value network are employed. Q The network and policy network work together, combining Expectile regression, a hybrid objective update mechanism, and composite policy objective function optimization to output continuous linear and angular velocity control commands; The system monitors the AGV's operating status in parallel and combines collision detection and intelligent control switching to achieve intelligent and safe navigation of the AGV.

2. The AGV global-local fusion navigation method based on LT-TD3 reinforcement learning according to claim 1, characterized in that, Based on LiDAR data, environmental features are extracted through obstacle edge detection, channel recognition, and open space detection to dynamically generate a set of navigation target points, including: The obstacle edge detection algorithm identifies the boundary position of obstacles in the environment by analyzing the distance jump characteristics of adjacent measurement points in the 180-degree scan data in front of the LiDAR. When the distance difference between two adjacent scan points meets the judgment condition... When the angle direction and radial distance are set in the AGV polar coordinate system, the edge obstacle navigation node is automatically generated. The channel detection algorithm identifies passageways in the environment by recognizing the spatial distribution pattern of laser scan data at distances from the AGV's set value. When a laser scan point is less than β meters away and the angular distance n between the next laser scan point also less than β meters away and the current point is greater than γ degrees, the algorithm detects the passageway. and The midpoint of the corresponding angle and the radial distance are Create a channel navigation node; The open space detection algorithm identifies free space in the environment by detecting a scanning area where the distance between multiple consecutive laser measurement points exceeds the distance threshold. When the measurement distance of M consecutive laser scanning points is greater than θ meters, the system establishes a free space navigation node in the middle direction of that scanning angle segment, at a distance of η meters from the current position of the AGV.

3. The AGV global-local fusion navigation method based on LT-TD3 reinforcement learning according to claim 1, characterized in that, The optimal navigation point is selected using a distance-weighted evaluation strategy, which includes: The dynamic node management mechanism maintains a valid set of navigation points by monitoring and updating the status of navigation nodes in real time. After every N control steps, it checks the AGV's position change and marks navigation points that are within the arrival threshold range as reached nodes and removes them from the candidate list. Simultaneously, it continuously monitors the safe distance between all navigation points and the latest perceived obstacle, moving navigation points that do not meet the obstacle avoidance threshold safety requirements to an error node list. Navigation points that the AGV cannot reach within a certain number of consecutive control steps are marked as invalid nodes and removed. A weighted distance evaluation formula is used to determine the final destination. Calculate the comprehensive score of each candidate navigation point, where D1 is the distance from the navigation point to the current position of the AGV, D2 is the distance from the navigation point to the final target position, and a and b are weighting coefficients. Select the navigation point with the smallest evaluation score as the next intermediate navigation target of the AGV.

4. The AGV global-local fusion navigation system based on LT-TD3 reinforcement learning according to claim 1, characterized in that, A three-layer convolutional neural network structure is used to extract spatial features from the 180-degree laser scanning data in front, mapping the original environmental obstacle information into a compact spatial feature representation; The target relative distance and angle information output by the global navigation module are linearly transformed. The feature vectors are encoded into 10-dimensional features and then normalized. The linear velocity and angular velocity control commands executed in the previous moment are linearly transformed. Encoded as a 10-dimensional action feature vector; The three types of heterogeneous features are simply concatenated to form a multidimensional integrated state vector that includes environmental perception, goal orientation, and action memory. A gated recurrent unit is used to perform sequence modeling on the integrated state vector. Through a gating mechanism, dynamic changes in the environment and long-term dependencies are captured, and a multidimensional state representation is output. .

5. The AGV global-local fusion navigation method based on LT-TD3 reinforcement learning according to claim 1, characterized in that, The State Value Network adopts the same front-end multimodal information processing architecture as the Actor Network, sharing the network structure of LiDAR data processing, target position encoding, historical action embedding, feature fusion and temporal memory processing. It outputs multidimensional state representation through gated recurrent units (GRUs), which are then mapped to a single scalar state value output V(s) through a fully connected layer. The State Value Network uses the Expectile Regression training method for parameter optimization and employs an asymmetric weighted squared error loss function. ,right Q Statistical learning of the value distribution characteristics, where τ∈(0,1) is the Expectile parameter controlling regression bias, and ψ is the trainable parameter of the state-value network. Q (s, a) represents the value from the target Q The value of network state actions The predicted output of the value network for the current state is given by E, which is the expected computation of samples from the experience replay buffer, expressed through an indicator function. Implement a weighting mechanism.

6. The AGV global-local fusion navigation method based on LT-TD3 reinforcement learning according to claim 1, characterized in that, Hybrid target update mechanisms include: A dynamic balance between conservative estimation and aggressive exploration is achieved by balancing the Bellman optimal equation and the Bellman expectation equation, using a hybrid objective value fusion calculation formula. By combining the immediate reward r with the discount factor γ, the state value network is estimated using a mixed weight parameter λ. and double Q The network minimum estimate is weighted and combined, where, Represents the j-th target Q network, ε represents the action output of the target policy network in the next state, and ε is the added target policy smoothing regularization noise. The dynamic parameter adjustment strategy uses an exponential decay method to adjust the mixed parameters. ,in, The parameter values ​​in the initial training stage are used to ensure learning stability, λ end The parameter values ​​are used to promote policy optimization in the later stages of training, where α is the decay rate and t is the current training step.

7. The AGV global-local fusion navigation method based on LT-TD3 reinforcement learning according to claim 1, characterized in that, The composite strategy objective function adopts a two-component architecture. Policy loss function ,Include Q The two complementary optimization components are the value maximization term and the advantage-weighted behavior cloning term. The first term maximizes the current policy. In double Q Network Minimum Function The expected cumulative return under the condition drives continuous improvement in policy performance. The second term applies regularization constraints to the policy output through an advantage-weighted behavior cloning mechanism. Among them, the adaptive coefficient Output actions based on the current strategy Q The absolute expected value is dynamically standardized and adjusted, where ν is a hyperparameter controlling the strength of behavioral clonal regularization. The L2 norm distance between the policy output and the actions of the data samples; the dominance weight calculation formula. Based on the advantage function The numerical values ​​are exponentially transformed, β is a temperature parameter used to control the stiffness of the weight distribution, and an upper limit for the weights is set. Truncate the calculation results.

8. The AGV global-local fusion navigation method based on LT-TD3 reinforcement learning according to claim 1, characterized in that, To achieve intelligent and safe navigation of AGVs, a hierarchical collision recovery strategy is also adopted. The hierarchical collision recovery strategy mechanism adopts a three-level recovery decision system based on increasing priority. When collision risk is detected in both the left and right recognition areas at the same time, the first-level recovery strategy is executed, and a straight-line forward movement command is output. When a collision risk is detected in the front recognition area, the second-level recovery strategy is executed. By comparing the minimum distance values ​​of the LiDAR in the left and right half-areas, the system automatically selects the direction with the larger distance value and outputs a maneuver command with a larger turning angle. When the system detects that the AGV position remains stationary for a set period of time within a preset time threshold, the third-level recovery strategy is executed. The system performs obstacle presence detection in the front area. If an obstacle is detected, a large-scale turning escape strategy is adopted. If there is no obstacle in the front area, a forward movement command is output.

9. The AGV global-local fusion navigation system based on LT-TD3 reinforcement learning according to claim 1, characterized in that, Collision detection and intelligent control switching enable intelligent and safe navigation of AGVs, including: By dividing the 180-degree frontal LiDAR sensing space around the AGV into three independent monitoring areas—the front recognition area, the left recognition area, and the right recognition area—each area employs an independent collision risk threshold judgment and status monitoring mechanism. When any area detects a collision risk or the AGV becomes stationary, the control authority of the reinforcement learning network is automatically suspended, and the AGV motion control is transferred to the collision recovery subsystem. Based on the current collision state type, the corresponding hierarchical recovery strategy is executed until the AGV successfully escapes the collision threat state. By continuously monitoring the safety status and AGV motion state changes of each recognition area, when no collision risk is detected in any of the monitoring areas within a preset number of steps and the AGV returns to normal motion state, the control is automatically transferred back to the reinforcement learning navigation and the normal global-local hybrid navigation mode is restored.

10. An AGV global-local fusion navigation system based on LT-TD3 reinforcement learning, characterized in that, It includes an optimal navigation point acquisition module, a feature fusion module, an optimization module, and a navigation module; The optimal navigation point acquisition module extracts environmental features based on LiDAR data through obstacle edge detection, channel recognition, and open space detection, dynamically generates a set of navigation target points, and selects the optimal navigation point using a distance-weighted evaluation strategy. The feature fusion module receives global optimal navigation point information, uses a multimodal input processing mechanism to fuse lidar data, target position information and historical action sequences, and forms a state vector after processing by a convolutional neural network and a gated recurrent unit. The optimization module takes the state vector as input and uses a state-value network and a dual-state optimization method. Q The network and policy network work together, combining Expectile regression, a hybrid objective update mechanism, and composite policy objective function optimization to output continuous linear and angular velocity control commands; The navigation module is used to monitor the AGV's operating status in parallel, and combines collision detection and intelligent control switching to achieve intelligent and safe navigation of the AGV.

Images