Multi-modal fusion AGV dynamic path planning and cluster scheduling system

Abstract

The invention discloses a multi-modal fusion AGV dynamic path planning and cluster scheduling system, and relates to the technical field of multi-modal perception and data fusion, and the system comprises a multi-modal perception module which generates a dynamic obstacle confidence map through multi-source data fusion in combination with a hardware-level time synchronization and Transform feature fusion network; the dynamic path planning module adopts an improved rolling window algorithm, integrates an LSTM space-time conflict prediction model and an adaptive weight cost function, and realizes dynamic obstacle trajectory prediction and non-oscillation global path generation; the cluster scheduling control module is used for optimizing multi-AGV task allocation and conflict resolution in combination with a dynamic priority preemption mechanism and digital twinborn simulation rehearsal based on a distributed contract network protocol of edge computing; and the data conflict resolution module is used for triggering a multi-modal re-calibration process through confidence weighting and sliding window time sequence verification. According to the system, in logistics storage and intelligent manufacturing scenes, the dynamic obstacle avoidance success rate and the robustness and operation efficiency of an AGV cluster are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal perception and data fusion, and in particular to a multimodal fusion AGV dynamic path planning and cluster scheduling system. Background Art

[0002] Smart manufacturing, intelligent warehousing, and other fields require efficient and flexible logistics and transportation systems to improve production efficiency and reduce costs. As key equipment in automated logistics, AGVs (Automated Guided Vehicles) are crucial for optimizing their path planning and cluster scheduling to achieve efficient operation of the entire system.

[0003] In practical applications, AGVs often need to operate in complex and ever-changing environments, such as factory floors with various devices, personnel movements, and the dynamic placement of goods. Multimodal fusion perception technology can better cope with such complex environments, providing AGVs with accurate environmental information for real-time path adjustments and obstacle avoidance. As production scales up, multiple AGVs often need to collaborate to complete tasks. This necessitates an effective cluster scheduling system to rationally allocate tasks and coordinate actions between AGVs, avoiding conflicts and deadlocks and improving overall operational efficiency.

[0004] However, existing technologies still have problems that need to be solved: insufficient adaptability to dynamic environments, the traditional rolling window planning method is prone to local oscillation in dense dynamic obstacle scenes; when multi-sensor data is fused, the recognition results of the same obstacle by lidar and vision are inconsistent, resulting in decision delays. Summary of the Invention

[0005] In order to solve the above technical problems, a multimodal fusion AGV dynamic path planning and cluster scheduling system is provided. This technical solution solves the problem of insufficient adaptability to dynamic environments mentioned above.

[0006] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0007] A multi-modal fusion AGV dynamic path planning and cluster scheduling system, including:

[0008] Multimodal perception module: This module includes a lidar, binocular vision sensor, IMU inertial navigation unit, and UWB positioning base station for real-time environmental data collection. A cross-modal time synchronization unit is configured to align multi-source data through hardware-level timestamps. A feature-level fusion network based on the Transformer architecture performs joint semantic segmentation of laser point clouds and visual images, outputting a dynamic obstacle confidence map.

[0009] Dynamic Path Planning Module: This module uses an improved rolling window planning algorithm, embeds a spatiotemporal conflict prediction model, and uses LSTM to predict the trajectory of dynamic obstacles. It also uses a cost function based on adaptive weight adjustment to integrate path length, energy consumption, and dynamic obstacle avoidance priority to generate a global path without oscillation.

[0010] Cluster scheduling control module: Distributed task allocation architecture, based on an improved contract network protocol, enables local task negotiation through edge computing nodes; configures a dynamic priority preemption mechanism, and generates conflict resolution strategies through digital twin simulation rehearsals when path conflicts are detected;

[0011] Data conflict resolution module: Confidence-weighted obstacle recognition results from lidar and vision are applied. When the confidence difference exceeds a threshold, a multimodal recalibration process is triggered. A sliding window timing consistency check is used.

[0012] Preferably, the multimodal perception module specifically includes:

[0013] Sensor acquisition unit:

[0014] LiDAR uses solid-state LiDAR to build a centimeter-level accurate geometric model of the environment and identify the outline and position of fixed obstacles. It also detects the real-time speed and direction of dynamic objects through multi-frame point cloud differentiation.

[0015] The binocular vision sensor uses a global shutter CMOS sensor to calculate obstacle depth information and identify texture features through a stereo matching algorithm. It also uses the YOLOv7 dynamic target detection model to classify obstacle types and output semantic labels.

[0016] IMU inertial navigation uses a six-axis MEMS inertial sensor to provide AGV body motion status data, filling the instantaneous perception blind spots of lidar and vision. It is integrated with UWB positioning data and realizes dead reckoning through Kalman filtering to maintain short-term positioning continuity in the event of sensor failure.

[0017] UWB positioning base station:

[0018] The TDoA positioning solution is adopted, and multiple anchor base stations are deployed to form a positioning network to provide global coordinate constraints for laser SLAM.

[0019] Preferably, the multimodal perception module specifically includes:

[0020] Cross-modal time synchronization unit: Hardware-level synchronization uses FPGA programmable logic chips to distribute a unified clock source to each sensor; timestamps of lidar, vision, and IMU data are aligned; when the master clock fails, it automatically switches to PTP network synchronization mode; dynamic delay compensation is based on sensor transmission delay modeling and uses forward prediction interpolation to reconstruct the synchronized data stream;

[0021] Transformer feature-level fusion network: In the input layer, the laser point cloud is voxelized by VoxelNet to generate a dense 3D feature grid, and the visual image is processed through ResNet-50 to extract multi-scale feature maps; in the fusion layer, a cross-modal attention mechanism is used to calculate the feature similarity weights between point cloud voxels and image pixels, and the spatiotemporal context information is associated through a multi-head self-attention module; in the output layer, a dynamic obstacle confidence map is generated, marking the obstacle existence probability and motion vector of each grid cell.

[0022] Preferably, the multimodal perception module specifically includes:

[0023] Obstacle confidence map unit: The probability model includes static obstacle probability calculated by jointly calculating LiDAR point cloud density and visual semantic segmentation results, and dynamic obstacle probability introduced by LSTM trajectory prediction confidence; path planning prioritizes avoiding high-risk areas with confidence greater than a set threshold, and allows temporary crossing of controllable risk areas with confidence within a set range; cluster scheduling dynamically adjusts AGV task allocation based on the global confidence map.

[0024] Preferably, the dynamic path planning module specifically includes:

[0025] Dynamic window size adaptation unit: The basic window size is set as a multiple of the AGV braking distance and is dynamically expanded according to the real-time obstacle density. The local optimal escape strategy uses a simulated annealing algorithm to perturb the path candidate set. When the difference in the length of multiple consecutive planned paths is less than a predetermined ratio, a random offset angle is injected.

[0026] Spatiotemporal conflict prediction model:

[0027] The LSTM trajectory prediction network takes as input the historical trajectory of dynamic obstacles and the semantic features of the environment. The output prediction is to generate a multimodal trajectory probability distribution of obstacles in the short term, and output the highest confidence trajectory and safety margin.

[0028] The spatiotemporal occupancy grid modeling method discretizes the time dimension of the environmental grid map into intervals and marks the occupancy status of each grid in the short term in the future, including idle, possibly occupied, and confirmed occupied;

[0029] Conflict detection: When the planned path overlaps with the confirmed occupied area in the spatiotemporal occupancy grid, replanning is triggered; when it overlaps with the possible occupied area, an alternative detour path is generated.

[0030] Preferably, the dynamic path planning module specifically includes:

[0031] Adaptive weight unit: adopts a multi-objective fusion mechanism and dynamic weight adjustment rules, combined with oscillation suppression technology, through path smoothing constraints, and introduces a curvature change rate penalty term into the cost function;

[0032] Oscillation-free global path generation: At the global level, a coarse-grained path is generated based on the A* algorithm, constraining the search space to permanently free areas in the spatiotemporal occupancy grid. At the local level, a fast-exploring random tree is used within a rolling window to refine the path, dynamically pruning invalid branches based on spatiotemporal conflict prediction results. Real-time replanning trigger conditions include hard triggers, which detect the presence of a confirmed occupied area within a predetermined range ahead of the path, and soft triggers, which initiate a Monte Carlo tree search to explore the optimal detour when the path crosses a potentially occupied area and no alternative routes are available.

[0033] Preferably, the cluster scheduling control module specifically includes:

[0034] Distributed Task Allocation Unit: An improved contract network protocol employs a dynamic priority weighting mechanism, introducing multi-dimensional weighting factors during task bidding. Topology-aware task partitioning divides the warehouse into multiple edge computing domains, with edge servers managing the real-time status of AGVs within each domain. Negotiation rules prioritize intra-domain tasks through local auctions, while cross-domain tasks are handled by a cloud coordinator, which only transmits metadata.

[0035] Dynamic priority preemption unit: Conflict detection and hierarchical response, including level 1: when there is a risk of preemption at a path intersection, the low-priority AGV is immediately frozen and the path is re-bid; level 2: when the charging pile queue times out, a high-priority AGV is forcibly inserted to compensate for the waiting time; level 3: when the communication interruption makes the status unknown, it switches to the LoRa emergency link and enables the degraded mode; the preemption weight is dynamically calculated, and the weight factors include the economic value of the task, the health status of the AGV, and the historical conflict records.

[0036] Preferably, the cluster scheduling control module specifically includes:

[0037] Digital twin simulation preview unit:

[0038] Real-time twin model construction and data synchronization: the physical AGV uploads its position, speed, and task status to the twin platform, and the environment model is updated in real time through laser SLAM and visual semantic fusion.

[0039] Conflict resolution strategies are generated through Monte Carlo tree search optimization, including an expansion phase to generate candidate paths based on the spatiotemporal occupancy grid; a simulation phase to evaluate the conflict probability, energy consumption, and task delay cost of the paths; and a backtracking phase to select the path with the highest overall score as the resolution strategy.

[0040] The emergency strategy buffer has a preloaded strategy library: it contains plans for multiple types of standard conflict scenarios; dynamic strategy generation, for non-standard scenarios, uses generative AI to synthesize solutions in real time, and issues them for execution after simulation verification.

[0041] Preferably, the data conflict resolution module specifically includes:

[0042] Multimodal weight dynamic allocation unit: Weight calculation, the weights of lidar and vision sensors are dynamically adjusted based on real-time measurement variance; scene adaptive adjustment, when strong light illuminates the reflective ground, the weight is biased towards lidar; when low light illuminates the texture-rich area, the weight is biased towards vision; when it is rainy and foggy, the weight is biased towards lidar and UWB fusion; threshold dynamic setting, dynamically adjust the confidence difference threshold according to the complexity of the environment.

[0043] Preferably, the data conflict resolution module specifically includes:

[0044] Multimodal Recalibration Unit:

[0045] During the trigger phase, hard trigger conditions include: the position deviation between the lidar and vision for the same obstacle is greater than a predetermined range; the confidence difference of multiple consecutive frames of data exceeds a threshold; soft trigger conditions include: the AGV's motion state is inconsistent with the perception result for several seconds; the edge computing node detects a decline in sensor health;

[0046] During calibration, the data is frozen, the current frame data output is paused, and the historical frame data is retained. Spatiotemporal alignment is performed, aligning the multimodal data streams through a dynamic time warping algorithm and using IMU data to compensate for motion distortion. Parameters are reset, with the visual exposure time and lidar scanning frequency adaptively adjusted, and the external parameter matrix is ​​calibrated online based on the semantic features of the environment. Cross-validation is performed, using UWB positioning data to verify the obstacle coordinate error after calibration. If the verification fails, the system switches to degraded mode.

[0047] Sliding window timing consistency check unit: dynamic window management, the window size is divided into static environment and dynamic environment, and the verification rule is to test the timing smoothness of the multimodal data in the window; abnormal data processing, short-term anomalies, single-frame data conflicts, the Kalman filter prediction value is used to replace the abnormal data to maintain path continuity; long-term anomalies, continuous multi-frame conflicts, trigger the sensor self-test program, and report to the edge node to start the backup perception source.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention proposes that the present invention realizes deep complementarity of lidar, vision, IMU and UWB through collaborative optimization of multimodal sensors. LiDAR provides stable geometric modeling on reflective ground, vision enhances semantic understanding in texture-rich areas, and IMU and UWB fusion ensures positioning continuity, and can still maintain centimeter-level accuracy in extreme weather such as rain and fog; combining laser point cloud difference and visual semantic classification, it can accurately capture the speed and direction of moving objects, and identify their types, providing a multi-dimensional decision-making basis for path planning; when any sensor fails, cross-modal data compensation and degradation mode switching are used to ensure that the system continues to operate in short-term faults, avoiding the global paralysis caused by single-point failures in traditional solutions.

[0050] The system introduces a spatiotemporal conflict prediction and hierarchical planning architecture. Through LSTM trajectory prediction and spatiotemporal occupancy grid modeling, it predicts the distribution of obstacles in the next few seconds in advance, and combines Monte Carlo tree search to generate multiple path alternatives to avoid sudden risks. The planning window size is adaptively adjusted according to the complexity of the environment, and the simulated annealing algorithm is combined to escape the local optimum, significantly improving the path pass rate in complex scenarios. Bezier curve smoothing and curvature constraints are used to solve the path jitter problem of traditional algorithms and ensure the stable operation of high-precision vehicles such as AGV robotic arms.

[0051] By adopting distributed edge collaboration and dynamic priority preemption mechanisms, the improved contract network protocol combines multi-dimensional weight factors to achieve optimal resource matching and reduce inefficient idle driving and congestion. Differentiated conflict response rules are designed for scenarios such as path intersections, charging queues, and communication interruptions. Through dynamic weight calculation and compensation mechanisms, efficiency and fairness are balanced. Local tasks are quickly processed by edge nodes, and cross-domain tasks are transmitted lightweight through metadata, which not only reduces cloud load but also ensures the real-time performance of global scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is an internal framework diagram of a multimodal fusion AGV dynamic path planning and cluster scheduling system;

[0053] Figure 2 This is the internal framework diagram of the multimodal perception module;

[0054] Figure 3 This is the internal framework diagram of the dynamic path planning module;

[0055] Figure 4 This is the internal framework diagram of the cluster scheduling control module;

[0056] Figure 5 This is the internal framework diagram of the data conflict resolution module. DETAILED DESCRIPTION

[0057] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0058] Reference Figure 1 As shown in FIG, a multimodal fusion AGV dynamic path planning and cluster scheduling system includes:

[0059] Multimodal perception module: This module includes a lidar, binocular vision sensor, IMU inertial navigation unit, and UWB positioning base station for real-time environmental data collection. A cross-modal time synchronization unit is configured to align multi-source data through hardware-level timestamps. A feature-level fusion network based on the Transformer architecture performs joint semantic segmentation of laser point clouds and visual images, outputting a dynamic obstacle confidence map.

[0060] Dynamic Path Planning Module: This module uses an improved rolling window planning algorithm, embeds a spatiotemporal conflict prediction model, and uses LSTM to predict the trajectory of dynamic obstacles. It also uses a cost function based on adaptive weight adjustment to integrate path length, energy consumption, and dynamic obstacle avoidance priority to generate a global path without oscillation.

[0061] Cluster scheduling control module: Distributed task allocation architecture, based on an improved contract network protocol, enables local task negotiation through edge computing nodes; configures a dynamic priority preemption mechanism, and generates conflict resolution strategies through digital twin simulation rehearsals when path conflicts are detected;

[0062] Data conflict resolution module: Confidence-weighted obstacle recognition results from lidar and vision are applied. When the confidence difference exceeds a threshold, a multimodal recalibration process is triggered. A sliding window timing consistency check is used.

[0063] It should be noted that the closed-loop link of perception → planning → scheduling → resolution → perception is:

[0064] The dynamic obstacle confidence map output by the multimodal perception module provides real-time environmental semantic constraints for path planning; the oscillation-free path generated by the planning module is input into the scheduling module, which combines the digital twin preview results to generate a global conflict probability heat map; the data conflict resolution module reversely optimizes the perception fusion weights through sliding window verification to form a self-calibration feedback mechanism.

[0065] Dynamic environmental adaptability, scene-level response:

[0066]

[0067] Reference Figure 2 As shown, the multimodal perception module specifically includes:

[0068] Sensor acquisition unit:

[0069] LiDAR uses solid-state LiDAR to build a centimeter-level accurate geometric model of the environment and identify the outline and position of fixed obstacles. It also detects the real-time speed and direction of dynamic objects through multi-frame point cloud differentiation.

[0070] The binocular vision sensor uses a global shutter CMOS sensor to calculate obstacle depth information and identify texture features through a stereo matching algorithm. It also uses the YOLOv7 dynamic target detection model to classify obstacle types and output semantic labels.

[0071] IMU inertial navigation uses a six-axis MEMS inertial sensor to provide AGV body motion status data, filling the instantaneous perception blind spots of lidar and vision. It is integrated with UWB positioning data and realizes dead reckoning through Kalman filtering to maintain short-term positioning continuity in the event of sensor failure.

[0072] UWB positioning base station:

[0073] Adopting the TDoA positioning solution, multiple anchor base stations are deployed to form a positioning network, providing global coordinate constraints for laser SLAM;

[0074] Cross-modal time synchronization unit: Hardware-level synchronization uses FPGA programmable logic chips to distribute a unified clock source to each sensor; timestamps of lidar, vision, and IMU data are aligned; when the master clock fails, it automatically switches to PTP network synchronization mode; dynamic delay compensation is based on sensor transmission delay modeling and uses forward prediction interpolation to reconstruct the synchronized data stream;

[0075] Transformer feature-level fusion network: At the input layer, the laser point cloud is voxelized by VoxelNet to generate a dense 3D feature grid, and the visual image is processed through ResNet-50 to extract multi-scale feature maps; at the fusion layer, a cross-modal attention mechanism is used to calculate the feature similarity weights between point cloud voxels and image pixels, and a multi-head self-attention module is used to associate spatiotemporal context information; at the output layer, a dynamic obstacle confidence map is generated, marking the obstacle existence probability and motion vector of each grid cell;

[0076] Obstacle confidence map unit: The probability model includes static obstacle probability calculated by jointly calculating LiDAR point cloud density and visual semantic segmentation results, and dynamic obstacle probability introduced by LSTM trajectory prediction confidence; path planning prioritizes avoiding high-risk areas with confidence greater than a set threshold, and allows temporary crossing of controllable risk areas with confidence within a set range; cluster scheduling dynamically adjusts AGV task allocation based on the global confidence map.

[0077] It should be noted that LiDAR and binocular vision are complementary:

[0078] In reflective ground scenes, the point cloud stability of the LiDAR compensates for the failure of depth calculation caused by mirror reflection in vision (such as stainless steel floors in warehouses); vision enhances dark area textures through HDR imaging, assisting the LiDAR in completing contours in low-reflectivity areas (such as black rubber mats).

[0079] For dynamic target tracking, LiDAR detects motion speed through multi-frame point cloud differentiation; visual systems use YOLOv7 to classify target types (such as distinguishing between humans and AGVs) and combine semantic labels to optimize motion prediction models;

[0080] IMU-UWB fusion fault-tolerant navigation:

[0081] Failure response strategy: If the LiDAR fails, the IMU provides angular velocity integral trajectory, and UWB constrains global drift. If vision fails, the LiDAR point cloud semantics is completed (such as recognizing QR codes through point cloud density), and the IMU compensates for motion blur.

[0082] Kalman filtering is innovative, introducing UWB positioning residual as an adaptive factor of observation noise to dynamically adjust the filter gain.

[0083] Cross-modal time synchronization:

[0084] Master-slave switching mechanism: When the master clock (rubidium atomic clock) fails, the UWB base station with the highest signal strength is automatically elected as the PTP master node; the switching process delay is less than 10ms, and the time jitter is controlled within ±5μs (meeting ISO26262 ASIL-D requirements);

[0085] Industrial verification of delay compensation: when the AGV is moving at 2m / s, the forward predictive interpolation method reduces the laser-vision data alignment error from 12cm to 3cm (measured at the CATL battery factory).

[0086] Transformer fusion network:

[0087] Cross-modal attention, feature association strategy:

[0088] Geometric alignment, projecting image pixels into point cloud voxel space and calculating Euclidean distance weights;

[0089] Semantic alignment, through cosine similarity matching of feature vectors of ResNet-50 and VoxelNet (e.g., associating shelf edges with point cloud line features);

[0090] Occlusion scene processing: when vision is blocked, the attention weight automatically favors the lidar geometric features (weight > 0.8);

[0091] Dynamic obstacle vector prediction, motion modeling, capturing historical trajectories (10-frame temporal association) based on multi-head attention, outputting velocity vectors (accuracy ±0.2m / s) and position predictions for the next 0.5 seconds;

[0092] Obstacle confidence map:

[0093] Static obstacle probability calculation:

[0094] Where, P static is the probability of the existence of a static obstacle, and its value range is [0,1]. The higher the value, the greater the possibility of the existence of a static obstacle at that location. α is the weight coefficient, which is used to dynamically adjust the weights of the lidar data and visual semantic segmentation data in calculating the probability of static obstacles. In strong light, α = 0.7, which means that the lidar data has a greater weight; in low light, α = 0.3, which means that the visual semantic segmentation data has a greater weight. N lidar is the number of laser point clouds in the grid, which is the number of laser points detected by the lidar sensor in a specific grid; N max is the maximum number of laser point clouds in a grid, which is the maximum number of laser points that a lidar sensor can detect in a specific grid; IoU vis is the overlap ratio between the visual semantic segmentation and the grid, which is the intersection-over-union ratio between the visual semantic segmentation result and the grid area. Its value range is [0,1]. The higher the value, the higher the degree of overlap.

[0095] Dynamic obstacle confidence, LSTM trajectory prediction error backpropagation updates the confidence. When the prediction error is greater than 0.5m, the confidence attenuation coefficient β = 0.8.

[0096] Dynamic cluster scheduling strategy and risk-level response:

[0097]

[0098] Reference Figure 3 As shown, the dynamic path planning module specifically includes:

[0099] Dynamic window size adaptation unit: The basic window size is set as a multiple of the AGV braking distance and is dynamically expanded according to the real-time obstacle density. The local optimal escape strategy uses a simulated annealing algorithm to perturb the path candidate set. When the difference in the length of multiple consecutive planned paths is less than a predetermined ratio, a random offset angle is injected.

[0100] Spatiotemporal conflict prediction model:

[0101] The LSTM trajectory prediction network takes as input the historical trajectory of dynamic obstacles and the semantic features of the environment. The output prediction is to generate a multimodal trajectory probability distribution of obstacles in the short term, and output the highest confidence trajectory and safety margin.

[0102] The spatiotemporal occupancy grid modeling method discretizes the time dimension of the environmental grid map into intervals and marks the occupancy status of each grid in the short term in the future, including idle, possibly occupied, and confirmed occupied;

[0103] Conflict detection: When the planned path overlaps with the confirmed occupied area in the spatiotemporal occupancy grid, replanning is triggered; when it overlaps with the possible occupied area, an alternative detour path is generated;

[0104] Adaptive weight unit: adopts a multi-objective fusion mechanism and dynamic weight adjustment rules, combined with oscillation suppression technology, through path smoothing constraints, and introduces a curvature change rate penalty term into the cost function;

[0105] Oscillation-free global path generation: At the global level, a coarse-grained path is generated based on the A* algorithm, constraining the search space to permanently free areas in the spatiotemporal occupancy grid. At the local level, a fast-exploring random tree is used within a rolling window to refine the path, dynamically pruning invalid branches based on spatiotemporal conflict prediction results. Real-time replanning trigger conditions include hard triggers, which detect the presence of a confirmed occupied area within a predetermined range ahead of the path, and soft triggers, which initiate a Monte Carlo tree search to explore the optimal detour when the path crosses a potentially occupied area and no alternative routes are available.

[0106] It should be noted that the window expansion, safe braking distance association, basic window size W base =1.5×D brake , where D brake The maximum braking distance of the AGV (typical value 2m) ensures that the path can terminate within the window boundary during emergency braking;

[0107] Dynamic expansion formula W=W base ·(1+θ·ρ obs ), α is adjusted dynamically according to the scene:

[0108]

[0109] Where W is the dynamically adjusted window size, which is adjusted based on the basic window size and the complexity of the surrounding environment to adapt to different working scenarios; W base is the basic window size, determined by the maximum braking distance of the AGV, ensuring that the path can terminate within the window boundary during emergency braking; θ is the dynamic adjustment coefficient, dimensionless. It is dynamically adjusted according to the complexity of the scene and is used to determine the degree of influence of obstacle density on the window size; ρ obs is the obstacle density, which reflects the number or distribution of obstacles per unit area or volume; D brake The maximum braking distance of the AGV is the maximum distance the AGV needs to stop completely in an emergency, with a typical value of 2 meters.

[0110] Locally optimal escape strategy:

[0111] When the difference in the length of the planned paths for three consecutive times is less than 5%, that is, |L i -L i-1 | / L i-1 ,<0.05, random bias angle is injected (range ±15°, step size 5°), combined with the annealing coefficient T=T0·0.9 of the simulated annealing algorithm k (T0=100, k is the number of iterations), perturb the path candidate set to escape the local optimum.

[0112] LSTM trajectory prediction network optimization and input feature engineering:

[0113] Historical trajectory, 10 frames of trajectory (time step 1 second, sampling rate 10Hz), including coordinates, velocity, and acceleration;

[0114] Environmental semantics: channel types (sidewalks / AGV lanes) and traffic signs (speed limits / no-entry zones) extracted through visual semantic segmentation;

[0115] Multimodal output, trajectory probability distribution uses a Gaussian mixture model to output three candidate trajectories (mean ± 2σ boundary), and the trajectory with confidence > 90% is selected as the safety boundary;

[0116] Dynamic obstacle classification (such as humans, forklifts) affects trajectory prediction model parameters (such as the maximum acceleration of forklifts is set to 1m / s 2 , for humans: 0.5m / s 2 ).

[0117] Spatiotemporal occupancy grid modeling, time discretization strategy:

[0118] The time axis is at intervals of 0.1 seconds, covering the next 3 seconds (30 time slices), and each grid (0.1m×0.1m) is marked as:

[0119]

[0120] The collision detection is real-time, with the spatiotemporal occupancy grid updated every 0.2 seconds, and the planning cycle compressed to 50ms, meeting the forward-looking requirements of the AGV at 3m / s operation (detection distance ≥ 6m).

[0121] Dynamic weight adjustment rules and cost function design:

[0122]

[0123] Where, L pathis the path length, which represents the total length of the planned path. It reflects the geometric length of the path and is an important indicator in path planning. A shorter path is usually more efficient. consumption Energy consumption, which indicates the energy required to complete the task, reflects the energy consumption during task execution and is a resource consumption indicator that needs to be considered in the optimization process; aviodance is the obstacle avoidance probability, which indicates the probability of an obstacle on the path and reflects the safety of the path. A higher obstacle avoidance probability indicates a safer path and can effectively avoid collisions. λ is the regularization parameter, which is used to control the weight of the oscillation suppression term in the total cost and determines the strength of the oscillation suppression. It is the second derivative of curvature with respect to arc length, reflecting the rate of change of curvature and used to measure the smoothness of the path. The smaller the value, the smoother the path.

[0124] Path smoothing technology, Bezier curve fitting, using 5th order Bezier curve to generate C 2 Continuous path, control point spacing ≤ 0.3m, curvature change rate | dκ / ds | mandatory constraint ≤ 0.15m -2 ;

[0125] Vibration suppression effect: in the test of Foxconn precision assembly workshop, the vibration amplitude of the AGV robot arm end is

[0126] ±1.2mm is reduced to ±0.2mm, meeting the micron-level precision requirements for chip handling.

[0127] Oscillation-free global path generation:

[0128] Global layer optimization, the A* algorithm is constrained in the "permanent idle" area of ​​STOG, and the heuristic function introduces a turning penalty term h(n) = h 传统 +0.1·θ 转向 (θ is the cumulative steering angle), reducing the sharp bends in the global path;

[0129] Local layer refinement: When RRT* expands within the rolling window, branches that intersect with the STOG "possibly occupied" area are dynamically pruned, and the candidate path sampling density is increased to 200 points / ㎡;

[0130] Real-time replanning strategy, hard trigger, when detecting the "confirmed occupied" area within 5m ahead, start incremental RRT* (retain 90% of existing nodes), replanning time is less than 100ms;

[0131] Soft triggering: When crossing the "possibly occupied" area, MCTS evaluates three alternative paths and selects the one with the lowest comprehensive cost (path length × conflict probability).

[0132] Reference Figure 4 As shown, the cluster scheduling control module specifically includes:

[0133] Distributed Task Allocation Unit: An improved contract network protocol employs a dynamic priority weighting mechanism, introducing multi-dimensional weighting factors during task bidding. Topology-aware task partitioning divides the warehouse into multiple edge computing domains, with edge servers managing the real-time status of AGVs within each domain. Negotiation rules prioritize intra-domain tasks through local auctions, while cross-domain tasks are handled by a cloud coordinator, which only transmits metadata.

[0134] Dynamic priority preemption unit: Conflict detection and hierarchical response, including level 1: when there is a risk of preemption at a path intersection, the low-priority AGV is immediately frozen and the path is re-bid; level 2: when the charging pile queue times out, a high-priority AGV is forced to be inserted to compensate for the waiting time; level 3: when the communication interruption makes the status unknown, it switches to the LoRa emergency link and activates the degraded mode; preemption weight is dynamically calculated, and the weight factors include the economic value of the task, the health status of the AGV, and the historical conflict records;

[0135] Digital twin simulation preview unit:

[0136] Real-time twin model construction and data synchronization: the physical AGV uploads its position, speed, and task status to the twin platform, and the environment model is updated in real time through laser SLAM and visual semantic fusion.

[0137] Conflict resolution strategies are generated through Monte Carlo tree search optimization, including an expansion phase to generate candidate paths based on the spatiotemporal occupancy grid; a simulation phase to evaluate the conflict probability, energy consumption, and task delay cost of the paths; and a backtracking phase to select the path with the highest overall score as the resolution strategy.

[0138] The emergency strategy buffer has a preloaded strategy library: it contains plans for multiple types of standard conflict scenarios; dynamic strategy generation, for non-standard scenarios, uses generative AI to synthesize solutions in real time, and issues them for execution after simulation verification.

[0139] It should be noted that the dynamic priority weight mechanism has multi-dimensional weight factors:

[0140]

[0141] Bidding formula: Bid Score = 0.4V econ +0.3H AGV +0.2C path +0.1R history

[0142] Where V econ It is an economic indicator that measures the cost-effectiveness of the bidding scheme; H AGV C is the performance index of the automatic guided vehicle, AGV operation efficiency, reliability, and maintenance cost;path It is the cost or complexity index of path planning, which measures the efficiency and economy of path planning in bidding schemes; R history Penalty points are recorded for historical conflicts, reflecting conflicts between bidders or proposals in the past. Each time a conflict occurs, the penalty points are reduced by 0.05.

[0143] Topology-aware task partitioning:

[0144] Warehouse zoning strategy, edge computing domain division, based on the UWB base station coverage range (30m radius) and combined with shelf layout to generate a cellular topology;

[0145] Intra-domain resource management: edge servers monitor AGV power, task queues, and path load rates in real time, dynamically adjusting the maximum capacity of AGVs within the domain (typically 15 units / domain);

[0146] For cross-domain task coordination, the cloud only transmits task metadata, such as target coordinates and cargo type, avoiding the transmission of original data; cross-domain path planning ensures conflict-free handover areas through spatiotemporal occupancy grid alignment.

[0147] Dynamic priority preemption unit:

[0148] Level 1 (intersection preemption): A freezing mechanism is implemented. Low-priority AGVs remain in the braking state (wheel motor reverse torque lock), while high-priority AGVs obtain exclusive right of way for 3 seconds. Re-bidding rules require frozen AGVs to resubmit a bid score that includes a "waiting time penalty" (the VeconVecon weight in the formula is reduced to 0.3).

[0149] Level 2 (charging pile preemption): Insertion threshold: When the queue time is greater than 5 minutes and the battery level of the AGV to be charged is less than 10%, forced insertion is triggered; compensation strategy: the interrupted AGV receives points compensation (1 point = 10 seconds of charging time), and the accumulated points can be exchanged for priority task permissions;

[0150] Level 3 (communication interruption emergency): LoRa degraded mode, switching to the 915MHz frequency band, transmitting AGV basic posture data (2Hz refresh rate), and extending the coverage radius to 500m; local decision logic, relying on IMU + visual SLAM to generate conservative paths (speed limited to 1m / s) to avoid sensitive areas of dynamic obstacles.

[0151] Digital twin simulation preview unit:

[0152] Accurate mapping of real-time twin models, data synchronization mechanism, AGV posture data uploaded at 10Hz frequency, and environmental models integrated with laser SLAM and visual semantics through edge nodes;

[0153] Fault-tolerant design: when network delay is greater than 50ms, Kalman filter prediction is enabled to fill data gaps.

[0154] To ensure physical-virtual consistency, twin calibration (ICP point cloud registration algorithm) is performed every 5 minutes to ensure that the virtual model offset is less than 3cm; dynamic environmental elements, such as moving shelves and temporary obstacles, trigger model updates through RFID tags.

[0155] Reference Figure 5 As shown, the data conflict resolution module specifically includes:

[0156] Multimodal weight dynamic allocation unit: Weight calculation, the weights of the lidar and vision sensors are dynamically adjusted based on the real-time measurement variance; scene adaptive adjustment, when strong light illuminates the reflective ground, the weight is biased towards lidar; when low light illuminates the textured area, the weight is biased towards vision; when rainy and foggy weather occurs, the weight is biased towards lidar and UWB fusion; threshold dynamic setting, dynamically adjust the confidence difference threshold according to the complexity of the environment;

[0157] Multimodal Recalibration Unit:

[0158] During the trigger phase, hard trigger conditions include: the position deviation between the lidar and vision for the same obstacle is greater than a predetermined range; the confidence difference of multiple consecutive frames of data exceeds a threshold; soft trigger conditions include: the AGV's motion state is inconsistent with the perception result for several seconds; the edge computing node detects a decline in sensor health;

[0159] During calibration, the data is frozen, the current frame data output is paused, and the historical frame data is retained. Spatiotemporal alignment is performed, aligning the multimodal data streams through a dynamic time warping algorithm and using IMU data to compensate for motion distortion. Parameters are reset, with the visual exposure time and lidar scanning frequency adaptively adjusted, and the external parameter matrix is ​​calibrated online based on the semantic features of the environment. Cross-validation is performed, using UWB positioning data to verify the obstacle coordinate error after calibration. If the verification fails, the system switches to degraded mode.

[0160] Sliding window timing consistency check unit: dynamic window management, the window size is divided into static environment and dynamic environment, and the verification rule is to test the timing smoothness of the multimodal data in the window; abnormal data processing, short-term anomalies, single-frame data conflicts, the Kalman filter prediction value is used to replace the abnormal data to maintain path continuity; long-term anomalies, continuous multi-frame conflicts, trigger the sensor self-test program, and report to the edge node to start the backup perception source.

[0161] It should be noted that the multimodal recalibration unit:

[0162] Trigger conditions and graded responses: Hard trigger conditions: The position deviation between the laser and vision for the same obstacle is greater than 0.5m (twice the AGV safety radius); the confidence difference for 5 consecutive frames is greater than the threshold, which is determined to be a systematic perception deviation;

[0163] Soft trigger conditions: the difference between the AGV body motion state (IMU angular velocity integral) and the visual optical flow estimation lasts for more than 3 seconds; the edge node detects the health of the sensor (such as the laser transmitter temperature is greater than 70°C) to trigger an early warning.

[0164] Calibration is performed, data freezing and spatiotemporal alignment are performed, freezing the current frame data and retaining the most recent 10 frames of historical data to build spatiotemporal context; dynamic time warping algorithms align multimodal data streams, and IMUs compensate for motion distortion;

[0165] Parameter reset and cross validation:

[0166]

[0167] In summary, the advantages of the present invention are:

[0168] Through the deep complementarity of lidar, vision, IMU, and UWB, the perception limitations of traditional solutions in complex scenarios such as reflections, low light, rain, and fog are overcome. Lidar ensures geometric modeling stability, vision enhances semantic understanding, and IMU-UWB fusion achieves positioning continuity. Intelligent degradation in the event of sensor failure ensures system robustness under extreme conditions. The spatiotemporal conflict prediction model, combined with a hierarchical planning architecture, predicts obstacle trajectories in advance and generates multimodal path solutions to avoid sudden risks. Dynamic window optimization and oscillation suppression technology balance path safety and motion smoothness, significantly improving traffic efficiency in narrow passages and mixed human and vehicle traffic scenarios.

[0169] The distributed edge collaborative architecture realizes intelligent task allocation, and the improved contract network protocol combines multi-dimensional weight factors to reduce resource competition and idle driving energy consumption; dynamic priority preemption rules are linked with digital twin rehearsals, and hierarchical responses are made to conflicts such as path intersections and charging queues to ensure priority execution of high-value tasks and system continuity; dynamic weight allocation adjusts sensor confidence in real time according to environmental complexity, and multi-modal recalibration under hard / soft trigger conditions can quickly repair perception deviations; sliding window timing verification and abnormal data replacement mechanism maintain path continuity and avoid frequent replanning caused by transient noise.

[0170] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-modal fusion AGV dynamic path planning and cluster scheduling system, characterized by: include: Multimodal perception module: including lidar, binocular vision sensor, IMU inertial navigation unit, and UWB positioning base station, used to collect environmental data in real time; A cross-modal time synchronization unit is configured to align multi-source data through hardware-level timestamps. A feature-level fusion network based on the Transformer architecture performs joint semantic segmentation of laser point clouds and visual images, and outputs a dynamic obstacle confidence map. Dynamic Path Planning Module: This module uses an improved rolling window planning algorithm, embeds a spatiotemporal conflict prediction model, and uses LSTM to predict the trajectory of dynamic obstacles. It also uses a cost function based on adaptive weight adjustment to integrate path length, energy consumption, and dynamic obstacle avoidance priority to generate a global path without oscillation. Cluster scheduling control module: Distributed task allocation architecture, based on an improved contract network protocol, enables local task negotiation through edge computing nodes; configures a dynamic priority preemption mechanism, and generates conflict resolution strategies through digital twin simulation rehearsals when path conflicts are detected; Data conflict resolution module: Confidence-weighted obstacle recognition results from lidar and vision are applied. When the confidence difference exceeds a threshold, a multimodal recalibration process is triggered. A sliding window timing consistency check is used.

2. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 1 is characterized in that: The multimodal perception module specifically includes: Sensor acquisition unit: LiDAR uses solid-state LiDAR to build a centimeter-level accurate geometric model of the environment and identify the outline and position of fixed obstacles. It also detects the real-time speed and direction of dynamic objects through multi-frame point cloud differentiation. The binocular vision sensor uses a global shutter CMOS sensor to calculate obstacle depth information and identify texture features through a stereo matching algorithm. It also uses the YOLOv7 dynamic target detection model to classify obstacle types and output semantic labels. IMU inertial navigation uses a six-axis MEMS inertial sensor to provide AGV body motion status data, filling the instantaneous perception blind spots of lidar and vision. It is integrated with UWB positioning data and realizes dead reckoning through Kalman filtering to maintain short-term positioning continuity in the event of sensor failure. UWB positioning base station: The TDoA positioning solution is adopted, and multiple anchor base stations are deployed to form a positioning network to provide global coordinate constraints for laser SLAM.

3. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 2 is characterized in that: The multimodal perception module specifically includes: Cross-modal time synchronization unit: Hardware-level synchronization uses FPGA programmable logic chips to distribute a unified clock source to each sensor; timestamps of lidar, vision, and IMU data are aligned; when the master clock fails, it automatically switches to PTP network synchronization mode; dynamic delay compensation is based on sensor transmission delay modeling and uses forward prediction interpolation to reconstruct the synchronized data stream; Transformer feature-level fusion network: In the input layer, the laser point cloud is voxelized by VoxelNet to generate a dense 3D feature grid, and the visual image is processed through ResNet-50 to extract multi-scale feature maps; in the fusion layer, a cross-modal attention mechanism is used to calculate the feature similarity weights between point cloud voxels and image pixels, and the spatiotemporal context information is associated through a multi-head self-attention module; in the output layer, a dynamic obstacle confidence map is generated, marking the obstacle existence probability and motion vector of each grid cell.

4. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 3 is characterized in that: The multimodal perception module specifically includes: Obstacle confidence map unit: The probability model includes static obstacle probability calculated by jointly calculating LiDAR point cloud density and visual semantic segmentation results, and dynamic obstacle probability introduced by LSTM trajectory prediction confidence; path planning prioritizes avoiding high-risk areas with confidence greater than a set threshold, and allows temporary crossing of controllable risk areas with confidence within a set range; cluster scheduling dynamically adjusts AGV task allocation based on the global confidence map.

5. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 4 is characterized in that: The dynamic path planning module specifically includes: Dynamic window size adaptation unit: The basic window size is set as a multiple of the AGV braking distance and is dynamically expanded according to the real-time obstacle density. The local optimal escape strategy uses a simulated annealing algorithm to perturb the path candidate set. When the difference in the length of multiple consecutive planned paths is less than a predetermined ratio, a random offset angle is injected. Spatiotemporal conflict prediction model: The LSTM trajectory prediction network takes as input the historical trajectory of dynamic obstacles and the semantic features of the environment. The output prediction is to generate a multimodal trajectory probability distribution of obstacles in the short term, and output the highest confidence trajectory and safety margin. The spatiotemporal occupancy grid modeling method discretizes the time dimension of the environmental grid map into intervals and marks the occupancy status of each grid in the short term in the future, including idle, possibly occupied, and confirmed occupied; Conflict detection: When the planned path overlaps with the confirmed occupied area in the spatiotemporal occupancy grid, replanning is triggered; when it overlaps with the possible occupied area, an alternative detour path is generated.

6. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 5 is characterized in that: The dynamic path planning module specifically includes: Adaptive weight unit: adopts a multi-objective fusion mechanism and dynamic weight adjustment rules, combined with oscillation suppression technology, through path smoothing constraints, and introduces a curvature change rate penalty term into the cost function; Oscillation-free global path generation: At the global level, a coarse-grained path is generated based on the A* algorithm, constraining the search space to permanently free areas in the spatiotemporal occupancy grid. At the local level, a fast-exploring random tree is used within a rolling window to refine the path, dynamically pruning invalid branches based on spatiotemporal conflict prediction results. Real-time replanning trigger conditions include hard triggers, which detect the presence of a confirmed occupied area within a predetermined range ahead of the path, and soft triggers, which initiate a Monte Carlo tree search to explore the optimal detour when the path crosses a potentially occupied area and no alternative routes are available.

7. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 6 is characterized in that: The cluster scheduling control module specifically includes: Distributed Task Allocation Unit: An improved contract network protocol employs a dynamic priority weighting mechanism, introducing multi-dimensional weighting factors during task bidding. Topology-aware task partitioning divides the warehouse into multiple edge computing domains, with edge servers managing the real-time status of AGVs within each domain. Negotiation rules prioritize intra-domain tasks through local auctions, while cross-domain tasks are handled by a cloud coordinator, which only transmits metadata. Dynamic priority preemption unit: Conflict detection and hierarchical response, including level 1: when there is a risk of preemption at a path intersection, the low-priority AGV is immediately frozen and the path is re-bid; level 2: when the charging pile queue times out, a high-priority AGV is forcibly inserted to compensate for the waiting time; level 3: when the communication interruption makes the status unknown, it switches to the LoRa emergency link and enables the degraded mode; the preemption weight is dynamically calculated, and the weight factors include the economic value of the task, the health status of the AGV, and the historical conflict records.

8. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 7 is characterized in that: The cluster scheduling control module specifically includes: Digital twin simulation preview unit: Real-time twin model construction and data synchronization: the physical AGV uploads its position, speed, and task status to the twin platform, and the environment model is updated in real time through laser SLAM and visual semantic fusion. Conflict resolution strategies are generated through Monte Carlo tree search optimization, including an expansion phase to generate candidate paths based on the spatiotemporal occupancy grid; a simulation phase to evaluate the conflict probability, energy consumption, and task delay cost of the paths; and a backtracking phase to select the path with the highest overall score as the resolution strategy. The emergency strategy buffer has a preloaded strategy library: it contains plans for multiple types of standard conflict scenarios; dynamic strategy generation, for non-standard scenarios, uses generative AI to synthesize solutions in real time, and issues them for execution after simulation verification.

9. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 8 is characterized in that: The data conflict resolution module specifically includes: Multimodal weight dynamic allocation unit: Weight calculation, the weights of lidar and vision sensors are dynamically adjusted based on real-time measurement variance; scene adaptive adjustment, when strong light illuminates the reflective ground, the weight is biased towards lidar; when low light illuminates the texture-rich area, the weight is biased towards vision; when it is rainy and foggy, the weight is biased towards lidar and UWB fusion; threshold dynamic setting, dynamically adjust the confidence difference threshold according to the complexity of the environment.

10. The multimodal fusion AGV dynamic path planning and cluster scheduling system according to claim 9, characterized in that: The data conflict resolution module specifically includes: Multimodal Recalibration Unit: During the trigger phase, hard trigger conditions include: the position deviation between the lidar and vision for the same obstacle is greater than a predetermined range; the confidence difference of multiple consecutive frames of data exceeds a threshold; soft trigger conditions include: the AGV's motion state is inconsistent with the perception result for several seconds; the edge computing node detects a decline in sensor health; During calibration, the data is frozen, the current frame data output is paused, and the historical frame data is retained. Spatiotemporal alignment is performed, aligning the multimodal data streams through a dynamic time warping algorithm and using IMU data to compensate for motion distortion. Parameters are reset, with the visual exposure time and lidar scanning frequency adaptively adjusted, and the external parameter matrix is ​​calibrated online based on the semantic features of the environment. Cross-validation is performed, using UWB positioning data to verify the obstacle coordinate error after calibration. If the verification fails, the system switches to degraded mode. Sliding window timing consistency check unit: dynamic window management, the window size is divided into static environment and dynamic environment, and the verification rule is to test the timing smoothness of the multimodal data in the window; abnormal data processing, short-term anomalies, single-frame data conflicts, the Kalman filter prediction value is used to replace the abnormal data to maintain path continuity; long-term anomalies, continuous multi-frame conflicts, trigger the sensor self-test program, and report to the edge node to start the backup perception source.

Images