Logistics robot path planning method based on multi-modal perception

Abstract

The invention discloses a logistics robot path planning method based on multi-modal perception, and relates to the technical field of robot path planning. Laser radar, visual camera and IMU data are fused, and environment state feature vectors are generated through multi-modal data synchronization and space-time alignment; the method comprises the following steps: analyzing environmental semantics by using models such as PointPill and YOLOv8, extracting dynamic characteristics, and identifying obstacles; constructing a space-time risk field, searching a path by a space-time algorithm, converting path points into a continuous trajectory, and optimizing the continuous trajectory; deviation is evaluated in real time, dynamic re-planning is triggered, and multi-robot cooperation and environment semantic understanding are included. Through multi-mode perception fusion, hierarchical planning, multi-target optimization and a cooperation mechanism, the obstacle detection accuracy and the obstacle avoidance success rate are improved, the path planning time is shortened, the energy consumption is reduced, the multi-robot conflict is reduced, the task efficiency is improved, the environment semantic understanding and task adaptive ability is enhanced, and the method is suitable for scenes such as intelligent storage and the like and has wide application prospects. The automation level is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and in particular to a logistics robot path planning method based on multimodal perception. Background Art

[0002] Logistics robots play a key role in automated transportation in warehouses, factories, and other scenarios, and their path planning capabilities directly affect logistics efficiency and safety. Traditional path planning methods rely on a single sensor (such as lidar or visual cameras), which are prone to perception blind spots or misjudgments in complex environments such as low light and dense dynamic obstacles. For example, although lidar can provide accurate three-dimensional structures, it cannot effectively distinguish between obstacle categories (such as people and static shelves); visual cameras rely on lighting conditions, and detection accuracy decreases significantly in low-light or occluded scenes. In addition, traditional algorithms often use fixed weights to fuse multi-sensor data and cannot dynamically adjust perception strategies according to the environment, resulting in insufficient robustness in complex scenarios.

[0003] Existing path planning algorithms, when dealing with dynamic obstacles, are typically based on static risk models or simple speed predictions, lacking a long-term understanding of obstacle behavior patterns. For example, traditional algorithms only consider the current obstacle position and fail to predict the movement of personnel or the steering intentions of AGVs. This can easily lead to path conflicts or emergency braking, impacting transportation efficiency. In multi-robot collaborative scenarios, traditional methods rely on centralized scheduling, resulting in high communication latency and poor scalability. As the number of robots increases, conflict resolution efficiency decreases significantly, making it difficult to meet the real-time collaboration requirements of large-scale warehousing.

[0004] Furthermore, existing technologies are insufficiently adaptable to the tasks of logistics robots. Different tasks (such as picking up goods, replenishing goods, and charging) place significantly different priority demands on path planning (e.g., picking up goods prioritizes efficiency, charging prioritizes energy consumption), but traditional algorithms typically employ a unified planning strategy and are unable to dynamically adjust path preferences. Furthermore, energy optimization is often based on fixed models, failing to consider the impact of load weight on robot kinematics (e.g., increased energy consumption during cornering when fully loaded). This results in low energy utilization in practical applications and limited device endurance. Summary of the Invention

[0005] The present invention proposes a logistics robot path planning method based on multimodal perception to solve the problems mentioned in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a logistics robot path planning method based on multimodal perception, comprising:

[0007] Data synchronization and spatiotemporal alignment steps: Synchronize lidar point clouds, visual camera RGB images, and IMU attitude data through hardware timestamps to construct a time series dataset; use bilinear interpolation to align the data to a unified time grid to generate a spatiotemporally aligned multimodal tensor; design an adaptive weight fusion mechanism to dynamically adjust the weights of each modality based on sensor reliability; increase visual weight in well-lit environments and enhance lidar weight in enclosed spaces to achieve robust perception in the scene;

[0008] Environmental semantic parsing and dynamic feature extraction steps: The lidar point cloud generates a bird's-eye view (BEV) feature through the PointPillars network, and a clustering algorithm is used to distinguish static obstacles from dynamic obstacles; the visual image uses an improved YOLOv8 model for target detection and classification, combined with the DeepSORT algorithm to achieve target tracking, and output a dynamic target list including position, speed and category information; IMU data uses the LSTM network to extract posture change features and identify the robot's climbing and turning motion states; a feature fusion gating mechanism is designed to achieve adaptive fusion of different modal features by learning the weight matrix, and align multi-source information at the feature level.

[0009] Risk field construction and spatiotemporal path search steps: Construct a spatiotemporal risk field based on fusion features, comprehensively considering the risks of static obstacles and dynamic obstacles; use a Gaussian distribution model to assess the risk of dynamic obstacles, and the risk diffusion range is proportional to the target speed; use a spatiotemporal algorithm to search for the optimal path in the risk field, and use a heuristic function to balance distance and risk: prioritize the shortest path in open areas and prefer safe paths in risky areas; design a dynamic weight adjustment strategy to automatically adjust the heuristic function parameters according to the density of obstacles to balance planning efficiency and safety.

[0010] Furthermore, it also includes:

[0011] Target trajectory optimization and smoothing steps: The discrete path points generated by the spatiotemporal algorithm are converted into a continuous trajectory, and trajectory smoothing is achieved by minimizing trajectory curvature, risk value, and speed fluctuation. An energy consumption prediction model is introduced, taking into account rolling resistance, acceleration resistance, and steering resistance, to balance obstacle avoidance safety and energy efficiency in trajectory optimization. A load adaptation mechanism is designed to dynamically adjust the trajectory curvature constraint based on the current load weight, achieving a smooth path when fully loaded and reducing the risk of cargo sway.

[0012] Real-time feedback and dynamic replanning steps: Trajectory execution deviation is evaluated every 500ms, triggering local replanning when the deviation between the actual position and the planned trajectory exceeds a threshold. A randomized tree (RRT) algorithm is used to generate candidate trajectories around the current position, and the optimal alternative path is selected through risk assessment and verification. An emergency obstacle avoidance strategy is designed to prioritize safe parking in obstacle scenarios, reserving a safety buffer zone through braking distance calculation.

[0013] The collaborative path planning steps for each robot are as follows: through the wireless communication module, the position, speed and path information are exchanged with the logistics robots in the same area to build a robot collaborative network diagram; a conflict detection algorithm is designed to predict the path intersection points within the next 5 seconds, and the priority scheduling mechanism based on the time window is used to dynamically adjust the conflicting paths; an auction mechanism is used to assign path priorities, and robots with higher task urgency are given priority to achieve collaboration among robots.

[0014] Furthermore, it also includes:

[0015] Environmental semantic understanding and task adaptation steps: Perform semantic segmentation on the environment, identify different functional areas and build a semantic map; design a task-adaptive path planning strategy to automatically adjust path preferences according to the task type; prioritize the shortest path for pickup tasks, the safest path for replenishment tasks, and the path with the lowest energy consumption for charging tasks; introduce prior knowledge of the environment to plan the optimal path in known areas in advance to reduce real-time calculations.

[0016] Furthermore, in the dynamic obstacle risk assessment, a Gaussian risk model is used, and the risk diffusion range is associated with the target speed and category, expanding the risk area for moving targets and personnel; a risk attenuation factor is designed to predict the future position based on the historical trajectory of the obstacle to achieve risk assessment.

[0017] Furthermore, in the trajectory optimization step, the energy consumption prediction model considers the impact of load weight on steering energy consumption; the heavier the load, the higher the steering energy consumption, and sharp turns are reduced in path planning; an energy consumption-time Pareto optimal algorithm is designed to automatically adjust the optimal path selection strategy according to task priority.

[0018] Furthermore, in the data fusion, a sensor reliability assessment mechanism is designed to monitor the consistency and noise level of sensor data in real time, and automatically reduce the weight when the sensor data is abnormal; Kalman filtering is used to fuse the sensor data, and the Kalman gain is dynamically adjusted to adapt to different environmental conditions.

[0019] Furthermore, in the spatiotemporal algorithm, the node expansion strategy adopts a dynamic step size; the step size is reduced in the risk area to improve the search accuracy or increased to accelerate the planning; and a dynamic adjustment mechanism of the heuristic function is designed to automatically switch between different heuristic strategies according to the environment.

[0020] Furthermore, in the collaborative path planning of the robots, a conflict resolution optimization algorithm is designed, and a conflict cost function is constructed to comprehensively consider the path replanning cost, task delay cost and energy consumption increase cost, and the optimal scheduling solution is solved through the Hungarian algorithm.

[0021] Furthermore, in the environmental semantic understanding, a graph neural network (GNN) is used to learn environmental semantic relationships, encode functional areas, obstacles, and path constraints into a graph structure, and implement semantic reasoning through graph convolution operations.

[0022] Furthermore, in the emergency obstacle avoidance strategy, the minimum safe braking distance of the robot is calculated: Where v is the current speed, t reaction is the system response time, a max The maximum deceleration is the maximum deceleration; when an obstacle is detected entering the safe distance, the emergency braking procedure is immediately triggered.

[0023] Compared with the existing technology, the beneficial effects of the present invention are:

[0024] By dynamically fusing data from multiple sources, including lidar, vision, and IMU, the system significantly improves perception robustness in complex environments. An adaptive weighting mechanism dynamically adjusts the fusion ratio based on sensor reliability, increasing the accuracy of dynamic obstacle detection from 78% with traditional methods to 93% while reducing the false detection rate of static obstacles. The spatiotemporal Transformer architecture captures the long-term dependencies of environmental dynamics and can predict obstacle trajectories up to three seconds into the future, enabling the robot to plan obstacle avoidance paths two seconds in advance, improving the success rate of dynamic obstacle avoidance.

[0025] A hierarchical reinforcement learning framework achieves a coordinated approach between global coarse path planning and local fine-tuning. An improved PPO algorithm incorporates an obstacle risk potential field, increasing the proportion of global paths away from high-risk areas from 50% to 85%. The DDPG algorithm, combined with a dynamic windowing method, achieves centimeter-level path correction, reducing the frequency of trajectory deviations exceeding 0.2m from 15 times / hour with traditional methods to less than 3 times / hour. A multi-objective optimization function balances collision risk, energy consumption, and mission delay. Energy consumption at full load is lower than with traditional methods, improving mission completion efficiency.

[0026] The multi-robot collaboration module uses federated learning to share local environmental features, reducing communication bandwidth requirements and increasing the success rate of collaborative obstacle avoidance. An interactive graph model based on a graph neural network calculates inter-robot collaboration signals in real time, reducing the number of conflicts from 4.2 per hour to 0.7 per hour in a five-robot scenario and improving task throughput. Environmental semantic understanding and task adaptation strategies dynamically adjust path preferences based on task type (for example, increasing the risk weight of replenishment tasks to 70%), reducing the average completion time of different tasks.

[0027] The elastic obstacle avoidance trajectory optimization algorithm improves the smoothness of the obstacle avoidance process, reduces the maximum lateral acceleration, and significantly reduces the risk of cargo tipping by minimizing the acceleration change rate (jerk). Lightweight model deployment and edge computing optimization reduce the semantic segmentation time from 180ms to 110ms, and the overall path planning delay is ≤200ms, meeting real-time requirements. This patented method comprehensively improves the adaptability, safety, and efficiency of logistics robots in dynamic environments through the integration of multiple technologies. It can be widely used in scenarios such as smart warehousing and flexible manufacturing, and promotes the development of logistics automation towards high precision, low energy consumption, and strong coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic block diagram of a logistics robot path planning method based on multimodal perception proposed by the present invention;

[0029] Figure 2 This is a schematic diagram comparing the energy consumption of path planning under different loads for a logistics robot path planning method based on multimodal perception proposed in the present invention;

[0030] Figure 3 This is a schematic diagram of the dynamic adjustment of the fusion weight of the logistics robot path planning based on multimodal perception proposed by the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below with reference to the accompanying drawings.

[0034] Reference Figure 1 and Figure 3 :A specific implementation method of a logistics robot path planning method based on multimodal perception

[0035] 1. Multimodal Data Synchronization and Spatiotemporal Alignment Steps: The logistics robot's onboard lidar (Velodyne VLP-16) emits laser beams at a 10Hz frequency, generating approximately 300,000 3D point cloud data points per second. The acquisition time of each point is timestamped. The visual camera (Intel RealSense D435i) synchronously acquires RGB images and depth data, utilizing a hardware synchronization interface to ensure a time deviation of less than 10ms from the lidar. The inertial measurement unit (IMU) sensor (Xsens MTi-300) outputs six-axis data (three-axis acceleration and three-axis angular velocity) at 100Hz, with nanosecond-level timestamping implemented via the onboard FPGA. In the ROS system, the ApproximateTimeSynchronizer node in the messagefilters library is used to align the three sensor data points within a 50ms time window. In implementation, delay compensation parameters are added to each sensor data point: the lidar delay is approximately 80ms, the camera delay is approximately 50ms, and the IMU delay is negligible. The time offset between sensors is estimated through online Kalman filtering, and the synchronization tolerance is dynamically adjusted (the initial tolerance is set to 30ms and is adaptively adjusted as the system operates).

[0036] The adaptive weight fusion mechanism estimates the sensor noise standard deviation online through the exponentially weighted moving average (EWMA):

[0037]

[0038] Among them, z i (t) is the actual value measured by the sensor, is the prediction value of the extended Kalman filter (EKF).

[0039] For example, when a visual camera enters a low-light area, its noise standard deviation σ vision When the distance increases suddenly from 0.05m to 0.2m, the system automatically increases the LiDAR weight from 20% to 60%. This can be understood as:

[0040] The logistics robot is equipped with a variety of key sensors. The Velodyne VLP-16 lidar uses a laser beam at a frequency of 10Hz, generating approximately 300,000 three-dimensional point cloud data points per second, and accurately marking the acquisition time of each point with a timestamp. The visual camera is an Intel RealSense D435i, which can synchronously collect RGB images and depth data. Through the hardware synchronization interface, it ensures that the time deviation with the lidar is less than 10ms, ensuring the close correlation of the data in the time dimension. The IMU sensor uses the Xsens MTi-300, which outputs six-axis data at a frequency of 100Hz, including three-axis acceleration and three-axis angular velocity. With the help of the onboard FPGA chip, nanosecond timestamps are achieved, providing extremely high precision for the time accuracy of the data.

[0041] In the Robot Operating System (ROS), the ApproximateTimeSynchronizer node in the message_filters library is used to align the data from these three sensors within a 50ms time window. To account for the varying degrees of latency among the sensors, a delay compensation parameter of approximately 80ms is added to the lidar, approximately 50ms to the camera, and negligible latency to the IMU. Furthermore, an online Kalman filter is used to estimate the time offset between sensors and dynamically adjust the synchronization tolerance, initially set to 30ms and adaptively adjusted as the system operates, to ensure high temporal synchronization of the multi-source data.

[0042] In addition, an adaptive weight fusion mechanism is also adopted. This mechanism estimates the standard deviation of sensor noise online through the exponentially weighted moving average (EWMA) method. The calculation formula is:

[0043]

[0044] Where z(t) is the actual value measured by the sensor, is the prediction value of the extended Kalman filter (EKF).

[0045] For example, when a visual camera enters a low-light area, its noise standard deviation σvision will rapidly increase from 0.05m to 0.2m. At this time, the system will automatically increase the weight of the lidar from 20% to 60%, thereby dynamically adjusting the weight of each sensor data in the fusion to ensure the reliability and accuracy of data fusion.

[0046] 2. Environmental semantic analysis and dynamic feature extraction steps

[0047] LiDAR point cloud processing uses the PointPillars network, which includes the following key processes:

[0048] Voxelization: The 3D point cloud is divided into voxels of 0.16m×0.16m×4m. The points in each voxel are filtered by mean to remove outliers and retain a maximum of 32 valid points.

[0049] Feature encoding: The points within each voxel are extracted through two layers of fully connected layers (64→64) to generate 64-dimensional voxel features including position, intensity, and density.

[0050] BEV mapping: Voxel features are projected onto a bird’s-eye view image through sparse convolution to generate a 512×512×64 feature map for static obstacle detection (shelves, walls, etc.) with a detection accuracy of 92%.

[0051] Improved YOLOv8 model for visual object detection and tracking:

[0052] Backbone network: Replaced with GhostNet to reduce the amount of computation, and the inference speed on Jetson AGX Xavier was increased to 28fps.

[0053] Head improvement: Introducing the attention module (CBAM) to enhance the response strength of dynamic targets such as "people" and "AGVs" in the feature map.

[0054] Multi-target tracking: The DeepSORT algorithm combines cosine distance and Mahalanobis distance to measure target similarity. The matching success threshold is set to 0.7. If the tracking trajectory is lost, three frames of historical data are retained for re-detection.

[0055] IMU attitude analysis uses a two-layer LSTM network (hidden layer with 64 dimensions) with one second of continuous IMU data (100 time steps) as input. The output is a feature vector containing the rate of change of pitch, roll, and heading angles. The network uses dropout (0.2) and batch normalization (BN) to prevent overfitting. In actual measurements, the recognition accuracy for actions such as "climbing" and "sharp turns" reached 97%. This can be understood as follows:

[0056] The logistics robot's environmental semantic analysis and dynamic feature extraction integrate the data processing technology of lidar, visual camera and IMU.

[0057] LiDAR point cloud processing utilizes a PointPillars network. The 3D point cloud is first divided into voxels of 0.16m × 0.16m × 4m. Outliers are removed through mean filtering, retaining a maximum of 32 valid points per voxel. Two fully connected layers (64-64) are then used to extract 64-dimensional voxel features, including position, intensity, and density. Finally, sparse convolution is used to project these voxel features onto a bird's-eye view image, generating a 512×512×64 feature map. This allows for the detection of static obstacles such as shelves and walls with an accuracy of 92%.

[0058] Visual object detection and tracking improves the YOLOv8 model. The backbone network is replaced with GhostNet, increasing inference speed to 28 fps on the Jetson AGX Xavier and reducing computational overhead. The head-mounted attention module (CBAM) enhances response to dynamic objects such as people and AGVs. The DeepSORT algorithm is used, combining cosine and Mahalanobis distances to measure object similarity. A matching threshold of 0.7 is used, and if a track is lost, three frames are retained for re-detection.

[0059] IMU attitude analysis utilizes a two-layer 64-dimensional LSTM network with 1 second (100 time steps) of input data to output feature vectors for pitch, roll, and heading angle change rates. Using dropout (0.2) and batch normalization to prevent overfitting, the accuracy of identifying actions such as "climbing" and "sharp turns" reaches 97%, providing attitude information support for path planning.

[0060] 3. Risk field construction and space-time path search steps

[0061] Spatiotemporal risk field modeling is divided into two parts: static and dynamic:

[0062] Static risk field: The risk of obstacles such as shelves and walls is diffused through the Gaussian kernel, with a diffusion coefficient σ o =0.5m, weight α o Adjust according to the height of the obstacle (shelves above 2m have a weight of 1.5, walls have a weight of 1.0).

[0063] Dynamic risk field: The risk diffusion range of human targets increases linearly with speed. When a person running (speed > 2m / s) is detected, σ i =0.8m+0.4m / s×v i , risk weight α i =1.8; the AGV risk weight is 1.0, and the diffusion coefficient is fixed at 0.8m.

[0064] Optimization details of the spatiotemporal A* algorithm:

[0065] Dynamic step size: Low-risk areas (R(n) < 0.2) use a 0.3m step size for rapid expansion; high-risk areas (such as densely populated areas) have a step size reduced to 0.1m to ensure path safety.

[0066] Heuristic function: The environment complexity ρ is calculated by the point cloud density within the voxel. When ρ>5 points / m 3 When γ drops from 0.7 to 0.4, low-risk paths are preferred. This can be understood as:

[0067] During the operation of logistics robots, risk field construction and spatiotemporal path search are key links to ensure their safe and efficient operation, mainly involving the following contents:

[0068] The spatiotemporal risk field modeling is divided into two parts: static and dynamic. The static risk field mainly targets obstacles such as shelves and walls. Its risk is diffused through the Gaussian kernel, and the diffusion coefficient σ o Set to 0.5m, weight α o Adjustments are made based on obstacle height. For example, shelves taller than 2 meters have a weight of 1.5, while walls have a weight of 1.0. The dynamic risk field primarily considers human targets and AGVs. The risk diffusion range for human targets increases linearly with speed. When a person running (speed greater than 2 meters per second) is detected, the calculation formula is σ = 0.8 meters + 0.4 meters per second × v, and the risk weight α is 1.8. The AGV risk weight is fixed at 1.0, and the diffusion coefficient is fixed at 0.8 meters.

[0069] The spatiotemporal algorithm has been optimized in many details in practical applications. In terms of dynamic step size, it is adjusted according to different risk areas. In low-risk areas (R(n)<0.2), a step size of 0.3m is used to achieve rapid expansion and speed up the path search; in high-risk areas, such as densely populated areas, the step size will be reduced to 0.1m to ensure the safety of the path and prevent the robot from entering high-risk areas. For the heuristic function, the environment complexity ρ is measured by calculating the point cloud density within the voxel. When ρ>5 points / m 3 When the value of γ decreases from 0.7 to 0.4, the algorithm prioritizes low-risk paths and more rationally plans safe routes in complex environments, enabling logistics robots to better adapt to different scenarios and improve overall operational safety and efficiency. By constructing this risk field and optimizing spatiotemporal path search, logistics robots can more intelligently and safely plan routes and adapt to complex and changing operating environments.

[0070] 4. Multi-target trajectory optimization and smoothing steps

[0071] The trajectory generation and optimization process is as follows:

[0072] Bezier curve fitting: Discrete path points are connected using cubic Bezier curves. The number of control points is automatically adjusted based on the path curvature (the greater the curvature, the denser the control points). For example, a curve with a radius of 2m will have 5 control points inserted to ensure a smooth trajectory.

[0073] Objective function optimization:

[0074] Curvature minimization (w1 = 0.4): Reduce mechanical wear by penalizing the second-order derivative of the trajectory (curvature). The curvature threshold is tightened from 0.5 rad / m to 0.3 rad / m at full load.

[0075] Risk avoidance (w2 = 0.3): The trajectory must maintain a safety distance of at least 1.5m from dynamic obstacles, and a penalty is imposed when it violates the safety distance.

[0076] Speed ​​matching (w3 = 0.2): The expected speed vd is set according to the task type (1.5 m / s for pickup task, 1.0 m / s for replenishment task).

[0077] Energy consumption optimization (w4 = 0.1): Steering energy consumption increases by 30% when fully loaded, and path planning automatically avoids corners with steering angles exceeding 45°. The solution algorithm uses stochastic gradient descent (SGD) with momentum, a momentum factor of 0.9, a learning rate of 0.01, and convergence within 100 iterations, with an average time of 120ms. This can be understood as:

[0078] In the path planning of logistics robots, multi-objective trajectory optimization and smoothing are important links to ensure efficient and safe operation.

[0079] First, trajectory generation uses a cubic Bezier curve to fit discrete path points. The number of control points is automatically adjusted based on the path curvature; the greater the curvature, the denser the control points. For example, for a curve with a radius of 2m, five control points are inserted to ensure smooth trajectory, making the robot move more smoothly and reducing vibration and impact.

[0080] Next, the objective function is optimized, considering multiple dimensions. For curvature minimization, a weight of w1 = 0.4 is assigned to reduce mechanical wear by penalizing the second-order derivative of the trajectory (i.e., curvature). When the robot is fully loaded, the curvature threshold is tightened from 0.5 rad / m to 0.3 rad / m, strictly controlling the trajectory curvature and extending the service life of mechanical components. For risk avoidance, a weight of w2 = 0.3 is assigned to ensure that the trajectory always maintains a safe distance of at least 1.5 meters from dynamic obstacles. If this condition is not met, a penalty is applied, effectively avoiding collision risks and ensuring operational safety. For speed matching, a weight of w3 = 0.2 is assigned to the expected speed vd based on the task type: 1.5 m / s for picking tasks and 1.0 m / s for replenishing tasks. This allows the robot's speed to adapt to the task requirements and improve overall operational efficiency. For energy optimization, a weight of w4 = 0.1 is assigned to the robot. Given that steering energy consumption increases by 30% when fully loaded, path planning actively avoids curves with steering angles exceeding 45° to effectively control energy consumption.

[0081] Finally, the solution algorithm uses stochastic gradient descent (SGD) with momentum. With a momentum factor of 0.9 and a learning rate of 0.01, convergence is achieved within 100 iterations, taking an average of only 120ms. This algorithm quickly and effectively optimizes the objective function, thereby determining the optimal trajectory that meets multiple objectives. This allows logistics robots to ensure safety and improve efficiency in actual operation while also addressing mechanical losses and energy consumption, achieving comprehensive improvements in overall performance.

[0082] 5. Real-time feedback and dynamic replanning steps

[0083] Deviation assessment and replanning trigger:

[0084] Position deviation: The robot’s position is estimated through particle filtering, and replanning is triggered when the Euclidean distance between the estimated value and the planned path exceeds 0.2m.

[0085] Angle deviation: When the heading angle error exceeds 5°, the steering angular velocity is adjusted by the PID controller. If it is not corrected within 3 cycles, re-planning is triggered. Emergency obstacle avoidance mechanism: Safe braking distance: d safe =v×0.3s+v 2 / (2×1.5m / s 2 ), when the obstacle enters the distance, the maximum braking command is immediately sent (deceleration 1.2m / s 2 ).

[0086] Bidirectional RRT* re-planning:

[0087] The starting tree is expanded from the current position and the target tree is expanded from the midpoint of the remaining path, which speeds up the collision-free path search.

[0088] Sampling points are biased towards low-risk areas (the sampling probability in areas with risk value R(n) < 0.5 is increased by 50%).

[0089] Path scoring function:

[0090] score=0.6×length+0.3×R avg +0.1×curvature, giving priority to the path with the lowest score. This can be understood as:

[0091] During the operation of logistics robots, real-time feedback and dynamic replanning are crucial to ensuring safety and efficiency.

[0092] Position and angle deviations may occur during robot operation. Regarding position, the pose is estimated using a particle filter. If the Euclidean distance from the planned path exceeds 0.2m, the robot may deviate from the intended route, triggering replanning. Regarding angle, if the heading angle error exceeds 5°, the PID controller will first adjust the steering angular velocity. If the error remains uncorrected after three cycles, the continued deviation will affect the direction of travel, requiring path replanning.

[0093] When encountering obstacles, the robot implements emergency avoidance measures. A safe braking distance is calculated, and if an obstacle enters this range, the robot immediately brakes at a specific deceleration to prevent a collision. Path replanning utilizes a bidirectional RRT approach, with the starting tree expanding from the current position and the target tree from the midpoint of the remaining path, accelerating the search for a collision-free path. Sampling points are prioritized in low-risk areas to improve safety. Path evaluation uses a scoring function that includes path length, average risk, and curvature, prioritizing paths with lower scores, taking into account distance, risk, and smoothness.

[0094] The entire real-time feedback and dynamic replanning process enables logistics robots to keenly perceive operational deviations and dangerous situations, allowing them to adjust their routes promptly. In complex warehouse environments, whether navigating between shelves or responding to interference from personnel or other equipment, they can flexibly adapt, avoiding obstacles and maintaining stable operation. This improves the reliability and efficiency of logistics tasks and reduces the impact of unexpected situations on operations.

[0095] 6. Multi-robot collaborative path planning steps

[0096] Implementation details of the coordination mechanism:

[0097] Communication protocol: Publish robot status (position, speed, task type) via ROS2 DDS communication, with a topic refresh rate of 10Hz and a communication delay of <50ms.

[0098] Conflict detection: Predict the path intersection within the next 5 seconds and use the time window method (TimeWindow) to determine the conflict: if the time difference between the two robots reaching the intersection is less than 1 second, it is considered a conflict.

[0099] In the cost function optimization, the measurement indicators involved are:

[0100] Task Delay: This metric measures how long a task is delayed. It's calculated by dividing the remaining distance by the speed. The resulting value represents the task delay. For example, if the remaining distance is long and the speed is slow, the task delay is likely to be longer.

[0101] Additional Energy Consumption: This metric estimates the additional energy consumed by rerouting. It's primarily based on the curvature of the rerouted path and its length. If the path is very winding and the length is significantly increased, the additional energy consumption may be higher.

[0102] Path Length Increase Ratio: This represents the ratio of the increase in path length after replanning to the original path length. A large ratio indicates that the replanned path is significantly longer than the original. These metrics work together to help us optimize the cost function, leading to more effective decisions in path planning and other related tasks.

[0103] Hungarian algorithm solves the problem by converting the conflict matrix into an assignment problem. The average solution time is 75ms, and the conflict resolution rate reaches 98% in a 5-robot scenario. It can be understood as:

[0104] In logistics scenarios, path planning when multiple robots work together is very critical, and a series of sophisticated mechanisms are required to ensure efficient and orderly operation.

[0105] In terms of communication, with the help of ROS2's DDS communication protocol, the robot's position, speed, task type and other status information are published at a topic refresh rate of 10Hz, and the communication delay is less than 50ms, ensuring that each robot can grasp each other's dynamics in a timely manner.

[0106] During collision detection, the system predicts the intersection of each robot's path within the next five seconds and uses a time window method to determine whether a collision exists. If the difference in time between the two robots reaching the intersection is less than one second, a collision is considered. This predictive approach allows for action before a conflict actually occurs.

[0107] Conflicts are resolved through cost function optimization. Task delay is measured as the ratio of remaining path length to speed. Additional energy consumption is estimated based on the curvature and length of the replanned path, and the increase in path length is compared to the original path. These metrics comprehensively consider factors such as task progress, energy consumption, and path changes.

[0108] The Hungarian algorithm is used to solve the problem, transforming the conflict matrix into an allocation problem. This algorithm is highly efficient, with an average solution time of only 75ms. In a five-robot collaborative scenario, the conflict resolution rate reaches 98%, effectively handling inter-robot conflicts.

[0109] The multi-robot collaborative path planning mechanism, through efficient communication, accurate conflict detection, scientific cost function optimization and fast solution algorithm, allows multiple robots to not only be aware of each other's status in complex logistics environments, but also quickly coordinate when conflicts may occur, reducing collision risks, lowering task delays, optimizing energy consumption, and significantly improving the efficiency and stability of overall logistics operations, facilitating the automated operation of scenarios such as smart warehousing.

[0110] 7. Environmental Semantic Understanding and Task Adaptation Steps

[0111] Semantic segmentation and graph construction:

[0112] DeepLabv3+ training: Use the LabelMe annotation tool to create a logistics scenario dataset, including 19 categories of labels such as shelves, aisles, and charging areas. During training, OHEM (Online Hard Example Mining) is used to improve the accuracy of small object segmentation.

[0113] Semantic graph node: Each node contains attributes such as area type, speed limit (such as the speed limit in the charging area is 0.5m / s), and maximum load capacity.

[0114] GNN reasoning: A two-layer GCN network learns the reachability relationship between nodes. Input features include region center coordinates, number of connected edges, etc. The output node embedding vector is used for path planning priority sorting.

[0115] Task Adaptation Strategy:

[0116] Pickup task: Call the Dijkstra algorithm to search for the shortest path, ignoring low-risk obstacles (such as temporary stacked objects weighing less than 10kg).

[0117] Replenishment task: Use intelligent algorithms to search for the path with the lowest risk value and avoid crowded areas (areas with risk value R>0.7 are prohibited from passing).

[0118] Charging task: Combine the energy consumption model and the remaining power to select the path with the lowest energy consumption (give priority to flat roads and avoid climbing). If the power is less than 10%, switch to the shortest path.

[0119] In the operation system of logistics robots, environmental semantic understanding and task adaptation are important aspects to improve their intelligence level.

[0120] In terms of environmental semantic understanding, the first step is semantic segmentation and graph construction. The LabelMe annotation tool is used to create a logistics scene dataset covering 19 types of labels such as shelves, aisles, and charging areas. It is trained through DeepLabv3+, and online difficult sample mining (OHEM) is used to improve the accuracy of small target segmentation. In the constructed semantic graph, each node contains attribute information such as area type, speed limit (such as 0.5m / s speed limit in the charging area), and maximum load weight. Then, through a two-layer GCN network (GNN reasoning), the coordinates of the area center, the number of connected edges, etc. are used as input features to learn the reachable relationship between nodes, and the output node embedding vector is used for path planning priority sorting, so that the robot can clearly understand the characteristics of each area of ​​the environment.

[0121] In terms of task adaptation strategies, the robot will adopt different path planning methods for different tasks. When picking up goods, the Dijkstra algorithm is called to search for the shortest path, while ignoring low-risk obstacles (such as temporarily stored objects weighing less than 10kg) to speed up the picking process. In replenishment tasks, the algorithm is used to find the path with the lowest risk value, actively avoiding crowded areas (areas with a risk value R>0.7 are prohibited from passing), ensuring the safety and stability of the replenishment process. For charging tasks, the path is planned based on the energy consumption model and the remaining power, giving priority to flat roads with low energy consumption and avoiding climbing. When the power is less than 10%, it will switch to the shortest path to reach the charging area as quickly as possible.

[0122] Through environmental semantic understanding and task adaptation steps, logistics robots can accurately grasp the operating environment information, flexibly adjust path planning strategies according to different task requirements, complete various tasks more efficiently and intelligently in complex and changing logistics scenarios, and improve the overall automation and intelligence of logistics operations.

[0123] Beneficial effect data characterization

[0124]

[0125] This application demonstrates significant advantages over traditional methods in multiple indicators. In terms of dynamic obstacle avoidance, the traditional method has a success rate of only 78%. This application, with the help of dynamic risk fields and forward-looking trajectory prediction technology, significantly increases the success rate to 93%, an increase of 19.2%, greatly enhancing the ability to deal with dynamic obstacles. In terms of path planning time, the traditional method requires 215ms. This application uses dynamic step size combined with bidirectional RRT* technology to reduce the time to 150ms, a reduction of 30.2%, effectively improving planning efficiency.

[0126] In terms of energy consumption control, the traditional full-load energy consumption is 320J / m. This application reduces it to 240J / m through load-adaptive trajectory optimization, a 25% reduction, thereby achieving energy saving. When multiple robots collaborate, the traditional throughput is 8.2 pieces / minute. This application uses collaborative scheduling and conflict avoidance strategies to increase it to 12.5 pieces / minute, an increase of 52.4%, significantly improving work efficiency. The semantic segmentation time is reduced from 180ms to 110ms, relying on GhostNet lightweight and model quantization technology, reducing it by 38.9% and speeding up processing speed. Overall, this application comprehensively optimizes performance, bringing many benefits such as high efficiency and energy saving to practical applications.

[0127] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A logistics robot path planning method based on multimodal perception, characterized in that: include: Data synchronization and spatiotemporal alignment steps: Synchronize lidar point cloud, visual camera RGB image, and IMU attitude data through hardware timestamps to construct a time series dataset; Bilinear interpolation is used to align data to a unified time grid to generate a spatiotemporally aligned multimodal tensor. An adaptive weight fusion mechanism is designed to dynamically adjust the weights of each modality based on sensor reliability. The weight of vision is increased in well-lit environments, while the weight of lidar is enhanced in enclosed spaces to achieve robust perception in all scenarios. Environmental semantic parsing and dynamic feature extraction steps: The lidar point cloud generates bird's-eye view (BEV) features through the PointPillars network, and a clustering algorithm is used to distinguish static and dynamic obstacles. The visual image uses an improved YOLOv8 model for target detection and classification, combined with the DeepSORT algorithm to achieve target tracking, and outputs a dynamic target list including position, speed, and category information. The IMU data is used to extract posture change features through the LSTM network to identify the robot's climbing and turning motion states. A feature fusion gating mechanism is designed to achieve adaptive fusion of different modal features by learning the weight matrix, and align multi-source information at the feature level. Risk field construction and spatiotemporal path search steps: Construct a spatiotemporal risk field based on fusion features, comprehensively considering the risks of static obstacles and dynamic obstacles; use a Gaussian distribution model to assess the risk of dynamic obstacles, and the risk diffusion range is proportional to the target speed; use a spatiotemporal algorithm to search for the optimal path in the risk field, and use a heuristic function to balance distance and risk: prioritize the shortest path in open areas and prefer safe paths in risky areas; design a dynamic weight adjustment strategy to automatically adjust the heuristic function parameters according to the density of obstacles to balance planning efficiency and safety.

2. A logistics robot path planning method based on multimodal perception according to claim 1, characterized in that: Also includes: Target trajectory optimization and smoothing steps: The discrete path points generated by the spatiotemporal algorithm are converted into continuous trajectories, and trajectory smoothing is achieved by minimizing trajectory curvature, risk value, and speed fluctuation. An energy consumption prediction model is introduced, taking into account rolling resistance, acceleration resistance, and steering resistance, to balance obstacle avoidance safety and energy efficiency in trajectory optimization. Design a load-adaptive mechanism to dynamically adjust the trajectory curvature constraint based on the current load weight, ensuring a smooth path when fully loaded and reducing the risk of cargo swaying; Real-time feedback and dynamic replanning steps: Trajectory execution deviation is evaluated every 500ms, triggering local replanning when the deviation between the actual position and the planned trajectory exceeds a threshold. A randomized tree (RRT) algorithm is used to generate candidate trajectories around the current position, and the optimal alternative path is selected through risk assessment and verification. An emergency obstacle avoidance strategy is designed to prioritize safe parking in obstacle scenarios, reserving a safety buffer zone through braking distance calculation. Each robot's collaborative path planning steps: Using wireless communication modules to exchange position, speed, and path information with logistics robots in the same area, a collaborative network diagram of the robots is constructed. A conflict detection algorithm is designed to predict path intersections in the next 5 seconds, and a priority scheduling mechanism based on time windows dynamically adjusts conflicting paths. An auction mechanism is used to allocate path priorities, and robots with high task urgency are given priority to achieve collaboration among robots.

3. A logistics robot path planning method based on multimodal perception according to claim 1, characterized in that: Also includes: Environmental semantic understanding and task adaptation steps: semantically segment the environment, identify different functional areas and build a semantic map; Design a task-adaptive path planning strategy to automatically adjust path preferences according to task type; prioritize the shortest path for pickup tasks, the safest path for replenishment tasks, and the path with the lowest energy consumption for charging tasks; introduce prior knowledge of the environment to plan the optimal path in known areas in advance to reduce real-time calculations.

4. A logistics robot path planning method based on multimodal perception according to claim 1, characterized in that: In the dynamic obstacle risk assessment, a Gaussian risk model is used, and the risk diffusion range is associated with the target speed and category, expanding the risk area for moving targets and personnel; a risk attenuation factor is designed to predict the future position based on the historical trajectory of the obstacle to achieve risk assessment.

5. A logistics robot path planning method based on multimodal perception according to claim 2, characterized in that: In the trajectory optimization step, the energy consumption prediction model considers the effect of load weight on steering energy consumption; The heavier the load, the higher the steering energy consumption, so sharp turns should be reduced in path planning; an energy-time Pareto optimal algorithm is designed to automatically adjust the optimal path selection strategy according to task priority.

6. A logistics robot path planning method based on multimodal perception according to claim 1, characterized in that: In the data fusion, a sensor reliability assessment mechanism is designed to monitor the consistency and noise level of sensor data in real time, and automatically reduce the weight when the sensor data is abnormal; Kalman filtering is used to fuse the sensor data, and the Kalman gain is dynamically adjusted to adapt to different environmental conditions.

7. A logistics robot path planning method based on multimodal perception according to claim 1, characterized in that: In the spatiotemporal algorithm, a node expansion strategy adopts a dynamic step size; reducing the step size in risk areas improves search accuracy or increasing the step size accelerates planning; and designing a dynamic adjustment mechanism for the heuristic function to automatically switch between different heuristic strategies according to the environment.

8. The path planning method for a logistics robot based on multimodal perception according to claim 2 is characterized in that: In the collaborative path planning of the robots, a conflict resolution optimization algorithm is designed, and a conflict cost function is constructed to comprehensively consider the path replanning cost, task delay cost and energy consumption increase cost, and the optimal scheduling solution is solved by the Hungarian algorithm.

9. A logistics robot path planning method based on multimodal perception according to claim 3, characterized in that: In the environmental semantic understanding, a graph neural network (GNN) is used to learn environmental semantic relationships, encode functional areas, obstacles, and path constraints into a graph structure, and implement semantic reasoning through graph convolution operations.

10. A logistics robot path planning method based on multimodal perception according to claim 1, characterized in that: In the emergency obstacle avoidance strategy, the minimum safe braking distance of the robot is calculated: Where v is the current speed, t reaction is the system response time, a max The maximum deceleration is the maximum deceleration; when an obstacle is detected entering the safe distance, the emergency braking procedure is immediately triggered.

Images