Application of obstacle recognition in pallet four-way vehicle based on multi-sensor fusion
By using multi-sensor fusion technology, combined with TOF lidar, vision sensors and position encoders, the problem of false signals in obstacle detection of four-way vehicles in complex environments has been solved, achieving high accuracy and robust obstacle recognition, and ensuring the safe and stable operation of four-way vehicles in high-density warehouse scenarios.
Patent Information
- Application Number
- CN202511893768.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-16
- Publication Date
- 2026-03-17
AI Technical Summary
The obstacle detection of existing four-way vehicles usually uses a single sensor, which is easily affected by factors such as track reflection and nighttime lighting, resulting in false signals and malfunctions, making it difficult to operate safely and stably in complex environments.
By employing multi-sensor fusion technology, combining TOF lidar, visual sensors, and position encoders, and through unified coordinate system transformation, time synchronization, and spatial consistency calibration, it achieves accurate obstacle identification and avoidance.
It improves the accuracy and robustness of obstacle recognition, ensuring the safe and stable operation of the four-way vehicle in complex environments, avoiding collisions, and meeting the space utilization and safety requirements of high-density warehousing scenarios.
Smart Images
Figure CN121680403A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automated dense storage technology, specifically to the application of obstacle recognition based on multi-sensor fusion in a four-way pallet truck. Background Technology
[0002] With the development of intelligent manufacturing, the market for intelligent warehousing is constantly expanding and maturing. This also places higher demands on intelligent warehousing. A major challenge is how to store more materials in limited space and ensure the safe and stable operation of intelligent four-way vehicles in aisles. Therefore, the selection and fusion technology of sensors for the four-way vehicles' environmental perception are crucial.
[0003] Currently, obstacle detection for four-way vehicles typically uses single obstacle detection methods such as photoelectric or laser sensors. However, single sensors are easily affected by factors such as track reflection and nighttime lighting, leading to false signals and thus malfunctions.
[0004] To address these issues, we propose the application of obstacle recognition based on multi-sensor fusion in a four-way pallet truck. Summary of the Invention
[0005] The purpose of this invention is to provide an application of obstacle recognition based on multi-sensor fusion in a pallet four-way vehicle, so as to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: the application of obstacle recognition based on multi-sensor fusion in a four-way pallet truck, and the specific application steps are as follows:
[0007] Step 1: Multi-sensor deployment: Install TOF lidar on all four sides of the four-way vehicle, using a 120° fan-shaped scanning mode, with a vertical detection angle covering 0-60°, to acquire three-dimensional distance information of obstacles in the surrounding environment; mount a vision sensor on the same side below the lidar, equipped with a wide-angle lens, to acquire color or grayscale images, focusing on identifying targets with texture features; integrate a position encoder into the four-way vehicle's drive wheel axle to record the wheel rotation angle and number of rotations in real time, and calculate the vehicle's real-time displacement and azimuth angle by combining preset wheel diameter parameters, providing its own coordinate reference for obstacle localization;
[0008] Step 2, Coordinate System 1 and Transformation: Unify and transform the coordinate system of the pallet four-way vehicle, the TOF lidar coordinate system, and the vision sensor coordinate system;
[0009] Step 3, Time Synchronization Mechanism: A high-precision clock module is used to provide unified time synchronization for the TOF LiDAR, vision sensor, and position encoder, ensuring that the timestamp error of the data acquisition of the three is ≤10ms. A unified sampling frequency is set, that is, every 100ms, a frame of vision image, a set of laser point cloud data and encoder position information are synchronously acquired to avoid data misalignment caused by differences in sampling frequency.
[0010] Step 4, Spatial Consistency Calibration: Before leaving the factory, joint calibration is performed using a calibration board. The checkerboard calibration board is placed at different positions around the four-way vehicle, and laser point clouds and visual images are collected simultaneously. The coordinate system transformation parameters are corrected through iterative optimization algorithms to ensure that the spatial position deviation of the same obstacle in the two sensor data is ≤5cm. During operation, the position encoder is used to correct it in real time. When the four-way vehicle is moving, the real-time displacement data fed back by the encoder is used to dynamically compensate for the position changes of the sensor coordinate system and maintain spatial consistency.
[0011] Step 5: Visual sensor target detection and feature extraction: The images acquired by the visual sensor are processed for noise reduction, distortion correction, and brightness equalization to enhance the contrast between the target and the background. Existing image recognition algorithms are integrated to classify and identify obstacles in the image, mark the pixel-level boundaries of people, goods, and protruding vehicles, and output the target category. The texture and color features of the target are extracted to construct a two-dimensional feature map containing the target's location, shape, and category information.
[0012] Step 6: TOF LiDAR Data Fusion and Verification: The point cloud collected by the TOF LiDAR is filtered and clustered to identify obstacle clusters with continuous spatial morphology. The three-dimensional bounding box of the obstacle is calculated in the four-way vehicle coordinate system. The three-dimensional distance information of the LiDAR is added to the visual feature map. The target boundary of the visual detection is corrected by spatial coordinate matching. The target classification accuracy is improved by combining the laser reflection intensity and visual color features.
[0013] Step 7: Position encoder-assisted trajectory interference judgment: Combining the real-time coordinates of the four-way vehicle fed back by the position encoder, the fused relative coordinates of the obstacle are converted into absolute coordinates in the global coordinate system of the warehouse. Based on the preset running trajectory of the four-way vehicle, the minimum distance between each point on the trajectory and the absolute position of the obstacle is calculated.
[0014] Preferably, in step one, the wide-angle lens has a field of view of ≥120° and focuses on identifying targets with texture features, including people, irregularly shaped goods, and equipment with markings.
[0015] Preferably, in step two, the pallet four-way vehicle coordinate system is established with the geometric center of the four-way vehicle as the origin (O), and a right-handed rectangular coordinate system is established. The X-axis is the vehicle's forward direction, the Y-axis is the horizontal direction, and the Z-axis is the vertical direction upward.
[0016] The TOF lidar coordinate system takes the radar scanning center as its origin (O1), and the transformation parameters (translation amount) between it and the origin of the four-way vehicle coordinate system are obtained through calibration. Rotation angle ), to collect the three-dimensional point cloud data of obstacles by radar ( Transform to the four-axis vehicle coordinate system using the coordinate transformation formula: (where R1 is the rotation matrix, derived from...) (Calculated)
[0017] The vision sensor coordinate system has the camera optical center as its origin (O2). The intrinsic parameter matrix (focal length, principal point coordinates) and extrinsic parameters (translation relative to the four-way vehicle coordinate system) are obtained through Zhang's calibration method. Rotation angle The image pixel coordinates are transformed into three-dimensional coordinates (x, y, z) in a four-way vehicle coordinate system through perspective transformation, thereby realizing the spatial positioning of the visual target.
[0018] Preferably, in step six, if the deviation between the visually marked cargo boundary and the three-dimensional boundary of the laser point cloud is >10cm, the boundary is corrected based on the laser data. For low-texture targets that are not visually recognized, the laser point cloud is used to supplement the feature map to avoid missed detection.
[0019] Preferably, in step seven, if the minimum distance is less than the safety threshold, it is determined that there is an interference risk and an obstacle avoidance action is triggered. For dynamic obstacles, the trajectory of the obstacle is predicted by combining historical location data, and it is determined in advance whether there is a collision risk within the next 5 seconds.
[0020] Preferably, in step seven, the formula for calculating the absolute coordinates is as follows: ,in This is the heading angle.
[0021] Compared with the prior art, the beneficial effects of the present invention are:
[0022] 1) By combining vision and laser technologies, the limitations of a single sensor in complex environments are overcome, greatly improving the accuracy of identifying various types of obstacles and enhancing the robustness of obstacle recognition.
[0023] 2) Through real-time positioning and trajectory interference analysis by the position encoder, the four-way vehicle can respond to obstacle risks within 0.5 seconds, avoiding collisions with personnel, equipment or protruding goods in narrow alleys, and ensuring operational safety;
[0024] 3) Through a unified spatiotemporal synchronization mechanism and high-precision coordinate transformation, it can still accurately identify protruding obstacles larger than 5cm in dense environments with only 1.2m between shelves, meeting the dual requirements of smart warehousing for space utilization and safety, and adapting to high-density warehousing scenarios. Attached Figure Description
[0025] Figure 1 This is a flowchart of the steps of the present invention. Detailed Implementation
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] Example:
[0028] Please see Figure 1 The present invention provides a technical solution:
[0029] The application of obstacle recognition based on multi-sensor fusion in a four-way pallet truck is as follows:
[0030] Step 1: Multi-sensor deployment: Install TOF LiDAR on all four sides of the four-way vehicle, using a 120° fan-shaped scanning mode, with a vertical detection angle covering 0-60°, to acquire three-dimensional distance information of obstacles in the surrounding environment. It is especially suitable for detecting hard obstacles such as shelf uprights, protruding pallets, and ground protrusions. Mount a vision sensor on the same side below the LiDAR, equipped with a wide-angle lens, to acquire color or grayscale images, focusing on identifying targets with texture features. Integrate a position encoder into the four-way vehicle's drive wheel axle to record the wheel rotation angle and number of revolutions in real time. Combined with preset wheel diameter parameters, calculate the vehicle's real-time displacement and azimuth angle, providing its own coordinate reference for obstacle positioning.
[0031] By dividing the functions of TOF LiDAR, vision sensors, and position encoders, the system detects hard obstacles, textured targets, and self-positioning respectively, compensating for the blind spots of single sensors and avoiding missed detections due to sensor limitations. The 360° / 120° scanning mode of the LiDAR and the wide-angle lens of the vision sensor ensure comprehensive capture of surrounding environmental information in scenarios such as narrow alleys and high-density shelves. The wheel-axle integrated design of the position encoder can provide the vehicle's own coordinates in real time, providing a precise benchmark for obstacle positioning and improving environmental adaptability. The equipment is deployed according to the core division of labor of LiDAR distance measurement, vision classification, and encoder positioning, allowing each sensor to focus on its core task, improving the relevance and efficiency of data collection, and laying the foundation for subsequent data fusion.
[0032] Step 2, Coordinate System 1 and Transformation: Unify and transform the coordinate system of the pallet four-way vehicle, the TOF lidar coordinate system, and the vision sensor coordinate system;
[0033] By unifying the coordinate systems of the four-way pallet vehicle, TOF LiDAR, and visual sensors, obstacle data collected by different sensors are mapped to the same spatial reference, avoiding position calculation deviations caused by coordinate system differences and ensuring spatial consistency of data. The coordinate conversion of the three-dimensional distance information of the LiDAR point cloud and the two-dimensional feature data of the visual image provides a unified fusion carrier, making subsequent data complementarity verification possible and creating conditions for improving obstacle recognition accuracy. After unifying the coordinate system, the relative and absolute positions of obstacles can be converted through standardized formulas, reducing the computational complexity of the algorithm, improving data processing speed, and adapting to the real-time operation needs of the four-way vehicle.
[0034] Step 3, Time Synchronization Mechanism: A high-precision clock module (such as GPS time synchronization or local crystal oscillator synchronization) is used to provide unified time synchronization for the TOF LiDAR, vision sensor, and position encoder, ensuring that the timestamp error of the data acquisition of the three is ≤10ms. A unified sampling frequency is set, that is, every 100ms, a frame of vision image, a set of laser point cloud data and encoder position information are synchronously acquired to avoid data misalignment caused by the difference in sampling frequency.
[0035] Unified timing ensures that the timestamp error of the three types of sensor data is controlled within 10ms, avoiding problems such as laser detecting obstacles but visual detection failing to capture them, or mismatch between position and obstacle information caused by time asynchrony. This ensures the consistency of data timing. The unified sampling frequency of 10Hz can meet the real-time perception requirements of the four-way vehicle for dynamic changes in obstacles, while avoiding data redundancy caused by excessively high sampling frequencies, thus balancing real-time performance and computational efficiency. For moving obstacles, synchronized time-series data can accurately capture their motion trajectory, providing continuous dynamic data support for subsequent trajectory interference judgment and avoiding motion prediction errors caused by timing misalignment.
[0036] Step 4, Spatial Consistency Calibration: Before leaving the factory, joint calibration is performed using a calibration board. The checkerboard calibration board is placed at different positions around the four-way vehicle, and laser point clouds and visual images are collected simultaneously. The coordinate system transformation parameters are corrected through iterative optimization algorithms to ensure that the spatial position deviation of the same obstacle in the two sensor data is ≤5cm. During operation, the position encoder is used to correct it in real time. When the four-way vehicle is moving, the real-time displacement data fed back by the encoder is used to dynamically compensate for the position changes of the sensor coordinate system and maintain spatial consistency.
[0037] The joint calibration of the calibration board before leaving the factory controls the spatial position deviation of the same obstacle within 5cm, ensuring the spatial matching accuracy of laser and vision data, providing a high-precision foundation for boundary correction and target positioning after data fusion. During operation, the position encoder dynamically compensates for changes in the coordinate system position, avoiding the disruption of spatial consistency due to vehicle movement, ensuring the stability of obstacle positioning during driving. Spatial calibration can correct sensor installation errors and position shifts caused by environmental vibrations, reducing the interference of external factors on data accuracy and improving the robustness of obstacle recognition.
[0038] Step 5: Visual sensor target detection and feature extraction: The images acquired by the visual sensor are processed for noise reduction, distortion correction, and brightness equalization to enhance the contrast between the target and the background. Existing image recognition algorithms are integrated to classify and identify obstacles in the image, mark the pixel-level boundaries of people, goods, and protruding vehicles, and output the target category. The texture and color features of the target are extracted to construct a two-dimensional feature map containing the target's location, shape, and category information.
[0039] By preprocessing such as noise reduction, distortion correction, and brightness equalization, the contrast between the target and the background is enhanced, and the impact of lighting changes and lens distortion on image recognition is reduced, providing high-quality data for accurate target detection. The image recognition algorithm is integrated to mark the pixel-level boundaries of obstacles and output the category, solving the problem that laser sensors have difficulty distinguishing obstacle types. It can accurately identify different targets such as people, goods, and protruding vehicles, providing a basis for subsequent differentiated obstacle avoidance strategies. The texture and color features of the target are extracted and a two-dimensional feature map is constructed, transforming the unstructured data of the visual image into structured information, which is convenient for subsequent fusion and comparison with the three-dimensional data of the LiDAR, improving data processing efficiency.
[0040] Step 6: TOF LiDAR Data Fusion and Verification: The point cloud collected by the TOF LiDAR is filtered and clustered to identify obstacle clusters with continuous spatial morphology. The three-dimensional bounding box of the obstacle is calculated in the four-way vehicle coordinate system. The three-dimensional distance information of the LiDAR is added to the visual feature map. The target boundary of the visual detection is corrected by spatial coordinate matching. The target classification accuracy is improved by combining the laser reflection intensity and visual color features.
[0041] By correcting the target boundary of visual detection using the 3D bounding box of laser point cloud, the problem of boundary misjudgment caused by illumination and occlusion is solved, and the target positioning accuracy is improved. For low-texture targets that are difficult for visual sensors to identify, laser point cloud can accurately capture and supplement them into the feature map, avoiding the risk of missed detection and achieving full-scene coverage recognition of obstacles. By combining laser reflection intensity and visual color features, the target category can be further subdivided, solving the problem of insufficient classification ability of single vision or laser sensor.
[0042] Step 7: Position encoder-assisted trajectory interference judgment: Combining the real-time coordinates of the four-way vehicle fed back by the position encoder, the fused relative coordinates of the obstacle are converted into absolute coordinates in the global coordinate system of the warehouse. Based on the preset running trajectory of the four-way vehicle, the minimum distance between each point on the trajectory and the absolute position of the obstacle is calculated.
[0043] By converting the relative coordinates of obstacles into global absolute coordinates and calculating the minimum distance based on the preset running trajectory of the four-way vehicle, collision risks can be predicted in advance, allowing sufficient obstacle avoidance response time for the four-way vehicle. The distance judgment mechanism based on the safety threshold can effectively avoid collisions with personnel, equipment, and protruding goods in narrow alleys, reducing the risks of goods damage, equipment failure, and personnel safety, and improving the safety of intelligent warehousing operations.
[0044] In step one, the wide-angle lens equipped with a field of view of ≥120° focuses on identifying targets with texture features, including people, irregularly shaped goods, and equipment with markings.
[0045] In step two, the pallet four-way vehicle coordinate system is established with the geometric center of the four-way vehicle as the origin (O). A right-handed rectangular coordinate system is established with the X-axis as the vehicle's forward direction, the Y-axis as the horizontal direction, and the Z-axis as the vertical direction upward. This system is used to uniformly describe the vehicle's own motion and its relative position to obstacles.
[0046] The TOF lidar coordinate system takes the radar scanning center as its origin (O1), and the transformation parameters (translation amount) between it and the origin of the four-way vehicle coordinate system are obtained through calibration. Rotation angle ), to collect the three-dimensional point cloud data of obstacles by radar ( Transform to the four-axis vehicle coordinate system using the coordinate transformation formula: (where R1 is the rotation matrix, derived from...) (Calculated)
[0047] The vision sensor coordinate system has the camera optical center as its origin (O2). The intrinsic parameter matrix (focal length, principal point coordinates) and extrinsic parameters (translation relative to the four-way vehicle coordinate system) are obtained through Zhang's calibration method. Rotation angle The image pixel coordinates are transformed into three-dimensional coordinates (x, y, z) in a four-way vehicle coordinate system through perspective transformation, thereby realizing the spatial positioning of the visual target.
[0048] In step six, if the deviation between the visually marked cargo boundary and the three-dimensional boundary of the laser point cloud is greater than 10cm, the boundary is corrected based on the laser data. For low-texture targets that are not visually recognized, the laser point cloud is used to supplement the feature map to avoid missed detection.
[0049] In step seven, if the minimum distance is less than the safety threshold, it is determined that there is an interference risk and an obstacle avoidance action is triggered. For dynamic obstacles, the trajectory of the obstacle is predicted by combining historical location data, and it is determined in advance whether there is a collision risk within the next 5 seconds.
[0050] In step seven, the formula for calculating the absolute coordinates is as follows: ,in This is the heading angle.
[0051] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.
[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. Use of multi-sensor fusion based obstacle recognition in pallet quad, characterized in that, The specific application steps are as follows: Step one, multi-sensor deployment: install the TOF laser radar on the four-way vehicle running direction on the four sides, adopt a 120° sector scanning mode, and vertically detect an angle of 0-60° to obtain three-dimensional distance information of obstacles in the surrounding environment; the visual sensor is mounted on the same side below the laser radar, equipped with a wide-angle lens, and collects color or grayscale images, focusing on identifying targets with texture features, integrates the position encoder into the four-way vehicle drive wheel shaft, and records the wheel rotation angle and the number of turns in real time, combines the preset wheel diameter parameters to calculate the real-time displacement and azimuth angle of the vehicle, and provides the own coordinate reference for obstacle positioning; Step two, coordinate system and conversion: unify and convert the four-way vehicle coordinate system, TOF laser radar coordinate system and visual sensor coordinate system; Step three, time synchronization mechanism: a high-precision clock module is used to synchronize the time of the TOF laser radar, the visual sensor and the position encoder, and ensure that the timestamp error of the data collection of the three is ≤10ms, and a unified sampling frequency is set, that is, a frame of visual image, a group of laser point cloud data and once encoder position information are collected synchronously every 100ms, so as to avoid data misplacement caused by different sampling frequencies; Step four, space consistency calibration: before leaving the factory, joint calibration is carried out through a calibration board, the checkerboard calibration board is placed at different positions around the four-way vehicle, laser point cloud and visual image are collected synchronously, coordinate system conversion parameters are corrected through iterative optimization algorithm, and the spatial position deviation of the same obstacle in the data of the two sensors is ensured to be ≤5cm, and in the running, the real-time displacement data fed back by the position encoder is used for dynamic compensation of the position change of the sensor coordinate system, so as to maintain the space consistency; Step five, target detection and feature extraction of visual sensor: the image collected by the visual sensor is subjected to noise reduction, distortion correction and brightness equalization processing, the contrast between the target and the background is enhanced, the existing pattern recognition algorithm is fused, the obstacles in the image are classified and identified, the pixel-level boundary of personnel, goods and convex trolley is marked, and the target category is output, the texture features and color features of the target are extracted, and a two-dimensional feature map containing target position, shape and category information is constructed; Step six, TOF laser radar data fusion and verification: the point cloud collected by the TOF laser radar is filtered and clustered, the obstacle cluster with continuous spatial form is identified, and the three-dimensional boundary box of the obstacle cluster in the four-way vehicle coordinate system is calculated, the three-dimensional distance information of the laser radar is attached to the visual feature map, the target boundary detected by the vision is corrected through space coordinate matching, and the target classification accuracy is improved by combining the laser reflection intensity and the visual color feature; Step seven, position encoder assisted trajectory interference judgment: the real-time coordinates of the four-way vehicle fed back by the position encoder are combined, the relative coordinates of the fused obstacles are converted into absolute coordinates in the warehouse global coordinate system, and the minimum distance between each point on the preset running track of the four-way vehicle and the absolute position of the obstacle is calculated.
2. Use of multi-sensor fusion based obstacle recognition in pallet trucks, according to claim 1, characterized in that: In step one, the field of view angle of the wide-angle lens is ≥120°, and the target with texture features includes personnel, special-shaped goods and equipment with marks.
3. Use of multi-sensor fusion based obstacle recognition in pallet trucks, according to claim 1, characterized in that: In the second step, the coordinate system of the pallet four-way vehicle takes the geometric center of the four-way vehicle as the origin (O) to establish a right-handed rectangular coordinate system, the X-axis is the forward direction of the vehicle, the Y-axis is the horizontal direction, and the Z-axis is the vertical direction upward. The TOF laser radar coordinate system takes the radar scanning center as the origin (O1), and obtains the conversion parameters (translation , rotation angle ) of the origin of the four-way vehicle coordinate system through calibration. The three-dimensional point cloud data (obstacles) ) collected by the radar is converted to the four-way vehicle coordinate system through the coordinate transformation formula: , (where R1 is a rotation matrix, calculated by ). The visual sensor coordinate system takes the camera optical center as the origin (O2), and the internal parameter matrix (focal length, principal point coordinates) and the external parameter (translation relative to the four-way vehicle coordinate system , rotation angle ) are obtained by Zhang's calibration method. The image pixel coordinates are converted into three-dimensional coordinates (x, y, z) in the four-way vehicle coordinate system through perspective transformation, realizing the spatial positioning of the visual target.
4. Use of multi-sensor fusion based obstacle recognition in pallet trucks, according to claim 1, characterized in that: In the sixth step, if the deviation between the visual marker cargo boundary and the three-dimensional boundary of the laser point cloud is greater than 10 cm, the laser data is used to correct the boundary, and for the low-texture target not recognized by vision, the laser point cloud is used to supplement the feature map to avoid missing detection.
5. Use of multi-sensor fusion based obstacle recognition in pallet trucks, according to claim 1, characterized in that: In the seventh step, if the minimum distance is less than the safety threshold, it is determined that there is a risk of interference, and an obstacle avoidance action is triggered. For dynamic obstacles, the historical position data is combined to predict the motion trajectory, and it is determined in advance whether there is a collision risk within 5s in the future.
6. Use of multi-sensor fusion based obstacle recognition in pallet trucks, according to claim 1, characterized in that: In the step seven, the calculation formula of the absolute coordinate is as follows: wherein is the heading angle.