AGV automatic posture adjusting method and system based on visual detection calibration

Abstract

The application relates to an AGV, in particular to an AGV automatic posture adjusting method and system based on visual detection calibration. The method comprises the following steps: data acquisition, data preprocessing, tray target detection, tray target segmentation, pose parameter calculation, path planning and real-time control. When the pose parameter calculation is performed, the main direction vector of the tray is determined through a principal component analysis algorithm, so that the inclination angle of the tray in a three-dimensional space is calculated, and the three-dimensional coordinate position of the tray center point under the AGV coordinate system is calculated in combination with camera calibration parameters and geometric relations, so that complete position and attitude parameters of the tray are obtained. The method improves the efficiency and accuracy of AGV tray fork taking operation, reduces goods damage and equipment failure rate, reduces labor cost and operation cost, and brings significant economic benefits to enterprises. Meanwhile, the high universality and expandability of the scheme help enterprises to quickly realize warehouse logistics automation upgrading and improve market competitiveness.

Description

Technical Field

[0001] This invention relates to an AGV, and more particularly to an automatic posture adjustment method and system for an AGV based on visual inspection and calibration. Background Technology

[0002] In recent years, with the rapid development of intelligent manufacturing, the automation of warehousing and logistics has placed higher demands on the autonomous operation capabilities of intelligent forklifts. In the field of pallet recognition and automatic posture adjustment, early methods primarily employed mechanical guidance and fixed limiting devices, which suffered from low positioning accuracy (above ±50mm) and poor adaptability. With advancements in sensor technology, visual recognition solutions based on 2D LiDAR and RGB cameras gradually emerged, improving accuracy to approximately ±20mm, but still susceptible to interference from ambient light and pallet reflections. Currently, this field has entered a technological stage centered on 3D visual perception and artificial intelligence. By employing ToF (Time-of-Flight) cameras, structured light, and other 3D sensing devices, combined with deep learning algorithms and multi-sensor fusion technology, millimeter-level high-precision pallet recognition and pose estimation have been achieved, significantly improving the system's adaptability to complex working conditions and diverse pallets. However, existing solutions still face technical bottlenecks in dynamic calibration accuracy, real-time optimization, and generalized recognition of heterogeneous pallets, necessitating the development of more efficient visual processing algorithms and intelligent control strategies to further overcome performance limitations.

[0003] In the process of pallet picking pose recognition in AGVs, existing mainstream technologies have significant limitations. While 2D vision-based systems are lower in cost, their reliance on two-dimensional image data leads to obvious performance defects in complex lighting conditions, high-density pallet stacking, or background interference scenarios. On the one hand, they suffer from high false alarm rates and low recognition rates; on the other hand, the lack of depth information makes it impossible to accurately obtain key three-dimensional posture parameters such as pallet tilt angle and height difference, severely limiting the accuracy of subsequent forklift path planning. While LiDAR solutions can achieve coarse environmental localization using point cloud data, they reveal two key problems in practical applications: first, insufficient accuracy in recognizing key features such as pallet holes and support leg edges, making it difficult to meet sub-centimeter accuracy requirements in high-level access or flexible stacking scenarios; second, severe noise interference occurs when encountering metal pallets or reflective surfaces, causing point cloud data distortion and further reducing the reliability and stability of positioning. Summary of the Invention

[0004] To address the above problems, this invention provides an automatic posture adjustment method for AGVs based on visual detection and calibration. The specific technical solution is as follows: An automatic posture adjustment method for AGVs based on visual inspection and calibration includes the following steps: Data acquisition: Continuously acquire three-dimensional depth data of the tray and its surrounding environment at a set frame rate. The data is presented in the form of a point cloud, with each point containing spatial coordinates (x, y, z) and reflection intensity information. Data preprocessing: The data is subjected to noise reduction and filtering smoothing processes in sequence; Pallet Target Detection: Based on deep learning target detection algorithms, feature extraction and analysis are performed on preprocessed 3D depth data to achieve accurate identification and localization of pallet targets, and obtain pallet bounding boxes and confidence scores; Pallet target segmentation: Based on the obtained pallet bounding box, DBSCAN clustering is performed to separate the pallet body from the attached items; Pose parameter calculation: The main direction vector of the pallet is determined by the principal component analysis algorithm, thereby calculating the tilt angle of the pallet in three-dimensional space. Combined with camera calibration parameters and geometric relationships, the three-dimensional coordinate position of the center point of the pallet in the AGV coordinate system is calculated to obtain the complete position and attitude parameters of the pallet. Path planning: Based on the calculated pallet pose information, combined with the current position and state of the AGV, a path planning algorithm is used to plan the movement trajectory of the AGV; Real-time control: Monitor the AGV's position and attitude information in real time and compare it with the planned path. Use PID control algorithm to precisely control the AGV's drive motor and steering mechanism, adjust the AGV's speed and direction, and make it travel strictly according to the planned path to achieve precise pallet picking operation.

[0005] Preferably, the denoising includes: using an adaptive threshold filtering algorithm to dynamically set a threshold according to the data distribution characteristics of different regions in order to remove noise points and retain the effective depth information of the tray and its edges.

[0006] Preferably, the filtering and smoothing process includes: using Gaussian filtering and convolution operations to perform a weighted average of the data in each pixel and its surrounding neighborhood, so as to make the depth data smoother and more continuous.

[0007] Preferably, the pose parameter calculation includes extracting the pallet main direction vector, and then combining the AGV odometer data with the pose estimation result to obtain the state vector.

[0008] Furthermore, when extracting the pallet principal direction vector, the PCA algorithm is used to calculate the pitch angle. Yaw angle Roll angle : ; In the formula: λ1, λ2, and λ3 are the eigenvalues ​​of the covariance matrix, where λ1>λ2>λ3; the coordinates (x,y,z) of the center point of the pallet are obtained by registering with the preset pallet CAD model through the ICP algorithm, with an accuracy of ±8mm.

[0009] Furthermore, when combining AGV odometer data, the pose estimation results are fused using extended Kalman filtering, and the state vector update equation is as follows: ; ; In the formula, F k H is the state transition matrix. k Let w be the observation matrix. k v k Process noise and measurement noise; X k Let X represent the state vector of the system at time step k. k-1 Z represents the state vector of the system at time step k-1. k This represents the observation at the k-th time step.

[0010] Preferably, the path planning includes: Global path: Generate the shortest path from the current point to the center point of the pallet using the A* algorithm, taking into account the rack spacing and safety distance; Local obstacle avoidance: A dynamic window method is used to adjust the velocity vector (v, ω) in real time when an obstacle is detected. ; In the formula, α, β, and γ are weighting coefficients, heading is the direction evaluation function, dist is the distance evaluation function, and velocity is the velocity evaluation function.

[0011] Preferably, the real-time control includes pose compensation, height supplementation, angle supplementation, and autonomous correction.

[0012] Preferably, the pose compensation is as follows: Based on the transformation relationship between the actual pose of the pallet and the AGV coordinate system, the height and angle of the fork teeth are adjusted. Height compensation: Δh = z_target - z_current - tray thickness; Angle compensation: Δθ = θ_z_target - θ_z_current; Autonomous correction: When the lateral deviation of the pallet is greater than 30mm, the AGV's lateral speed is adjusted via the PID controller. ; In the formula, =0.8、 =0.1、 =0.05 is the tuning parameter, and e(t) is the offset error.

[0013] An AGV automatic posture adjustment system based on vision detection and calibration includes: A 3D ToF camera is used to acquire three-dimensional depth information of the tray and its surrounding environment in real time. The data processing module is used to perform preprocessing operations such as noise reduction and filtering on the collected raw data to improve data quality. The pose calculation module performs in-depth analysis of the preprocessed data through algorithms to accurately calculate the position and attitude parameters of the tray in three-dimensional space. The AGV motion control module is used to plan and control the AGV's motion trajectory based on the calculated pose parameters, so as to achieve precise picking operations.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an AGV automatic posture adjustment method based on vision detection and calibration. This method improves the efficiency and accuracy of AGV pallet picking operations, reduces cargo damage and equipment failure rates, and lowers labor and operating costs, bringing significant economic benefits to enterprises. Furthermore, the high versatility and scalability of this solution help enterprises quickly upgrade their warehousing and logistics automation, enhancing their market competitiveness. Attached Figure Description

[0015] Figure 1 This is a system diagram of this application; Figure 2 This is a flowchart of the point cloud data processing in this application; Figure 3 This is the AGV motion control logic diagram; Figure 4 These are comparison charts for verification in typical scenarios. Detailed Implementation

[0016] The present invention will now be further described with reference to the accompanying drawings.

[0017] This application presents an efficient and accurate AGV pallet forklift posture recognition method and system based on a 3D ToF (Time of Flight) camera. The system mainly consists of a 3D ToF camera, a data processing module, a posture calculation module, and an AGV motion control module. The 3D ToF camera, as the core sensor, is responsible for real-time acquisition of the 3D depth information of the pallet and its surrounding environment. The data processing module performs preprocessing operations such as noise reduction and filtering on the acquired raw data to improve data quality. The posture calculation module uses a specific algorithm to perform depth analysis on the preprocessed data to accurately calculate the pallet's position and posture parameters in 3D space. The AGV motion control module plans and controls the AGV's motion trajectory based on the calculated posture information to achieve precise forklift operations.

[0018] 1. 3D ToF camera layout design: A 3D ToF camera is mounted directly in front of the AGV forklift, ensuring that the camera's field of view covers the area where the pallet may appear. By precisely adjusting the camera's mounting angle and height, it can acquire the pallet's three-dimensional information from the optimal perspective, reducing obstructions and blind spots, and improving the comprehensiveness and accuracy of recognition.

[0019] 2. Data Processing Module: A 3D ToF camera continuously acquires 3D depth data of the tray and its surrounding environment at a set frame rate. This data is presented in the form of a point cloud, where each point contains spatial coordinates (x, y, z) and reflection intensity information. It's like scattering countless points with position and brightness markers in 3D space to depict the shape of the tray and surrounding objects.

[0020] (1) Noise Removal: Due to the interference of ambient light, multipath effect and other factors during the data acquisition process of 3D ToF camera, noise exists in the depth data. An adaptive threshold filtering algorithm is adopted to dynamically set the threshold according to the data distribution characteristics of different areas, effectively removing noise points and retaining the effective depth information of the tray and its edges.

[0021] (2) Filtering and smoothing: To further improve data quality, Gaussian filtering algorithm is used to smooth the denoised depth data. Gaussian filtering uses convolution operation to perform weighted averaging on the data of each pixel and its surrounding neighborhood, making the depth data smoother and more continuous, reducing abrupt changes caused by measurement errors or local interference, and providing a reliable data basis for subsequent pose calculation.

[0022] 3. Pose Calculation Module (1) Pallet Target Detection: Based on deep learning target detection algorithm, feature extraction and analysis are performed on preprocessed 3D depth data to achieve accurate identification and localization of pallet targets. A large amount of 3D depth data labeled with pallets is used to train the deep learning model, enabling it to automatically learn the feature patterns of pallets and quickly and accurately detect the position and boundary of pallets in complex environments.

[0023] (2) Pose parameter calculation: After detecting the pallet target, the pallet point cloud data is further analyzed using a point cloud processing algorithm. Principal component analysis (PCA) algorithm is used to determine the main direction vector of the pallet, thereby calculating the tilt angle of the pallet in three-dimensional space. At the same time, combined with camera calibration parameters and geometric relationships, the three-dimensional coordinate position of the pallet center point in the AGV coordinate system is calculated to obtain the complete position and attitude parameters of the pallet.

[0024] 4. AGV Motion Control Module (1) Path planning: Based on the calculated pallet pose information and the current position and state of the AGV, the path planning algorithm is used to plan the AGV's motion trajectory. Considering the kinematic constraints and safety requirements of the AGV, the planned path is ensured to be smooth, without collision risk, and can efficiently guide the AGV to the pallet fork position.

[0025] (2) Real-time control: During the movement of the AGV, its position and attitude information are monitored in real time and compared with the planned path. The drive motor and steering mechanism of the AGV are precisely controlled by the PID control algorithm to adjust the speed and direction of the AGV so that it travels strictly according to the planned path, and finally achieves the operation of accurately picking up the pallet.

[0026] like Figures 1 to 4 As shown, an automatic posture adjustment method for AGVs based on visual inspection and calibration includes the following steps: 1. System Initialization and Preparation After the AGV is started, the 3D ToF camera goes through four steps: camera self-test, parameter loading, environmental scanning, and camera calibration.

[0027] (1) Camera self-test: Check whether the hardware connection is normal and whether the sensor is working properly.

[0028] (2) Parameter loading: Load camera parameters, including resolution, frame rate, measurement range, etc., from the preset configuration file. At the same time, load the AGV's kinematic parameters, such as maximum speed, acceleration, turning radius, etc., to provide basic data for subsequent pose calculation and motion control.

[0029] (3) Environmental scanning: The 3D ToF camera performs a preliminary scan of the surrounding environment to obtain three-dimensional depth information. By analyzing the scan data, obstacles that may affect pallet recognition, such as shelves, other AGVs, etc., are identified and their position information is recorded.

[0030] (4) Camera calibration: Using a standard calibration plate of known size, data are collected at different positions and angles to establish the transformation relationship between the camera coordinate system and the AGV coordinate system.

[0031] 2. Pallet Target Detection and Positioning (1) Real-time data acquisition A 3D ToF camera continuously acquires 3D depth data of the tray and its surrounding environment at a set frame rate. This data is presented as a point cloud, with each point containing spatial coordinates (x, y, z) and reflection intensity information, used to depict the shape of the tray and surrounding objects.

[0032] (2) Data preprocessing Noise Reduction Filtering: Adaptive threshold filtering and Gaussian filtering algorithms are used to process the acquired point cloud data. Adaptive threshold filtering dynamically sets the threshold according to the data distribution in different regions to remove noise points caused by environmental interference; Gaussian filtering smooths the remaining data, reducing local abrupt changes caused by measurement errors and making the data more continuous and accurate.

[0033] Region of Interest (ROI) Extraction: Based on the AGV's movement trajectory and pre-defined areas where pallets may appear, the system extracts the region of interest from the pre-processed point cloud data. For example, if the AGV typically travels within a certain shelf aisle, the system will designate that aisle area as the ROI, performing further analysis only on the data in that area, reducing data processing volume and improving computational efficiency.

[0034] (3) Pallet target detection Deep learning model recognition: The extracted ROI point cloud data is input into a pre-trained deep learning object detection model. This model learns from a large amount of labeled 3D point cloud data of pallets and can automatically identify the characteristic patterns of the pallets, such as their shape, size, and edge features, thereby locating the pallets in the point cloud data.

[0035] Detection result verification: The detection results output by the model are verified by calculating the similarity between the detected pallet area and the preset pallet template, and analyzing the continuity of the pallet edges, etc., to ensure the accuracy of the detection results. If the detection results are unreliable, the system will re-collect data or adjust the detection parameters for re-detection.

[0036] 3. Calculation of tray pose parameters (1) Point cloud feature extraction Key features are extracted from the detected pallet point cloud data, including the pallet's center point coordinates, edge contour points, and normal vectors. The center point coordinates determine the pallet's approximate position in space, the edge contour points describe the pallet's shape and size, and the normal vectors indicate the orientation of the pallet's surface.

[0037] (2) Calculation of pose parameters Position Calculation: Based on the camera calibration parameters and the spatial coordinates of the point cloud data, the coordinates of the pallet's center point in the camera coordinate system are converted to the coordinates in the AGV coordinate system, thus obtaining the pallet's precise position in three-dimensional space. For example, if the camera measures the pallet's center coordinates in the camera coordinate system as (100, 200, 300), its coordinates in the AGV coordinate system can be calculated using the calibration conversion relationship, providing an accurate target position for the AGV to position the pallet.

[0038] Attitude calculation: Principal component analysis (PCA) algorithm is used to analyze pallet point cloud data to determine the main orientation vector of the pallet, thereby calculating the tilt angle of the pallet in three-dimensional space, including pitch angle, yaw angle and roll angle.

[0039] 4. AGV Motion Planning and Control (1) Motion path planning Path Generation: Based on the calculated pallet pose information and the AGV's current position and state, combined with obstacle information obtained from the environmental perception module, a path planning algorithm (such as the RRT algorithm) is used to plan the AGV's motion path. The planned path needs to consider the AGV's kinematic constraints, such as minimum turning radius and maximum acceleration, to ensure the path is feasible and efficient. For example, in a warehouse environment with multiple shelves and AGVs, an optimal path is planned that avoids obstacles and quickly reaches the pallet forklift position.

[0040] Path evaluation and optimization: The planned path is evaluated, taking into account factors such as path length, travel time, and energy consumption. The path is optimized by adjusting path nodes or using local path optimization algorithms to improve the operating efficiency and energy utilization of AGVs.

[0041] (2) Real-time motion control Position and attitude tracking: During the movement of the AGV, the position, speed and attitude information of the AGV are monitored in real time by sensors such as encoders and gyroscopes installed on the AGV, and compared with the planned path to calculate the deviation.

[0042] Control command generation: Based on the deviation information, a PID control algorithm is used to generate control commands, adjusting the AGV's drive motor speed and steering mechanism angle to ensure the AGV travels along the planned path. For example, if the AGV deviates from the planned path, the PID control algorithm will calculate appropriate motor speed adjustments and steering angles based on the magnitude and trend of the deviation, gradually bringing the AGV back to the correct path.

[0043] Dynamic obstacle avoidance and adjustment: During movement, if the environmental perception module detects new obstacles or changes in the position of existing obstacles, the system will adjust the movement path and control commands in a timely manner to ensure that the AGV safely and accurately reaches the pallet fork position.

[0044] 5. Pallet picking and task completion (1) Precise forking operation Once the AGV reaches the pallet picking position, it adjusts the height and angle of the picking device based on the pallet's height and orientation information to perfectly match the pallet. Then, it controls the picking device to slowly insert into the bottom of the pallet, completing the picking action.

[0045] (2) Task execution and feedback Task execution: After picking up the pallet, the AGV transports the pallet to the target location, such as a shelf storage location or goods sorting area, according to the preset task flow. During the transportation process, the status of the pallet and the operation of the AGV are continuously monitored to ensure the successful completion of the task.

[0046] Task Feedback: Upon completion of the task, the AGV will send the task execution results back to the host computer system, including information such as the starting position, target position, and transport time of the pallet. Simultaneously, the system will evaluate and record the task, providing data support for subsequent task scheduling and system optimization.

[0047] (3) System Reset and Standby After completing the current task, the AGV returns to the standby area, and the system performs a reset operation, including turning off unnecessary sensors and clearing temporary data, awaiting the next task instruction. At this time, the system enters a low-power state to save energy, while maintaining monitoring of the surrounding environment to respond promptly to new task requirements.

[0048] Through the above-described complete workflow logic, the AGV pallet forklift posture recognition system based on a 3D ToF camera can efficiently and accurately complete pallet forklift tasks, solve key problems existing in current technologies, and improve the efficiency and reliability of logistics automation and intelligent warehousing.

[0049] A novel solution based on 3D ToF cameras: Breaking through traditional recognition technology approaches, this solution applies 3D ToF cameras to the field of AGV pallet forklift posture recognition, fully leveraging their three-dimensional imaging advantages and providing an innovative approach to solving key technical problems in this field.

[0050] Deep learning and point cloud processing fusion algorithm: In the pose calculation process, the deep learning target detection algorithm is innovatively combined with the point cloud processing algorithm to achieve accurate detection of pallet targets and efficient calculation of pose parameters, thereby improving the accuracy and real-time performance of recognition. Specific implementation examples: (I) Project Background A large e-commerce logistics center is facing the dual pressures of surging order volume and rising labor costs. The existing AGV pallet retrieval system has the following pain points: (1) Insufficient recognition accuracy: Traditional 2D vision solutions are prone to misjudging the position of the pallet under complex lighting conditions (such as direct strong light or supplemental lighting at night), with a forklift failure rate as high as 15%; (2) Poor compatibility of irregular pallets: There are various specifications of pallets in the warehouse (such as European standard wooden pallets, customized plastic pallets, and irregular pallets with raised structures), and the existing system has a recognition rate of only 60% for non-standard pallets; (3) Bottleneck in work efficiency: Manual intervention in pallet adjustment accounts for 25% of the time, and the average time for a single forklift is 45 seconds, which is difficult to meet the demand of processing 2,000 orders per hour during major promotions such as "Double Eleven".

[0052] (II) Implementation of the technical solution 1. Hardware System Configuration (1) 3D ToF camera selection: Orbbec Gemini 335L industrial-grade camera was adopted, and the key parameters are as follows: Distance measurement performance: working distance 0.3-5m, depth accuracy at 2m ±5mm, angular resolution 0.125°; Environmental adaptability: IP65 protection rating, supports wide temperature range of -20℃ to 50℃, and has an anti-ambient light interference capability of ≥100klux; Interfaces and computing power: Gigabit Ethernet port + USB 3.0 dual interfaces, built-in NPU computing power of 4 TOPS, supporting real-time point cloud processing.

[0053] (2) Installation layout: Mount the camera 30cm above the front end of the AGV fork teeth, with a tilt angle of 15°, to ensure that the field of view covers the pallet placement area (1.2m×1.5m). Equipped with an LED ring light (wavelength 850nm), the brightness is automatically adjusted to 1500 lux in low-light environments.

[0054] Data Acquisition and Preprocessing (1) Dynamic ROI extraction: Based on the target pallet coordinates provided by the AGV navigation system, the initial ROI (length × width × height = 1.5m × 1.8m × 0.8m) is defined in the point cloud data. By using voxel grid filtering, the point cloud density was reduced from 500,000 points / frame to 50,000 points / frame, and the processing time was shortened from 120ms to 15ms.

[0055] (2) Multimodal denoising: Statistical filtering: Remove outliers that deviate from the mean by more than three standard deviations (such as ground reflection noise); Radius filtering: Removes isolated points with fewer than 10 points in the neighborhood; Temporal filtering: Combining five consecutive frames of data, Kalman filtering is used to smooth the point cloud at the edge of the tray.

[0056] Phase 2: Pallet Target Detection and Segmentation (1) Deep learning model training: The PointNet++ network architecture was used for training on a self-built dataset (containing 20,000 pallet point clouds covering 12 pallet types), with Chamfer Distance as the loss function. The model outputs the tray bounding box (center point coordinates, length, width, height, and rotation angle) and confidence score, achieving 98.2% mAP@0.5 on the test set.

[0057] (2) Instance segmentation optimization: DBSCAN clustering is performed on the detected pallet areas to separate the pallet body from the attached materials (such as stretch film and labels). Abnormal protrusions on the tray surface are removed by RANSAC plane fitting, which improves the stability of pose calculation.

[0058] Phase 3: Pose Estimation and Optimization (1) Coarse pose calculation: The PCA algorithm is used to extract the principal direction vector of the pallet and calculate the pitch angle. Yaw angle Roll angle : ; Where λ1, λ2, and λ3 are the eigenvalues ​​of the covariance matrix, with λ1>λ2>λ3; the coordinates (x,y,z) of the center point of the pallet are obtained by registering with the preset pallet CAD model using the ICP algorithm, with an accuracy of ±8mm.

[0059] (2) Fine posture correction: Combining AGV odometer data, the pose estimation results are fused using Extended Kalman Filter (EKF), and the state vector update equation is as follows: ; ; Where F k H is the state transition matrix. k Let w be the observation matrix. k v k This refers to process noise and measurement noise.

[0060] The state vector symbol in Kalman filtering, X k Let X represent the state vector of the system at the k-th time step (the current time). k-1 Z represents the state vector of the system at time step k-1 (the previous time step). k This represents the observation at the k-th time step.

[0061] Motion control strategy (1) Path planning: Global path: Generate the shortest path from the current point to the center point of the pallet using the A* algorithm, taking into account the rack spacing (1.8m) and the safety distance (0.5m). Local obstacle avoidance: The Dynamic Window Method (DWA) is used to adjust the velocity vector (v, ω) in real time when an obstacle (such as a moving picking cart) is detected. ; Where α, β, and γ are weighting coefficients, heading is the direction evaluation function, dist is the distance evaluation function, and velocity is the velocity evaluation function.

[0062] (2) Precise forklift control: Position compensation: Adjust the height and angle of the fork teeth according to the transformation relationship between the actual position of the pallet and the AGV coordinate system. Height compensation: Δh = z_target - z_current - tray thickness (150mm); Angle compensation: Δθ = θ_z_target - θ_z_current; Autonomous correction: When the lateral deviation of the pallet is greater than 30mm, the AGV's lateral speed is adjusted via the PID controller. ; in =0.8、 =0.1、 =0.05 is the tuning parameter, e(t) is the offset error. d represents the derivative, t is time, and dt is the derivative with respect to time.

[0063] This application has the following advantages: (1) To overcome the technical deficiency that existing 2D vision systems cannot acquire three-dimensional pose information (tilt angle, height difference, etc.) of the pallet, millimeter-level measurement of the pallet's six-degree-of-freedom pose is achieved through high-precision ToF point cloud data.

[0064] (2) Overcome the identification failure problem of traditional schemes under complex lighting, high-density stacking, metal reflection and other working conditions, and improve the robustness of the system through multi-spectral fusion and dynamic noise suppression algorithm.

[0065] (3) Overcome the limitation of insufficient accuracy of lidar in identifying key features such as tray hole position and support leg edge, and use high-resolution ToF point cloud to achieve sub-centimeter level feature extraction.

[0066] (4) Solve the attitude deviation caused by the accumulation of calibration error and computation delay in the existing system, and achieve millisecond-level real-time response through multi-sensor fusion calibration and edge computing architecture.

[0067] This invention constructs a closed-loop control system of "perception-computation-execution" to achieve a positioning accuracy of ±10mm within a 2-meter working range, significantly improving the success rate and operational efficiency of AGV pallet picking in complex industrial environments.

[0068] Pose recognition based on 3D ToF cameras has shown significant advantages in AGV applications.

[0069] First, ToF cameras can acquire depth, distance, and orientation information of objects in real time, providing high-precision 3D perception. This allows them to accurately calculate the 3D coordinates and tilt angle of the fork openings in tasks such as pallet recognition, thereby improving the success rate of fork pickup. Compared to traditional RFID or LiDAR, which can only provide planar position data, ToF cameras perform significantly better.

[0070] Secondly, ToF cameras possess strong adaptability to dynamic environments. They are not limited by fixed paths and can adapt to dynamically placed trays or temporarily changed routes. Furthermore, employing active infrared technology, they operate stably in both low-light and high-light conditions. In addition, ToF cameras support multi-task integration, including tray recognition, obstacle avoidance (detecting low or suspended obstacles through area array imaging), and SLAM-assisted positioning optimization. Traditional AGVs require multiple sensors to perform navigation, obstacle avoidance, and pose recognition tasks separately. Moreover, ToF cameras boast a frame rate of 30-60fps, far exceeding the scanning speed of traditional mechanical LiDAR, making them ideal for real-time pose adjustment in high-speed AGVs.

[0071] Finally, compared with multi-sensor solutions, ToF cameras have higher hardware integration, reduce wiring complexity, and lower long-term maintenance costs. For example, Myer Microvision's PalletPro system only requires a single ToF camera to complete tray recognition, replacing the traditional complex multi-sensor solution.

[0072] In summary, 3D ToF cameras provide a more efficient, accurate, and economical solution for AGV pose recognition.

[0073] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without inventive effort, and these embodiments will all fall within the scope of protection of the claims of the present invention.

Claims

1. An automatic posture adjustment method for AGVs based on visual inspection and calibration, characterized in that, Includes the following steps: Data acquisition: Continuously acquire three-dimensional depth data of the tray and its surrounding environment at a set frame rate. The data is presented in the form of a point cloud, with each point containing spatial coordinates (x, y, z) and reflection intensity information. Data preprocessing: The data is subjected to noise reduction and filtering / smoothing processes in sequence; Pallet Target Detection: Based on deep learning target detection algorithms, feature extraction and analysis are performed on preprocessed 3D depth data to achieve accurate identification and localization of pallet targets, and obtain pallet bounding boxes and confidence scores; Pallet target segmentation: Based on the obtained pallet bounding box, DBSCAN clustering is performed to separate the pallet body from the attached items; Pose parameter calculation: The main direction vector of the pallet is determined by the principal component analysis algorithm, thereby calculating the tilt angle of the pallet in three-dimensional space. Combined with camera calibration parameters and geometric relationships, the three-dimensional coordinate position of the center point of the pallet in the AGV coordinate system is calculated to obtain the complete position and attitude parameters of the pallet. Path planning: Based on the calculated pallet pose information, combined with the current position and state of the AGV, a path planning algorithm is used to plan the movement trajectory of the AGV; Real-time control: Monitor the AGV's position and attitude information in real time and compare it with the planned path. Use PID control algorithm to precisely control the AGV's drive motor and steering mechanism, adjust the AGV's speed and direction, and make it travel strictly according to the planned path to achieve precise pallet picking operation.

2. The AGV automatic posture adjustment method based on visual detection and calibration according to claim 1, characterized in that, The noise reduction includes: using an adaptive threshold filtering algorithm to dynamically set a threshold based on the data distribution characteristics of different regions in order to remove noise points and retain the effective depth information of the tray and its edges.

3. The AGV automatic posture adjustment method based on visual detection and calibration according to claim 1, characterized in that, The filtering and smoothing process includes: using Gaussian filtering and convolution operations to perform a weighted average of the data in each pixel and its surrounding neighborhood, so as to make the depth data smoother and more continuous.

4. The AGV automatic posture adjustment method based on visual detection and calibration according to claim 1, characterized in that, The pose parameter calculation includes extracting the pallet's main direction vector, and then combining the AGV odometer data with the pose estimation results to obtain the state vector.

5. The AGV automatic posture adjustment method based on visual detection calibration according to claim 4, characterized in that, When extracting the principal direction vector of the pallet, the PCA algorithm is used to calculate the pitch angle. Yaw angle Roll angle : ； In the formula: λ1, λ2, and λ3 are the eigenvalues ​​of the covariance matrix, where λ1>λ2>λ3; the coordinates (x,y,z) of the center point of the pallet are obtained by registering with the preset pallet CAD model through the ICP algorithm, with an accuracy of ±8mm.

6. The AGV automatic posture adjustment method based on visual detection calibration according to claim 5, characterized in that, When combining AGV odometer data, the pose estimation results are fused using an extended Kalman filter. The state vector update equation is as follows: ； ； In the formula, F k H is the state transition matrix. k Let w be the observation matrix. k v k Process noise and measurement noise; X k Let X represent the state vector of the system at time step k. k-1 Z represents the state vector of the system at time step k-1. k This represents the observation at the k-th time step.

7. The AGV automatic posture adjustment method based on visual detection and calibration according to claim 1, characterized in that, The path planning includes: Global path: Generate the shortest path from the current point to the center point of the pallet using the A* algorithm, taking into account the rack spacing and safety distance; Local obstacle avoidance: A dynamic window method is used to adjust the velocity vector (v, ω) in real time when an obstacle is detected. ； In the formula, α, β, and γ are weighting coefficients, heading is the direction evaluation function, dist is the distance evaluation function, and velocity is the velocity evaluation function.

8. The AGV automatic posture adjustment method based on visual detection calibration according to claim 1, characterized in that, The real-time control includes pose compensation, height supplementation, angle supplementation, and autonomous correction.

9. The AGV automatic posture adjustment method based on visual detection calibration according to claim 8, characterized in that, Position compensation: Adjust the height and angle of the fork teeth according to the transformation relationship between the actual position of the pallet and the AGV coordinate system. Height compensation: Δh = z_target - z_current - tray thickness; Angle compensation: Δθ = θ_z_target - θ_z_current; Autonomous correction: When the lateral deviation of the pallet is greater than 30mm, the AGV's lateral speed is adjusted via the PID controller. ； In the formula, =0.8、 =0.1、 =0.05 is the tuning parameter, and e(t) is the offset error.

10. An AGV automatic posture adjustment system based on visual inspection and calibration, characterized in that, include: A 3D ToF camera is used to acquire three-dimensional depth information of the tray and its surrounding environment in real time. The data processing module is used to perform preprocessing operations such as noise reduction and filtering on the collected raw data to improve data quality. The pose calculation module performs in-depth analysis of the preprocessed data through algorithms to accurately calculate the position and attitude parameters of the tray in three-dimensional space. The AGV motion control module is used to plan and control the AGV's motion trajectory based on the calculated pose parameters, so as to achieve precise picking operations.

Images