A tray posture detection method and a correction method based on 3D vision

Abstract

The application discloses a tray posture detection method and a correction method based on 3D vision and belongs to the technical field of control. Three-dimensional point cloud data of a goods location tray is detected by a 3D vision acquisition module, an inclination angle is detected by an inclination detection module, the center position of the goods location is obtained in combination with a positioning module, the center position and the offset of the tray are obtained by processing, and comparison is made with a threshold value and an allowable value. If the requirements are met, the goods can be directly taken, if the requirements are not met, the goods can be taken after fine adjustment, and if the threshold value is exceeded, the situation is reported to a human for disposal. Multiple information is fused to comprehensively solve the posture and intelligently correct the posture, so that the success rate, safety and efficiency of the goods taking operation of the stacking machine are improved.

Description

Technical Field

[0001] This invention relates to control technology, specifically a pallet posture detection and correction method based on 3D vision. Background Technology

[0002] In automated storage and retrieval systems (AS / RS), stacker cranes are the core handling equipment, responsible for automatically storing and retrieving palletized goods between high-level racks. In actual operation, due to factors such as pallet placement errors, uneven goods placement, cumulative errors from multiple stacker crane operations, rack structure deformation, or ground subsidence, pallets on the racks may not be in a properly positioned state, exhibiting horizontal tilt or offset. Furthermore, the stacker crane's loading platform may also experience slight tilting during lifting and operation, and its positioning system also has inherent positioning errors.

[0003] Traditional stacker crane picking processes rely on encoders or barcode laser positioning for operation. When the pallet is significantly tilted, the stacker crane forks may collide with the pallet during extension, leading to equipment damage, goods falling, or pallet jamming, seriously affecting system operating efficiency and safety. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system that can detect pallet skew in a non-contact, high-precision, and real-time manner, and integrate the stacker crane body status information for comprehensive pose calculation and intelligent correction, so as to improve the success rate, safety and efficiency of stacker crane picking operations.

[0005] A pallet posture detection and correction method based on 3D vision is invented by employing a 3D vision acquisition module, a tilt detection module, a stacker crane positioning module, and a communication connection processing and control unit.

[0006] A pallet posture detection method based on 3D vision is disclosed. The stacker crane includes a positioning module. A 3D vision acquisition module and a tilt angle detection module are installed on the loading platform of the stacker crane. The 3D vision acquisition module's field of view covers the target pallet and its location. The tilt angle detection module measures the tilt angle (αz, αx) of the loading platform along the z-axis (fork extension / retraction direction) and the x-axis (stack crane travel direction). The positioning module provides the center position (X0, Y0) of the location in the coordinates of the horizontal travel x-axis and the vertical lifting y-axis. A processing and control unit is communicatively connected to all the above modules. The method includes the following steps: Step S1, triggering data acquisition and point cloud acquisition: When the stacker crane moves to the detection position, the 3D camera acquires the depth image of the target location and converts it into three-dimensional point cloud data; Step S2, Tray Region Segmentation and Extraction: The point cloud data is processed, and a pre-trained deep learning model is used to segment the point cloud clusters corresponding to the front of the tray legs and the tray corner contours from the scene. Step S3: Pallet pose calculation, which specifically includes the following steps: a. Extract the point cloud of the segmented front of the tray leg and fit it; b. Extract the edge points of the tray and fit the main edge line of the tray; c. Calculate the direction vector of the main edge line on the horizontal plane. The angle between the vector and the fork direction in the ideal horizontal coordinate system of the loading platform is the horizontal skew angle θ of the pallet. d. Calculate the position (Xc, Yc, Zc) of the geometric center of the tray point cloud cluster in the camera coordinate system; Step S4, Data Fusion and Truth Determination: Includes the following steps: a. Transform the tilt angles (αz, αx), horizontal skew angle θ, and geometric center (Xc, Yc, Zc) according to the installation relationship between the camera and the loading platform to calculate the pallet skew angle θ' and pallet center position (Xr, Yr) in the ideal horizontal coordinate system of the loading platform. The skew angle transformation formula is: θ'z = arctan(tanθ / cos αz); θ'x = arctan(tanθ / cos αx); The pallet center position transformation formula is: Xr = Xc / cosαx; Yr = Yc / cosαz; When αz > 0, the loading platform is offset upwards; when αx > 0, the loading platform is offset clockwise. b. Positioning error fusion: Calculate the deviation of the pallet center position (ΔX, ΔY). Conversion formulas: ΔX = Xr - X0; ΔY = Yr - Y0.

[0007] Furthermore, the positioning error of the stacker crane positioning module is (±δx, ±δy), and the deviation of the pallet center position (ΔX, ΔY) is corrected as follows: ΔX=Xr-X0±δx; ΔY=Yr-Y0±δy.

[0008] Furthermore, the tilt detection module is a dual-axis tilt sensor.

[0009] Furthermore, the 3D vision acquisition module uses a 3D camera based on the Time-of-Flight (TOF) principle.

[0010] A 3D vision-based pallet posture correction method uses any of the aforementioned detection methods to compare the calculated actual pallet skew angle θ'z or θ'x with θ0, and simultaneously compares the calculated pallet center position (Xr, Yr) with the warehouse center position (X0, Y0) to obtain the actual position deviation (ΔX, ΔY). If |ΔX| and |ΔY| exceed the allowable range, but if θ'z < θ0 or θ'x < θ0, then the stacker crane's horizontal travel mechanism and lifting mechanism are controlled to make micro-movements to compensate for the position deviation (ΔX, ΔY).

[0011] Furthermore, the Z-direction distance Zc of the point cloud center is compared with the calibrated distance value Z0 to obtain the fork extension distance deviation ΔZ = Z0 - Zc; if |ΔZ| exceeds the allowable range but is less than the safe range, the forks are controlled to jog multiple times to compensate for the fork extension distance Zc; if |ΔZ| exceeds the safe range, the machine is stopped and reported for manual intervention.

[0012] Furthermore, if θ'z>θ0 or θ'x>θ0, the system will be shut down and reported for manual intervention.

[0013] Furthermore, the method also includes post-correction confirmation: after the stacker crane drive system executes the correction command, it triggers the 3D camera again for verification and detection. After confirming that the pallet pose meets the picking requirements, it then executes the final action of fork extension and picking.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. High precision and robustness: This invention uses a 3D camera to directly acquire depth information, which is not affected by lighting and color, and can accurately calculate the three-dimensional pose of the tray, including the tilt angle and position offset.

[0015] 2. Systematic Correction: This invention innovatively integrates the pallet visual inspection results, the real-time tilt angle of the loading platform, and the positioning error of the stacker crane system to obtain a more realistic global pallet pose information, making the correction more scientific and accurate.

[0016] 3. Proactive safety: This invention anticipates and proactively adjusts the forks before they contact the pallet, preventing collisions and jamming risks and protecting the safety of equipment and goods.

[0017] 4. Strong adaptability: The system of the present invention can automatically compensate for changes in the stacker crane's own state (such as the tilt angle caused by uneven load on the loading platform) and positioning errors accumulated over long-term operation, thereby improving the long-term stability and adaptability of the system.

[0018] 5. Improved efficiency: This invention reduces the time spent on repeated attempts and troubleshooting due to failed pickups, thereby improving the overall inbound and outbound efficiency of the automated warehouse. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the tilt sensor and its installation according to the method of the present invention; Figure 2 This is a schematic diagram of the 3D TOF camera and its installation according to the method of the present invention; Figure 3 This is a schematic diagram of a 3D TOF detection tray; Figure 4 This is an overall flowchart of the method of the present invention.

[0020] Among them, 1 is the tilt sensor; 2 is the 3D TOF camera. Detailed Implementation

[0021] The present invention adopts the following technical solution, which will be explained in conjunction with the accompanying drawings. Example 1

[0022] like Figure 1 , Figure 2 The following are included: (1) 3D vision acquisition module: fixedly installed on the stacker crane loading platform and moves with the loading platform. A 3D camera based on the TOF (Time-of-Flight) principle is used, and its field of view can cover part of the target pallet and its location.

[0023] (2) Cargo platform tilt angle detection module: Installed on the main body of the cargo platform structure, it is used to measure the tilt angle of the cargo platform in real time on the Z-axis (along the fork extension direction) and X-axis (along the stacker crane travel direction). A high-precision dual-axis tilt sensor is used.

[0024] (3) Stacker positioning module: The original walking, lifting and positioning control system of the stacker crane consists of an encoder or laser ranging positioning system for horizontal walking (X-axis) and vertical lifting (Y-axis). The coordinate values ​​provided by the system have a known positioning error range (±δx, ±δy), and are generally considered to be in place within ±5mm.

[0025] (4) Central Processing and Control Unit (PLC): Communicates with all the above modules and executes the following core algorithm flow: Step S1: Trigger data acquisition and point cloud acquisition. When the stacker crane moves to the preset detection position near the target storage location according to the instruction, it controls the 3D camera to acquire the depth image of the target storage location and convert it into three-dimensional point cloud data.

[0026] Step S2: Tray Region Segmentation and Extraction. After preprocessing the point cloud data (filtering and denoising), a pre-trained deep learning model is used to segment the point cloud clusters corresponding to the front of the tray legs and the tray corner contours from the scene.

[0027] Step S3: Calculate the tray pose.

[0028] Extract the point cloud of the segmented front of the tray leg and fit it.

[0029] Extract the edge points of the tray and fit the main edge line of the tray.

[0030] Calculate the direction vector of the main edge line on the horizontal plane, and the angle between it and the theoretical coordinate axis of the stacker crane (fork direction) is the horizontal skew angle θ of the pallet.

[0031] Calculate the position (Xc, Yc, Zc) of the geometric center of the tray point cloud cluster in the camera coordinate system.

[0032] Step S4: Multi-source data fusion and truth value calculation.

[0033] Tilt compensation: Read the tilt sensor data (αz, αx) of the loading platform at this time. Transform the pallet pose (skew angle θ and center position) calculated in step S3 according to the installation relationship between the camera and the loading platform. Eliminate the measurement error caused by the loading platform not being level, and calculate the pallet skew angle θ'z, θ'x and relative position (Xr, Yr) in the ideal horizontal loading platform coordinate system.

[0034] When αz>0, the loading platform moves upward; when αx>0, the loading platform rotates clockwise. The deflection angle conversion formulas are: θ'z = arctan(tanθ / cosαz); θ'x = arctan(tanθ / cosαx); The formulas for converting the relative positions of the centers are: Xr = Xc / cosαx; Yr = Yc / cosαz; Positioning error fusion: Obtain the theoretical positioning coordinates (X0, Y0) and its positioning error tolerances (±δx, ±δy) of the current loading platform from the stacker crane positioning module. Superimpose the relative position (Xr, Yr) of the pallet with the theoretical positioning coordinates, and considering the error tolerances, calculate the actual positional deviation of the pallet in the global coordinate system. (ΔX, ΔY) conversion formula: ΔX=Xr-X0±δx; ΔY=Yr-Y0±δ.

[0035] Step S5: Deviation judgment and correction instruction generation.

[0036] The calculated actual pallet skew angle θ'z / θ'x is compared with θ0 (the preset alarm skew threshold). Simultaneously, the calculated actual pallet center position is compared with the theoretical center position of the storage location to obtain the positional deviation (ΔX, ΔY). The Z-axis distance Zc from the point cloud center is compared with the calibrated distance Z0 to obtain the fork extension distance deviation ΔZ = Z0 - Zc.

[0037] If |ΔX|, |ΔY|, and |ΔZ| exceed the allowable range, it is determined to be an "abnormal skew / offset". Based on (ΔX, ΔY, |ΔZ|), a correction control command is generated. The command may include: controlling the stacker crane's horizontal travel mechanism and lifting mechanism to make micro-movements, compensating for position deviations (ΔX, ΔY), and correcting the fork extension distance |ΔZ|.

[0038] If θ'z>θ0 or θ'x>θ0 or |ΔZ| exceeds the safe range, the system should be shut down and reported for manual intervention.

[0039] Step S6: Perform correction and confirmation. After the stacker crane drive system executes the correction command, it can trigger the 3D TOF camera again for quick verification and detection. After confirming that the pallet position meets the picking requirements, the final action of fork extension and picking is then performed. Example 2

[0040] Specifically, a stacker crane needs to retrieve goods from 5 columns, 3 layers, and 1 row. The theoretical pallet coordinates for this location are (X0=150mm, Y0=50mm, Z0=250mm). The preset safety skew threshold is θ0=1.5°. After automatic positioning, the positioning error is δx=+3mm and δy=-2mm. The data from the platform tilt sensor is (αz=3°, αx=2°). Before retrieval, the 3D camera takes a picture: the pallet deflection angle θ=1.1° and the relative center position is (Xc=158mm, Yc=59mm, Zc=253mm).

[0041] After deflection angle conversion: θ'z = arctan(tan1.1 / cos3) = 1.101° θ'x=arctan(tan1.1 / cos2)=1.1006° After the relative position of the center is transformed: Xr = Xc / cosαx = 158 / cos2 = 158.1mm Yr=Yc / cosαz=59 / cos3=59.1mm After fusion and conversion: ΔX = Xr - X0 ± δx = 158.1 - 150 + 3 = 11.1 mm ΔY = Yr - Y0 ± δ = 59.1 - 50 - 2 = 7.1 mm ΔZ = Z0 - Zc = 250 - 253 = -3mm Based on the above calculations, if θ'z < θ0, θ'x < θ0, and |ΔX| and |ΔY| exceed the allowable range of ±5, then the correction procedure is executed. The stacker crane performs secondary micro-movements |ΔX| and |ΔY|. After the secondary movements are in place, a new photo is taken to determine if the pickup is within the allowable range, and the pickup action is executed to complete the pickup task. Example 3

[0042] Specifically, a stacker crane needs to retrieve goods from 5 columns, 3 layers, and 1 row. The theoretical pallet coordinates for this location are (X0=150mm, Y0=50mm, Z0=250mm). The preset safety skew threshold is θ0=1.5°. After automatic positioning, the positioning error is δx=+3mm and δy=-2mm. The data from the platform tilt sensor is (αz=3°, αx=2°). Before retrieval, the 3D camera takes a picture: the pallet deflection angle θ=1.1° and the relative center position is (Xc=158mm, Yc=59mm, Zc=275mm).

[0043] After deflection angle conversion: θ'z = arctan(tan1.1 / cos3) = 1.101° θ'x=arctan(tan1.1 / cos2)=1.1006° After the relative position of the center is transformed: Xr = Xc / cosαx = 158 / cos2 = 158.1mm Yr=Yc / cosαz=59 / cos3=59.1mm After fusion and conversion: ΔX = Xr - X0 ± δx = 158.1 - 150 + 3 = 11.1 mm ΔY = Yr - Y0 ± δ = 59.1 - 50 - 2 = 7.1 mm ΔZ = Z0 - Zc = 250 - 275 = -25mm Based on the above calculations, θ'z < θ0, θ'x < θ0, but ΔZ < -20, which exceeds the allowable safe range for the extension fork. Therefore, the machine should be stopped and reported for manual intervention. Example 4

[0044] Specifically, a stacker crane needs to retrieve goods from 5 columns, 3 layers, and 1 row. The theoretical pallet coordinates for this location are (X0=150mm, Y0=50mm, Z0=250mm). The preset safety skew threshold is θ0=1.5°. After automatic positioning, the positioning error is: δx=+3mm, δy=-2mm. The data from the platform tilt sensor is (αz=3°, αx=2°). Before retrieval, the 3D camera takes a picture: the pallet deflection angle θ=1.6°, and the relative center position is (Xc=158mm, Yc=59mm, Zc=253mm).

[0045] After deflection angle conversion: θ'z = arctan(tan1.6 / cos3) = 1.602° θ'x=arctan(tan1.6 / cos2)=1.601° After the relative position of the center is transformed: Xr = Xc / cosαz = 158 / cos3 = 158.1mm Yr = Yc / cosαx = 59 / cos2 = 59.1mm After fusion and conversion: ΔX = Xr - X0 ± δx = 158.1 - 150 + 3 = 11.1 mm ΔY = Yr - Y0 ± δ = 59.1 - 50 - 2 = 7.1 mm ΔZ = Z0 - Zc = 250 - 253 = -3mm Based on the above calculations, θ'z>θ0 and θ'x>θ0, which exceed the allowable safety range, the system must be shut down and reported for manual intervention. Example 5

[0046] Specifically, a stacker crane needs to retrieve goods from 5 columns, 3 layers, and 1 row. The theoretical pallet coordinates for this location are (X0=150mm, Y0=50mm, Z0=250mm). The preset safety skew threshold is θ0=1.5°. After automatic positioning, the positioning error is δx=+3mm and δy=-2mm. The data from the platform tilt sensor is (αz=3°, αx=2°). Before retrieval, the 3D camera takes a picture: the pallet deflection angle θ=1.1° and the relative center position is (Xc=151mm, Yc=52mm, Zc=253mm).

[0047] After deflection angle conversion: θ'z = arctan(tan1.1 / cos3) = 1.101° θ'x=arctan(tan1.1 / cos2)=1.1006° After the relative position of the center is transformed: Xr = Xc / cosαx = 158 / cos2 = 158.1mm Yr=Yc / cosαz=59 / cos3=59.1mm After fusion and conversion: ΔX = Xr - X0 ± δx = 158.1 - 150 + 3 = 11.1 mm ΔY = Yr - Y0 ± δ = 59.1 - 50 - 2 = 7.1 mm ΔZ = Z0 - Zc = 250 - 253 = -3mm Based on the above calculations: θ'z

Claims

1. A pallet posture detection method based on 3D vision, wherein the stacker crane includes a positioning module, characterized in that, A 3D vision acquisition module and a tilt angle detection module are installed on the loading platform of the stacker crane. The 3D vision acquisition module's field of view covers the target pallet and its location. The tilt angle detection module measures the tilt angle (αz, αx) of the loading platform along the z-axis (fork extension / retraction direction) and the x-axis (stack crane travel direction). The positioning module provides the center position (X0, Y0) of the location in the coordinates of the horizontal travel x-axis and the vertical lifting Y-axis. The processing and control unit is communicatively connected to all the above modules. The method includes the following steps: Step S1, triggering data acquisition and point cloud acquisition: When the stacker crane moves to the detection position, the 3D camera acquires the depth image of the target location and converts it into three-dimensional point cloud data; Step S2, Tray Region Segmentation and Extraction: The point cloud data is processed, and a pre-trained deep learning model is used to segment the point cloud clusters corresponding to the front of the tray legs and the tray corner contours from the scene. Step S3: Pallet pose calculation, which specifically includes the following steps: a. Extract the point cloud of the segmented front of the tray leg and fit it; b. Extract the edge points of the tray and fit the main edge line of the tray; c. Calculate the direction vector of the main edge line on the horizontal plane. The angle between the vector and the fork direction in the ideal horizontal coordinate system of the loading platform is the horizontal skew angle θ of the pallet. d. Calculate the position (Xc, Yc, Zc) of the geometric center of the tray point cloud cluster in the camera coordinate system; Step S4: Data fusion and truth calculation, including the following steps: a. Transform the tilt angles (αz, αx), horizontal skew angle θ, and geometric center (Xc, Yc, Zc) according to the installation relationship between the camera and the loading platform to calculate the pallet skew angle θ' and pallet center position (Xr, Yr) in the ideal horizontal coordinate system of the loading platform. The skew angle transformation formula is: θ'z = arctan(tanθ / cos αz); θ'x = arctan(tanθ / cos αx); The pallet center position transformation formula is: Xr = Xc / cosαx; Yr = Yc / cosαz; When αz > 0, the loading platform is offset upwards; when αx > 0, the loading platform is offset clockwise. b. Positioning error fusion: Calculate the deviation of the pallet center position (ΔX, ΔY). Conversion formulas: ΔX = Xr - X0; ΔY = Yr - Y0.

2. The pallet posture detection method based on 3D vision according to claim 1, characterized in that, The positioning error of the stacker crane positioning module is (±δx, ±δy). At this time, the deviation of the pallet center position (ΔX, ΔY) is corrected as follows: ΔX=Xr-X0±δx; ΔY=Yr-Y0±δy.

3. The pallet posture detection method based on 3D vision according to claim 1, characterized in that, The tilt detection module is a dual-axis tilt sensor.

4. The pallet posture detection method based on 3D vision according to claim 1, characterized in that, The 3D vision acquisition module uses a 3D camera based on the Time-of-Flight (TOF) principle.

5. A method for correcting tray posture based on 3D vision, using the detection method of any one of claims 1 to 3, characterized in that, The calculated actual pallet skew angle θ'z or θ'x is compared with θ0, and the calculated pallet center position (Xr, Yr) is compared with the warehouse center position (X0, Y0) to obtain the actual position deviation (ΔX, ΔY). If |ΔX| and |ΔY| exceed the allowable range, but if θ'z < θ0 or θ'x < θ0, the stacker crane's horizontal travel mechanism and lifting mechanism are controlled to make micro-movements to compensate for the position deviation (ΔX, ΔY).

6. The pallet posture correction method based on 3D vision according to claim 5, characterized in that, The Z-axis distance Zc of the point cloud center is compared with the calibrated distance Z0 to obtain the fork extension distance deviation ΔZ = Z0 - Zc. If |ΔZ| exceeds the allowable range but is less than the safe range, the forks are controlled to jog multiple times to compensate for the fork extension distance Zc. If |ΔZ| exceeds the safe range, the machine is stopped and reported for manual intervention.

7. The pallet posture correction method based on 3D vision according to claim 5, characterized in that, If θ'z > θ0 or θ'x > θ0, then the system should be stopped and reported for manual intervention.

8. The pallet posture correction method based on 3D vision according to claim 5, characterized in that, The method also includes post-correction confirmation: after the stacker crane drive system executes the correction command, it triggers the 3D camera again for verification and detection. After confirming that the pallet position meets the picking requirements, it then executes the final action of fork extension and picking.

Images