Goods allocation cross beam detection and tray pose offset detection method

Abstract

The invention provides a goods allocation cross beam detection and tray pose offset detection method, and relates to the field of automatic control and visual identification. According to the invention, three-dimensional vision and an intelligent algorithm are combined to realize comprehensive detection of the cross beam state of the double-extension-position goods shelf, goods placing space occupation detection, pallet pose offset and pallet jack deformation, so that the purpose of remarkably improving the operation safety of the stacker and the warehouse-in and warehouse-out operation precision is achieved. Comprising the following steps: (1) calibrating hands and eyes; step (2), teaching point location establishment; (3) correcting the point cloud data; step (4), goods shelf cross beam detection; step (5), goods placing space invasion and occupation detection; step (6), tray pose detection; step (7), tray jack detection; step (8), detection result feedback; and the detection result is sent to the stacker PCL through TCP communication, and the PLC executes goods forking and placing operation according to the returned detection result.

Description

Technical Field

[0001] This application belongs to the field of automation control and visual recognition, specifically for the implementation of a new method for detecting pallet beams and pallet position offset in the management of goods entering and leaving automated warehouses. Background Technology

[0002] With the rapid development of computer vision technology, it has been widely applied in various industrial inspection and automation control fields. For example, in industrial automation and logistics warehousing, cameras are commonly used for the identification and positioning of cargo locations or goods to improve operational accuracy and efficiency. Similarly, in automated storage and retrieval systems (AS / RS), stacker cranes play a crucial role in high-density storage and high-speed inbound and outbound operations. The accuracy of stacker crane operations and their safety depend on their ability to accurately detect and judge the structure and condition of the racking, as well as the position and posture of pallets.

[0003] Existing technologies for rack and pallet detection mainly include: For beam detection, laser rangefinders are typically used to scan the beam position and loading area to determine the beam deformation, overall tilt angle, and whether there is interference in the loading area. However, this method usually only provides single-point information and cannot comprehensively detect beam deformation and the loading area. For pallet position detection, stacker crane travel encoders and limit switches are typically used to roughly determine whether the pallet is placed correctly. However, when the pallet has a Z-axis offset (towards the stacker crane), it is difficult to detect by the encoder alone, which can easily cause interference or even collisions during subsequent handling. For example, a previously published domestic patent application, application number CN202510065030.9, entitled "A Method for Pallet Position and Offset Detection in an Automated Warehouse," discloses a millimeter-level precision pallet position and offset detection scheme. However, this patent application lacks effective detection methods for common issues such as recessed pallet insertion holes and encroachment on loading space. If such problems are not detected in a timely and accurate manner, serious collisions are more likely to occur when pallets are picked up or put down; moreover, this patent application only supports single-extension racks and cannot meet the detection requirements of dual-extension racks that are both near and far.

[0004] The existing technologies mentioned above lack effective solutions for problems such as stacker cranes being unable to properly pick up and place pallets due to issues like shelf deformation, dented pallet insertion holes, and encroachment on loading space, as well as for double-extension storage units. There are still many technical problems that urgently need to be solved.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This application proposes a method for detecting pallet beams and pallet position offset, which aims to solve the problems existing in the prior art by combining three-dimensional vision and intelligent algorithms, in order to achieve comprehensive detection of the status of double-extension rack beams, the encroachment of loading space, pallet position offset, and the deformation of pallet insertion holes, thereby significantly improving the safety of stacker crane operations and the accuracy of inbound and outbound operations.

[0007] To achieve the above design objectives, the method for detecting pallet beams and pallet position offset involves installing several 3D cameras at different distances (far and near) on the stacker crane to acquire point cloud data of the rack beams, the storage space, and the pallets at the far and near racks; this includes the following steps:

[0008] Step (1), hand-eye alignment;

[0009] Step (2): Establish teaching points;

[0010] Step (3), point cloud data correction;

[0011] Step (4) Inspection of shelf beams;

[0012] Step (5) Inspection of encroachment on storage space;

[0013] Step (6), tray pose detection;

[0014] Step (7), Tray socket inspection:

[0015] Step (8), feedback of test results;

[0016] The detection results are sent to the stacker crane PCL via TCP communication, and the PLC performs the forklift placement operation based on the returned detection results.

[0017] In step (1), the camera is installed at a pre-defined location on the stacker crane, and the hand-eye calibration matrix is ​​obtained through coordinate system calibration. It includes the following steps,

[0018] Step (1.1): Set up a checkerboard calibration plate in the working area of ​​the stacker crane. The position of the calibration plate is fixed and its precise coordinates in the world coordinate system are known. After the camera is installed, ensure that the entire calibration plate can be captured in the camera's field of view.

[0019] Step (1.2): Control the stacker crane to move in multiple positions and postures along the X, Y, and Z axes, and collect multiple sets of images and pose data; each movement process requires the use of a checkerboard corner detection algorithm to calculate the pose matrix of the calibration plate in the camera coordinate system. And record the end effector pose matrix of the stacker crane at that moment. ;

[0020] Step (1.3): Through the system of equations For the above multiple groups ( , The data is processed using the least squares method.

[0021] The hand-eye transformation matrix is ​​calculated by solving the problem. .

[0022] In step (2), the photographing pose Pc is converted into a photographing matrix. Convert the forklift placement pose Pd into a forklift matrix. The transformation relationship is as follows:

[0023] definition , , , , , ,

[0024] The following single-axis rotation matrix is ​​used:

[0025] , , ;

[0026] The following composite rotation matrix is ​​used in the order of Z–Y–X axes:

[0027] ;

[0028] ;

[0029] The above formula T reflects the conversion of the photographic pose Pc into a photographic matrix. Convert the forklift placement pose Pd into a forklift take-up matrix. The result.

[0030] Step (3.1) includes,

[0031] Step (3.1): Obtain the standard posture of the crossbeam;

[0032] A set of baseline beam point clouds was captured in the initial state of the shelf. Based on the beam cube region Cube_H = {Xmin, Xmax, Ymin, Ymax, Zmin, Zmax}, a subset of the beam point cloud was extracted from the original point cloud:

[0033]

[0034] Step (3.2): Calculate the beam plane;

[0035] Because the camera is tilted, the beam plane is not horizontal in the camera coordinate system, so its true direction needs to be calculated.

[0036] Point cloud Perform PCA decomposition: ;in: Let the principal axis vector be the length direction of the beam. The vertical vector of the beam. It is the normal vector of the beam plane (the eigenvector corresponding to the smallest eigenvalue). ;

[0037] Step (3.3): Solve for the rotation matrix M that aligns the normal vector of the beam plane with the Z-axis;

[0038] Rotate the normal vector n of the beam plane to the Z-axis direction (0,0,1), and calculate the axis of rotation as follows: The rotation angle is: The rotation matrix is ​​constructed as follows:

[0039] , ;

[0040] Step (3.4), data correction;

[0041] Tilted point cloud obtained by camera photography Perform coordinate rotation correction , This is the corrected point cloud.

[0042] In step (4), the beam point cloud is extracted from the beam cube region Cube_H and projected onto the xoy plane to obtain the point cloud set. Principal component analysis (PCA) was used to calculate the three-dimensional covariance matrix. (3x3), take the unit eigenvector v corresponding to the largest eigenvalue as the principal axis vector of the beam. The angle between the principal axis vector and the normal vector n=(1,0,0) (i.e., the tilt angle) is as follows:

[0043] ;

[0044] Divide the beam into several sections (such as slices) along the main direction of the beam. Draw a slice every 0.5 cm to 2 cm and calculate the midpoint of each slice. Connect all the centerline points to obtain a center curve L1. Connect the two ends of the centerline to obtain a straight line L2. Calculate the maximum distance D between L1 and L2, which is the U-shaped deformation of the beam.

[0045] Step (4) includes,

[0046] Step (4.1): Project the point onto the principal axis coordinates, such as the scalar s along the principal axis, for each Calculate the projected position of the point along the principal axis: ,remember , ;

[0047] Step (4.2): Divide the slices, number of slices The k-th slice contains elements that satisfy the following formula: The set of points k (k=0,...N-1) is denoted as . ;

[0048] Step (4.3): Calculate the center point of each slice for each non-empty slice. Calculate the center point ;

[0049] Step (4.4): Take the first point of the center curve. With the last point Let line L2 pass through these two points. The equation of line L2 is: The distance from any point in L1 to L2 is: Then the maximum distance This is the amount of U-shaped deformation.

[0050] Step (5) includes,

[0051] Step (5.1): Extract the beam point cloud subset based on Cube_T as follows: ;

[0052] Step (5.2): Use a KD-tree to perform a neighborhood search based on a Euclidean distance threshold. Perform clustering;

[0053] Perform a radius search on an unlabeled point p, adding neighboring points to the same cluster, iteratively expanding until no more points are added, and outputting the set cluster { ,..., };

[0054] Step (5.3): For each cluster Calculate bounding boxes and geometric features. , The resulting object dimensions are as follows: , , .

[0055] In step (6), firstly, based on the point cloud data extracted from the tray cube region Cube_T, the following calculation is performed to obtain... ;

[0056] Then, load the tray template point cloud. The iterative nearest point ICP algorithm is used to process the tray template. Point cloud of the tray area Registration was performed to obtain the following registration matrix. :

[0057] ;

[0058] Hand-eye calibration matrix Photo pose matrix and registration matrix Perform matrix operations to obtain the final cross position. The transformation relationship is shown in the following formula:

[0059] ;

[0060] in, The top left 3x3 is a rotation matrix. Right front three The translation component is represented by the matrix; converting this matrix to coordinate form (X, Y, Z, RX, RY, RZ) gives the stacker crane's fork pick-up position; X = X= X= ;

[0061] The conversion methods for RX, RY, and RZ are as follows:

[0062] , , .

[0063] In step (7), based on the pallet structure parameters and storage location information, three cubic point cloud regions related to the socket are extracted: Cube_S (internal region of the socket), Cube_SU (reference region above the socket, used to detect the height of the upper edge of the socket opening), and Cube_SD (reference region below the socket, used to detect the height of the lower edge of the socket opening). The expressions are as follows:

[0064]

[0065]

[0066]

[0067] Point cloud inside the socket The system counts the number of points. If the number is less than a set threshold, it indicates that the socket is empty; if the number exceeds the threshold, it is considered that there is a foreign object blocking the socket. The expression is as follows:

[0068] ;

[0069] Point cloud of the region above the jack based on y-value. Sort: Take the first 10% of the smallest point sets: Calculate the average of the y-values ​​at these points as the height of the top edge of the socket: Point cloud of the area below the socket. Perform similar steps to obtain the height of the lower edge of the socket: ;

[0070] The height difference between the upper and lower edges is the actual height of the socket opening, calculated using the following formula: ; Compare the actual height of the socket opening with the standard socket height The difference calculation shows that the current socket collapse value is: .

[0071] The method may include the following steps (9), multi-extension rack adaptation;

[0072] Step (9.1): Establish teaching points in the new extension position;

[0073] The implementation process is as follows (2);

[0074] Step (9.2): Calculate the point cloud correction matrix of the new extension position. After completion, write the point cloud correction matrix into the configuration file. The implementation process is as in step (3).

[0075] Step (9.3): Collect point cloud data of the current position at the shooting location, and manually select the point cloud image.

[0076] The configuration files define the following cube regions: Cube_H (beam cube region), Cube_P (cargo space cube region), Cube_T (pallet cube region), Cube_S (pallet socket cube region), Cube_SU (above socket cube region), and Cube_SD (below beam cube region). The maximum and minimum values ​​of the cube regions in the XYZ directions are then written to the configuration file.

[0077] In summary, the advantages and beneficial effects of this application are as follows:

[0078] 1. This application offers more comprehensive and reliable advantages for beam detection. Existing technologies can only perform single-point ranging or local two-dimensional imaging, making it difficult to accurately identify the overall deformation of the beam. This application utilizes three-dimensional point clouds to extract the beam's principal axis, calculates beam variables and tilt angles by fitting straight lines, and outputs indicators that can be used for structural safety assessment, significantly improving the reliability of rack condition monitoring.

[0079] 2. This application provides more accurate detection of encroachment on the cargo storage space. Compared with existing technologies that rely on photoelectric sensors, this application can effectively identify foreign objects or goods in the cargo storage space through point cloud filtering and clustering. It can also calculate the location, quantity, and size of the encroaching objects, ensuring that the stacker crane can safely and smoothly complete the loading and unloading operations in complex aisle environments.

[0080] 3. This application offers higher accuracy in detecting pallet pose. Existing technologies generally rely on encoders and limit switches, which cannot identify pallet offset in the Z-axis direction. This application achieves millimeter-level detection of pallet position and orientation through point cloud registration, with particularly strong ability to identify Z-axis offset, thus effectively avoiding interference or collisions caused by pallet misalignment.

[0081] 4. This application provides a more intelligent detection method for pallet sockets. Existing technologies using two-dimensional image recognition methods struggle to detect socket displacement deformation or foreign objects within the socket. This application, based on point cloud feature extraction and analysis, can accurately calculate the amount of pallet socket displacement deformation and identify foreign objects within the socket, thereby assisting in precise fork arm alignment and reducing the risk of socket damage and forklift failure.

[0082] 5. This application has a wider range of applicability. Compared with the limitations of existing technologies that are only applicable to single-extension racks, this application supports the detection of both near and far extension racks, and can be extended to support multi-extension racks through parameter expansion, thus having greater applicability and flexibility.

[0083] 6. This invention can achieve comprehensive detection of rack beam status, cargo space encroachment, pallet position and socket deformation in the complex operating environment of automated warehouses, significantly improving the safety, stability and accuracy of stacker crane operations, and has important application value and promotion significance. Attached Figure Description

[0084] The present application will now be further described in conjunction with the following figures;

[0085] Figure 1 This is a schematic diagram of a testing scenario for implementing stacker crane inbound and outbound management in an automated warehouse using this application;

[0086] Figure 2 This is a flowchart of the pallet beam detection and pallet position offset detection method described in this application; Detailed Implementation

[0087] Example 1, as Figure 1As shown, in order to realize visual recognition and detection for inbound and outbound management in automated warehouses using stacker cranes, this application proposes a method for detecting pallet beams and pallet position offset. Several 3D cameras with different extension positions at far and near are installed on the stacker crane to obtain relevant point cloud data of rack beams, storage space, and pallets at far and near racks.

[0088] like Figure 2 As shown, the method for detecting the pallet beam and pallet position offset includes the following steps:

[0089] Step (1), hand-eye alignment;

[0090] Hand-eye calibration is performed using an eye-to-hand method, and the transformation relationship between the camera coordinate system and the stacker crane coordinate system is calculated to map the pose of the pallet in the camera coordinate system to the stacker crane coordinate system.

[0091] Specifically, the camera is installed at a pre-defined location on the stacker crane, and a hand-eye calibration matrix is ​​obtained through coordinate system calibration. It includes the following steps:

[0092] Step (1.1): Set up a checkerboard calibration plate in the working area of ​​the stacker crane. The position of the calibration plate is fixed and its precise coordinates in the world coordinate system are known. After the camera is installed, ensure that the entire calibration plate can be captured in the camera's field of view.

[0093] Step (1.2): Control the stacker crane to move in multiple positions and postures along the X, Y, and Z axes, and collect multiple sets of images and pose data; each movement process requires the use of a checkerboard corner detection algorithm to calculate the pose matrix of the calibration plate in the camera coordinate system. And record the end effector pose matrix of the stacker crane at that moment. ;

[0094] Step (1.3): Through the system of equations For the above multiple groups ( , The data is processed using the least squares method.

[0095] The hand-eye transformation matrix is ​​calculated by solving the problem. ;

[0096] Step (2): Establish teaching points;

[0097] During the teaching process, the 3D camera's photo pose Pc(x1,y1,z1,Rx1,Ry1,Rz1) for acquiring point cloud data of the target object and the stacker crane's fork-and-cart pose Pd(x2,y2,z2,Rx2,Ry2,Rz2) for determining the gripping point position and orientation are determined respectively; where x, y, and z are spatial coordinates, and Rx, Ry, and Rz are rotation angles around the X-axis, Y-axis, and Z-axis, respectively.

[0098] The camera pose Pc is converted into a camera matrix. Convert the forklift placement pose Pd into a forklift matrix. The transformation relationship is as follows:

[0099] definition , , , , , ,

[0100] The following single-axis rotation matrix is ​​used:

[0101] , , ;

[0102] The following composite rotation matrix is ​​used in the order of Z–Y–X axes:

[0103] ;

[0104] ;

[0105] The above formula T reflects the conversion of the photographic pose Pc into a photographic matrix. Convert the forklift placement pose Pd into a forklift take-up matrix. The result;

[0106] Step (3), point cloud data correction;

[0107] Step (3.1): Obtain the standard posture of the crossbeam;

[0108] A set of baseline beam point clouds was captured in the initial state of the shelf. Based on the beam cube region Cube_H = {Xmin, Xmax, Ymin, Ymax, Zmin, Zmax}, a subset of the beam point cloud was extracted from the original point cloud:

[0109]

[0110] Step (3.2): Calculate the beam plane;

[0111] Because the camera is tilted, the beam plane is not horizontal in the camera coordinate system, so its true direction needs to be calculated.

[0112] Point cloud Perform PCA decomposition: ;in: Let the principal axis vector be the length direction of the beam. The vertical vector of the beam. It is the normal vector of the beam plane (the eigenvector corresponding to the smallest eigenvalue). ;

[0113] Step (3.3): Solve for the rotation matrix M that aligns the normal vector of the beam plane with the Z-axis;

[0114] Rotate the normal vector n of the beam plane to the Z-axis direction (0,0,1), and calculate the axis of rotation as follows: The rotation angle is: The rotation matrix is ​​constructed as follows:

[0115] , ;

[0116] Step (3.4), data correction;

[0117] Tilted point cloud obtained by camera photography Perform coordinate rotation correction , This is the corrected point cloud;

[0118] Step (4) Inspection of shelf beams;

[0119] The beam point cloud data is acquired by 3D camera, the main direction of the beam is extracted by algorithm, the beam variables and tilt angle are calculated by fitting a straight line, and relevant indicators are output for safety assessment.

[0120] Specifically, the point cloud of the beam is extracted from the cube region Cube_H and projected onto the xoy plane to obtain the point cloud set. Principal component analysis (PCA) was used to calculate the three-dimensional covariance matrix. (3x3), take the unit eigenvector v corresponding to the largest eigenvalue as the principal axis vector of the beam. The angle between the principal axis vector and the normal vector n=(1,0,0) (i.e., the tilt angle) is as follows:

[0121] ;

[0122] Divide the beam into several sections (such as slices) along the main direction of the beam. Draw a slice every 0.5 cm to 2 cm and calculate the midpoint of each slice. Connect all the centerline points to obtain a center curve L1. Connect the two ends of the centerline to obtain a straight line L2. Calculate the maximum distance D from L1 to L2, which is the U-shaped deformation of the beam.

[0123] The specific process is as follows:

[0124] Step (4.1): Project the point onto the principal axis coordinates, such as the scalar s along the principal axis, for each Calculate the projected position of the point along the principal axis: ,remember , ;

[0125] Step (4.2): Divide the slices, number of slices The k-th slice contains elements that satisfy the following formula: The set of points k (k=0,...N-1) is denoted as . ;

[0126] Step (4.3): Calculate the center point of each slice for each non-empty slice. Calculate the center point ;

[0127] Step (4.4): Take the first point of the center curve. With the last point Let line L2 pass through these two points. The equation of line L2 is: The distance from any point in L1 to L2 is: Then the maximum distance This is the amount of U-shaped deformation;

[0128] In the process of inspecting the above-mentioned shelf beams, point cloud data and principal component analysis (PCA) are used to obtain slices along the main direction of the beam's main axis and calculate the beam's U-shaped deformation and tilt angle, rather than using traditional single-point lasers.

[0129] Step (5) Inspection of encroachment on storage space;

[0130] The point cloud of the storage area is filtered and clustered to identify the presence of foreign objects, pallets, and goods within the space. This allows for the calculation of the location, quantity, and size of intruding objects, preventing interference from pallet placement.

[0131] Point cloud data is extracted from the cube region Cube_P of the cargo space. Euclidean clustering is used to segment the point cloud of the cargo space, and the minimum bounding boxes are used to calculate the number and size information of the objects.

[0132] Includes the following steps:

[0133] Step (5.1): Extract the beam point cloud subset based on Cube_T as follows: ;

[0134] Step (5.2): Use a KD-tree to perform a neighborhood search based on a Euclidean distance threshold. Perform clustering;

[0135] Perform a radius search on an unlabeled point p, adding neighboring points to the same cluster, iteratively expanding until no more points are added, and outputting the set cluster { ,..., };

[0136] Step (5.3): For each cluster Calculate bounding boxes and geometric features. , The resulting object dimensions are as follows: , , ;

[0137] The above-mentioned cargo space encroachment detection process utilizes point cloud filtering and Euclidean clustering to effectively and automatically identify foreign objects, pallets, or goods in the cargo space. Furthermore, by calculating the size of the foreign objects, it can solve the problem of limited detection range of devices such as photoelectric sensors in aisle environments.

[0138] Step (6), tray pose detection;

[0139] The point cloud registration method is used to match the position and orientation of the pallet in the storage location with millimeter-level accuracy. In particular, it can identify the offset of the pallet in the Z-axis direction (the direction of the stacker crane) to ensure the accuracy of stacker crane picking and placing.

[0140] Specifically, firstly, based on the point cloud data extracted from the tray cube region Cube_T, the following calculations are performed to obtain... ;

[0141] Then, load the tray template point cloud. The iterative nearest point ICP algorithm is used to process the tray template. Point cloud of the tray area Registration was performed to obtain the following registration matrix. :

[0142] ;

[0143] Hand-eye calibration matrix Photo pose matrix and registration matrix Perform matrix operations to obtain the final cross position. The transformation relationship is shown in the following formula:

[0144] ;

[0145] in, The top left 3x3 is a rotation matrix. Right front three The translation component is represented by the matrix. Converting this matrix to coordinate form (X, Y, Z, RX, RY, RZ) gives the stacker crane's fork pick-up position; X = X= X= ;

[0146] The conversion methods for RX, RY, and RZ are as follows:

[0147] , , ;

[0148] The pallet pose detection process described above uses ICP point cloud registration to achieve millimeter-level accuracy, especially in detecting pallet offset in the Z-axis direction (towards the stacker crane), which is difficult to achieve with existing technologies such as encoders or limit switches.

[0149] Step (7), Tray socket inspection:

[0150] Extract the point cloud features of the tray socket, calculate the downward deformation of the socket, and detect whether there are foreign objects in the hole to assist the fork arm in achieving accurate alignment and prevent socket damage or fork failure.

[0151] Specifically, based on the pallet structure parameters and storage location information, three cubic point cloud regions related to the socket are extracted: Cube_S (internal region of the socket), Cube_SU (reference region above the socket, used to detect the height of the upper edge of the socket opening), and Cube_SD (reference region below the socket, used to detect the height of the lower edge of the socket opening). The expressions are as follows:

[0152]

[0153]

[0154]

[0155] Point cloud inside the socket The system counts the number of points. If the number is less than a set threshold, it indicates that the socket is empty; if the number exceeds the threshold, it is considered that there is a foreign object blocking the socket. The expression is as follows:

[0156] ;

[0157] Point cloud of the region above the jack based on y-value. Sort: Take the first 10% of the smallest point sets: Calculate the average of the y-values ​​at these points as the height of the top edge of the socket: Point cloud of the area below the socket. Perform similar steps to obtain the height of the lower edge of the socket: ;

[0158] The height difference between the upper and lower edges is the actual height of the socket opening, calculated using the following formula: ; Compare the actual height of the socket opening with the standard socket height The difference calculation shows that the current socket collapse value is: ;

[0159] The above-mentioned pallet insertion hole detection and deformation calculation process innovatively calculates the collapse amount by using point cloud difference between the upper and lower regions of the insertion hole and can detect foreign objects in the hole, thereby effectively ensuring the alignment and insertion safety of the fork arm.

[0160] Step (8), feedback of test results;

[0161] The detection results are sent to the stacker crane PCL via TCP communication, and the PLC performs corresponding operations based on the returned detection results.

[0162] Step (9), adapting multi-position racks;

[0163] This application supports the testing requirements of both near and far-reaching double-extension racks, and can be extended to multi-extension racks through parameter configuration, thus expanding its applicability; specifically, it includes the following steps:

[0164] Step (9.1): Establish teaching points in the new extension position;

[0165] The implementation process is as described in step (2) above;

[0166] Step (9.2): Calculate the point cloud correction matrix of the new extension position. After completion, write the point cloud correction matrix into the configuration file. The implementation process is as described in step (3) above.

[0167] Step (9.3): Collect point cloud data of the current extension position at the shooting position, and manually select the horizontal line on the point cloud image.

[0168] The system defines the following cube regions: Beam Cube_H, Cargo Space Cube_P, Pallet Cube_T, Pallet Socket Cube_S, Socket Above Cube_SU, and Beam Above Cube_SD. The maximum and minimum values ​​of the cube regions in the XYZ directions are then written to the configuration file.

[0169] Therefore, through the above technical solution, this application can realize the status of the rack beams and the empty space for loading goods in complex operating environments.

[0170] Comprehensive detection of space occupancy, pallet position, and socket deformation significantly improves the safety, stability, and adaptability of stacker crane operations.

[0171] As described above, the embodiments given in conjunction with the accompanying drawings are merely preferred solutions for achieving the objectives of this invention. Those skilled in the art can draw inspiration from this and directly derive other alternative structures that conform to the design concept of this invention. Other structural features derived therefrom should also fall within the scope of the solutions described in this invention.

Claims

1. A method for detecting pallet beams and pallet position offset, characterized in that: Several 3D cameras with different extension positions (far and near) are installed on the stacker crane to obtain point cloud data of the rack beams, loading space, and pallets at the far and near racks. The method is characterized by the following steps: Step (1), hand-eye alignment; Step (2): Establish teaching points; Step (3), point cloud data correction; Step (4) Inspection of shelf beams; Step (5) Inspection of encroachment on storage space; Step (6), tray pose detection; Step (7), Tray socket inspection: Step (8), feedback of test results; The detection results are sent to the stacker crane PCL via TCP communication, and the PLC performs the forklift placement operation based on the returned detection results.

2. The method for detecting pallet beams and pallet position offset according to claim 1, characterized in that: In step (1), the camera is installed at a pre-defined location on the stacker crane, and the hand-eye calibration matrix is ​​obtained through coordinate system calibration. It includes the following steps, Step (1.1): Set up a checkerboard calibration plate in the working area of ​​the stacker crane. The position of the calibration plate is fixed and its precise coordinates in the world coordinate system are known. After the camera is installed, ensure that the entire calibration plate can be captured in the camera's field of view. Step (1.2): Control the stacker crane to move in multiple positions and postures along the X, Y, and Z axes, and collect multiple sets of images and pose data; each movement process requires the use of a checkerboard corner detection algorithm to calculate the pose matrix of the calibration plate in the camera coordinate system. And record the end effector pose matrix of the stacker crane at that moment. ; Step (1.3): Through the system of equations For the above multiple groups ( , The hand-eye transformation matrix was calculated using the least squares method. .

3. The method for detecting pallet beams and pallet position offset according to claim 2, characterized in that: In step (2), the photographing pose Pc is converted into a photographing matrix. Convert the forklift placement pose Pd into a forklift matrix. The transformation relationship is as follows: definition , , , , , , The following single-axis rotation matrix is ​​used: ， ， ； The following composite rotation matrix is ​​used in the order of Z–Y–X axes: ； ； The above formula T reflects the conversion of the photographic pose Pc into a photographic matrix. Convert the forklift placement pose Pd into a forklift take-up matrix. The result.

4. The method for detecting pallet beams and pallet position offset according to claim 3, characterized in that: Step (3.1) includes, Step (3.1): Obtain the standard posture of the crossbeam; A set of baseline beam point clouds was captured in the initial state of the shelf. Based on the beam cube region Cube_H = {Xmin,Xmax, Ymin, Ymax, Zmin, Zmax}, a subset of the beam point cloud was extracted from the original point cloud: Step (3.2): Calculate the beam plane; Because the camera is tilted, the beam plane is not horizontal in the camera coordinate system, so its true direction needs to be calculated. Point cloud Perform PCA decomposition: ;in: Let the principal axis vector be the length direction of the beam. The vertical vector of the beam. It is the normal vector of the beam plane (the eigenvector corresponding to the smallest eigenvalue). ; Step (3.3): Solve for the rotation matrix M that aligns the normal vector of the beam plane with the Z-axis; Rotate the normal vector n of the beam plane to the Z-axis direction (0,0,1), and calculate the axis of rotation as follows: The rotation angle is: The rotation matrix is ​​constructed as follows: ， ； Step (3.4), data correction; Tilted point cloud obtained by camera photography Perform coordinate rotation correction: , This is the corrected point cloud.

5. The method for detecting pallet beams and pallet position offset according to claim 4, characterized in that: In step (4), the beam point cloud is extracted from the beam cube region Cube_H and projected onto the xoy plane to obtain the point cloud set. Principal component analysis (PCA) was used to calculate the three-dimensional covariance matrix. (3x3), take the unit eigenvector v corresponding to the largest eigenvalue as the principal axis vector of the beam. The angle between the principal axis vector and the normal vector n=(1,0,0) (i.e., the tilt angle) is as follows: ； Divide the beam into several sections (such as slices) along the main direction of the beam. Draw a slice every 0.5 cm to 2 cm and calculate the midpoint of each slice. Connect all the centerline points to obtain a center curve L1. Connect the two ends of the centerline to obtain a straight line L2. Calculate the maximum distance D between L1 and L2, which is the U-shaped deformation of the beam.

6. The method for detecting pallet beams and pallet position offset according to claim 5, characterized in that: Step (4) includes, Step (4.1): Project the point onto the principal axis coordinates, such as the scalar s along the principal axis, for each Calculate the projected position of the point along the principal axis: ,remember , ; Step (4.2): Divide the slices, number of slices The k-th slice contains elements that satisfy the following formula: The set of points k (k=0,...N-1) is denoted as . ; Step (4.3): Calculate the center point of each slice for each non-empty slice. Calculate the center point ; Step (4.4): Take the first point of the center curve. With the last point Let line L2 pass through these two points. The equation of line L2 is: The distance from any point in L1 to L2 is: Then the maximum distance This is the amount of U-shaped deformation.

7. The method for detecting pallet beams and pallet position offset according to claim 6, characterized in that: Step (5) includes, Step (5.1): Extract the beam point cloud subset based on Cube_T as follows: ; Step (5.2): Use a KD-tree to perform a neighborhood search based on a Euclidean distance threshold. Perform clustering; Perform a radius search on an unlabeled point p, adding neighboring points to the same cluster, iteratively expanding until no more points are added, and outputting the set cluster { ,..., }; Step (5.3): For each cluster Calculate bounding boxes and geometric features. , The resulting object dimensions are as follows: , , .

8. The method for detecting pallet beams and pallet position offset according to claim 7, characterized in that: In step (6), firstly, based on the point cloud data extracted from the tray cube region Cube_T, the following calculation is performed to obtain... ; Then, load the tray template point cloud. The iterative nearest point ICP algorithm is used to process the tray template. Point cloud of the tray area Registration was performed to obtain the following registration matrix. : ； Hand-eye calibration matrix Photo pose matrix and registration matrix Perform matrix operations to obtain the final cross position. The transformation relationship is shown in the following formula: ； in, The top left 3x3 is a rotation matrix. Right front three The translation component is represented by the matrix; converting this matrix to coordinate form (X, Y, Z, RX, RY, RZ) gives the stacker crane's fork pick-up position; X = X= X= ; The conversion methods for RX, RY, and RZ are as follows: ， ， 。 9. The method for detecting pallet beams and pallet position offset according to claim 8, characterized in that: In step (7), based on the pallet structure parameters and storage location information, three cubic point cloud regions related to the socket are extracted: Cube_S (internal region of the socket), Cube_SU (reference region above the socket, used to detect the height of the upper edge of the socket opening), and Cube_SD (reference region below the socket, used to detect the height of the lower edge of the socket opening). The expressions are as follows: Point cloud inside the socket The system counts the number of points. If the number is less than a set threshold, it indicates that the socket is empty; if the number exceeds the threshold, it is considered that there is a foreign object blocking the socket. The expression is as follows: ； Point cloud of the region above the jack based on y-value. Sort: Take the first 10% of the smallest point sets: Calculate the average of the y-values ​​at these points as the height of the top edge of the socket: ; Point cloud of the area below the socket Perform similar steps to obtain the height of the lower edge of the socket: ; The height difference between the upper and lower edges is the actual height of the socket opening, calculated using the following formula: ; Compare the actual height of the socket opening with the standard socket height The difference calculation shows that the current socket collapse value is: .

10. The method for detecting pallet beams and pallet position offset according to claim 9, characterized in that: It also includes step (9), multi-position shelf adaptation; Step (9.1): Establish teaching points in the new extension position; The implementation process is as follows (2); Step (9.2): Calculate the point cloud correction matrix of the new extension position. After completion, write the point cloud correction matrix into the configuration file. The implementation process is as in step (3). Step (9.3): Collect point cloud data of the current position at the shooting location, and manually select the point cloud image. The configuration files define the following cube regions: Cube_H (beam cube region), Cube_P (cargo space cube region), Cube_T (pallet cube region), Cube_S (pallet socket cube region), Cube_SU (above socket cube region), and Cube_SD (below beam cube region). The maximum and minimum values ​​of the cube regions in the XYZ directions are then written to the configuration file.

Images