Positioning and Measurement Methods for AGV Soft Pack Stacked Objects

Abstract

This invention discloses a method for locating and measuring stacked AGV soft-pack objects, comprising: mounting a depth camera on the right fork tip of the AGV fork to pre-collect pallet image data in the AGV soft-pack stacking scenario; upon reaching the picking point, acquiring RGB image and depth data of the area where the pallet to be picked is located, processing it, and inputting it into a pallet detection model to locate the pallet insertion holes; controlling the AGV to travel along a movement path to the picking point; then acquiring the right-side RGB image of the pallet to be picked, processing it, segmenting it, adjusting the height and rotation angle of the AGV forks, and completing the picking action after straight-line travel to compensate for the distance. This invention identifies and locates the pallet at the picking point and the picking point respectively, adjusting the fork height and rotation angle to complete the end-of-line operation. This satisfies the application scenarios of squeezing, deformation, tilting, and partial occlusion during soft-pack stacking, improving the robustness and environmental adaptability of the end-of-line operation system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology for industrial vehicles, and in particular to a method for locating and measuring AGV soft-pack stacked objects. Background Technology

[0002] AGVs (Automatic Guided Vehicles) are key equipment in automated logistics and transportation systems and flexible manufacturing systems, playing a vital role in automated logistics warehouses. In intelligent material handling, they offer complete and mature solutions for task management, vehicle traffic management, vehicle path planning, vehicle navigation, and vehicle motion control. For the end-point operations of AGV tasks, intelligent detection and identification of the position, status, and type of the work object are required. Currently, this mainly relies on 3D vision for pallet image acquisition or LiDAR for point cloud data acquisition. Combined with corresponding image processing algorithms and point cloud data processing, the forklift pallet is identified, its pose is obtained, and the forklift entry direction is automatically adjusted to achieve unmanned intelligent pallet handling, solving the problem of large offset angles when unmanned forklifts dock with pallets.

[0003] However, in AGV soft-pack stacking scenarios, the pallet carrying soft-pack goods (such as fertilizer and feed) is subject to unpredictable states such as compression, deformation, tilting, and partial obstruction due to the stacked soft-pack goods. Currently, the main approach is to use an AGV storage management system to read a fixed height in the storage location and adjust the forklift lifting height. This cannot meet the needs of soft-pack stacking scenarios with varying heights, as the stacked soft-pack goods may settle locally over time, causing the pallet to tilt irregularly. Using tags for identification and positioning requires manual intervention, resulting in a significant workload, and also fails to meet the requirements of AGV soft-pack stacking scenarios with dynamically changing heights. Summary of the Invention

[0004] The purpose of this invention is to provide a method for positioning and measuring AGV soft-pack stacked objects, which enables the identification, measurement and positioning of the insertion hole position in the pallet, thereby enabling the control of the forklift and fork movement to achieve accurate picking.

[0005] The specific technical solution of the present invention is as follows:

[0006] The method for locating and measuring AGV soft-pack stacked objects includes the following steps:

[0007] S1. Install the depth camera on the right fork tip of the AGV fork, and calibrate the external parameters of the depth camera with the center of the AGV body as a reference.

[0008] S2. Pre-collect pallet image data in the AGV soft package stacking scenario, and train it into a pallet detection model using deep learning methods;

[0009] When the S3 and AGV arrive at the picking point, the depth camera collects the RGB image and depth data of the area where the pallet to be picked up is located, and then performs brightness enhancement processing after synchronizing and aligning the RGB image and depth data.

[0010] S4. Input the RGB image after brightness enhancement into the tray detection model in step S2 to obtain the tray area image, and then locate the tray hole according to its depth data.

[0011] S5. Based on the extrinsic parameters of the depth camera and the real-time pose of the AGV body in world coordinates in step S1, calculate the pose of the pallet in world coordinates; then obtain the movement path of the AGV from the picking point to the picking point according to the Bezier curve, and control the forklift to travel to the picking point according to the movement path.

[0012] S6. After the AGV arrives at the pickup point, adjust the fork height so that the depth camera can capture the right RGB image of the pallet to be picked up. Process the right RGB image according to step S3. Then, segment the processed right RGB image to obtain the pixel coordinates of the center point of the middle leg and the right leg of the pallet. Calculate the height and tilt angle of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera. Adjust the height and rotation angle of the AGV forks accordingly. After straight-line travel to compensate for the distance, complete the pickup action.

[0013] In a further embodiment, in step S1, the extrinsic parameters of the depth camera include horizontal offset dx, relative height dy, vertical offset dz, yaw angle, roll angle, and pitch angle; the relative height dy is the relative height between the center of the depth camera and the horizontal plane above the fork.

[0014] In a further embodiment, in step S2, the pallet image data includes RGB image data of the pallet at a yaw angle of -30 to 30°, a horizontal offset of -400 to 400 mm, and a distance of 0.8 to 2.5 m from the fork tip at any height when the AGV vehicle is at the pickup pre-position point and pickup point.

[0015] In a further embodiment, step S3, the synchronization process includes timestamp generation and timestamp comparison. Timestamp generation refers to the depth camera automatically generating timestamps when acquiring each frame of image data. Timestamp comparison refers to determining the degree of time alignment between the timestamps of the RGB image and the depth data by analyzing their timestamps.

[0016] The alignment process involves converting two-dimensional points in the depth data from pixel coordinates to world coordinates based on the depth camera transformation matrix, and then converting them back to pixel coordinates in the RGB image.

[0017] In a further embodiment, step S3 involves the following brightness enhancement process:

[0018] S31. Convert the RGB image to be processed to CMY space, and subtract the minimum value from C, M and Y of each pixel, that is, C-=min(C,M,Y), M-=min(C,M,Y), Y-=min(C,M,Y);

[0019] S32. Then convert the RGB image processed in step S31 to HSV space, and calculate the maximum value vMin and minimum value vMax of V, and requantize the V value: newPixel=(oldPixel / 255-vMin) / (vMax-vMin);

[0020] S33. Following steps S32 and S31, reverse the image to obtain a color-enhanced image.

[0021] In a further embodiment, step S4, the positioning of the tray insertion hole, is as follows:

[0022] S41. Convert the depth data from the depth camera into point cloud information, and use the standard-shaped tray socket point cloud as the template socket point cloud P. t The real-time detected deformation jack point cloud is P d ;

[0023] S42. Using the ICP algorithm, the deformed jack point cloud P d Point cloud P with standard jack t Registration is performed to obtain an optimal transformation matrix T, such that P d With P t The difference is minimal;

[0024]

[0025] in, and These are the point cloud data for deformable sockets and template sockets, respectively. N is the number of point pairs in the deformable socket point cloud and template socket point cloud, and i is the variable for each point cloud pair.

[0026] S43. Use the optimal transformation matrix T to correct the deformed socket and restore it to its standard shape P. c =T·P d .

[0027] In a further step, the method for obtaining the pixel coordinates of the center points of the middle leg and the right leg of the tray in step S6 is as follows:

[0028] S61. Perform mean grayscale processing on the RGB image on the right through the three RGB channels, that is, the RGB pixels of the right image are:

[0029] Gray(u,v)=(R(u,v)+G(u,v)+B(u,v)) / 3

[0030] S62. Set a threshold K to perform image binarization, set pixels with pixel values ​​greater than threshold K to white, and set pixels with pixel values ​​less than threshold K to black.

[0031] S63. Filter out noise in the binarized image, then draw the outer rectangle of the white area, and select the area with the largest area in the outer rectangle as the right jack area.

[0032] S64. Calculate the pixel coordinates of the center points of the middle leg and the right leg based on the width of the tray socket area and the width of the middle leg and the right leg of the tray.

[0033] In a further step, in step S6, the height of the AGV fork is adjusted as H = dy + 1 / 2H0 - ΔH, where dy is the relative height dy in the extrinsic parameters of the depth camera, that is, the relative height between the center of the depth camera and the horizontal plane of the fork; H0 is the thickness of the cross-section of the fork tip of the AGV fork; and ΔH is the height of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera.

[0034] Where H0 is the thickness of the cross-sectional area of ​​the fork tip of the AGV fork, and this parameter is determined by the vehicle model; H can have positive and negative values, with positive values ​​indicating the height that needs to be raised and negative values ​​indicating the height that needs to be lowered.

[0035]

[0036] Where d is the depth value of the center point O of the middle leg of the tray; pitch and roll are the pitch and roll angles of the depth camera relative to the vehicle body, respectively; p x p y It is the pixel coordinate of the center point O of the middle leg of the tray, f x f y C represents the pixel focal length in the horizontal and vertical directions of the depth camera, respectively. x C y These represent the coordinates of the depth camera's optical center in the pixel coordinate system.

[0037] In a further embodiment, in step S6, the formula for calculating the tilt angle of the center of the tray's middle leg relative to the horizontal plane of the depth camera's center is as follows:

[0038]

[0039] In the above formula: x 11 y 11 Point P 11 Pixel coordinates, x 21 y 21 Point P 21 Pixel coordinates, x1, x r Representing points P1 and P2 respectively r The pixel coordinates of the x-axis and y-axis are...12 y 22 They represent point P respectively. 12 P 22 pixel coordinates y-axis coordinates; f y This represents the normalized focal length of the depth camera's horizontal axis.

[0040] Where P 11 P 12 P represents the two endpoints of the middle leg of the tray in the vertical direction. 21 P 22 H represents the two endpoints of the right leg of the tray in the vertical direction; H represents P. 11 P 12 Height difference; P1, P r These are two points on the upper edge of the tray that lie on the same straight line.

[0041] This invention mounts a depth camera at the right fork tip of the AGV. The AGV body identifies and positions itself at the pre-picking point and the picking point. Using RGB image recognition and depth data feature extraction from the pallet, it performs high-precision measurements of position, height, and tilt angle. It also calculates the relative height of the fork tip raising or lowering and the rotation of the fork tip according to the pallet tilt angle, forming a local path curve algorithm and fork tip adjustment actions to complete end-of-line operations such as picking and placing goods. This meets the requirements of soft-pack stacking scenarios and adapts to applications involving compression, deformation, tilting, and partial obstruction, improving the robustness and environmental adaptability of the end-of-line operation system.

[0042] By calculating the height and tilt angle of the center of the pallet's middle leg relative to the horizontal plane of the depth camera center, the AGV can adjust the lifting or lowering height of the forks when it reaches the picking point; and adjust the rotation angle of the forks according to the tilt angle to compensate for the straight-line travel distance, thus completing the picking action in the soft-pack stacking scenario; after picking up the goods, the horizontal height of the forks is adjusted to complete the end-of-line operation process.

[0043] The method of this invention is applicable to soft-pack stacking applications where pallets are subjected to compression, deformation, tilting, or partial obstruction. A depth camera is installed at the tip of the right fork, and high-precision measurements of identification, positioning, and elevation angle are performed sequentially at the forward point and the pickup point, forming a two-stage operation and expanding the application scenarios of AGV end-of-line operations. Detailed Implementation

[0044] To facilitate understanding of the present invention, a more comprehensive description of the invention will be provided below in conjunction with specific embodiments. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0046] Example:

[0047] The method for locating and measuring AGV soft-pack stacked objects includes the following steps:

[0048] S1. Install the depth camera on the right fork tip of the AGV fork, and calibrate the external parameters of the depth camera with the center of the AGV body as a reference.

[0049] In step S1, since the depth camera itself has a field of view of 90° horizontal and 60° vertical, and to save on end-effector costs, this invention selects the right fork tip for installation (the installation method for the left side is similar). The effective measurement range of the depth camera is 0.3-5m; the accuracy decreases with increasing distance, while the depth camera has a smaller accuracy error at close range.

[0050] Therefore, at the pre-pickup point, the distance between the fork tip and the center of the pallet is approximately 1.5-2.5m, and the depth is relatively far from the goods on the pallet. The camera's field of view can capture the entire pallet area. The depth error is reduced by using a multi-frame depth data fusion method, and the pallet position information and yaw angle information are calculated. At the pickup point, the distance between the fork tip and the center of the pallet is approximately 0.8-1.5m, and the depth camera's field of view can capture part of the pallet area. The depth data collected by the depth camera has relatively high accuracy and can accurately measure the pallet tilt angle.

[0051] The extrinsic parameters of the depth camera include horizontal offset dx, relative height dy, vertical offset dz, yaw angle, roll angle, and pitch angle; the relative height dy is the relative height between the center of the depth camera and the horizontal plane above the fork.

[0052] The method for calibrating the extrinsic parameters of a depth camera includes the following steps:

[0053] S11. The depth camera is mounted on the right fork tip, so that the depth camera plane is perpendicular to the horizontal plane.

[0054] S12. Use an auxiliary 4*5 grid checkerboard calibration board tool and place it 1m in front of the right fork tip of the fork.

[0055] S13. Based on the vertical field of view of the depth camera being 60°, the forks need to be raised by 0.57m (1m*tan30°) to calculate the height. The height of the checkerboard calibration plate should be appropriately increased and recorded so that the 4*5 checkerboard grid in the checkerboard calibration plate is completely captured by the camera.

[0056] S14. Acquire images of the chessboard calibration board using a depth camera, and use image processing methods to detect the corner points of the chessboard, obtaining the pixel coordinates of the 12 chessboard corner points.

[0057] S15. Based on Zhang Zhengyou's calibration method, calculate the transformation matrix between the depth camera coordinate system and the chessboard pixel coordinate system, and use it as the extrinsic parameter matrix from the depth camera to the chessboard grid coordinates.

[0058] S16. By measuring the translation of the coordinate system of the AGV vehicle body center and the coordinate system of the chessboard calibration plate on the x and y axes, and the lifting height of the fork tip, the transformation matrix between the two is obtained, which is used as the external parameter matrix from the chessboard grid to the vehicle center.

[0059] S17. Based on the two extrinsic parameter matrices, calculate the extrinsic parameter matrix from the camera to the center of the AGV body to achieve the extrinsic parameter calibration of the depth camera;

[0060] To reduce the errors in the calibration of extrinsic parameters, the above steps are repeated multiple times to obtain extrinsic parameter matrices between multiple camera coordinate systems and the vehicle center coordinate system. The final extrinsic parameter matrix is ​​obtained by averaging the values.

[0061] S2. Pre-collect pallet image data in the AGV soft package stacking scenario, and train it into a pallet detection model using deep learning methods;

[0062] The pallet image data includes RGB image data of the pallet at a yaw angle of -30 to 30°, a horizontal offset of -400 to 400 mm, and a distance of 0.8 to 2.5 m from the fork tip at any height when the AGV vehicle is at the pickup pre-position point and pickup point.

[0063] The RGB image data of the aforementioned 10,000 tray areas were calibrated using the open-source tool LabImg. Then, a small object detection layer network was added to the existing open-source YOLOv8 backbone network architecture. The 10,000 calibrated tray classification data were used, and the calibrated data was trained in multiple rounds using data augmentation methods. The deep learning model of the domestic platform of this invention refers to the domestic Rockchip RK3568 platform. The model was simulated and run on the server to obtain inference results. The model was converted into an RKNN model using the RK Toolkit tool, and the model output results were obtained by designing quantized inference.

[0064] When the S3 and AGV arrive at the picking point, the depth camera collects the RGB image and depth data of the area where the pallet to be picked up is located, and then performs brightness enhancement processing after synchronizing and aligning the RGB image and depth data.

[0065] The synchronization process includes timestamp generation and timestamp comparison. Timestamp generation refers to the automatic generation of timestamps by the depth camera when acquiring each frame of image data. Timestamp comparison refers to determining the degree of time alignment between the timestamps of the RGB image and the depth data by analyzing their timestamps.

[0066] The alignment process involves converting two-dimensional points in the depth data from pixel coordinates to world coordinates based on the depth camera transformation matrix, and then converting them back to pixel coordinates in the RGB image.

[0067] The brightness enhancement process steps are as follows:

[0068] S31. Convert the RGB image to be processed to CMY space, and subtract the minimum value from C, M and Y of each pixel, that is, C-=min(C,M,Y), M-=min(C,M,Y), Y-=min(C,M,Y);

[0069] S32. Then convert the RGB image processed in step S31 to HSV space, and calculate the maximum value vMin and minimum value vMax of V, and requantize the V value: newPixel=(oldPixel / 255-vMin) / (vMax-vMin);

[0070] S33. Following steps S32 and S31, reverse the image to obtain a color-enhanced image.

[0071] RGB images provide detailed color and texture information, while depth data provides distance and spatial structure information. Only when these two types of data are fully synchronized can this information be accurately combined to identify objects, determine their location and size, and perform complex 3D spatial analysis and calculations. Therefore, synchronized processing of RGB images and depth data is necessary.

[0072] Due to differences in the optical center position, intrinsic parameters, extrinsic parameters, and manufacturing processes of RGB and depth cameras, the field of view of RGB and depth images acquired under the same distance and angle conditions differ. The field of view of the RGB image is smaller, while that of the depth image is larger. The alignment principle is to transform the two-dimensional points on the depth image from pixel coordinates to world coordinates based on the camera transformation matrix, and then convert them back to pixel coordinates in the RGB image.

[0073] Let the RGB intrinsic parameter matrix be K. rgb The depth camera intrinsic parameter matrix is ​​K d World coordinates to RGB camera coordinates extrinsic parameter T w2rgb The extrinsic parameter from world coordinates to depth camera coordinates is T. w2d extrinsic parameters T from depth camera to RGB camera coordinates d2rgb(T can be obtained through calibration or the depth camera SDK), and is calculated as follows:

[0074] 1) Point P in the depth pixel coordinate system d The coordinates after transformation to the depth camera coordinate system are: Z d It is the depth value of the current point;

[0075] 2) P in the depth camera coordinate system D The coordinates after conversion to world coordinates are:

[0076] 3) P in world coordinates w The coordinates after conversion to the RGB camera coordinate system are:

[0077] 4) P in the RGB camera coordinate system C Convert to RGB pixel coordinate system: Z rgb It is the depth value of the RGB pixel at the current point, i.e. Consider that the camera planes of the RGB and depth cameras are on the same plane, i.e., Z. d and Z rgb The values ​​are equal; the conversion from pixels in the depth image to pixels in the RGB image has been completed.

[0078]

[0079] S4. Input the RGB image after brightness enhancement into the tray detection model in step S2 to obtain the tray area image, and then locate the tray hole according to its depth data.

[0080] The steps for positioning the tray insertion holes are as follows:

[0081] S41. Convert the depth data from the depth camera into point cloud information, and use the standard-shaped tray socket point cloud as the template socket point cloud P. t The real-time detected deformation jack point cloud is P d ;

[0082] S42. Using the ICP algorithm, the deformed jack point cloud P d Point cloud P with standard jack t Registration is performed to obtain an optimal transformation matrix T, such that P d With P t The difference is minimal;

[0083]

[0084] in, and These are the point cloud data for deformable sockets and template sockets, respectively. N is the number of point pairs in the deformable socket point cloud and template socket point cloud, and i is the variable for each point cloud pair.

[0085] S43. Use the optimal transformation matrix T to correct the deformed socket and restore it to its standard shape P. c =T·P d .

[0086] S5. Based on the extrinsic parameters of the depth camera and the real-time pose of the AGV body in world coordinates in step S1, calculate the pose of the pallet in world coordinates; then obtain the movement path of the AGV from the picking point to the picking point according to the Bezier curve, and control the forklift to travel to the picking point according to the movement path.

[0087] To achieve accurate pallet picking, AGVs need to perform local path planning based on the pallet's pose and make adaptive control adjustments according to the new trajectory. The Bézier curve is known; for details, refer to "Research on Intelligent Vehicle Obstacle Avoidance Path Planning Methods" (Journal of Nanjing University of Science and Technology, Vol. 47, No. 2), which explores obstacle avoidance path planning methods for intelligent vehicles and conducts research on quintic Bézier curves, cubic Bézier curves, quintic spline curves, and cubic spline curve obstacle avoidance paths.

[0088] Based on the relationship between the pixel coordinate system and the camera coordinate system during the AGV's end-effector operation, and according to the camera model, in the camera coordinate system, the xz plane is parallel to the ground, the z-axis points directly in front of the camera, and the y-axis is perpendicular to the ground and points downwards. The normalized plane is the z=1 plane in front of the camera. P is any point on the pallet, with coordinates (X,Y,Z) in the forklift camera coordinate system. p is the point corresponding to P in the normalized plane, with pixel coordinates (x,y). p and P have the following mapping relationship.

[0089]

[0090] Where, f x ,f y For normalized focal length, c x ,c y Z represents the coordinates of the principal point, Z represents the distance from the object to the camera (i.e., the depth measured by the depth camera), and K represents the intrinsic parameter matrix of the camera.

[0091] The coordinate transformation from the world 3D coordinate system to the pixel coordinate system is performed. In a real-world scenario, R and t information are obtained based on the calibration results to estimate the tray pose.

[0092]

[0093] R1 and t1 are the extrinsic parameters calibrated during the installation of the depth camera, while R2 and t2 are the real-time pose parameters of the AGV body relative to the world coordinate system. Through rigid body motion calculations, the coordinates PO(Xw, Yw, Zw) of the center point PO of the middle support column of the pallet in the world coordinate system are obtained. The world coordinates of the other two supports are calculated using a similar method.

[0094] S6. After the AGV arrives at the pickup point, adjust the fork height so that the depth camera can capture the right RGB image of the pallet to be picked up. Process the right RGB image according to step S3. Then, segment the processed right RGB image to obtain the pixel coordinates of the center point of the middle leg and the right leg of the pallet. Calculate the height and tilt angle of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera. Adjust the height and rotation angle of the AGV forks accordingly. After straight-line travel to compensate for the distance, complete the pickup action.

[0095] In this embodiment, the compensation distance is set to 0.8m. When the distance between the fork tip and the center of the pallet is 0.8m, the depth camera's field of view can capture part of the pallet (including the right side area, including the center leg and the right insertion hole).

[0096] The method for obtaining the pixel coordinates of the center points of the middle leg and right leg of the tray is as follows:

[0097] S61. Perform mean grayscale processing on the RGB image on the right through the three RGB channels, that is, the RGB pixels of the right image are:

[0098] Gray(u,v)=(R(u,v)+G(u,v)+B(u,v)) / 3

[0099] S62. Set a threshold K to perform image binarization, set pixels with pixel values ​​greater than threshold K to white, and set pixels with pixel values ​​less than threshold K to black.

[0100] S63. Filter out noise in the binarized image, then draw the outer rectangle of the white area, and select the area with the largest area in the outer rectangle as the right jack area.

[0101] S64. Calculate the pixel coordinates of the center points of the middle leg and the right leg based on the width of the tray socket area and the width of the middle leg and the right leg of the tray.

[0102] The height of the AGV fork is adjusted as follows: H = dy + 1 / 2H0 - ΔH, where dy is the relative height in the extrinsic parameters of the depth camera, that is, the relative height between the center of the depth camera and the horizontal plane of the fork; H0 is the thickness of the cross-section of the fork tip of the AGV fork; ΔH is the height of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera, which is affected by the pitch angle and roll angle of the depth camera.

[0103] Where H0 is the thickness of the cross-sectional area of ​​the AGV fork tip, which is determined by the vehicle model; since the depth camera is installed at the fork tip, it can be assumed that the height of the depth camera center and the horizontal plane of the fork is half the height of the fork tip cross-section. In this embodiment, the thickness of the AGV fork tip cross-section is 30mm, that is, the relative height between the depth camera and the horizontal plane of the fork is 15mm.

[0104] H can have positive and negative values. A positive value indicates the height that needs to be raised, and a negative value indicates the height that needs to be lowered.

[0105]

[0106] Where d is the depth value of the center point O of the middle leg of the tray; pitch and roll are the pitch and roll angles of the depth camera relative to the vehicle body, respectively; p x p y It is the pixel coordinate of the center point O of the middle leg of the tray, f x f y C represents the pixel focal length in the horizontal and vertical directions of the depth camera, respectively. x C y These represent the coordinates of the depth camera's optical center in the pixel coordinate system.

[0107] In a further embodiment, the formula for calculating the tilt angle of the center of the tray's middle leg relative to the horizontal plane of the depth camera's center is as follows:

[0108] When two endpoints on the same vertical line are selected on the cross-section of the pallet, the height difference between them is H, and their spatial relationship is expressed by the following formula:

[0109]

[0110] If you choose two pairs of points P perpendicular to the middle leg... 11 ,P 12 The right leg is perpendicular to point P. 21 ,P 22 Two points P on the upper edge of the tray that are on the same straight line 11 and P 21 And the two ends P1 and P2 of the upper edge of the tray. r The following spatial relationships exist:

[0111]

[0112] Then point P 11 and P 21 The spatial distance W is:

[0113]

[0114] X, Y, and Z represent the three-dimensional spatial coordinates of different points in camera coordinates, cy Indicates the coordinates of the camera's optical center;

[0115] The tilt angle of the tray is:

[0116]

[0117] In the above formula: x 11 y 11 Point P 11 Pixel coordinates, x 21 y 21 Point P 21 Pixel coordinates, x1, x r Representing points P1 and P2 respectively r The x-axis coordinates and y-axis coordinates of the pixels. 12 y 22 They represent point P respectively. 12 P 22 The pixel coordinates y-axis coordinates; f y This represents the normalized focal length of the depth camera's horizontal axis.

[0118] Where P 11 P 12 P represents the two endpoints of the middle leg of the tray in the vertical direction. 21 P 22 The two endpoints of the right leg of the pallet are perpendicular to each other; H is the height of the pallet, and L is the width of the pallet; P1, P r These are two points on the upper edge of the tray that lie on the same straight line.

[0119] By calculating the height and tilt angle of the center of the pallet's middle leg relative to the horizontal plane of the depth camera center, the AGV can adjust the lifting or lowering height of the forks when it reaches the picking point, and adjust the rotation angle of the forks according to the tilt angle, and compensate for the straight travel distance (the compensation distance in this embodiment is 0.8m) to complete the picking action in the soft package stacking scenario; after picking up the goods, the horizontal height of the forks is adjusted to complete the end operation process.

[0120] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for positioning and measuring stacked AGV soft-pack objects, characterized in that: Includes the following steps: S1. Install the depth camera on the right fork tip of the AGV fork, and calibrate the external parameters of the depth camera with the center of the AGV body as a reference. S2. Pre-collect pallet image data in the AGV soft package stacking scenario, and train it into a pallet detection model using deep learning methods; When S3 and AGV arrive at the picking point, the depth camera collects RGB images and depth data of the area where the pallet to be picked up is located, and performs synchronization and alignment processing on the RGB images and depth data before performing brightness enhancement processing. S4. Input the RGB image after brightness enhancement into the tray detection model in step S2 to obtain the tray area image, and then locate the tray hole according to its depth data. S5. Based on the extrinsic parameters of the depth camera and the real-time pose of the AGV body in world coordinates in step S1, calculate the pose of the pallet in world coordinates; then obtain the movement path of the AGV from the picking point to the picking point according to the Bezier curve, and control the forklift to travel to the picking point according to the movement path. S6. After the AGV arrives at the pickup point, adjust the fork height so that the depth camera can capture the right RGB image of the pallet to be picked up. Process the right RGB image according to step S3. Then, segment the processed right RGB image to obtain the pixel coordinates of the center point of the middle leg and the right leg of the pallet. Calculate the height and tilt angle of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera. Adjust the height and rotation angle of the AGV forks accordingly. After straight-line travel to compensate for the distance, complete the pickup action. Therefore, the height of the AGV forks is adjusted accordingly: H = dy + 1 / 2H0 - H, where dy is the relative height dy in the extrinsic parameters of the depth camera, that is, the relative height between the center of the depth camera and the horizontal plane on the fork; H0 is the thickness of the cross-sectional area of ​​the fork tip of the AGV fork. H is the height of the center of the tray's middle leg relative to the horizontal plane of the depth camera's center. Where d is the depth value of the center point O of the middle leg of the tray; pitch and roll are the pitch and roll angles of the depth camera relative to the vehicle body, respectively; p x p y It is the pixel coordinate of the center point O of the middle leg of the tray, f x f y C represents the pixel focal length in the horizontal and vertical directions of the depth camera, respectively. x C y These represent the coordinates of the depth camera's optical center in the pixel coordinate system.

2. The positioning and measurement method according to claim 1, characterized in that: In step S1, the extrinsic parameters of the depth camera include horizontal offset dx, relative height dy, vertical offset dz, yaw angle, roll angle, and pitch angle; the relative height dy is the relative height between the center of the depth camera and the horizontal plane above the fork.

3. The positioning and measurement method according to claim 1, characterized in that: In step S2, the pallet image data includes RGB image data of the pallet at a yaw angle of -30 to 30°, a horizontal offset of -400 to 400 mm, and a distance of 0.8 to 2.5 m from the fork tip at any height when the AGV vehicle is at the pickup pre-position point and pickup point.

4. The positioning and measurement method according to claim 1, characterized in that: In step S3, the synchronization process includes timestamp generation and timestamp comparison, wherein timestamp generation refers to the depth camera automatically generating timestamps when acquiring each frame of image data; Timestamp matching refers to determining the degree of temporal alignment between RGB images and depth data by analyzing their timestamps. The alignment process involves converting two-dimensional points in the depth data from pixel coordinates to world coordinates based on the depth camera transformation matrix, and then converting them back to pixel coordinates in the RGB image.

5. The positioning and measurement method according to claim 1, characterized in that: In step S3, the brightness enhancement process is as follows: S31. Convert the RGB image to be processed to CMY space, and subtract the minimum value from the C, M and Y values ​​of each pixel, i.e., C-=min(C,M,Y), M-=min(C,M,Y), Y-=min(C,M,Y). S32. Next, convert the RGB image processed in step S31 to HSV space, calculate the maximum value vMin and the minimum value vMax of V, and requantize the V value: newPixel=(oldPixel / 255-vMin) / (vMax-vMin); S33. Following steps S32 and S31, reverse the image to obtain a color-enhanced image.

6. The positioning and measurement method according to claim 1, characterized in that: In step S4, the steps for positioning the tray insertion hole are as follows: S41. Convert the depth data from the depth camera into point cloud information, and use the standard-shaped tray socket point cloud as the template socket point cloud P. t The real-time detected deformation jack point cloud is P d ; S42. Using the ICP algorithm, the deformed jack point cloud P d Point cloud P with standard jack t Registration is performed to obtain an optimal transformation matrix. , making and The difference is minimal; in, and These are the point cloud data for deformable sockets and template sockets, respectively. N is the number of point pairs in the deformable socket point cloud and template socket point cloud, and i is the variable for each point cloud pair. S43. Using the optimal transformation matrix The deformed sockets were corrected to restore them to their standard shape. .

7. The positioning and measurement method according to claim 1, characterized in that: In step S6, the method for obtaining the pixel coordinates of the center points of the middle leg and the right leg of the tray is as follows: S61. Perform mean grayscale processing on the RGB image on the right through the three RGB channels, that is, the RGB pixels of the right image are: Gray(u,v)=(R(u,v)+G(u,v)+B(u,v)) / 3 S62. Set a threshold K to perform image binarization, set pixels with pixel values ​​greater than threshold K to white, and set pixels with pixel values ​​less than threshold K to black. S63. Filter out noise in the binarized image, then draw the outer rectangle of the white area, and select the area with the largest area in the outer rectangle as the right jack area. S64. Calculate the pixel coordinates of the center points of the middle leg and the right leg based on the width of the tray socket area and the width of the middle leg and the right leg of the tray.

8. The positioning and measurement method according to claim 1, characterized in that: In step S6, the formula for calculating the tilt angle of the center of the tray's middle leg relative to the horizontal plane of the depth camera's center is as follows: In the above formula: x 11 y 11 Point P 11 Pixel coordinates, x 21 y 21 Point P 21 Pixel coordinates, x1, x r Representing points P1 and P2 respectively r The x-axis coordinates and y-axis coordinates of the pixels. 12 y 22 They represent point P respectively. 12 P 22 The pixel coordinates y-axis coordinates; f y This represents the normalized focal length of the depth camera's horizontal axis. Where P 11 P 12 P represents the two endpoints of the middle leg of the tray in the vertical direction. 21 P 22 The two endpoints of the right leg of the tray are vertically aligned; H is the height of the tray; P1, P... r These are two points on the upper edge of the tray that lie on the same straight line.

Images