Method for positioning and measuring AGV (Automatic Guided Vehicle) soft package stacking object

Abstract

The invention discloses an AGV soft package stacking object positioning and measuring method comprising the following steps: installing a depth camera on a right fork tip of an AGV pallet fork, and pre-collecting pallet image data in an AGV soft package stacking scene; when a goods taking front point is reached, RGB images and depth data of an area where a tray to be forked is located are collected and input into the tray detection model after being processed, and tray insertion holes are positioned; the AGV is controlled to run to a goods taking point according to the moving path; and acquiring a right RGB image of the to-be-forked tray, performing segmentation after processing, adjusting the height and the rotation angle of an AGV pallet fork, and performing linear driving for a compensation distance to complete a cargo taking action. The tray is recognized and positioned at the goods taking front point and the goods taking point, the height and the rotating angle of the pallet fork are adjusted, and tail end operation is completed. Therefore, the application scenes of extrusion, deformation, inclination and small-part shielding during soft package stacking are met, and the robustness and environmental adaptability of a tail end operation system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial vehicle automatic driving, and in particular to a method for positioning and measuring an AGV soft package stacking object. Background Art

[0002] AGV (Automatic Guided Vehicle) is a key equipment in automated logistics and transportation systems and flexible production systems, and plays an important role in automated logistics warehouses. In the process of intelligent material handling, it has complete and mature solutions for job task management, vehicle traffic management, vehicle path planning, vehicle navigation and vehicle motion control. Among them, for the terminal operation of AGV task execution, intelligent detection and identification of the position, state and type of the work object are required. At present, it mainly relies on 3D vision for pallet image acquisition or laser radar to acquire point cloud data, combined with the corresponding image processing algorithm and point cloud data processing to identify the forklift cargo pallet, and obtain its posture, and then automatically adjust the fork entry direction to realize unmanned intelligent pallet handling, and solve the problem of large offset angle of the unmanned forklift docking pallet position.

[0003] However, for AGV soft package stacking scenarios, the posture of the pallet carrying soft package goods (such as fertilizers, feed, etc.) is uncertain, such as being squeezed, deformed, tilted, or partially blocked by multiple layers of stacked soft package goods. Currently, the AGV warehouse management system is mainly used to read the fixed height in the warehouse to adjust the fork tip lifting height, which cannot meet the needs of soft package stacking scenarios with height changes, because the soft package stack will have local settlement over time, and the pallet will have irregular tilts. There is also the use of label-assisted identification and positioning, which requires manual participation, a large amount of additional work, and cannot meet the needs of highly dynamic AGV soft package stacking scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a positioning and measuring method for AGV soft package stacking objects, so as to realize the identification, measurement and positioning of the hole position in the pallet, and then realize the control of the movement of the forklift and the fork to achieve the purpose of accurate forking.

[0005] The specific technical solutions of the present invention are as follows:

[0006] The positioning and measuring method of the AGV soft package stacking object includes the following steps:

[0007] S1. Install the depth camera on the right fork tip of the AGV fork, and calibrate the external parameters of the depth camera using the center of the AGV body as a reference.

[0008] S2: Collect pallet image data in the AGV soft package stacking scenario in advance and train it into a pallet detection model through deep learning methods;

[0009] S3. When the AGV arrives at the forward pickup point, the depth camera collects the RGB image and depth data of the area where the pallet to be picked up is located, and synchronizes and aligns the RGB image and depth data before performing brightness enhancement processing.

[0010] S4, inputting the RGB image after brightness enhancement processing into the pallet detection model in step S2 to obtain a pallet area image, and then locating the pallet insertion hole according to its depth data;

[0011] S5, according to the external parameters of the depth camera in step S1 and the real-time posture of the AGV body in the world coordinate, calculate the posture of the pallet in the world coordinate; then obtain the moving path of the AGV from the pre-pickup point to the pick-up point according to the Bezier curve, and control the forklift to travel to the pick-up point according to the moving path;

[0012] S6. After the AGV arrives at the pickup point, adjust the height of the fork so that the depth camera can capture the right RGB image of the pallet to be picked up, and process the right RGB image according to step S3; then segment the processed right RGB image to obtain the pixel coordinates of the center points of the middle leg and the right leg of the pallet; calculate the height and inclination angle of the center of the middle leg of the pallet relative to the horizontal plane of the depth camera center, and adjust the height and rotation angle of the AGV fork accordingly, and complete the pickup action after driving in a straight line to compensate the distance.

[0013] In a further solution, in step S1, the external parameters of the depth camera include a horizontal offset dx, a relative height dy, a vertical offset dz, a yaw angle, a roll angle and a pitch angle; the relative height dy is the relative height between the center of the depth camera and the horizontal plane on the fork.

[0014] In a further solution, in step S2, the pallet image data includes RGB image data of the pallet at a yaw angle of -30 to 30°, a horizontal offset of -400 to 400 mm, a distance from the fork tip of 0.8 to 2.5 m and any height when the AGV body is at the pre-pickup point and the pick-up point.

[0015] In a further solution, in step S3, the synchronization processing includes timestamp generation and timestamp comparison, wherein timestamp generation refers to the automatic generation of timestamps by the depth camera when collecting each frame of image data; timestamp comparison refers to determining the degree of time alignment of the RGB image and the depth data by analyzing their timestamps;

[0016] The alignment process is to transform the two-dimensional points on the depth data from pixel coordinates to world coordinates according to the depth camera transformation matrix, and then transform them into pixel coordinates of the RGB image.

[0017] In a further solution, in step S3, the brightness enhancement processing steps are as follows:

[0018] S31, convert the RGB image to be processed into CMY space, and subtract the minimum value from each pixel point C, M, and Y, that is, C-=min(C,M,Y), M-=min(C,M,Y), and Y-=min(C,M,Y);

[0019] S32, convert the RGB image processed in step S31 to the HSV space, calculate the maximum value vMin and the minimum value vMax of V, and requantize the V value: newPixel=(oldPixel / 255-vMin) / (vMax-vMin);

[0020] S33. According to steps S32 and S31, the image is reversely converted to obtain a color-enhanced image.

[0021] In a further solution, in step S4, the steps of positioning the tray insertion hole are as follows:

[0022] S41, convert the depth data of the depth camera into point cloud information, and use the standard shape of the tray socket point cloud as the template socket point cloud P t , the real-time detected deformation socket point cloud is P d ;

[0023] S42, through the ICP algorithm, the deformation socket point cloud P d With standard jack point cloud P t Perform registration and obtain an optimal transformation matrix T so that P d With P t The difference is minimal;

[0024]

[0025] in, and are the point cloud data of the deformed socket and the template socket, respectively, N is the number of point pairs of the deformed socket point cloud and the template socket point cloud, and i is the variable of each pair of point clouds;

[0026] S43, using the optimal transformation matrix T to correct the deformed socket and restore it to the standard shape P c =T·P d .

[0027] In a further solution, in step S6, the method for obtaining the pixel coordinates of the center points of the middle leg and the right leg of the tray is as follows:

[0028] S61, perform mean grayscale processing on the right RGB image through the three RGB channels, that is, the RGB pixels of the right image are:

[0029] Gray(u,v)=(R(u,v)+G(u,v)+B(u,v)) / 3

[0030] S62, setting a threshold K to perform image binarization, setting pixels with values ​​greater than the threshold K to white, and setting pixels with values ​​less than the threshold K to black;

[0031] S63, filtering out noise points in the binary image, then drawing a circumscribed rectangle of the white area, and selecting the area with the largest area in the circumscribed rectangle as the right jack area;

[0032] S64. Calculate the pixel coordinates of the center points of the middle leg and the right leg according to the width of the tray insertion area and the widths of the middle leg and the right leg of the tray.

[0033] In a further solution, in step S6, the height of the AGV fork is adjusted to H=dy+1 / 2H0-ΔH, where dy is the relative height dy in the external parameter of the depth camera, that is, the relative height between the center of the depth camera and the horizontal plane on the fork; H0 is the cross-sectional thickness of the fork tip of the AGV fork; ΔH is the height of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera;

[0034] Among them, H0 is the cross-sectional thickness of the fork tip of the AGV fork, which is determined by the vehicle model; H has positive and negative values, a positive value indicates the height that needs to be lifted, and a negative value indicates the height that needs to be lowered.

[0035]

[0036] Where d is the depth value of point O, the center of the middle leg of the pallet; pitch and roll are the pitch angle and roll angle calibrated by the depth camera relative to the vehicle body; p x 、p y is the pixel coordinate of the center point O of the middle leg of the tray, f x 、f y Respectively represent the pixel focal length of the depth camera in the horizontal and vertical directions, C x , C y Respectively represent the coordinates of the optical center of the depth camera in the pixel coordinate system.

[0037] In a further solution, in step S6, the calculation formula of the inclination angle of the center of the middle leg of the tray relative to the horizontal plane of the center of the depth camera is as follows:

[0038]

[0039] In the above formula: x 11 ,y 11 Indicates point P 11 Pixel coordinate, x 21 ,y 21 Indicates point P 21 Pixel coordinates, x1, x r Respectively represent points P1, P r The x-axis coordinate of the pixel coordinate, y12 ,y 22 Respectively represent point P 12 , P 22 The pixel coordinate y-axis coordinate; f y Indicates the normalized focal length of the depth camera's horizontal coordinate;

[0040] Where P 11 , P 12 P is the two end points of the middle leg of the tray in the vertical direction. 21 , P 22 are the two end points of the right leg of the tray in the vertical direction; H is P 11 , P 12 The height difference between P1 and P r The two points on the upper edge of the tray are located on the same straight line.

[0041] The present invention installs a depth camera at the right fork tip of the fork, and the AGV body is identified and located at the pre-pickup point and the pick-up point respectively. The RGB image recognition and depth data feature extraction of the pallet are used to perform high-precision position positioning and height and tilt angle measurement respectively. The fork tip is raised or lowered to a relative height, and the fork tip is rotated according to the tilt angle of the pallet to form a local path curve algorithm and fork tip adjustment action, thereby completing the terminal operation of picking up and releasing goods and other functions. This meets the soft package stacking scene, adapts to the application scenes of extrusion, deformation, tilt, and a small amount of occlusion, and improves the robustness and environmental adaptability of the terminal operation system.

[0042] By calculating the height and inclination angle of the center of the middle leg of the pallet relative to the horizontal plane of the depth camera center, the AGV body can adjust the lifting or lowering height of the fork accordingly when it arrives at the pickup point; and adjust the rotation angle of the fork according to the inclination angle, and compensate the distance for straight-line driving to complete the pickup action in the soft package stacking scenario; after picking up the goods, adjust the horizontal height of the fork to complete the terminal operation process.

[0043] The method of the present invention is suitable for soft package stacking application scenarios, where the pallet is squeezed, deformed, tilted, partially blocked, etc. The depth camera is installed at the right fork tip of the fork, and the identification, positioning and high-precision measurement of the elevation angle are performed at the leading point and the pickup point respectively, forming a two-stage operation action, expanding the application scenario of AGV terminal operation. DETAILED DESCRIPTION

[0044] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively below in conjunction with specific embodiments. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly understood.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0046] Example:

[0047] The positioning and measuring method of the AGV soft package stacking object includes the following steps:

[0048] S1. Install the depth camera on the right fork tip of the AGV fork, and calibrate the external parameters of the depth camera using the center of the AGV body as a reference.

[0049] In step S1, since the depth camera's field of view can reach 90° horizontal field of view and 60° vertical field of view, and in order to save the cost of the terminal device, the present invention selects the right fork tip for installation (the installation method of the left fork tip is similar). The effective measurement range of the depth camera is 0.3-5m. The farther the measurement distance, the worse the accuracy. The depth camera has a smaller accuracy error at close range.

[0050] Therefore, at the front point of picking up goods, the distance between the fork tip and the center of the pallet is about 1.5-2.5m, and the depth is relatively far away from the goods on the pallet. The camera field of view can capture the area of ​​the entire pallet, and the depth error is reduced through the multi-frame depth data fusion method to calculate the pallet position information and yaw angle information; at the picking up point, the distance between the fork tip and the center of the pallet is about 0.8-1.5m. The depth camera field of view can capture the area of ​​part of the pallet. The depth data collected by the depth camera is relatively accurate and can measure the tilt angle of the pallet with high precision.

[0051] The external parameters of the depth camera include horizontal offset dx, relative height dy, vertical offset dz, yaw angle, roll angle and pitch angle; the relative height dy is the relative height between the center of the depth camera and the horizontal plane on the fork.

[0052] The calibration method for calibrating the external parameters of the depth camera includes the following steps:

[0053] S11, the depth camera is installed on the right fork tip of the fork so that the plane of the depth camera is perpendicular to the horizontal plane;

[0054] S12. Use an auxiliary 4*5 grid checkerboard calibration plate tool and place it 1m in front of the right fork tip of the fork;

[0055] S13. According to the vertical field of view of the depth camera of 60°, the fork needs to be lifted to a height of 0.57m (1m*tan30°). The height of the checkerboard calibration board is appropriately raised and recorded so that the 4*5 checkerboards in the checkerboard calibration board are completely captured by the camera.

[0056] S14, collecting a checkerboard calibration plate image through a depth camera, and performing checkerboard corner point detection using an image processing method to obtain pixel coordinates of 12 checkerboard corner points;

[0057] S15. Based on the Zhang Zhengyou calibration method, calculate the transformation matrix between the depth camera coordinate system and the chessboard pixel coordinate system as the external parameter matrix from the depth camera to the chessboard grid coordinate system;

[0058] S16, by measuring the translation of the coordinate system of the AGV body center and the chessboard calibration plate on the x and y axes, and the fork tip lifting height, a transformation matrix between the two is obtained as an external parameter matrix from the chessboard grid to the center of the vehicle;

[0059] S17. Based on the two extrinsic parameter matrices, the extrinsic parameter matrix from the camera to the center of the AGV body is calculated to achieve extrinsic parameter calibration of the depth camera;

[0060] In order to reduce the error in external parameter calibration, repeat the above steps multiple times to obtain the external parameter matrix between multiple camera coordinate systems and the vehicle center coordinate system, and obtain the final external parameter matrix by taking the average value.

[0061] S2: Collect pallet image data in the AGV soft package stacking scenario in advance and train it into a pallet detection model through deep learning methods;

[0062] The pallet image data includes RGB image data of the pallet at a yaw angle of -30 to 30°, a horizontal offset of -400 to 400 mm, and a distance from the fork tip of 0.8 to 2.5 m and at any height when the AGV body is at the pre-pickup point and the pick-up point.

[0063] The open source tool LabImg is used to calibrate the RGB image data of the above-mentioned 10,000 pallets area; then, a small target detection layer network is added through the existing open source YOLOv8 backbone network architecture, and the above-mentioned calibrated 10,000 pallets classification data are used, and the calibrated data are subjected to multiple rounds of training in a data enhancement manner; the deep learning model of the domestic platform of the present invention refers to the domestic Rockchip RK3568 platform, the model is simulated and run on the server to obtain the reasoning results, the model is converted into an RKNN model through the RK Toolkit tool model, and quantitative reasoning is designed to obtain the model output results.

[0064] S3. When the AGV arrives at the forward pickup point, the depth camera collects the RGB image and depth data of the area where the pallet to be picked up is located, and synchronizes and aligns the RGB image and depth data before performing brightness enhancement processing.

[0065] The synchronization process includes timestamp generation and timestamp comparison, wherein timestamp generation refers to the automatic generation of timestamps by the depth camera when collecting each frame of image data; timestamp comparison refers to determining the degree of time alignment of RGB images and depth data by analyzing their timestamps;

[0066] The alignment process is to transform the two-dimensional points on the depth data from pixel coordinates to world coordinates according to the depth camera transformation matrix, and then transform them into pixel coordinates of the RGB image.

[0067] The brightness enhancement processing steps are as follows:

[0068] S31, convert the RGB image to be processed into CMY space, and subtract the minimum value from each pixel point C, M, and Y, that is, C-=min(C,M,Y), M-=min(C,M,Y), and Y-=min(C,M,Y);

[0069] S32, convert the RGB image processed in step S31 to the HSV space, calculate the maximum value vMin and the minimum value vMax of V, and requantize the V value: newPixel=(oldPixel / 255-vMin) / (vMax-vMin);

[0070] S33. According to steps S32 and S31, the image is reversely converted to obtain a color-enhanced image.

[0071] Since RGB images provide detailed color and texture information, while depth data provides distance and spatial structure information, only when these two types of data are fully synchronized can this information be accurately combined to identify objects, determine their positions and sizes, and perform complex three-dimensional spatial analysis and calculations. Therefore, it is necessary to synchronize the RGB images and depth data.

[0072] Due to the differences in the optical center position, internal parameters, external parameters and processes of the RGB camera and the depth camera, the field of view of the RGB image and the depth image collected at the same distance and angle are different. The field of view of the RGB image is small, and the field of view of the depth image is large. The alignment principle is to convert the two-dimensional points on the depth image from pixel coordinates to world coordinates according to the camera conversion matrix, and then convert them to pixel coordinates of the RGB image.

[0073] Let the RGB internal parameter matrix be K rgb , the depth camera internal parameter matrix is ​​K d , world coordinate to RGB camera coordinate external parameter T w2rgb , the external parameter from world coordinates to depth camera coordinates is T w2d , depth camera to RGB camera coordinate external parameter T d2rgb(T can be obtained through calibration or depth camera SDK), calculated as follows:

[0074] 1) Point P in the depth pixel coordinate system d The coordinates converted to the depth camera coordinate system are: Z d is the depth value of the current point;

[0075] 2) P in the depth camera coordinate system D The coordinates converted to world coordinates are:

[0076] 3) P in world coordinates w The coordinates converted to the RGB camera coordinate system are:

[0077] 4)P C Convert to RGB pixel coordinate system: Z rgb is the depth value of the RGB pixel at the current point, that is Consider that the camera planes of the RGB and depth cameras are in the same plane, i.e., Z d and Z rgb The values ​​are equal; the conversion of pixels on the depth image to pixels on the RGB image is completed:

[0078]

[0079] S4, inputting the RGB image after brightness enhancement processing into the pallet detection model in step S2 to obtain a pallet area image, and then locating the pallet insertion hole according to its depth data;

[0080] The steps for locating the tray jack are as follows:

[0081] S41, convert the depth data of the depth camera into point cloud information, and use the standard shape of the tray socket point cloud as the template socket point cloud P t , the real-time detected deformation socket point cloud is P d ;

[0082] S42, through the ICP algorithm, the deformation socket point cloud P d With standard jack point cloud P t Perform registration and obtain an optimal transformation matrix T so that P d With P t The difference is minimal;

[0083]

[0084] in, and are the point cloud data of the deformed socket and the template socket, respectively, N is the number of point pairs of the deformed socket point cloud and the template socket point cloud, and i is the variable of each pair of point clouds;

[0085] S43, using the optimal transformation matrix T to correct the deformed socket and restore it to the standard shape P c =T·P d .

[0086] S5, according to the external parameters of the depth camera in step S1 and the real-time posture of the AGV body in the world coordinate, calculate the posture of the pallet in the world coordinate; then obtain the moving path of the AGV from the pre-pickup point to the pick-up point according to the Bezier curve, and control the forklift to travel to the pick-up point according to the moving path;

[0087] In order for AGV to accurately pick up pallets, it is necessary to perform local path planning based on the position of the pallet and make adaptive control adjustments according to the new trajectory. The Bezier curve is known, and the specific reference is "Research on Intelligent Vehicle Obstacle Avoidance Path Planning Method" (Journal of Nanjing University of Science and Technology, Vol. 47, No. 2), which explores the intelligent vehicle obstacle avoidance path planning method and conducts research on quintic Bezier curves, cubic Bezier curves, quintic spline curves and cubic spline curves.

[0088] According to the relationship between the pixel coordinate system and the camera coordinate system during the end operation of the AGV, according to the camera model, in the camera coordinate system, the xz plane is parallel to the ground, the z axis is in the direction of the camera, and the y axis is perpendicular to the ground and facing downward. The normalized plane is the z=1 plane in front of the camera. P is any point on the pallet, and its coordinates in the forklift camera coordinate system are (X, Y, Z). p is the point corresponding to P in the normalized plane, and its pixel coordinates are (x, y). p and P have the following mapping relationship

[0089]

[0090] Among them, f x ,f y is the normalized focal length, c x ,c y is the coordinate of the principal point, Z is the distance from the object to the camera (that is, the depth measured by the depth camera), and K is the intrinsic parameter matrix of the camera.

[0091] Coordinate transformation between the world 3D coordinate system and the pixel coordinate system. In the actual scene, the R and t information is obtained according to the calibration results to estimate the position and posture of the pallet:

[0092]

[0093] Among them, R1 and t1 are the external parameters calibrated when the depth camera is installed, and R2 and t2 are the real-time posture parameters of the AGV body relative to the world coordinate system. Through rigid body motion calculation, the coordinates PO (Xw, Yw, Zw) of the center point PO of the middle support column of the pallet in the world coordinate system are obtained. The world coordinates of the other two supports are calculated in a similar way.

[0094] S6. After the AGV arrives at the pickup point, adjust the height of the fork so that the depth camera can capture the right RGB image of the pallet to be picked up, and process the right RGB image according to step S3; then segment the processed right RGB image to obtain the pixel coordinates of the center points of the middle leg and the right leg of the pallet; calculate the height and inclination angle of the center of the middle leg of the pallet relative to the horizontal plane of the depth camera center, and adjust the height and rotation angle of the AGV fork accordingly, and complete the pickup action after driving in a straight line to compensate the distance.

[0095] In this embodiment, the compensation distance is set to 0.8m. When the distance between the fork tip and the center of the pallet is 0.8m, the field of view of the depth camera can capture part of the pallet (including the right side area including the center leg and the right plug hole).

[0096] The method for obtaining the pixel coordinates of the center points of the middle leg and the right leg of the tray is as follows:

[0097] S61, perform mean grayscale processing on the right RGB image through the three RGB channels, that is, the RGB pixels of the right image are:

[0098] Gray(u,v)=(R(u,v)+G(u,v)+B(u,v)) / 3

[0099] S62, setting a threshold K to perform image binarization, setting pixels with values ​​greater than the threshold K to white, and setting pixels with values ​​less than the threshold K to black;

[0100] S63, filtering out noise points in the binary image, then drawing a circumscribed rectangle of the white area, and selecting the area with the largest area in the circumscribed rectangle as the right jack area;

[0101] S64. Calculate the pixel coordinates of the center points of the middle leg and the right leg according to the width of the tray insertion area and the widths of the middle leg and the right leg of the tray.

[0102] Among them, the height of the AGV fork is adjusted to H=dy+1 / 2H0-ΔH, where dy is the relative height dy in the external parameters of the depth camera, that is, the relative height between the center of the depth camera and the horizontal plane on the fork; H0 is the cross-sectional thickness of the fork tip of the AGV fork; ΔH is the height of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera, which is affected by the pitch angle (pitch) and roll angle (roll) of the depth camera.

[0103] Wherein, H0 is the thickness of the cross section of the fork tip of the AGV fork, and this parameter is determined by the vehicle model; since the depth camera is installed at the fork tip, it can be considered that the height between the center of the depth camera and the horizontal plane of the fork is half of the height value of the cross section of the fork tip. In this embodiment, the thickness of the cross section of the fork tip of the AGV is 30 mm, that is, the relative height between the depth camera and the horizontal plane of the fork is 15 mm.

[0104] H has positive and negative values. A positive value indicates the height that needs to be raised, and a negative value indicates the height that needs to be lowered.

[0105]

[0106] Where d is the depth value of point O, the center of the middle leg of the pallet; pitch and roll are the pitch angle and roll angle calibrated by the depth camera relative to the vehicle body; p x 、p y is the pixel coordinate of the center point O of the middle leg of the tray, f x 、f y Respectively represent the pixel focal length of the depth camera in the horizontal and vertical directions, C x , C y Respectively represent the coordinates of the optical center of the depth camera in the pixel coordinate system.

[0107] In a further solution, the calculation formula of the inclination angle of the center of the middle leg of the tray relative to the horizontal plane of the center of the depth camera is as follows:

[0108] When the upper and lower endpoints on the same vertical line of the pallet cross section are selected, the height difference is H, and the spatial relationship is expressed by the following formula:

[0109]

[0110] If we select two pairs of points P in the vertical direction of the middle leg 11 ,P 12 , right leg vertically opposite point P 21 ,P 22 , two points P on the upper edge of the tray that are on the same straight line 11 and P 21 , and the two end points P1 and P on the upper edge of the tray r . There are the following spatial relationships:

[0111]

[0112] Then point P 11 and P 21 The spatial distance W is:

[0113]

[0114] X, Y, and Z represent the three-dimensional space coordinates of different points under the camera coordinates.y Represents the coordinates of the camera optical center;

[0115] The tilt angle of the tray is:

[0116]

[0117] In the above formula: x 11 ,y 11 Indicates point P 11 Pixel coordinate, x 21 ,y 21 Indicates point P 21 Pixel coordinates, x1, x r Respectively represent points P1, P r The x-axis coordinate of the pixel coordinate, y 12 ,y 22 Respectively represent point P 12 , P 22 The pixel coordinate y-axis coordinate; f y Indicates the normalized focal length of the depth camera's horizontal coordinate;

[0118] Where P 11 , P 12 P is the two end points of the middle leg of the tray in the vertical direction. 21 , P 22 are the two end points of the right leg of the pallet in the vertical direction; H is the height of the pallet, L is the width of the pallet; P1, P r The two points on the upper edge of the tray are located on the same straight line.

[0119] By calculating the height and inclination angle of the center of the middle leg of the pallet relative to the horizontal plane of the depth camera center, the AGV body can adjust the lifting or lowering height of the fork accordingly when it arrives at the pickup point, and adjust the rotation angle of the fork according to the inclination angle, and drive in a straight line to compensate the distance (the compensation distance in this embodiment is 0.8m) to complete the pickup action in the soft package stacking scenario; when the goods are picked up, the horizontal height of the fork is adjusted to complete the terminal operation process.

[0120] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. The positioning and measurement method of AGV soft package stacking objects is characterized by: The following steps are involved: S1. Install the depth camera on the right fork tip of the AGV fork, and calibrate the external parameters of the depth camera using the center of the AGV body as a reference. S2: Collect pallet image data in the AGV soft package stacking scenario in advance and train it into a pallet detection model through deep learning methods; S3. When the AGV arrives at the forward pickup point, the depth camera collects the RGB image and depth data of the area where the pallet to be picked up is located, and synchronizes and aligns the RGB image and depth data before performing brightness enhancement processing. S4, inputting the RGB image after brightness enhancement processing into the pallet detection model in step S2 to obtain a pallet area image, and then locating the pallet insertion hole according to its depth data; S5, according to the external parameters of the depth camera in step S1 and the real-time posture of the AGV body in the world coordinate, calculate the posture of the pallet in the world coordinate; then obtain the moving path of the AGV from the pre-pickup point to the pick-up point according to the Bezier curve, and control the forklift to travel to the pick-up point according to the moving path; S6. After the AGV arrives at the pickup point, adjust the height of the fork so that the depth camera can capture the right RGB image of the pallet to be picked up, and process the right RGB image according to step S3; then segment the processed right RGB image to obtain the pixel coordinates of the center points of the middle leg and the right leg of the pallet; calculate the height and inclination angle of the center of the middle leg of the pallet relative to the horizontal plane of the depth camera center, and adjust the height and rotation angle of the AGV fork accordingly, and complete the pickup action after driving in a straight line to compensate the distance.

2. The positioning and measuring method according to claim 1, characterized in that: In step S1, the external parameters of the depth camera include horizontal offset dx, relative height dy, vertical offset dz, yaw angle, roll angle and pitch angle; the relative height dy is the relative height between the center of the depth camera and the horizontal plane on the fork.

3. The positioning and measuring method according to claim 1, characterized in that: In step S2, the pallet image data includes RGB image data of the pallet at a yaw angle of -30 to 30°, a horizontal offset of -400 to 400 mm, and a distance from the fork tip of 0.8 to 2.5 m and at any height when the AGV body is at the pre-pickup point and the pick-up point.

4. The positioning and measuring method according to claim 1, characterized in that: In step S3, the synchronization process includes timestamp generation and timestamp comparison, wherein timestamp generation refers to the depth camera automatically generating a timestamp when collecting each frame of image data; Timestamp comparison refers to analyzing the timestamps of RGB images and depth data to determine their temporal alignment; The alignment process is to transform the two-dimensional points on the depth data from pixel coordinates to world coordinates according to the depth camera transformation matrix, and then transform them into pixel coordinates of the RGB image.

5. The positioning and measuring method according to claim 1, characterized in that: In step S3, the brightness enhancement processing steps are as follows: S31, convert the RGB image to be processed into CMY space, and subtract the minimum value from each pixel point C, M, and Y, that is, C-=min(C,M,Y), M-=min(C,M,Y), and Y-=min(C,M,Y); S32, convert the RGB image processed in step S31 to the HSV space, calculate the maximum value vMin and the minimum value vMax of V, and requantize the V value: newPixel=(oldPixel / 255-vMin) / (vMax-vMin); S33. According to steps S32 and S31, the image is reversely converted to obtain a color-enhanced image.

6. The positioning and measuring method according to claim 1, characterized in that: In step S4, the steps of positioning the tray insertion hole are as follows: S41, convert the depth data of the depth camera into point cloud information, and use the standard shape of the tray socket point cloud as the template socket point cloud P t , the real-time detected deformation socket point cloud is P d ; S42, through the ICP algorithm, the deformation socket point cloud P d With standard jack point cloud P t Perform registration and obtain an optimal transformation matrix T so that P d With P t The difference is minimal; in, and P t i are the point cloud data of the deformed socket and the template socket, respectively, N is the number of point pairs of the deformed socket point cloud and the template socket point cloud, and i is the variable of each pair of point clouds; S43, using the optimal transformation matrix T to correct the deformed socket and restore it to the standard shape P c =T·P d .

7. The positioning and measuring method according to claim 1, characterized in that: In step S6, the method for obtaining the pixel coordinates of the center points of the middle leg and the right leg of the tray is as follows: S61, perform mean grayscale processing on the right RGB image through the three RGB channels, that is, the RGB pixels of the right image are: Gray(u,v)=(R(u,v)+G(u,v)+B(u,v)) / 3 S62, setting a threshold K to perform image binarization, setting pixels with values ​​greater than the threshold K to white, and setting pixels with values ​​less than the threshold K to black; S63, filtering out noise points in the binary image, then drawing a circumscribed rectangle of the white area, and selecting the area with the largest area in the circumscribed rectangle as the right jack area; S64. Calculate the pixel coordinates of the center points of the middle leg and the right leg according to the width of the tray insertion area and the widths of the middle leg and the right leg of the tray.

8. The positioning and measuring method according to claim 1, characterized in that: In step S6, the height of the AGV fork is adjusted to H=dy+1 / 2H0-ΔH, where dy is the relative height dy in the external parameter of the depth camera, that is, the relative height between the center of the depth camera and the horizontal plane on the fork; H0 is the cross-sectional thickness of the fork tip of the AGV fork; ΔH is the height of the center of the middle leg of the pallet relative to the horizontal plane of the center of the depth camera; Where d is the depth value of point O, the center of the middle leg of the pallet; pitch and roll are the pitch angle and roll angle calibrated by the depth camera relative to the vehicle body; p x 、p y is the pixel coordinate of the center point O of the middle leg of the tray, f x 、f y Respectively represent the pixel focal length of the depth camera in the horizontal and vertical directions, C x , C y Respectively represent the coordinates of the optical center of the depth camera in the pixel coordinate system.

9. The positioning and measuring method according to claim 1, characterized in that: In step S6, the calculation formula of the inclination angle of the center of the middle leg of the tray relative to the horizontal plane of the center of the depth camera is as follows: In the above formula: x 11 ,y 11 Indicates point P 11 Pixel coordinate, x 21 ,y 21 Indicates point P 21 Pixel coordinates, x1, x r Respectively represent points P1, P r The x-axis coordinate of the pixel coordinate, y 12 ,y 22 Respectively represent point P 12 , P 22 The pixel coordinate y-axis coordinate; f y Indicates the normalized focal length of the depth camera's horizontal coordinate; Where P 11 , P 12 P is the two end points of the middle leg of the tray in the vertical direction. 21 , P 22 are the two vertical endpoints of the right leg of the pallet; H is the height of the pallet; P1, P r The two points on the upper edge of the tray are located on the same straight line.

Images