Method, device, equipment and storage medium for placing goods on high-level shelves by driverless forklift

Abstract

The present invention belongs to the technical field of automatic control, and discloses a method, device, equipment and storage medium for an unmanned forklift to place goods on a high-level shelf. The method includes: obtaining image information of a forklift pallet and a shelf through a depth camera on the fork arm of the unmanned forklift; determining point cloud data of the forklift pallet and the shelf according to the image information of the forklift pallet and the shelf; determining spatial position information of the forklift pallet and the shelf according to the point cloud data; determining a distance difference between the front surfaces of the forklift pallet and the shelf according to the spatial position information; and controlling the pose adjustment of the fork arm according to the distance difference to complete the goods placement. Through the above method, it is possible to reduce the situation that the gantry shakes due to too high lifting when the unmanned forklift places goods on the high-level shelf, making the goods placement of the unmanned forklift more accurate, reducing the possibility of safety accidents, and improving the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic control, and particularly to a method, device, equipment and storage medium for placing goods on a high-level shelf by an unmanned forklift. Background Art

[0002] With the proposal of Industry 4.0 and intelligent manufacturing, the industrial field is continuously developing from traditional manufacturing to digital, intelligent and unmanned directions. In the era of everything being intelligent, unmanned operation is constantly attracting our attention. In the intelligent warehousing industry, the application demand for unmanned forklifts is increasing. Due to the too high extension of the forklift mast, the mast shakes, resulting in too large deviation in the goods placing truck, inaccurate goods placing and prone to safety accidents.

[0003] The above content is only used to assist in understanding the technical solution of the present invention, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for placing goods on a high-level shelf by an unmanned forklift, aiming to solve the technical problem in the prior art that the forklift mast extends too high, which easily causes the mast to shake, resulting in too large deviation in the goods placing truck, inaccurate goods placing and prone to safety accidents.

[0005] To achieve the above purpose, the present invention provides a method for placing goods on a high-level shelf by an unmanned forklift, and the method includes the following steps:

[0006] Obtain the image information of the forklift tray and the shelf through the depth camera on the forklift arm of the unmanned forklift;

[0007] Determine the point cloud data of the forklift tray and the shelf according to the image information of the forklift tray and the shelf;

[0008] Determine the spatial position information of the forklift tray and the shelf according to the point cloud data;

[0009] Determine the distance difference between the front surfaces of the forklift tray and the shelf according to the spatial position information;

[0010] Control the pose adjustment of the forklift arm according to the distance difference to complete the goods placing.

[0011] Optionally, the determining the point cloud data of the forklift tray and the shelf according to the image information of the forklift tray and the shelf includes:

[0012] Determine the point cloud information of the forklift tray and the shelf according to the image information of the forklift tray and the shelf;

[0013] Match the point cloud information of the forklift tray and the shelf with a preset checkerboard to obtain a set of matching three-dimensional point clouds;

[0014] Determine the rotation matrix and translation matrix from the depth camera to the unmanned forklift body according to the set of matched three-dimensional points;

[0015] Determine the point cloud data of the forklift tray and the shelf according to the rotation matrix, the translation matrix, and the point cloud information of the forklift tray and the shelf.

[0016] Optionally, the determining the point cloud data of the forklift tray and the shelf according to the rotation matrix, the translation matrix, and the point cloud information of the forklift tray and the shelf includes:

[0017] Determine the point cloud data of the forklift tray and the shelf in the camera coordinate system according to the point cloud information of the forklift tray and the shelf;

[0018] Convert the point cloud data in the camera coordinate system to the point cloud data in the forklift coordinate system according to the rotation matrix and the translation matrix;

[0019] Use the point cloud data in the forklift coordinate system as the point cloud data of the forklift tray and the shelf.

[0020] Optionally, the matching the point cloud information of the forklift tray and the shelf with a preset checkerboard to obtain a set of matched three-dimensional point clouds includes:

[0021] Determine the grayscale image of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf;

[0022] Determine the checkerboard corner points according to the grayscale image of the forklift tray and the shelf;

[0023] Fit a checkerboard plane formula according to the grayscale image of the forklift tray and the shelf and the checkerboard corner points;

[0024] Search and match the points on the grayscale image of the forklift tray and the shelf according to the checkerboard plane formula to obtain a set of two-dimensional point clouds with the same abscissa and ordinate;

[0025] Determine the homography matrix according to the set of two-dimensional point clouds;

[0026] Determine the set of matched three-dimensional point clouds according to the homography matrix and the checkerboard plane formula.

[0027] Optionally, the determining the point cloud data of the forklift tray and the shelf according to the point cloud data in the forklift coordinate system includes:

[0028] Determine the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud data in the forklift coordinate system;

[0029] Determine the region of interest on the point cloud image corresponding to the gray-scale images of the forklift pallet and the shelf in the forklift coordinate system based on the point cloud image corresponding to the gray-scale images of the forklift pallet and the shelf in the forklift coordinate system;

[0030] Traverse the region of interest to obtain the point cloud data of the forklift pallet and the shelf.

[0031] Optionally, the determining the spatial position information of the forklift pallet and the shelf according to the point cloud data includes:

[0032] Perform discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the target point cloud data of the pallet legs and the shelf;

[0033] Use the RANSAC algorithm on the target point cloud data to obtain the point cloud plane information;

[0034] Determine the mean surface point cloud data of the forklift pallet and the surface point cloud data of the shelf according to the point cloud plane information;

[0035] Determine the spatial position information of the forklift pallet and the shelf according to the mean surface point cloud data and the surface point cloud data.

[0036] Optionally, the controlling the fork arms to perform pose adjustment according to the distance difference to complete goods placement includes:

[0037] Compare the distance difference with the pose adjustment threshold to obtain a comparison result;

[0038] When the comparison result is that the distance difference is greater than the pose adjustment threshold, control the fork arms to perform pose adjustment according to the distance difference;

[0039] When the comparison result is that the distance difference is less than or equal to the pose adjustment threshold, do not adjust the pose of the fork arms.

[0040] In addition, to achieve the above object, the present invention also proposes an unmanned forklift high-level shelf goods placement device, and the unmanned forklift high-level shelf goods placement device includes:

[0041] An image acquisition module, configured to obtain image information of the forklift pallet and the shelf through a depth camera on the fork arms of the unmanned forklift;

[0042] A point cloud extraction module, configured to determine the point cloud data of the forklift pallet and the shelf according to the image information of the forklift pallet and the shelf;

[0043] A position determination module, configured to determine the spatial position information of the forklift pallet and the shelf according to the point cloud data;

[0044] A difference calculation module, configured to determine the distance difference between the forklift tray and the front surface of the shelf according to the spatial position information;

[0045] An adjustment and goods placement module, configured to control the posture adjustment of the fork arms according to the distance difference to complete goods placement.

[0046] In addition, to achieve the above object, the present invention further provides an unmanned forklift high-level shelf goods placement device, which includes: a memory, a processor, and an unmanned forklift high-level shelf goods placement program stored on the memory and executable on the processor. The unmanned forklift high-level shelf goods placement program is configured to implement the steps of the unmanned forklift high-level shelf goods placement method as described above.

[0047] In addition, to achieve the above object, the present invention further provides a storage medium, on which an unmanned forklift high-level shelf goods placement program is stored. When the unmanned forklift high-level shelf goods placement program is executed by a processor, it implements the steps of the unmanned forklift high-level shelf goods placement method as described above.

[0048] The present invention obtains the image information of the forklift tray and the shelf through the depth camera on the fork arms of the unmanned forklift; determines the point cloud data of the forklift tray and the shelf according to the image information of the forklift tray and the shelf; determines the spatial position information of the forklift tray and the shelf according to the point cloud data; determines the distance difference between the forklift tray and the front surface of the shelf according to the spatial position information; controls the posture adjustment of the fork arms according to the distance difference to complete goods placement. In this way, it realizes determining the spatial position information of the forklift tray and the shelf according to the image information of the forklift tray and the shelf collected by the depth camera on the fork arms, thereby determining the distance difference between the forklift tray and the front surface of the shelf, and then adjusting the posture of the fork arms according to the distance difference to complete precise goods placement. This can reduce the situation where the gantry shakes due to too high lifting when the unmanned forklift places goods on the high-level shelf, making the goods placement of the unmanned forklift more precise, reducing the possibility of safety accidents, and improving the user experience. Description of the Drawings

[0049] Figure 1 is a schematic structural diagram of an unmanned forklift high-level shelf goods placement device in the hardware operating environment related to the embodiment solution of the present invention;

[0050] Figure 2 is a schematic flowchart of the first embodiment of the unmanned forklift high-level shelf goods placement method of the present invention;

[0051] Figure 3 is a schematic diagram of the installation position of the depth camera in an embodiment of the unmanned forklift high-level shelf goods placement method of the present invention;

[0052] Figure 4Schematic flowchart of the second embodiment of the method for placing goods on a high rack by the unmanned forklift of the present invention;

[0053] Figure 5 Block diagram of the structure of the first embodiment of the device for placing goods on a high rack by the unmanned forklift of the present invention.

[0054] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] Refer to Figure 1 , Figure 1 Schematic diagram of the structure of the unmanned forklift high rack goods placement device for the hardware operating environment involved in the embodiment solution of the present invention.

[0057] As Figure 1 shown, the unmanned forklift high rack goods placement device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless-fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art can understand that Figure 1 the structure shown in

[0059] As Figure 1 shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and an unmanned forklift high rack goods placement program.

[0060] InFigure 1 In the high-level rack goods placement device of the driverless forklift shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the high-level rack goods placement device of the driverless forklift of the present invention can be arranged in the high-level rack goods placement device of the driverless forklift. The high-level rack goods placement device of the driverless forklift calls the high-level rack goods placement program stored in the memory 1005 through the processor 1001 and executes the high-level rack goods placement method provided by the embodiments of the present invention.

[0061] Embodiments of the present invention provide a high-level rack goods placement method for a driverless forklift, referring to Figure 2 , Figure 2 which is a schematic flowchart of the first embodiment of a high-level rack goods placement method for a driverless forklift of the present invention.

[0062] In this embodiment, the high-level rack goods placement method for the driverless forklift includes the following steps:

[0063] Step S10: Obtain the image information of the forklift tray and the rack through the depth camera on the fork arm of the driverless forklift.

[0064] It should be noted that the execution subject of this embodiment is a controller, which can be the processor of the driverless forklift, or the control unit, or other devices that can implement this function. This embodiment does not limit this.

[0065] It should be understood that currently, driverless forklifts are widely used in industry for intelligent goods management and goods placement in warehouses or other scenarios. However, when the driverless forklift places goods, the gantry of the forklift may shake because the gantry of the forklift extends too high, which will lead to a large deviation and inaccurate goods placement when placing goods, resulting in safety problems such as easy occurrence of goods tipping over and high-altitude goods falling. The solution in this embodiment realizes the calculation of the position and distance difference between the forklift tray and the rack according to the images collected by the depth camera by installing depth cameras on both sides of the fork arm of the driverless forklift, so as to accurately adjust the pose and achieve safer and more accurate high-level rack goods placement for the driverless forklift.

[0066] In specific implementation, two depth cameras are arranged on the fork arm of the driverless forklift. As Figure 3 shown, the specific installation positions of the depth cameras are the left side of the left fork arm and the right side of the right fork arm of the driverless forklift. Among them, the depth camera refers to a TOF camera, which continuously sends light pulses to the target and then uses a sensor to receive the light returned from the object, and obtains the distance of the target object by detecting the round-trip time of the light pulse.

[0067] It should be noted that the image information of the forklift tray and the rack refers to the specific information of the images collected by the two depth cameras respectively.

[0068] Step S20: Determine the point cloud data of the forklift tray and the shelf according to the image information of the forklift tray and the shelf.

[0069] It should be understood that the forklift tray is the storage tray above the fork arms of the driverless forklift, and the shelf is the target shelf where the goods need to be placed.

[0070] In a specific implementation, determining the point cloud data of the forklift tray and the shelf according to the image information of the forklift tray and the shelf means: obtaining the grayscale images of the forklift tray and the shelf according to the image information of the forklift tray and the shelf, and then performing checkerboard annotation based on the grayscale images of the forklift tray and the shelf, so as to obtain the three-dimensional point cloud data of the forklift tray and the shelf.

[0071] Step S30: Determine the spatial position information of the forklift tray and the shelf according to the point cloud data.

[0072] It should be noted that the spatial position information refers to the specific coordinates of the forklift tray and the shelf in the spatial position and related information such as the occupied position.

[0073] It should be understood that determining the spatial position information of the forklift tray and the shelf according to the point cloud data means: performing operations such as filtering and smoothing on the point cloud data, and then using the RANSAC algorithm, that is, an algorithm that can filter the point cloud data to obtain valid data, and finally obtaining the surface point cloud mean data of the forklift tray and the surface point cloud data of the shelf, so as to determine the spatial position information of the forklift tray and the shelf.

[0074] Further, in order to obtain accurate spatial position information, step S30 includes: performing discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the target point cloud data of the tray legs and the shelf; using the RANSAC algorithm for the target point cloud data to obtain point cloud plane information; determining the surface point cloud mean data of the forklift tray and the surface point cloud data of the shelf according to the point cloud plane information; determining the spatial position information of the forklift tray and the shelf according to the surface point cloud mean data and the surface point cloud data.

[0075] In a specific implementation, both discrete point filtering and normal vector filtering are steps of filtering, that is, denoising the point cloud data, and then performing point cloud smoothing and clustering to obtain the point cloud data of the area of the tray legs and the shelf, that is, the target point cloud data.

[0076] It should be noted that using the RANSAC algorithm for the target point cloud data to obtain the point cloud plane information means: using the RANSAC algorithm for the target point cloud data, that is, effectively filtering the abnormal data and noise from the target point cloud data to obtain the point cloud plane information, which is the point cloud data of the forklift pallet legs and the plane of the shelf.

[0077] It should be understood that determining the surface point cloud mean data of the forklift pallet and the surface point cloud data of the shelf according to the point cloud plane information means: extracting again according to the point cloud plane information to obtain the mean z coordinate of the point cloud data on the surface of the forklift pallet, that is, the average height position of the plane where the forklift pallet is located. The surface point cloud data of the shelf is also the point cloud data of the plane where the shelf is located.

[0078] In this way, filtering and denoising are realized based on the point cloud data, and then the point cloud data of the planes corresponding to the forklift pallet and the shelf are obtained, so as to accurately obtain the spatial position information of the forklift pallet and the shelf, and further accurately determine the adjustment pose of the forklift's fork arms.

[0079] Step S40: Determine the distance difference between the front surface of the forklift pallet and the shelf according to the spatial position information.

[0080] In a specific implementation, determining the distance difference between the front surface of the forklift pallet and the shelf according to the spatial position information means: according to the z coordinate data of the surface point cloud mean data of the forklift pallet and the z coordinate data of the surface point cloud data of the shelf in the spatial position information, calculating the difference d, which is the value to be adjusted by the fork arms, that is, the distance difference.

[0081] Step S50: Control the fork arms to adjust their poses according to the distance difference to complete the goods placement.

[0082] It should be noted that controlling the fork arms to adjust their poses according to the distance difference to complete the goods placement means determining the pose range and target position that the fork arms need to adjust according to the distance difference, so as to complete the goods placement.

[0083] Further, in order to minimize the adjustment times of the fork arms as much as possible, step S50 includes: comparing the distance difference with the pose adjustment threshold to obtain a comparison result; when the comparison result is that the distance difference is greater than the pose adjustment threshold, controlling the fork arms to adjust their poses according to the distance difference; when the comparison result is that the distance difference is less than or equal to the pose adjustment threshold, not adjusting the poses of the fork arms.

[0084] It should be understood that the comparison result refers to the comparison result of the size relationship between the distance difference and the pose adjustment threshold. Here, the pose adjustment threshold is a threshold with an arbitrary value preset by the user, and this embodiment does not limit it.

[0085] In a specific implementation, when the comparison result is that the distance difference is greater than the pose adjustment threshold, controlling the fork arm to perform pose adjustment according to the distance difference means that when the distance difference is greater than the pose adjustment threshold, it is determined that the fork arm needs to be adjusted. Therefore, the fork arm of the driverless forklift is adjusted according to the distance difference, so that the pose of the fork arm can safely place the goods on the shelf.

[0086] It should be noted that when the comparison result is that the distance difference is less than or equal to the pose adjustment threshold, not adjusting the pose of the fork arm means that when the distance difference is less than or equal to the pose adjustment threshold, it is determined that the data of the height difference between the fork arm and the shelf does not affect the safety and accuracy of placing the goods. Therefore, the pose of the fork arm can be not adjusted and the goods can be placed directly.

[0087] In this way, when the position difference between the fork arm and the shelf is within the allowable error range, no adjustment is made, which can reduce unnecessary pose adjustments, making the process of placing goods by the driverless forklift more simple and fast.

[0088] In this embodiment, the depth camera on the fork arm of the driverless forklift is used to obtain the image information of the forklift tray and the shelf; the point cloud data of the forklift tray and the shelf is determined according to the image information of the forklift tray and the shelf; the spatial position information of the forklift tray and the shelf is determined according to the point cloud data; the distance difference between the front surfaces of the forklift tray and the shelf is determined according to the spatial position information; and the fork arm is controlled to perform pose adjustment according to the distance difference to complete the goods placement. In this way, the spatial position information of the forklift tray and the shelf is determined according to the image information of the forklift tray and the shelf collected by the depth camera on the fork arm, so as to determine the distance difference between the front surfaces of the forklift tray and the shelf, and then the pose of the fork arm is adjusted according to the distance difference to complete accurate goods placement. This can reduce the situation that the gantry shakes when the forklift places goods on the high-rise shelf due to too high lifting, making the goods placement of the driverless forklift more accurate, reducing the possibility of safety accidents, and improving the user experience.

[0089] Reference Figure 4 , Figure 4 is a schematic flowchart of the second embodiment of a method for placing goods on a high-rise shelf by a driverless forklift according to the present invention.

[0090] Based on the above first embodiment, the method for placing goods on a high-rise shelf by a driverless forklift in this embodiment includes in the step S20:

[0091] Step S201: Determine the point cloud information of the forklift tray and the shelf according to the image information of the forklift tray and the shelf.

[0092] It should be understood that determining the point cloud information of the forklift tray and the shelf according to the image information of the forklift tray and the shelf means: based on the image information of the forklift tray and the shelf, perform grayscale processing according to the characteristics of the depth camera, and then obtain the relevant information of the point cloud data of the forklift tray and the shelf in the coordinate system of the depth camera.

[0093] Step S202: Match the point cloud information of the forklift tray and the shelf with a preset checkerboard to obtain a set of matching three-dimensional point clouds.

[0094] It should be noted that the preset checkerboard refers to the initial checkerboard preset by the user, which is used to match with the grayscale image of the forklift tray and the shelf to obtain a set of matching three-dimensional point clouds. Among them, the set of matching three-dimensional point clouds refers to the set composed of points with the same x and y coordinates in the grayscale image of the forklift tray and the shelf.

[0095] Furthermore, in order to accurately obtain the set of matching three-dimensional point clouds, step S202 includes: determining the grayscale image of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf; determining the checkerboard corner points according to the grayscale image of the forklift tray and the shelf; fitting the checkerboard plane formula according to the grayscale image of the forklift tray and the shelf and the checkerboard corner points; searching and matching the points on the grayscale image of the forklift tray and the shelf according to the checkerboard plane formula to obtain a set of two-dimensional point clouds with the same abscissa and ordinate; determining the homography matrix according to the set of two-dimensional point clouds; and determining the set of matching three-dimensional point clouds according to the homography matrix and the checkerboard plane formula.

[0096] It should be noted that determining the grayscale image of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf means: first extracting the grayscale images corresponding to the positions of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf.

[0097] It should be understood that determining the checkerboard corner points according to the grayscale image of the forklift tray and the shelf means: determining the checkerboard corner points of the preset checkerboard on the grayscale image according to the grayscale image of the forklift tray and the shelf, that is, determining the checkerboard corner points corresponding to the four preset checkerboards on the grayscale image of the forklift tray and the shelf.

[0098] It should be noted that fitting the checkerboard plane formula according to the grayscale image of the forklift tray and the shelf means: respectively fitting the plane formulas of the checkerboard in the coordinate systems of the two depth cameras, and then taking these two plane formulas as the checkerboard plane formula.

[0099] It should be understood that finding and matching points on the grayscale images of the forklift tray and the shelf according to the checkerboard plane formula, and obtaining a two-dimensional point cloud set with the same abscissa and ordinate means: finding and matching each point on the grayscale images of the forklift tray and the shelf according to the checkerboard plane formula, and taking the points with the same x and y coordinates of the checkerboard as a set, thereby obtaining several two-dimensional point cloud sets.

[0100] In a specific implementation, the homography matrix refers to the projection matrix from the real physical coordinates to the ideal pixel points. Determining the matching three-dimensional point cloud set according to the homography matrix and the checkerboard plane formula means: obtaining the corresponding three-dimensional coordinate xy values through the homography matrix H and the two-dimensional coordinates of the checkerboard corner points on the image, that is, obtaining the matching three-dimensional point cloud set.

[0101] In this way, a method of specifically using checkerboard annotation is implemented to obtain a two-dimensional point cloud set from the points on the grayscale images of the forklift tray and the shelf, and then the two-dimensional point cloud set is matched to obtain the matching three-dimensional point cloud set.

[0102] Step S203: Determine the rotation matrix and the translation matrix from the depth camera to the unmanned forklift body according to the matching three-dimensional point set.

[0103] It should be understood that the rotation matrix is a matrix that has the effect of changing the direction of a vector but not its magnitude when multiplying a vector and maintains the properties, while the translation matrix refers to the matrix used for calculation when implementing matrix translation. And the rotation matrix and the translation matrix do not change the original matrix.

[0104] Step S204: Determine the point cloud data of the forklift tray and the shelf according to the rotation matrix, the translation matrix, and the point cloud information of the forklift tray and the shelf.

[0105] In a specific implementation, determining the point cloud data of the forklift tray and the shelf according to the rotation matrix and the translation matrix means: calibrating the point cloud data according to the rotation matrix and the translation matrix, thereby obtaining the point cloud data of the forklift tray and the shelf.

[0106] Furthermore, in order to accurately obtain the point cloud data of the forklift tray and the shelf, step S204 includes: determining the point cloud data of the camera coordinate system of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf; converting the point cloud data of the camera coordinate system into the point cloud data of the forklift coordinate system according to the rotation matrix and the translation matrix; determining the point cloud data of the forklift tray and the shelf according to the point cloud data of the forklift coordinate system.

[0107] It should be noted that determining the point cloud data of the forklift tray and the shelf in the camera coordinate system based on the point cloud information of the forklift tray and the shelf means: determining the point cloud data of the positions of various parts of the forklift tray and the shelf in the coordinate system of the depth camera based on the point cloud information of the forklift tray and the shelf, which is used as the point cloud data in the camera coordinate system.

[0108] It should be understood that converting the point cloud data in the camera coordinate system to the point cloud data in the forklift coordinate system according to the rotation matrix and the translation matrix means: converting the point cloud data on the camera coordinate system to the forklift coordinate system through the rotation matrix and the translation matrix, that is, performing coordinate transformation on each point cloud data, and the obtained data is the point cloud data in the forklift coordinate system.

[0109] In a specific implementation, determining the point cloud data of the forklift tray and the shelf based on the point cloud data in the forklift coordinate system means: determining the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system based on the point cloud data in the forklift coordinate system, and then determining the region of interest, so as to traverse the region of interest to obtain the point cloud data of the forklift tray and the shelf.

[0110] In this way, it is realized to accurately determine the point cloud data of the forklift tray and the shelf by means of coordinate transformation.

[0111] Furthermore, in order to be able to determine accurate point cloud data according to the rotation matrix and the translation matrix, the steps of determining the point cloud data of the forklift tray and the shelf based on the point cloud data in the forklift coordinate system include: determining the point cloud data of the forklift tray and the shelf based on the point cloud data in the forklift coordinate system, including: determining the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud data in the forklift coordinate system; determining the region of interest on the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system; traversing the region of interest to obtain the point cloud data of the forklift tray and the shelf.

[0112] It should be noted that on the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system, the distribution image of the point cloud in each image region is determined according to the rotation matrix and the translation matrix, and then an ROI region, that is, the region of interest, is drawn, and the point cloud data on the point cloud image is obtained by traversing the region of interest, and finally the accurate point cloud data of the forklift tray and the shelf can be obtained.

[0113] In this way, it is realized to accurately screen out the region of interest and the point cloud data of the forklift tray and the shelf from the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system, making the subsequent calculation more accurate and reducing the data processing amount.

[0114] In this embodiment, the point cloud information of the forklift tray and the shelf is determined based on the image information of the forklift tray and the shelf; the point cloud information of the forklift tray and the shelf is matched with a preset checkerboard to obtain a set of matched three-dimensional point clouds; the rotation matrix and the translation matrix from the depth camera to the body of the driverless forklift are determined according to the set of matched three-dimensional points; and the point cloud data of the forklift tray and the shelf is determined according to the rotation matrix, the translation matrix, and the point cloud information of the forklift tray and the shelf. In this way, the point cloud information of the forklift tray and the shelf is obtained based on the image information of the forklift tray and the shelf, then it is matched and calibrated with a preset checkerboard to obtain a set of matched three-dimensional points, then the rotation matrix and the translation matrix are determined based on the set of matched three-dimensional points, and finally the point cloud data of the forklift tray and the shelf is determined based on the rotation matrix and the translation matrix, realizing the acquisition of the point cloud data of the forklift tray and the shelf through the method of checkerboard annotation, making the acquisition of the point cloud data simpler and more accurate, making the subsequent calculation of the distance difference more accurate, and improving the accuracy of the automatic goods placement of the forklift.

[0115] In addition, an embodiment of the present invention further provides a storage medium, on which a program for placing goods on a high-level shelf by a driverless forklift is stored. When the program for placing goods on a high-level shelf by a driverless forklift is executed by a processor, the steps of the method for placing goods on a high-level shelf by a driverless forklift as described above are realized.

[0116] Since this storage medium adopts all the technical solutions of the above-mentioned all embodiments, it has at least all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be elaborated here one by one.

[0117] Refer to Figure 5 , Figure 5 which is the structural block diagram of the first embodiment of the device for placing goods on a high-level shelf by a driverless forklift of the present invention.

[0118] As Figure 5 shown, the device for placing goods on a high-level shelf by a driverless forklift proposed by an embodiment of the present invention includes:

[0119] An image acquisition module 10, configured to acquire the image information of the forklift tray and the shelf through a depth camera on the fork arm of the driverless forklift.

[0120] A point cloud extraction module 20, configured to determine the point cloud data of the forklift tray and the shelf according to the image information of the forklift tray and the shelf.

[0121] A position determination module 30, configured to determine the spatial position information of the forklift tray and the shelf according to the point cloud data.

[0122] A difference calculation module 40, configured to determine the distance difference between the forklift tray and the front surface of the shelf according to the spatial position information.

[0123] The goods discharging module 50 is used to control the pose adjustment of the fork arm according to the distance difference to complete the goods discharging.

[0124] In this embodiment, the depth camera on the fork arm of the driverless forklift is used to obtain the image information of the forklift tray and the shelf; the point cloud data of the forklift tray and the shelf is determined according to the image information of the forklift tray and the shelf; the spatial position information of the forklift tray and the shelf is determined according to the point cloud data; the distance difference between the front surface of the forklift tray and the shelf is determined according to the spatial position information; the pose of the fork arm is controlled according to the distance difference to complete the goods discharging. In this way, the spatial position information of the forklift tray and the shelf is determined according to the image information of the forklift tray and the shelf collected by the depth camera on the fork arm, so as to determine the distance difference between the front surface of the forklift tray and the shelf, and then the pose of the fork arm is adjusted according to the distance difference to complete accurate goods discharging. This can reduce the situation that the gantry shakes when the forklift discharges goods at high-level shelves, make the goods discharging of the driverless forklift more accurate, reduce the possibility of safety accidents, and improve the user experience.

[0125] In one embodiment, the point cloud extraction module 20 is further configured to determine the point cloud information of the forklift tray and the shelf according to the image information of the forklift tray and the shelf; match the point cloud information of the forklift tray and the shelf with a preset checkerboard to obtain a matching three-dimensional point cloud set; determine the rotation matrix and translation matrix from the depth camera to the driverless forklift body according to the matching three-dimensional point set; determine the point cloud data of the forklift tray and the shelf according to the rotation matrix, the translation matrix and the point cloud information of the forklift tray and the shelf.

[0126] In one embodiment, the point cloud extraction module 20 is further configured to determine the point cloud data of the forklift tray and the shelf in the camera coordinate system according to the point cloud information of the forklift tray and the shelf; convert the point cloud data in the camera coordinate system into the point cloud data in the forklift coordinate system according to the rotation matrix and the translation matrix; use the point cloud data in the forklift coordinate system as the point cloud data of the forklift tray and the shelf.

[0127] In one embodiment, the point cloud extraction module 20 is further configured to determine the grayscale image of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf; determine the checkerboard corner points according to the grayscale image of the forklift tray and the shelf; fit the checkerboard plane formula according to the grayscale image of the forklift tray and the shelf and the checkerboard corner points; search and match the points on the grayscale image of the forklift tray and the shelf according to the checkerboard plane formula to obtain a two-dimensional point cloud set with the same abscissa and ordinate; determine the homography matrix according to the two-dimensional point cloud set; determine the matching three-dimensional point cloud set according to the homography matrix and the checkerboard plane formula.

[0128] In one embodiment, the point cloud extraction module 20 is further configured to determine a point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud data in the forklift coordinate system; determine an interested region on the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system; traverse the interested region to obtain the point cloud data of the forklift tray and the shelf.

[0129] In one embodiment, the position determination module 30 is further configured to perform discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the target point cloud data of the tray leg and the shelf; use the RANSAC algorithm on the target point cloud data to obtain point cloud plane information; determine the surface point cloud mean data of the forklift tray and the surface point cloud data of the shelf according to the point cloud plane information; determine the spatial position information of the forklift tray and the shelf according to the surface point cloud mean data and the surface point cloud data.

[0130] In one embodiment, the adjustment and loading module 50 is further configured to compare the distance difference with the pose adjustment threshold to obtain a comparison result; when the comparison result is that the distance difference is greater than the pose adjustment threshold, control the fork arms to perform pose adjustment according to the distance difference; when the comparison result is that the distance difference is less than or equal to the pose adjustment threshold, do not adjust the pose of the fork arms.

[0131] Since this device adopts all the technical solutions of the above-mentioned all embodiments, it has at least all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be elaborated one by one here.

[0132] It should be understood that the above is only for illustration and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can set according to needs, and the present invention does not limit this.

[0133] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is made here.

[0134] In addition, for the technical details not described in detail in this embodiment, reference can be made to the method for placing goods on a high-level shelf by an unmanned forklift provided in any embodiment of the present invention, which will not be elaborated here.

[0135] In addition, it should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such a process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or system including such element.

[0136] The serial numbers of the above-described embodiments of the present invention are for description only and do not represent the superiority or inferiority of the embodiments.

[0137] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as Read Only Memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0138] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the description of the present invention and the drawings, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for placing goods on a high - level shelf by an automated forklift, characterized in that, the method for placing goods on a high - level shelf by an automated forklift includes: acquiring image information of the forklift tray and the shelf through a depth camera on the fork arm of the automated forklift; determining the point cloud information of the forklift tray and the shelf according to the image information of the forklift tray and the shelf; determining the grayscale image of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf; determining the checkerboard corner points according to the grayscale image of the forklift tray and the shelf; fitting a checkerboard plane formula according to the grayscale image of the forklift tray and the shelf and the checkerboard corner points; searching and matching the points on the grayscale image of the forklift tray and the shelf according to the checkerboard plane formula to obtain a two - dimensional point cloud set with the same abscissa and ordinate; determining a homography matrix according to the two - dimensional point cloud set; determining a matching three - dimensional point cloud set according to the homography matrix and the checkerboard plane formula; determining the rotation matrix and the translation matrix from the depth camera to the body of the automated forklift according to the matching three - dimensional point set; determining the point cloud data of the camera coordinate system of the forklift tray and the shelf according to the point cloud information of the forklift tray and the shelf; converting the point cloud data of the camera coordinate system into the point cloud data of the forklift coordinate system according to the rotation matrix and the translation matrix; determining the point cloud data of the forklift tray and the shelf according to the point cloud data of the forklift coordinate system; determining the spatial position information of the forklift tray and the shelf according to the point cloud data; determining the distance difference between the front surfaces of the forklift tray and the shelf according to the spatial position information; controlling the pose adjustment of the fork arm according to the distance difference to complete the goods placement.

2. The method according to claim 1, characterized in that, the determining the point cloud data of the forklift tray and the shelf according to the point cloud data of the forklift coordinate system includes: determining the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud data of the forklift coordinate system; determining the region of interest on the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system according to the point cloud image corresponding to the grayscale image of the forklift tray and the shelf in the forklift coordinate system; traversing the region of interest to obtain the point cloud data of the forklift tray and the shelf.

3. The method according to claim 1, characterized in that, the determining the spatial position information of the forklift tray and the shelf according to the point cloud data includes: performing discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the target point cloud data of the tray legs and the shelf; using the RANSAC algorithm on the target point cloud data to obtain point cloud plane information; determining the mean surface point cloud data of the forklift tray and the surface point cloud data of the shelf according to the point cloud plane information; determining the spatial position information of the forklift tray and the shelf according to the mean surface point cloud data and the surface point cloud data.

4. The method according to any one of claims 1 to 3, characterized in that, the controlling the pose adjustment of the fork arm according to the distance difference to complete the goods placement includes: Compare the distance difference with the pose adjustment threshold to obtain a comparison result; When the comparison result is that the distance difference is greater than the pose adjustment threshold, control the fork arm to adjust its pose according to the distance difference; When the comparison result is that the distance difference is less than or equal to the pose adjustment threshold, do not adjust the pose of the fork arm.

5. An automatic forklift high-level rack goods placement device, characterized in that, the automatic forklift high-level rack goods placement device includes: an image acquisition module, configured to obtain image information of a forklift tray and a rack through a depth camera on the fork arm of the automatic forklift; a point cloud extraction module, configured to determine point cloud information of the forklift tray and the rack according to the image information of the forklift tray and the rack; determine a grayscale image of the forklift tray and the rack according to the point cloud information of the forklift tray and the rack; determine checkerboard corner points according to the grayscale image of the forklift tray and the rack; fit a checkerboard plane formula according to the grayscale image of the forklift tray and the rack and the checkerboard corner points; search and match points on the grayscale image of the forklift tray and the rack according to the checkerboard plane formula to obtain a two-dimensional point cloud set with the same abscissa and ordinate; determine a homography matrix according to the two-dimensional point cloud set; determine a matching three-dimensional point cloud set according to the homography matrix and the checkerboard plane formula; determine a rotation matrix and a translation matrix from the depth camera to the automatic forklift body according to the matching three-dimensional point set; determine camera coordinate system point cloud data of the forklift tray and the rack according to the point cloud information of the forklift tray and the rack; convert the camera coordinate system point cloud data into forklift coordinate system point cloud data according to the rotation matrix and the translation matrix; determine point cloud data of the forklift tray and the rack according to the forklift coordinate system point cloud data; a position determination module, configured to determine spatial position information of the forklift tray and the rack according to the point cloud data; a difference calculation module, configured to determine a distance difference between the forklift tray and the front surface of the rack according to the spatial position information; an adjustment goods placement module, configured to control the fork arm to adjust its pose according to the distance difference to complete goods placement.

6. An automatic forklift high-level rack goods placement equipment, characterized in that, the equipment includes: a memory, a processor, and an automatic forklift high-level rack goods placement program stored on the memory and executable on the processor, and the automatic forklift high-level rack goods placement program is configured to implement the automatic forklift high-level rack goods placement method according to any one of claims 1 to 4.

7. A storage medium, characterized in that, an automatic forklift high-level rack goods placement program is stored on the storage medium, and when the automatic forklift high-level rack goods placement program is executed by a processor, it implements the automatic forklift high-level rack goods placement method according to any one of claims 1 to 4.

Images